DEV Community: JhonLiu

How One Open-Source IoT Platform Unified China Largest Oil Field — 928 Gateways, 114K Sensors, 600x Faster

JhonLiu — Tue, 11 Aug 2026 03:25:37 +0000

How One Open-Source IoT Platform Unified China's Largest Oil Field — 928 Gateways, 114K Sensors, 600× Faster

Daqing Oil Field produces 30 million tons of crude annually. When they needed to modernize their sensor network, they chose an open-source platform built by a 10-person team. Here's the story.

The Scale of the Problem

Daqing Oil Field spans 6,000 square kilometers — roughly the size of Delaware. It has:

16 oil extraction plants
928 industrial gateways (mostly Modbus RTU/TCP and a proprietary CNPC protocol called A11)
114,809 sensor points measuring pressure, temperature, flow rate, vibration, and 20 other parameters
30+ years of accumulated legacy equipment

When we first visited in 2020, the data infrastructure looked like this:

Field Operator checks gauge → writes value on clipboard
  → enters into Excel at end of shift
  → supervisor reviews next morning
  → alarm triggered 4 hours after anomaly occurred

Collection frequency: every 10 minutes on a good day. End-to-end latency: 15-30 minutes. False alarm rate: over 20%. Storage: data older than 10 days was purged.

An oil pump failure caught 4 hours late costs $50,000-200,000 in lost production.

The Approach: Protocol-First

We didn't try to replace the gateways. Instead, we built protocol adapters that spoke their language:

Siemens S7-1200 (Modbus TCP)  ──┐
ABB AC500 (Modbus RTU)         ──┤
A11 CNPC Gateways (proprietary) ──┼──→ DGIOT Unified Pipeline
OPC UA Servers (newer units)   ──┤
Manual Input (legacy gauges)   ──┘

Each protocol adapter is an Erlang gen_statem process. A single gateway's Modbus connection is one lightweight process consuming ~2KB of memory. 928 gateways = <2MB RAM for the entire protocol layer.

The A11 Protocol Challenge

The hardest part was A11 — a proprietary CNPC protocol with almost no documentation. We had to reverse-engineer it from packet captures and conversations with retired engineers. The protocol uses a unique 4-byte magic number (0x6A6A5A5A) and variable-length frames with custom CRC. It took three weeks of on-site work, but once we cracked it, we could talk to 60% of Daqing's legacy gateways natively.

The Architecture in Production

928 Gateways
    │  Modbus RTU/TCP, A11 CNPC, OPC UA
    ↓
DGIOT Edge (Runs on existing gateway hardware)
    │  Protocol adaptation + stream computation + 7-day offline buffer
    ↓
MQTT Broker (EMQX) — 1M+ msg/sec
    │
    ├──→ TDengine (time-series, 10:1 compression, 652M data points)
    ├──→ Parse Server (device metadata, auth, tenant isolation)
    └──→ Stream Engine (15 algorithms, real-time alerting)
    │
    ↓
Vue 3 Dashboard (operator view, maintenance view, director view)

The Results

Metric	Before	After	Improvement
Collection frequency	10 min	1 sec	600×
End-to-end latency	15-30 min	<3 sec	600×
Storage retention	10 days	2 years	73×
False alarm rate	22%	4.7%	4.7× better
Gateways managed	1 per system	928 unified	—
Protocols supported	3	15 at Daqing, 300+ globally	—
Uptime	~95%	99.9999%	—

The Real Impact

$2.3 million saved in the first year — not from license fees (DGIOT is free), but from:

Reduced downtime: Pumps caught at first sign of bearing wear, not after failure
Eliminated manual rounds: Operators stopped driving 2 hours to check a gauge
Optimized maintenance: Predictive instead of scheduled — 40% fewer truck rolls

Why Open Source Won

Daqing initially wanted a commercial SCADA system. Three vendors quoted:

Vendor A: $1.2M license + $200K/year maintenance
Vendor B: $850K + $180K/year
Vendor C: $3M for a "full digital transformation"

All three required replacing the existing gateways — a $15M hardware project. DGIOT ran on the existing hardware. The software was free. The only cost was integration and training.

When the project lead asked "What if your company goes out of business?", we pointed to the GitHub repo: "The code is Apache 2.0. It outlives us."

That closed the deal.

Lessons for Industrial IoT Founders

Don't compete on features. Compete on protocols. Every protocol you support removes a reason to say no.
Open source is a trust mechanism for hardware companies. They've been burned by vendor lock-in. Apache 2.0 is the ultimate guarantee.
Edge computing is non-negotiable. Oil fields have terrible connectivity. Your platform must work offline for days.
The hard part isn't the tech — it's the protocol reverse-engineering. Budget 2-4 weeks per proprietary protocol. It's worth it.
Pricing: free software, paid integration. The software is open source. The 3-week on-site integration, training, and custom protocol work is what you charge for.

DGIOT is open source (Apache 2.0) at github.com/dgiot/dgiot. The Daqing deployment has been running in production since 2021.

Liu Shouxin is the founder of DGIOT. He previously architected IoT platforms at EMQ, Huawei, and TCL.

How We Built an IoT Platform That Handles 30 Million Concurrent Connections — With a Team of 10

JhonLiu — Tue, 11 Aug 2026 03:25:00 +0000

How We Built an IoT Platform That Handles 30 Million Concurrent Connections — With a Team of 10

DGIOT is an open-source industrial IoT platform. We run 928 gateways across 16 oil fields, process 652 million data points, and maintain 99.9999% uptime. Here's the architecture that makes it possible.

The Problem

In 2021, we got a call from Daqing Oil Field — China's largest oil producer. They had a problem:

928 industrial gateways from different vendors
114,809 sensor points speaking 15 different protocols
Data collection every 10 minutes (they needed seconds)
15-30 minute end-to-end latency (they needed <3 seconds)
False alarm rate above 20%

The existing system was a patchwork of vendor-specific tools, each with its own database, UI, and authentication. Operators had to log into 8 different systems just to check if a pump was overheating.

They asked: "Can you unify this?"

What We Built

DGIOT is an Erlang/OTP-based platform that acts as a universal translator for industrial protocols. Think of it as a Rosetta Stone for machines.

Modbus ─┐
OPC UA ─┤
MQTT  ──┼──→ Unified Pipeline ──→ TDengine ──→ Dashboard
IEC104 ─┤
A11   ──┘

The key insight: industrial protocols are just state machines. Once you model each protocol as a gen_statem FSM in Erlang, you can handle hundreds of them concurrently with almost zero overhead.

The Architecture: DLAS

We designed a four-layer architecture that separates concerns cleanly:

Layer 1: DATA — Ingestion

Parse Server (23 classes) handles device metadata, user auth, tenant isolation
TDengine stores 652M time-series data points with 10:1 compression
EMQX handles MQTT message routing at 1M+ msg/sec
Mnesia/ETS provides in-memory caching for hot data

Layer 2: LOGIC — Ontology Engine

This is our secret weapon. We built a 252-entity OWL ontology that models industrial equipment:

Pump ⊑ Equipment ⊓ ∃hasPart.Bearing ⊓ ∃measures.Pressure
Bearing ⊑ Component ⊓ ∃hasFailureMode.Overheat
Overheat → triggers(Alert) ∧ reduces(RemainingLife, 0.8)

These rules are compiled to Erlang pattern matches. When a pressure sensor reads 2.35 MPa, the platform doesn't just display a number — it knows that pump P-103's bearing is trending toward failure, cross-references the maintenance schedule, and generates a work order.

Layer 3: ACTION — Shadow State Machines

Every physical device has a "shadow" in software — a gen_statem process that mirrors its real-world state:

init()    → auth → online → {normal, alarm, offline}
normal()  → evaluate(Rules) → publish(MQTT)
alarm()   → escalate → notify → acknowledge
offline() → retry(5) → alert

The shadow absorbs network interruptions. If a gateway goes offline for 30 seconds, the shadow holds the last known state and retries. The dashboard never shows "disconnected" unless the device has been unreachable for 2+ minutes.

Layer 4: EDGE — iotStudio

A Python + Vue 3 edge agent that runs on ARM gateways. It does:

Protocol adaptation: Modbus RTU → MQTT bridge
Stream computation: Moving average, anomaly detection, deadband filtering — 15 algorithms
Offline autonomy: 7-day buffer if cloud connection drops
OTA updates: Rolling firmware deployment to 928 gateways

The Numbers

Metric	Before DGIOT	After DGIOT
Collection frequency	10 min	1 second (600×)
End-to-end latency	15-30 min	<3 seconds
Storage retention	10 days	2 years
False alarm rate	>20%	<5%
Concurrent connections	~1,000	30M verified
Protocols supported	3	300+

Why Erlang?

We get this question a lot. The answer is simple: Erlang was built for this.

Preemptive scheduling: A slow Modbus device can't block a fast MQTT stream
Hot code reloading: We deploy updates without shutting down 928 gateways
Let-it-crash philosophy: A failing protocol adapter restarts in milliseconds without affecting others
Built-in distribution: Adding a new node to the cluster is one command

The same reasons WhatsApp used Erlang for 2 billion users. Industrial IoT has the same requirements: massive concurrency, high availability, and soft real-time.

Open Source

DGIOT is Apache 2.0 licensed. Full source at github.com/dgiot/dgiot.

We made this choice deliberately. Industrial IoT is too fragmented for any single company to solve. The only way to cover 300+ protocols is to let the community add them. Every new protocol adapter benefits everyone.

git clone https://github.com/dgiot/dgiot.git
cd dgiot && docker-compose up -d
# Open http://localhost:5080

Lessons Learned

Protocols are the moat. Every new protocol adapter opens a new industry. We went from 3 to 300 in 4 years — each one brought new customers.
Ontology beats rules. Hard-coded alarm thresholds break. An ontology that understands "pump→bearing→overheat" generalizes across equipment types.
Open source is the distribution. We don't have a sales team. Our customers find us through GitHub, then ask for enterprise support.
Erlang is a competitive advantage. In a world of Node.js and Python IoT platforms, Erlang's concurrency model lets us handle 100x the load with the same hardware.
Community > code. The 70K developers in our community have built integrations we never would have thought of.

Liu Shouxin is the founder of DGIOT, an open-source industrial IoT platform. He previously worked at EMQ, Huawei, and Kaspersky. He lives in Hangzhou and Dallas.