A remote station loses connectivity for twenty minutes. When the connection returns, its gateway uploads buffered pressure and flow readings.
The backend receives the data successfully—but the dashboard replaces a current measurement with an older one, repeated messages create duplicate records, and historical pressure changes trigger new operational alerts.
The transport recovered. The application handled recovery incorrectly.
Reliable pipeline telemetry requires explicit rules for event time, duplicate delivery, data quality and replay. This guide explains how to design those rules between field equipment and the monitoring application.
What Does a Pipeline Telemetry Backend Need to Do?
Pipeline telemetry collects measurements and equipment states from distributed assets and delivers them to monitoring software.
Typical inputs include pressure, flow, temperature, pump status and valve position.
A useful backend must preserve:
- Which asset produced the observation.
- When the observation occurred.
- Whether the measurement is usable.
- Whether the record has already been processed.
- Whether it belongs in the current-state view, historical storage or both.
These requirements become especially important when remote links are intermittent.
1. Define a Message Contract
Start with a documented record format rather than allowing every gateway to publish unrelated payloads.
Here is an illustrative application schema:
{
"schemaVersion": 1,
"stationId": "ST-014",
"assetId": "PT-014-02",
"tag": "discharge_pressure",
"value": 6.42,
"unit": "bar",
"sourceTimestamp": "2026-10-22T08:30:00Z",
"gatewayTimestamp": "2026-10-22T08:30:01Z",
"bootId": "session-7f21",
"sequence": 184203,
"quality": "good",
"deliveryMode": "live"
}
This is an example, not a protocol standard.
Define each field precisely:
| Field | Meaning |
|---|---|
sourceTimestamp |
Time the measurement was acquired |
gatewayTimestamp |
Time the gateway handled the observation |
bootId |
Identifier for the session in which sequence numbers are assigned |
sequence |
Record position within a documented stream |
quality |
Usability of the measurement |
deliveryMode |
Whether the record is live or replayed |
If the instrument does not provide a timestamp, document which component assigns one. Gateway polling time and instrument acquisition time are not necessarily equivalent.
Normalize engineering units before applying shared calculations, while retaining source information where traceability requires it.
Protocol selection is a separate architectural decision from the message contract. For background on connecting field acquisition with upstream messaging, see OPC UA vs MQTT for industrial IoT.
2. Separate Historical Storage from Current State
A historian and a live dashboard answer different questions.
Historical storage: What happened, and when?
Current state: What is the newest usable observation, and is it still fresh?
A delayed record can be valid historical data without being suitable for the current-state view.
For example:
Current stored observation: 08:35
Incoming observation: 08:20
Incoming arrival time: 08:40
The incoming record belongs at 08:20 in historical storage. It should not replace the 08:35 observation simply because it arrived later.
A simplified processing pattern is:
def process_record(record):
validate_schema(record)
inserted = historian.insert_if_absent(
key=record_identity(record),
record=record
)
if not inserted:
return "duplicate"
if not timestamp_is_usable(record):
return "stored_for_review"
if record.quality != "good":
update_quality_status(record)
return "stored_with_quality_issue"
current = current_state.get(record.asset_id, record.tag)
if current is None or record.source_timestamp > current.source_timestamp:
current_state.update(record)
return "stored"
This is architectural pseudocode. A production implementation also needs concurrency handling, validation policies and defined behavior for equal timestamps or conflicting observations.
3. Design Duplicate Handling Explicitly
Retries can deliver a record more than once.
Without duplicate handling, repeated deliveries can inflate event counts, distort aggregates or create repeated incidents.
One possible identity is:
stationId + assetId + tag + bootId + sequence
That key works only if the producer guarantees its uniqueness within the defined scope.
Check what happens when:
- A gateway restarts.
- A sequence counter resets.
- A device is replaced.
- A batch is uploaded twice.
- The same identity arrives with different content.
Enforce idempotent storage using a unique constraint, conditional write or equivalent mechanism.
If storing a record also triggers a notification or work order, those side effects need duplicate protection too. Deduplicating the historian alone does not prevent repeated downstream actions.
4. Buffer at the Edge with a Capacity Plan
Where supported, an edge gateway can persist observations while the central application is unreachable.
A store-and-forward implementation needs:
- Durable local storage.
- A defined acknowledgement policy.
- Controlled retries.
- Storage limits.
- A documented overflow policy.
- Queue health reporting.
Estimate capacity using the actual acquisition rate.
Suppose a gateway stores 100 measurements every 10 seconds, averaging 200 bytes per serialized record:
100 × 200 bytes ÷ 10 seconds
= 2,000 bytes per second
For a 24-hour outage:
2,000 × 86,400
= 172,800,000 bytes
That is approximately 173 MB of raw records, before storage overhead and capacity margin.
Measure disk usage with representative data. Database indexes, transaction logs and other implementation details can increase the requirement.
Monitor Recovery as Well as Storage
Track queue depth, age of the oldest queued record, disk usage and upload failures.
After reconnection, the system must drain the backlog while continuing to acquire new observations. Define how replay and live traffic share available bandwidth.
Also specify what happens when the buffer fills. Any discarded or aggregated observations should be accounted for through a visible data-loss or overflow event.
5. Keep Missing, Invalid and Stale Data Distinct
A missing pressure value should not become zero.
Zero is a measurement with a specific meaning. Missing data means the application does not have a usable observation.
| State | Interpretation |
|---|---|
| Good | Measurement is usable under the defined policy |
| Uncertain | Measurement has a documented limitation |
| Invalid | Measurement is unsuitable for normal processing |
| Stale | Last usable observation exceeds its freshness window |
| Missing | No observation is available |
Freshness is separate from source quality. A measurement can have been valid when acquired but be too old to describe the current process.
Display the last known value with its age and status. Do not silently present it as current.
Quality handling should also carry through to derived measurements, reports and alert rules.
6. Make Alerts Aware of Freshness and Replay
A historical pressure excursion should not automatically become a new live alarm when buffered data arrives.
Define a policy for each rule.
| Rule category | Typical handling |
|---|---|
| Current process exception | Evaluate sufficiently fresh observations with acceptable quality |
| Communication timeout | Evaluate expected arrival behavior |
| Historical analysis | Process event-time data, including replay |
| Data quality incident | Evaluate invalid values, clock issues or missing observations |
A simplified live-rule condition might be:
eligible = (
record.quality == "good"
and timestamp_is_trusted(record)
and is_fresh(record)
and record.delivery_mode == "live"
and operating_mode_allows_rule(record)
)
Treat this as an example policy, not a universal requirement. Some applications may accept sufficiently fresh replayed observations. Define that behavior deliberately.
Process thresholds and persistence periods require engineering review for the actual installation.
Also distinguish a process anomaly from a confirmed leak. A pressure change alone does not establish a leak; interpretation may require synchronized measurements, operational context and an appropriate detection method.
7. Attach Measurements to an Asset Model
A tag name is rarely enough for an operator.
Associate observations with their station, pipeline segment, equipment record, engineering unit and maintenance context.
That model supports useful navigation from an alert to the affected equipment and its history.
For teams implementing the application layer, Iotellect’s pipeline monitoring solution provides a low-code foundation for connecting instruments and gateways and creating dashboards, maps, alerts and operational workflows.
The deployment still needs a defined telemetry contract, validated instruments and tested application logic.
8. Test the Recovery Path
A healthy connection tests only part of the design.
Include these scenarios in integration testing:
- Deliver the same record twice.
- Deliver records out of order.
- Replay a batch after newer live observations.
- Restart a gateway and reset its sequence counter.
- Introduce invalid measurements.
- Shift a source clock.
- Exhaust the local buffer.
- Interrupt an upload before acknowledgement.
For each scenario, verify historical storage, current-state behavior and downstream side effects.
The expected result should be explicit. For example: a duplicate causes no additional business action, an older observation does not replace newer current state, and buffer overflow produces a visible data-loss event.
For deployments that also require server redundancy, review Iotellect’s fault-tolerant SCADA documentation. Include server failover in your recovery tests alongside gateway restarts and communication outages.
Frequently Asked Questions
Why Store Both Source Time and Ingestion Time?
Source time describes when the observation occurred. Ingestion time describes when the backend received it. Keeping both helps distinguish process history from delivery behavior.
Does Edge Buffering Guarantee Complete Data Recovery?
No. Recovery depends on storage capacity, persistence behavior, acknowledgement handling and the outage duration. Test the complete implementation.
Does Successful Message Delivery Mean Processing Happened Exactly Once?
No. Application retries and side effects still require explicit idempotency controls.
Is Pipeline Telemetry the Same as Leak Detection?
No. Telemetry supplies measurements. Leak detection interprets suitable data using a defined and validated method.
Top comments (1)
The bootId plus sequence pair is the part I'd underline. I'm running a small overnight monitor on two lab boxes right now, and those two fields are what let the morning report say what kind of gap it was. A new boot id after the gap means the process restarted. Same boot id with missing sequence numbers means records went missing. And if the device also keeps its own count of what it failed to send, you can split "never left the device" from "left and never arrived". Without the sequence, all three look like the same hole in the chart.
Do you also keep the gateway's own count of what it failed to send, or is the sequence gap the only signal the backend gets?