Poseidon: Monitoring a Simulated Oilfield with Kafka, Spark and the Elastic Stack

AJOUHY ABDELAH — Sat, 14 Nov 2026 07:09:47 +0000

Introduction

Oil and gas operations generate large volumes of industrial data, collected continuously from sensors, equipment and production systems across wells and facilities. Turning this data into actionable insights requires reliable ingestion, processing and analytics.

To address this challenge, I built DatAbde, my own data platform, and used it to develop Poseidon, an oil and gas project that addresses these industry challenges through a simulated oilfield.

Using Apache Kafka and Spark, Poseidon processes industrial telemetry through a Medallion-based ETL pipeline, feeding two analytical workflows: near-real-time monitoring with the Elastic Stack, and historical business reporting with Spark, Delta tables on MinIO and Metabase.

This article explores how the platform supports the monitoring and analysis of oilfield operations.

Industrial Data Ingestion and Processing

Poseidon relies on Ignition Gateway to simulate four production sites, each generating industrial telemetry every second.

Randomized operational scenarios reproduce varying production conditions and well behaviors, providing a realistic simulation environment.

Figure 1 - Production History [Ignition Designer]

Figure 2 - Remote Site Monitoring [Ignition Designer]

Figure 3 - Site A Monitoring [Ignition Designer]

Telemetry is published over MQTT, received by EMQX, and forwarded to Apache Kafka, which acts as the central ingestion layer. Apache Spark then consumes these streams and processes them through a Medallion-based ETL pipeline.

The resulting datasets feed two separate analytical workflows: near-real-time monitoring and historical business reporting, as shown in Figure 4.

Figure 4 - Poseidon Data Architecture

Near-Real-Time Operational Monitoring

Once processed by Apache Spark, operational metrics are continuously indexed in Elasticsearch, making production data available for analysis within seconds.

Kibana provides a unified view of the simulated oilfield, combining production volumes, well pressures, temperatures and flow rates.

Its dashboards let users monitor the four wells, compare their performance and track operational changes over time.

Figure 5 - Wells Monitoring [Kibana]

Figure 6 - Production Monitoring [Kibana]

Historical Production Analysis and Business Reporting

While near-real-time monitoring focuses on current operations, historical analysis provides a broader view of production performance over time.

To support this, I developed a separate Apache Spark batch workflow that reads processed data from the Gold Kafka topic and generates production and well downtime reports at hourly granularity.

These reports include production volumes, Gas-Oil Ratio (GOR), Barrels of Oil Equivalent (BOE), Water-Oil Ratio (WOR) and downtime durations.

Figure 7 - Production Report [Metabase]

Figure 8 - Well Status and Duration [Metabase]

The resulting datasets are stored as Delta tables on MinIO object storage, complementing near-real-time monitoring with historical analysis and business reporting.

Conclusion

Poseidon demonstrates the value of combining Apache Spark’s distributed processing capabilities with Elasticsearch’s search and analytical capabilities, powered by Apache Lucene. While Spark handles large-scale data processing and transformation, Elasticsearch provides fast indexing, filtering and exploration of operational data, making it particularly well suited for near-real-time monitoring.

Beyond operational monitoring and historical reporting, MinIO provides object storage for generated reports and curated datasets, making them available for future distributed machine learning and deep learning workloads.

Above all, by centralizing data from all production sites in a single platform, operators gain a unified view of their entire operation instead of monitoring each site in isolation.

Thank you for reading.