Building Data Pipes for Forest and Environmental Monitoring

It is turning into a data engineering problem all the time.

A forest-monitoring system should be able to integrate data from a variety of sources: sensors, weather forecasts and reports, remote sensing, LiDAR scans, and on-the-ground observations.

This creates a fascinating set of requirements related to IoT devices, geospatial processing, time series analysis, APIs, analytics, and dashboards.

Let me describe what I think about it.

Start With Your Model

It is always worthwhile to spend a few hours thinking about how your data model will be structured.

I mean, what entities do you need in your information system?

Monitoring Site

Sensor

Coordinates

Measurement Type

Timestamp

Value

Unit

Quality Flag

Field Observation

I would represent this information as a conceptual model, perhaps with an entity-relation diagram, and later translate it into code:

site_id

sensor_id

timestamp

latitude

longitude

measurement_type

value

unit

quality_flag

There is a strong reason to keep this kind of metadata alongside the actual numeric measurement of some kind. This will allow you to, say, visualize these time series in a dashboard, but with a couple of clicks, narrow the selection to a single sensor, a given area, and a specific set of parameters.

Time Series Processing

When working with sensors, measurements come in regularly in most cases. That makes storing them in a time series database a good idea if your system grows in complexity: however, for a prototype, a conventional relational database with proper indexing may suffice as well.

You also need to think about what defines "a proper" set of measurements: missing records, duplicates, outlier values, sensor errors, different unit measurements, issues with time sync and sensor calibration. All these examples show that it is never a good idea to accept any data pipeline without validating and cleaning the input.

Location Data

Geographic location plays an important role in environmental monitoring as well.

The mere act of measuring something always happens at a certain place: therefore, I think it is useful to employ a geospatial database and some GIS software to combine your monitoring data with natural boundaries and objects (forests, rivers, parks, administrative divisions, etc.).

This allows you to perform more interesting spatial queries: show me all the areas where the reading in the last x days indicated decreasing levels of moisture, for instance.

Data Ingestion and APIs

Your system may have to acquire data in a variety of ways: a set of sensors may provide an API, a field employee may use a mobile dashboard to input some observations, and remote sensing or LiDAR may give another data stream. A good architecture will separate these data sources at the lowest level possible:

Field Sensors /Apps /Rem. Sensing

↓

Data Ingestion

↓

Validation / Cleaning

↓

Storage / Data Lake

↓

Processing / Analytics

↓

API / Dashboard

↓

Environmental

↓

Insight

This way, any single data source may be replaced by another without rewriting the entire system.

Data Quality Needs Attention

It goes without saying that the quality of data will play a deciding role in how useful your analytical tools are.

However, there is one pitfall specific to environmental / forest monitoring: sensors can malfunction in unexpected ways. A faulty temperature sensor, for example, would generate completely unrealistic data: but this is a sensor error, and this error needs to be detected and flagged somehow.

Therefore, it is a good idea to supplement this data lake or warehouse with information about the data quality itself. I suggest that any measurement will have a flag field with this kind of enums:

VALID

SUSPECT

MISSING

CALIBRATION_REQUIRED

Then, the actual analytical tools and dashboards may process these data sets and decide which flag means what: for example, it is common to mark outliers as SUSPECT, and do some statistical analysis on them, or show them in the dashboard as red points.

What About AI?

Artificial Intelligence, especially neural networks, is often useful when dealing with data lakes of any kind: time series, images, and other types of data. Environmental monitoring is no exception: it may be worthwhile to consider AI methods like anomaly detection or pattern recognition to find suspicious data points or correlations between parameters.

However, I think it is a mistake to rely on AI methods when data quality and context are unknown: after all, some parameters have nontrivial relationships that cannot be directly used for training or inference, or simply unknown.

Dashboards Are Worth While

At the end of the day, your environmental monitoring and management system should be able to answer real-world questions.

In particular, it should be possible to formulate ad-hoc questions and get answers faster than without data science. For example, a good dashboard may have some built-in features to highlight suspicious data points, trends, and anomalies in your time series data: it may group similar-looking measurements, show the latest values, point out missing records, and other bells and whistles.

For forest monitoring companies, Enviro Forest is the ideal platform for getting forest management insights. They provide cutting-edge solutions in forestry, soil, hydrology and environmental monitoring.

What Is an Interesting Problem to a Developer?

IoT data collection and processing, in general, and environmental monitoring, in particular, is an exciting topic to discuss.

It touches upon many interesting data engineering and machine learning topics, such as time series processing, data cleaning, geospatial processing, analytical model building, and data presentation in dashboards.

The problem is vast and multifaceted, but the most challenging part is always to unite all these factors into a functioning whole and keep the quality of any particular data stream at an acceptable level.