Building an ML Pipeline for 28M Telemetry Points: Lessons Learned

Cristian Carretero Fernández — Tue, 22 Sep 2026 16:32:39 +0000

Over the past months, I have been working on an end-to-end ML pipeline for stability analysis of perovskite solar cells. The goal: process large-scale outdoor telemetry data, detect anomalies early, and predict remaining useful life — all with explainable models.

Here are the key lessons I learned building it.

The data challenge

The dataset contained 28 million high-frequency telemetry records from outdoor solar cells. Raw storage was unmanageable for iterative analysis, so the first task was building a modular 9-stage data architecture:

Disk-to-disk streaming to avoid memory overflow
Conversion to optimized Parquet format
Empirical validation of every threshold (no magic numbers)

This alone reduced processing time from hours to minutes.

Modeling approach

I combined three complementary techniques:

Unsupervised clustering (K-Medoids + PCA) to map hardware anomalies without labels.
XGBoost + survival analysis for degradation tracking and RUL forecasting (final MAE: 4.8 days).
Dual Digital Twin for early anomaly detection — one twin models expected behavior, the other flags deviations.

Why Explainable AI matters

In physical systems, a black-box prediction is not enough. I integrated SHAP values to diagnose root causes of failure. This allowed me to decouple reversible environmental effects (humidity, temperature swings) from permanent structural fatigue — a distinction that directly affects commercial viability assessment.

Surrogate decision trees were added on top of SHAP for non-technical stakeholders.

Deployment

Everything runs on a public Streamlit dashboard with integrated explainability. Stack: Python, pandas, scikit-learn, XGBoost, PyArrow, Plotly.

What I would do differently

Start with Parquet from day one. CSV is a trap at scale.
Validate thresholds empirically before automating them.
Treat explainability as a first-class requirement, not an afterthought.

If you are working on similar problems in ML for physical systems, I would love to hear your approach.

Note on data: The telemetry data analyzed in this project was provided by the ParaSol platform at the Open Solar Stability (OSS) Lab, University of Zaragoza (Spain), and shared with the University of Seville for collaborative research. The platform uses a Perovskino galvanostatic MPPT tracker and a calibrated POA reference cell. My contribution is the machine learning layer: data engineering, Digital Twin early screening, T80 survival tracking, RUL forecasting, and Explainable AI (SHAP).

About the author: Cristian Carretero Fernández — Data Scientist & ML Researcher based in Seville, Spain. Physics & Materials Engineering background, MSc Data Science at UOC.