MLOps in Action with Scalable Self-Updating Infection Spreading Prediction Pipeline

Client Profile
The client was a public-sector healthcare organization focused on regional epidemiological monitoring and preparedness. They needed an automated forecasting system to predict the spread of illness across administrative districts based on hospital-reported case data. Key priorities included low-maintenance deployment, seamless integration with existing health data pipelines, and the ability to scale across geographic units. Their use case demanded robust MLOps infrastructure to ensure consistent model retraining, evaluation, and deployment—minimizing manual oversight while maintaining high model performance.

Challenge

1. Fully Autonomous Operation Without Manual QA
The system had to function entirely without developer oversight or manual quality assurance. All stages—data ingestion, retraining, evaluation, and deployment—needed to operate reliably under automation. This required robust orchestration, fault tolerance, and safety checks to ensure stable performance in production without human intervention.

2. Unstructured and Shifting Input Data
Incoming datasets lacked column documentation and were subject to schema drift. This made it difficult to interpret features consistently over time. The pipeline had to be schema-agnostic and self-validating, capable of identifying malformed fields and filling in missing time windows without introducing model bias or corruption.

3. Zero-Downtime Model Updates
Each retraining cycle produced a potential model candidate for deployment. To avoid disrupting live inference, updated models had to be atomically swapped into production via a live REST API. This required coordinated model loading, version control, and service-level health checks to ensure uninterrupted availability.

4. Model Versioning and Trust in Retraining Outcomes
With monthly retraining in place, it was critical to avoid blindly promoting underperforming models. Each model had to be versioned and evaluated. Only demonstrably better models were deployed. All versions and metrics were logged to enable auditability and rollback.

5. Geospatial Alignment Between Training and Inference
Predictions were generated per administrative region (tract), determined by user-submitted coordinates. To avoid silent failures, coordinate-to-tract mapping logic had to be embedded identically in both training and inference pipelines. Any misalignment would have compromised the geographic accuracy of forecasts.

Solution

Model & API Architecture
The core solution used an LSTM-based time series model, trained on historical infection case data and deployed as a REST API via Flask. The model was serialized (e.g., in .h5 format) and exposed through an endpoint that accepted geographic coordinates and returned localized predictions. The API supported stateless inference for seamless integration into external systems.

Automation Pipeline
A production-grade MLOps pipeline was implemented to fully automate model lifecycle management. The system periodically ingested newly published public health data and triggered retraining of the LSTM forecasting model. Each retrained version was evaluated on MSE (loss), MAPE, MAE, and RMSE. Only models that demonstrated measurable improvements across these metrics were automatically promoted to production. The deployment process was seamless and version-controlled, enabling uninterrupted, hands-off model delivery via a stateless REST API.

Fallback and Monitoring
To ensure operational resilience, the system included real-time health checks, structured logging, and fault isolation. Each model version was archived along with its evaluation metrics, supporting performance audits and enabling automated rollback in case of degradation. These mechanisms ensured stability and reliability even during edge cases or partial failures.

Geospatial Integration
A geoprocessing module converted incoming latitude/longitude coordinates to administrative tract IDs, ensuring geographic consistency between training and live inference. This logic was embedded into both batch and real-time pipelines to eliminate misalignment.

Features

*Real-Time Forecasts *
The system delivered current epidemiological data and next-day forecasts for specific locations. When users provided geographic coordinates, the backend mapped them to the relevant administrative unit (tract) and returned the total number of cases and a next-day prediction. Forecasts were generated by the latest retrained LSTM model, ensuring that insights reflected up-to-date trends.

*Radius-Based Statistics *
In addition to point-based predictions, the system supported geographic aggregation. By defining a custom radius, users could retrieve case statistics across multiple nearby tracts. This functionality was designed for healthcare planners and institutions seeking broader situational awareness beyond a single location.

*API-Based Access *
All forecasting features were exposed via a RESTful API, enabling real-time requests and integration into external systems. The API handled geographic inputs (latitude/longitude), routed them through the backend model pipeline, and returned structured outputs in JSON. This delivery layer ensured low-latency performance and compatibility with mobile, web, and enterprise platforms.

*Versioned Forecasting *
Each forecast was served by a specific model version, with all versions tracked alongside their performance metrics (WAPE, MAPE, MAE, RMSE). This versioning ensured transparency, enabled historical comparisons, and supported rollbacks in case of model degradation. Clients could trust that every prediction reflected a validated model with auditable performance history.

Development Process

1. Initial Model Development & Deployment Setup
The project centered on developing the infrastructure to support a large-scale time series forecasting model capable of frequent, automated updates. The pipeline was designed to ingest raw epidemiological data, preprocess it flexibly, retrain the model on a regular schedule, and serve forecasts through a scalable, stateless API. Emphasis was placed on creating a robust foundation for continuous model delivery, minimizing manual maintenance while preserving high inference availability and version control.

Model and Preprocessing Context:

• Type: LSTM (Long Short-Term Memory), well-suited for capturing temporal dependencies in multi-year time series data
• Input features: Set of epidemiological indicators (e.g., confirmed cases, test counts, hospitalization rates, age distribution) per administrative tract. Geographic coordinates (latitude/longitude) were used to map user input to the appropriate tract, enabling location-based predictions.
• Training data: Spanned 2–3 years, with occasional missing windows handled via trend-based extrapolation method.
• Data challenges: No schema documentation—field meanings were reverse-engineered manually.

2. Automation & Retraining
They were built into the system to ensure it operated continuously without developer oversight. A scheduler triggered the pipeline monthly by checking a public health portal for newly published data. If new data was available, the following steps were executed automatically:

• Download the dataset
• Preprocess features (schema-agnostic logic)
• Retrain the LSTM model
• Evaluate using specified metrics
• Promote and deploy the new model if metrics improved, update current metrics; otherwise, retain the existing one

All metrics were stored in a versioned JSON file and used to validate performance over time, enabling autonomous model lifecycle management.

3. Deployment & Serving
Deployment and serving were handled through a stateless REST API built with Flask. The trained LSTM model was serialized (e.g., in .h5 format) and exposed via an endpoint that accepted geographic coordinates as input. These coordinates were processed through an embedded geospatial module to resolve the corresponding administrative tract.

The API returned both the current total disease case count and a forecast for the next day. Additionally, users could request aggregated statistics within a defined radius to obtain broader regional data. This architecture enabled seamless integration with external systems, including mobile health apps and institutional dashboards.

4. Monitoring & Reliability
The system followed production-grade MLOps practices to ensure consistent availability and resilience. Core workflows—including data ingestion, preprocessing, retraining, and evaluation—ran independently of the inference pipeline. This separation meant that failures in upstream automation didn’t impact live predictions.

Health checks monitored each service component in real time, while structured logging enabled proactive issue detection and root-cause analysis. This architecture supported continuous uptime with minimal intervention, even during partial system failures or scheduled model retraining cycles.

5. Model Versioning & Traceability
To ensure full transparency and traceability of every forecast, we implemented a robust versioning and deployment workflow using MLflow, Git, and Docker. Each trained model was logged in MLflow alongside:

• Performance metrics (WAPE, MAPE, MAE, RMSE)
• Training hyperparameters
• Git commit hash
• Data version reference

Metrics were additionally stored in JSON format to support direct comparisons during regular retraining cycles. The final model artifact (.h5 format) was stored locally or in S3 and served through a Dockerized REST API, managed by Docker Compose for reproducibility across environments.

Every prediction made in production was tied to a specific model version—enabling full auditability, reproducibility, and rollback when needed. This ensured reliable operational oversight and compliance with best practices in model governance.

Technical Highlights

• LSTM (.h5 model format)
• FAST API
• Tensorflow
• Python
• APScheduler (Advanced Python Scheduler)
• Fiona and Shapely (libraries for processing geospatial data)
• S3, Docker, MLFlow, Git

Impact

• The deployed model maintained a mean absolute percentage error (MAPE) of 5.35%, indicating strong predictive accuracy across regional forecasts.
• A mean absolute error (MAE) of ~2.5 cases per prediction meant the system could provide actionable, location-specific insights with minimal noise.
• The low RMSE (~0.066) and consistent error metrics ensured that retraining decisions were based on stable, trustable performance.

These results directly supported the fully automated retraining pipeline:

• Model updates required no human validation, thanks to reliable and interpretable metrics.
• Retrained models were promoted only when they clearly outperformed previous versions, preventing drift or silent degradation.
• The system’s version tracking, metric logging, and rollback logic functioned effectively because model outputs remained within predictable, bounded error ranges.