DEV Community: Ernest Kabahima

Guide to Scalable ML Pipelines

Ernest Kabahima — Fri, 31 Jul 2026 17:48:08 +0000

Guide to Scalable ML Pipelines

Machine learning projects often start small: a notebook, a CSV file, and a model that works well enough on one dataset. The trouble begins when the project needs to grow into something repeatable, reliable, and production-ready. A scalable ML pipeline solves that problem by turning one-off experiments into a system that can ingest data, validate it, train models, package them, deploy them, and monitor them over time. papers.ssrn

In this article, we will build that idea step by step. We will start with the core concepts of scalable ML pipelines, then move into practical code labs that show how to build each stage in Python. By the end, you will have a clear picture of how to move from a simple local training script to a more production-friendly machine learning workflow. github

What is an ML pipeline?

An ML pipeline is the sequence of steps that takes raw data and produces a usable machine learning model. In a scalable setup, the pipeline is modular so that each stage can be developed, tested, and run independently instead of being buried inside one large script. That modular design makes the system easier to maintain, easier to debug, and easier to scale when data volume or team size increases. papers.ssrn

A typical pipeline includes:

data ingestion,
validation and preprocessing,
feature engineering,
training and tuning,
evaluation and approval,
packaging and deployment,
monitoring and retraining. medium

The most important thing to understand is that ML in production is not just about model accuracy. It is also about repeatability, traceability, reliability, and operational safety. papers.ssrn

Why scalability matters

A pipeline becomes “scalable” when it can handle growing data, more frequent training, more users, and more production complexity without breaking down. That does not always mean huge distributed systems or expensive infrastructure. Sometimes scalability just means the pipeline is organized well enough to grow gradually without being rewritten from scratch. medium

A scalable ML pipeline helps you:

reproduce training runs,
version data and model artifacts,
automate repetitive tasks,
reduce human error,
detect issues after deployment,
support future growth. papers.ssrn

If you skip pipeline design early on, you usually end up with notebooks that cannot be rerun, manual deployment steps, and no way to tell whether the model in production is still behaving correctly. papers.ssrn

Core design principles

A strong ML pipeline is built on a few principles.

Modularity

Each stage should do one thing well. Data ingestion should not also train the model. Training should not also manage deployment. This separation makes the workflow easier to test and replace. papers.ssrn

Reproducibility

You should be able to rerun the pipeline and get the same or nearly the same results using the same code, data version, and dependencies. Reproducibility is one of the biggest reasons to introduce version control, data tracking, and experiment logging. papers.ssrn

Automation

The pipeline should run with as little manual effort as possible. That may mean scheduled jobs, triggers from new data, or CI/CD workflows that test and deploy automatically. github

Observability

Once the model is live, you need visibility into performance, latency, errors, and drift. Monitoring is what keeps a “working” model from slowly becoming a bad one. papers.ssrn

Governance

For serious production use, you also need approvals, lineage, and audit trails. This matters for regulated industries, internal accountability, and debugging model behavior later. papers.ssrn

Code Lab 1: Build a local training pipeline

The easiest place to start is a single Python script that loads data, validates it, trains a model, and saves the result. This first lab gives you a baseline pipeline you can improve later. vishaluttammane.medium

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import joblib

df = pd.read_csv("data.csv")

assert "target" in df.columns, "Missing target column"
assert not df.isnull().values.any(), "Missing values detected"

X = df.drop(columns=["target"])
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

preds = model.predict(X_test)
print("accuracy:", accuracy_score(y_test, preds))

joblib.dump(model, "model.pkl")

This example is simple on purpose. It shows the basic shape of an ML pipeline without hiding the logic behind too many abstractions. You can already see the beginning of a scalable design: data validation, train/test splitting, model training, evaluation, and artifact saving. vishaluttammane.medium

Code Lab 2: Add preprocessing

In many real projects, raw data is not ready for modeling. Missing values, scale differences, and mixed feature types require preprocessing. A scalable pipeline should handle these transformations in a repeatable way so training and inference use the same logic. medium

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer

numeric_features = X.select_dtypes(include=["int64", "float64"]).columns

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

preprocessor = ColumnTransformer(
    transformers=[("num", numeric_transformer, numeric_features)]
)

pipeline = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("model", RandomForestClassifier(random_state=42)),
])

pipeline.fit(X_train, y_train)
preds = pipeline.predict(X_test)
print("accuracy:", accuracy_score(y_test, preds))

This version is better because preprocessing is no longer a one-off step hidden inside the training code. It is part of the pipeline itself, which makes the whole workflow easier to reuse and maintain. labellerr

Code Lab 3: Track experiments

A scalable ML team needs to know what happened in each training run. Which data version was used? What accuracy did the model achieve? Which hyperparameters were changed? Without experiment tracking, it becomes very hard to compare models or reproduce results. papers.ssrn

import json
from datetime import datetime

metrics = {
    "accuracy": float(accuracy_score(y_test, preds)),
    "timestamp": datetime.utcnow().isoformat()
}

with open("metrics.json", "w") as f:
    json.dump(metrics, f, indent=2)

This is a simple version of experiment tracking, but it teaches the right idea: every run should leave behind a record. In larger systems, tools such as MLflow can store runs, parameters, artifacts, and metrics more systematically. papers.ssrn

Code Lab 4: Package the model for serving

Training is only half the story. At some point, the model has to make predictions for real users or systems. The usual pattern is to expose the model behind an API so other software can call it. vishaluttammane.medium

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI()
model = joblib.load("model.pkl")

class InputData(BaseModel):
    features: list[float]

@app.post("/predict")
def predict(data: InputData):
    arr = np.array([data.features])
    prediction = model.predict(arr)
    return {"prediction": prediction.tolist()}

This lab introduces the separation between training and serving. That separation is one of the most important ideas in scalable ML architecture because it allows models to be updated independently of the applications that consume them. medium

Code Lab 5: Add GitHub Actions CI

Once your pipeline works locally, the next step is automation. GitHub Actions can run your tests every time code is pushed so broken changes do not move further down the pipeline. github

name: ml-ci
on:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: pytest

This is the beginning of CI/CD for ML. At first, it may only run unit tests. Over time, you can expand it to include data validation, training checks, model packaging, and deployment steps. github

Code Lab 6: Dockerize the pipeline

Containerization makes the pipeline easier to move between environments. A Docker image packages your Python runtime, dependencies, and application code into a consistent unit that can run on a laptop, server, or cloud platform. codezup

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"]

Docker is especially helpful when you want the same environment for local testing, CI, and production. It reduces the “works on my machine” problem and gives you a cleaner deployment unit. codezup

Deployment and monitoring

Once the model is deployed, the work is not finished. In production, the pipeline should monitor latency, errors, input drift, and prediction drift. When those metrics change, you may need to retrain, rollback, or investigate the data source. papers.ssrn

Useful monitoring signals include:

API latency,
failure rate,
feature distribution drift,
prediction drift,
business outcome changes. papers.ssrn

If your model is not monitored, it may quietly degrade while still appearing to work. That is one of the biggest operational risks in machine learning systems. papers.ssrn

Best practices for scalable pipelines

A strong pipeline usually follows a few habits:

keep steps modular,
version data and code,
automate tests and validation,
log experiments and metrics,
deploy through controlled release steps,
monitor after deployment,
retrain only when needed. papers.ssrn

Another important practice is to start small. Many teams try to build a fully distributed MLOps platform too early. In practice, it is often better to create a simple, reliable pipeline first and add complexity only when the project truly needs it. medium

Common mistakes

The most common mistakes are surprisingly simple:

training logic locked inside notebooks,
no dataset versioning,
manual model promotion,
no experiment tracking,
no production monitoring,
too much infrastructure too soon. papers.ssrn

These mistakes make models hard to reproduce and hard to trust. Scalability is not only about hardware; it is about building a system that stays understandable as it grows. papers.ssrn

Conclusion

A scalable ML pipeline is a structured way to move from raw data to a live, monitored model without losing control of the process. The best way to build one is to start with a simple local workflow, then add preprocessing, tracking, automation, containerization, deployment, and monitoring in stages. papers.ssrn

If you treat each stage as a small, testable piece, you will end up with a pipeline that is much easier to maintain and much easier to scale. That is the real goal of MLOps: not just building models, but building systems that can keep delivering them reliably over time. papers.ssrn

# Automating Python Deployments to a VPS with GitHub Actions

Ernest Kabahima — Fri, 31 Jul 2026 17:36:13 +0000

If you are learning deployment for the first time, the best path is to start simple: run your Python app on a VPS, keep it alive with systemd, put Nginx in front of it, and only then automate the whole process with GitHub Actions. This two-part approach makes CI/CD much easier to understand because you first learn how deployment works by hand, then you learn how to automate it. onehost

Introduction

Deployment is the process of moving your application from development into a live environment where other people can access it. For a beginner, that usually means a VPS, a reverse proxy like Nginx, and a process manager such as systemd to keep the app running reliably. onehost

This article is split into two parts:

Part 1: Set up the VPS, run the Python app, configure Nginx, and keep the system live.
Part 2: Add GitHub Actions to automate testing and deployment.

Part 1: VPS and Nginx Lab

The goal in Part 1 is to get a working production-style setup without automation first. Once you understand this manual setup, CI/CD in Part 2 will make a lot more sense.

1. Choose your Python app

Start with a simple Python web app using Flask, FastAPI, or Django. Keep the app small so you can focus on deployment rather than application complexity, and make sure it has a health endpoint such as /health so you can test whether it is alive after deployment.

A minimal app should:

respond to web requests,
read environment variables,
have clearly defined dependencies,
expose a health check route.

2. Prepare the VPS

Use an Ubuntu VPS for the simplest beginner-friendly setup. Connect with SSH, install Python and Git, and create a dedicated user for deployment rather than using the root account.

A basic VPS setup usually includes:

installing Python,
installing git,
creating a virtual environment,
opening only the ports you need,
configuring SSH access securely.

3. Keep the app live with systemd

If you run python app.py manually in a terminal, the app stops when you disconnect or reboot the server. systemd fixes this by running the app as a service, starting it automatically on boot, and restarting it if it crashes. onehost

A typical systemd service gives you:

automatic start on boot,
automatic restart on failure,
centralized logs,
easier service management with systemctl.

This is one of the most important beginner deployment lessons because it turns a fragile script into a real service.

4. Put Nginx in front

Nginx acts as a reverse proxy: it receives traffic from the internet and forwards requests to your Python app running on localhost. This is the standard way to expose a web app on a VPS because it keeps the app private while Nginx handles public traffic on ports 80 and 443. space-node

Why use Nginx?

It handles domain routing.
It can terminate HTTPS.
It can proxy traffic cleanly to your app.
It keeps your Python app off the public internet.

In practice, your app listens on something like 127.0.0.1:8000, and Nginx listens on your public domain or IP address.

5. Test the manual setup

Before moving to automation, make sure the manual setup is stable.

Check that:

the app starts correctly,
the app survives a restart,
Nginx routes requests properly,
the health endpoint works,
logs are readable through journalctl or your app logs.

At this point, you have a real production-style deployment, even though it is not automated yet.

Part 2: CI/CD Lab

Now that the app is running manually, you can automate the process with GitHub Actions. The goal here is to make every push to GitHub test the app and then deploy it to the VPS safely. advantch

1. Understand GitHub Actions

GitHub Actions is GitHub’s automation system. It runs workflows on events such as pushes, pull requests, or manual triggers, and those workflows can install dependencies, run tests, and deploy code.

A workflow usually has:

a trigger,
one or more jobs,
steps inside each job.

2. Add build and test steps

Start automation with testing, not deployment. Your workflow should install Python dependencies and run tests so bad code never reaches the server.

Typical CI steps:

checkout the repository,
set up Python,
install dependencies,
run unit tests,
run lint checks if you have them.

If tests fail, deployment should stop immediately.

3. Deploy to the VPS

Once testing passes, GitHub Actions can connect to your VPS over SSH and deploy the new code. The workflow can either pull the latest commit on the server or send a build artifact, depending on how your app is structured.

A simple deployment flow looks like this:

Push code to main.
GitHub Actions runs tests.
If tests pass, it SSHs into the VPS.
The server updates the code.
systemd restarts the app.
Nginx continues serving traffic.

This pattern is easy to understand and works well for small to medium Python apps.

4. Store secrets safely

Never hardcode passwords or private keys in your repository. Use GitHub Secrets for SSH keys, server addresses, and other sensitive values. This keeps your pipeline secure and prevents credentials from leaking into source control.

Good secret management means:

no private keys in Git,
no passwords in code,
minimal access on the VPS,
clear separation between app config and source code.

5. Keep the system live during deploys

A good deployment does not just copy files; it keeps the service available as much as possible. systemd helps by restarting the app after an update, and Nginx keeps the public side stable while the backend is refreshed.

For beginners, the simplest practical target is:

short downtime or near-zero downtime,
a fast restart,
a health check after deployment,
logs ready for troubleshooting.

6. Add production safety

Once deployment works, make it safer.

Add:

rollback steps,
health checks,
deployment logs,
branch protection,
staging before production if possible.

The goal is not just “it deploys,” but “it deploys reliably.”

Crossroads: Docker, Terraform, and Kubernetes

After you understand the VPS workflow, you reach a crossroads where you decide whether to stay simple or move into more advanced deployment tooling. This is where Docker, Terraform, and Kubernetes become relevant. developer.hashicorp

Docker

Docker packages your app and dependencies into a container, making the runtime more consistent across machines. This helps reduce environment drift and makes the app easier to move between local development, testing, and production.

Use Docker when:

environment consistency matters,
you want portable builds,
your app depends on multiple services.

Terraform

Terraform lets you define infrastructure as code, so you can provision and manage infrastructure with configuration files instead of clicking through a cloud console. It is useful when you want your VPS, networking, and cloud resources to be reproducible and version-controlled. developer.hashicorp

Use Terraform when:

infrastructure changes often,
you want repeatable cloud provisioning,
your team needs auditable infrastructure changes.

Kubernetes

Kubernetes is an orchestration platform for running containerized apps at scale. It is powerful, but it also adds a lot of complexity, so it is usually not the best starting point for a beginner with a single Python app and one VPS. docs.docker

Use Kubernetes when:

you have multiple services,
you need auto-scaling or advanced orchestration,
you are ready for the operational overhead.

Other tools

Other useful tools include:

Docker Compose for multi-container apps on one server,
Ansible for server configuration,
cloud-init for server bootstrap automation,
managed cloud platforms when you want less server maintenance.

Choosing the right path

For most beginners, the best path is:

Build the Python app.
Deploy it to a VPS.
Run it with systemd.
Put Nginx in front.
Add GitHub Actions for CI/CD.
Only then explore Docker, Terraform, or Kubernetes.

This order works because each step builds on the one before it. You learn the fundamentals first, then you add tooling only when you have a real need.

Conclusion

The simplest reliable deployment system is often the best one to start with. For a beginner, Python + VPS + systemd + Nginx gives you the foundation, and GitHub Actions adds automation once you understand the manual process.

# 🛠️ Fixing “No Valid Android SDK Root Found” Error in WSL (Ubuntu)

Ernest Kabahima — Thu, 13 Nov 2025 17:05:35 +0000

When developing Android or Ionic apps inside WSL (Windows Subsystem for Linux), you might hit this frustrating error:

[ERROR] Error while getting native targets for android: No valid Android SDK root found.

This usually happens when running:

npm run dev:android
ionic cap run android
ionic cordova run android

Don’t worry — it’s a common issue when working across Windows and WSL.
Here’s what causes it and how to fix it cleanly.

Why This Happens

There are two main culprits behind this error:

1. WSL Can’t See the Android SDK

Your Android SDK is installed on Windows, not inside WSL.
Typically, it lives here:

C:\Users\<Your-Username>\AppData\Local\Android\Sdk

In WSL, that same folder appears under:

/mnt/c/Users/<Your-Username>/AppData/Local/Android/Sdk

However, if your Windows username contains a space (like FIRST NAME), Linux treats it as two separate paths unless you quote it properly.
That’s why the Android build tools inside WSL can’t find your SDK.

2. Missing Android Command-Line Tools

Even if the SDK path is correct, you might still lack the essential Android tools (sdkmanager, avdmanager, etc.) that WSL needs to manage devices and builds.
Without these, the SDK can’t be validated — hence the “no valid SDK root” message.

Step-by-Step Fix

Follow these steps in your WSL terminal 👇

1. Set Correct Android SDK Paths

Run these commands (replace your username if needed):

echo 'export ANDROID_HOME="/mnt/c/Users/Ernest Kabahima/AppData/Local/Android/Sdk"' >> ~/.bashrc
echo 'export ANDROID_SDK_ROOT="$ANDROID_HOME"' >> ~/.bashrc
echo 'export PATH="$PATH:$ANDROID_HOME/emulator:$ANDROID_HOME/platform-tools:$ANDROID_HOME/cmdline-tools/latest/bin"' >> ~/.bashrc
source ~/.bashrc

Confirm the path:

echo $ANDROID_HOME

You should see:

/mnt/c/Users/Ernest Kabahima/AppData/Local/Android/Sdk

2. Install Android Command-Line Tools

If sdkmanager isn’t recognized, install it with:

sudo apt update
sudo apt install google-android-cmdline-tools-11.0-installer -y

This installs the latest command-line tools inside /usr/lib/android-sdk/.

3. Link WSL Tools to Your Windows SDK

Make sure your Windows SDK has the proper structure:

mkdir -p "/mnt/c/Users/Ernest Kabahima/AppData/Local/Android/Sdk/cmdline-tools/latest"
sudo ln -s /usr/lib/android-sdk/cmdline-tools/* "/mnt/c/Users/Ernest Kabahima/AppData/Local/Android/Sdk/cmdline-tools/latest/"

Now WSL and Windows share the same SDK tools.

4. Accept SDK Licenses

This ensures the tools can run properly:

yes | sdkmanager --licenses

5. Verify Your Setup

Check versions and SDK content:

adb version
sdkmanager --list | head -20

If both work, your SDK is ready.

6. Run Your Project Again

Now re-run your dev command:

npm run dev:android

If you still get a native-run error, use:

npm run dev:android -- --no-native-run

That tells the system to fall back to Cordova’s runner.

Quick Recap

Problem	Cause	Fix
No valid Android SDK root	WSL can’t find Windows SDK	Use quoted path `/mnt/c/...`
sdkmanager not found	Missing SDK tools	`sudo apt install google-android-cmdline-tools-11.0-installer`
native-run errors	Emulator or device not found	Start emulator or use `--no-native-run`

Bonus Tip: Avoid Spaces in SDK Paths

If possible, move your Android SDK to a path without spaces, such as:

C:\Android\Sdk

Then update your environment variables accordingly.
This avoids quoting issues entirely and makes WSL integration smoother.

Conclusion

This issue boils down to two things:

WSL not interpreting Windows paths with spaces
Missing command-line tools for Android builds

By setting your environment variables properly and installing the SDK tools, you can run and build Android apps inside WSL just like a native Linux environment — no more “No valid Android SDK root” headaches.

"Exploring Top MLOps Platforms for Efficient Machine Learning Lifecycle Management"

Ernest Kabahima — Fri, 31 Jan 2025 14:50:58 +0000

These platforms help with automating and streamlining various stages of the machine learning lifecycle, from development to deployment and monitoring, and many integrate with popular cloud services like AWS, Google Cloud, and Azure. The choice of platform depends on specific needs such as scalability, integration with existing tools, and the complexity of the machine learning operations.

There are several MLOps platforms available, each catering to different aspects of machine learning lifecycle management, such as model development, deployment, monitoring, and governance. Here are some of the popular MLOps platforms:

Kubeflow

An open-source platform designed to facilitate the development, deployment, and monitoring of machine learning models on Kubernetes.
Key Features:

Supports end-to-end pipelines.
Model training, serving, and monitoring.
Built on Kubernetes for scalability.

MLflow

An open-source platform for managing the complete machine learning lifecycle, including experimentation, reproducibility, and deployment.
Key Features:

Experiment tracking.
Model versioning and packaging.

Model deployment.

TensorFlow Extended (TFX)

An end-to-end platform for deploying production machine learning pipelines using TensorFlow.
Key Features:

Model training, evaluation, and deployment.
Data validation and transformation.
Model monitoring and versioning.

Seldon

A platform focused on deploying, monitoring, and managing machine learning models at scale.
Key Features:

Model deployment and scaling.
Model monitoring and explainability.
Integrates with Kubernetes and supports many model types.

Amazon SageMaker

A fully managed service by AWS that provides an integrated environment for developing, training, and deploying machine learning models.
Key Features:

Managed Jupyter notebooks.
Automated hyperparameter tuning.
End-to-end deployment and monitoring.

Azure ML

A cloud-based platform from Microsoft Azure for building, training, and deploying machine learning models.
Key Features:

Experiment tracking.
Automated ML and hyperparameter tuning.
Model deployment and monitoring.

Google AI Platform

A cloud-based machine learning platform by Google that helps with building and deploying models at scale.
Key Features:

Integration with TensorFlow and other ML frameworks.
Model deployment and monitoring.
Automated machine learning.

Databricks

A cloud platform that provides collaborative notebooks and unified analytics for data engineering and machine learning.
Key Features:

Managed Spark clusters for scalable ML.
Collaborative notebooks for team development.
Integration with MLflow for managing models.

Comet.ml

An experiment tracking platform for machine learning, focused on collaboration, model versioning, and visualization.
Key Features:

Tracking and visualizing experiments.
Model and dataset versioning.
Collaboration for teams.

Weights & Biases (W&B)

A platform for tracking experiments, visualizing results, and managing datasets and models.
Key Features:

Experiment tracking and visualization.
Model versioning.
Collaborative team features.

DataRobot

An enterprise AI platform that automates machine learning and helps with model deployment and monitoring.
Key Features:

Automated model training and selection.
Model monitoring and deployment.
Interpretability and explainability features.

MLflow

A popular open-source platform for managing the machine learning lifecycle, especially for tracking experiments and managing models.
Key Features:

Experiment tracking.
Model packaging and versioning.

Model deployment and serving.

Flyte

An open-source workflow orchestration platform that supports building, deploying, and managing ML workflows.

Key Features:

Supports batch and streaming data workflows.
Scalable orchestration and execution.
Built-in integrations with ML frameworks like TensorFlow and PyTorch.

Pachyderm

An open-source data versioning platform designed for machine learning workflows.
Key Features:

Data lineage tracking.
Scalable ML pipelines.
Version-controlled data for reproducibility.

Polyaxon

A platform for building, training, and deploying machine learning and deep learning models.
Key Features:
Experiment tracking and model versioning.
Scalable deployment.
Kubernetes-native architecture.

How to Safeguard Against Disruptions from Data Schema Changes

Ernest Kabahima — Fri, 31 Jan 2025 14:00:41 +0000

Data schema changes are inevitable as applications grow and evolve. However, if not handled properly, they can lead to broken functionality, data loss, or downtime. To prevent these disruptions, developers must follow a structured approach that ensures seamless transitions while maintaining data integrity. Below, we discuss key strategies to safeguard against disruptions caused by schema changes.

Use Migrations Instead of Direct Changes

Making direct changes to a live database, such as adding or removing columns manually, can be risky. Instead, developers should use database migration tools that provide a structured way to apply and track schema changes. Some popular migration tools include:

Alembic (for SQLAlchemy/Python)
Flyway (for various databases including PostgreSQL and MySQL)
Liquibase (for version-controlled database changes)
Knex.js (for Node.js applications)

These tools allow for version control, rollback options, and a safer way to update the database without disrupting the application.

Make Schema Changes Backward-Compatible

Ensuring backward compatibility is crucial when updating a database, especially in production environments. Backward-compatible changes allow existing functionalities to continue working while transitioning to the new schema. Some key practices include:

Adding new fields without removing old ones immediately to prevent breaking queries.
Using default values when adding new columns to avoid unexpected NULL values.
Gradually phasing out deprecated fields only after all application parts are updated.
Implement API and Database Versioning
Versioning ensures different parts of the system remain compatible during schema transitions.
Database versioning: Use version numbers or timestamps for database migrations to track and roll back changes if needed.
API versioning: Create new API versions (e.g., /v2/products) instead of modifying existing ones immediately.
Grace periods: Allow clients time to adapt before deprecating older versions.
Automate Testing & Schema Validation
Testing prevents unexpected errors before rolling out schema changes.
Schema validation tools (e.g., JSON Schema) ensure data adheres to the expected format.
Automated database tests in CI/CD pipelines detect issues before deployment.
Load testing helps assess performance impact from schema modifications.

Maintain Database Backups & Audits

Despite careful planning, things can go wrong. Regular backups and audit logs help mitigate risks.Perform full backups before applying schema modifications for quick recovery.

Use point-in-time recovery for databases that support it. Maintain an audit trail to track schema changes and modifications.

Implement Feature Flags for Schema-Dependent Features : Feature flags allow gradual rollout of new database features instead of immediate deployment.

Use feature toggles to enable or disable schema-dependent features dynamically.

Test schema updates in a staging environment before full deployment.

Roll back changes easily by disabling the feature flag rather than modifying the database directly.

Document Schema Changes Clearly

A lack of documentation can lead to confusion and errors when working with evolving schemas. To ensure team members understand the changes:

Maintain a schema changelog recording each modification, its purpose, and impact.

Update API and database documentation to reflect new structures.

Communicate changes with stakeholders to avoid misalignment.

Test Schema Changes in a Staging Environment

Before deploying schema changes to production, always test them in a staging environment that mirrors production.

Replicate real-world data scenarios to assess potential impacts.

Ensure application compatibility with the new schema before final deployment.

Monitor query performance to detect slow queries introduced by schema modifications.

Overcoming Resistance to Change in Database Scalability

Resistance to change is common when implementing database scalability improvements. However, ensuring a smooth transition requires effective strategies:

Communicate the vision: Clearly articulate the benefits of scalable databases to all stakeholders.

Provide training: Offer comprehensive support and education on the new systems and processes.

Showcase quick wins: Demonstrate immediate improvements to motivate and build confidence in the changes.

By addressing concerns early and involving stakeholders in the process, resistance can be minimized, and scalability efforts can succeed without unnecessary delays.

Final Thoughts

Database schema changes are necessary for evolving applications, but they must be managed carefully to prevent disruptions. By using migrations, ensuring backward compatibility, automating testing, maintaining backups, implementing feature flags, and documenting changes, developers can minimize risks and ensure smooth transitions. Proper planning and a structured approach help safeguard against potential issues, ensuring that schema changes do not negatively impact the system or its users. Additionally, addressing resistance to change through effective communication, training, and showcasing quick wins will ensure scalability improvements are adopted smoothly.

Simple Event Registration API with MongoDB and Flask

Ernest Kabahima — Thu, 28 Nov 2019 06:30:17 +0000

let's start

Simple Event Registration API is a demonstration of how to create, read and write data with MongoDB using Flask

MongoDB

MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need. Its document model is simple for developers to learn and use, while still providing all the capabilities needed to meet the most complex requirements at any scale but You can get started here

Use cases for MongoDB include the following;

To install MongoDB Click here if not installed

Flask

Flask is a micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries.

If you are new to python and Flask You can get started here with the following articles
first, get to understand python secondly start playing with Flask

Postman

This is a tool that makes the game of building API's fun, its a must-have before even thinking of building an API. Postman is currently one of the most popular tools used in API testing. It started in 2012 as a side project by Abhinav Asthana to simplify API workflow in testing and development.

API stands for Application Programming Interface which allows software applications to communicate with each other via API calls.

More about APIs Click Here

Set up

Install python on your machine if not installed you can check by using the following command python3 --version
check if the virtual environment is installed if not install it Linux for Windows

1.Create Virtual environment

python3 -m venv venv

venv - means the name of the virtual environment it can be anything
Read More

2.Activate Virtual environment

source venv/bin/activate or . venv/bin/activate

2.Installing the following

Flask
pip install flask
Pymongo
pip install pymongo
check using pip list

Create a file called app.py with the following command Touch app.py
paste the following code

from flask import Flask, jsonify, request
from flask_pymongo import PyMongo

app = Flask(__name__)
app.config["MONGO_URI"] = "mongodb://localhost:27017/regdb"
mongo = PyMongo(app)


@app.route('/', methods = ['GET'])
def get_all_attendees():
"""To get all the items from the database.its a Get request and we use the databasename.find()"""
    attendees = mongo.db.attendees
    output = []

    for r in attendees.find():
        output.append({
                    'name':r['name'],
            'address':r['address'],
                    'Phone_Number':r['p_number']
                    })
    return jsonify({'result' : output})

@app.route('/attendee/<name>', methods = ['GET'])
def get_one_attendee(name):
"""To get one item from the database.its a Get request and We use the databasename.find_one"""
    attendees = mongo.db.attendees
    attendees.find_one({'name':name})

    output.append({
                    'name':r['name'],
            'address':r['address'],
                    'Phone_Number':r['p_number']
                    })
    return jsonify({'result' : output})

@app.route('/register', methods = ['POST'])
def register():
"""To add an item to the database. its a post request and we use databasename.insert()"""
    attendees = mongo.db.attendees
    name = request.json['name']
    language=request.json['language']
    reg_id = attendees.insert({
                    'name':r['name'],
            'address':r['address'],
                    'Phone_Number':r['p_number']
                    })
    r = attendees.find_one({'_id':reg_id})
        output = ({
                    'name':r['name'],
            'address':r['address'],
                    'Phone_Number':r['p_number']
                    })
        return jsonify({'result' : output})

if __name__== '__main__':
    app.run(debug=True)

Go ahead and test your API with Postman

Thanks, Ernest

DEV Community: Ernest Kabahima

Guide to Scalable ML Pipelines

Guide to Scalable ML Pipelines

What is an ML pipeline?

Why scalability matters

Core design principles

Modularity

Reproducibility

Automation

Observability

Governance

Code Lab 1: Build a local training pipeline

Code Lab 2: Add preprocessing

Code Lab 3: Track experiments

Code Lab 4: Package the model for serving

Code Lab 5: Add GitHub Actions CI

Code Lab 6: Dockerize the pipeline

Deployment and monitoring

Best practices for scalable pipelines

Common mistakes

Conclusion

# Automating Python Deployments to a VPS with GitHub Actions

Introduction

Part 1: VPS and Nginx Lab

1. Choose your Python app

2. Prepare the VPS

3. Keep the app live with systemd

4. Put Nginx in front

5. Test the manual setup

Part 2: CI/CD Lab

1. Understand GitHub Actions

2. Add build and test steps

3. Deploy to the VPS

4. Store secrets safely

5. Keep the system live during deploys

6. Add production safety

Crossroads: Docker, Terraform, and Kubernetes

Docker

Terraform

Kubernetes

Other tools

Choosing the right path

Conclusion

# 🛠️ Fixing “No Valid Android SDK Root Found” Error in WSL (Ubuntu)

Why This Happens

1. WSL Can’t See the Android SDK

2. Missing Android Command-Line Tools

Step-by-Step Fix

1. Set Correct Android SDK Paths

2. Install Android Command-Line Tools

3. Link WSL Tools to Your Windows SDK

4. Accept SDK Licenses

5. Verify Your Setup

6. Run Your Project Again

Quick Recap

Bonus Tip: Avoid Spaces in SDK Paths

Conclusion

"Exploring Top MLOps Platforms for Efficient Machine Learning Lifecycle Management"

Kubeflow

MLflow

TensorFlow Extended (TFX)

Seldon

Amazon SageMaker

Azure ML

Google AI Platform

Databricks

Weights & Biases (W&B)

DataRobot

MLflow

Flyte

Pachyderm

Polyaxon

How to Safeguard Against Disruptions from Data Schema Changes

Use Migrations Instead of Direct Changes

Make Schema Changes Backward-Compatible

Maintain Database Backups & Audits

Document Schema Changes Clearly

Simple Event Registration API with MongoDB and Flask

let's start

MongoDB