I built an ML retraining pipeline that's allowed to say no

harshi606 — Mon, 03 Aug 2026 19:59:44 +0000

Here's a thing that bugged me while I was learning MLOps: almost every tutorial ends the same way. Train a model, print the accuracy, done. Cool, but... then what? What happens in three weeks when new data shows up? Does the model just get retrained and shoved into production because a cron job said so? What if the new one is actually worse?

Nobody seemed to want to answer that part. So I built something that does.

I come from a backend/software engineering background, not classical ML, so honestly the model itself was the least interesting part of this to me. I used a plain Random Forest. I'm not trying to win a Kaggle competition. What I actually wanted to figure out was: how do you build a system that can look at a freshly trained model and say "no, not good enough" — and mean it?

The actual problem

Say you've got a churn prediction model in production. A month goes by, you've got new customer data, and you retrain. Now you have a new model. Is it better? Maybe! Maybe not. Maybe it just overfit to some weird pocket of last month's data. If you don't have a real answer to "is it better," you shouldn't be letting it anywhere near production.

So I built a pipeline with an actual gate in it. A promotion gate. The new model has to prove itself before it's allowed to replace what's currently live.

How the gate works

Two conditions, both have to pass:

The new model has to clear a minimum bar on its own (I used F1 ≥ 0.70). Doesn't matter what's in production — if it's bad, it's rejected, full stop.
It has to beat whatever's currently in production, by a real margin. Not tie. Not "basically the same." I set the bar at 0.5% absolute F1 improvement.

That second one matters more than it sounds like it should. If you promote on literally any improvement, even 0.0001, you get this annoying situation where production flips back and forth based on random noise instead of anything meaningful. Requiring an actual margin stops that.

python
def promote_if_better(candidate_version, candidate_metrics, model_name):
candidate_score = candidate_metrics["f1"]

if candidate_score < MIN_ACCEPTABLE_F1:
    return {"promoted": False, "reason": "below the floor, not even close"}

prod_version, prod_metrics = get_current_production_metrics(model_name)

if prod_version is None:
    # nothing to compare against, this is the first one
    set_alias(model_name, "production", candidate_version)
    return {"promoted": True}

improvement = candidate_score - prod_metrics["f1"]

if improvement >= PROMOTION_MIN_IMPROVEMENT:
    set_alias(model_name, "previous-production", prod_version)  # keep it, just in case
    set_alias(model_name, "production", candidate_version)
    return {"promoted": True, "improvement": improvement}

return {"promoted": False, "improvement": improvement}

That previous-production alias is doing quiet but important work. Before I let a new model take over, I tag the old one so I can find it again. That's what makes rollback a single function call instead of me digging through run history at 2am trying to remember which version was good.

I didn't just trust it, I tried to break it

Writing the logic is one thing. Believing it actually works is another. So before I let myself trust it, I ran it through a few scenarios by hand:

First model ever, nothing to compare to → should just promote. It did.
A model that ties with production → should get rejected. It did.
A model that's genuinely 3% better → should promote, and the old one should stick around for rollback → it did, and rollback actually worked.

Only after all three checked out did I go write proper integration tests for it — four of them, running against a real (but temporary) MLflow store so they're testing actual behavior, not just asserting that some constant is greater than zero.

python
def test_weak_candidate_is_rejected(mlflow_tracking):
v1 = register_and_promote(f1=0.80)
v2 = register_candidate(f1=0.801) # barely ties

decision = promote_if_better(v2, {"f1": 0.801})

assert decision["promoted"] is False
# and production should still be v1, unchanged

Genuinely satisfying watching that test suite go green after knowing I'd already proven the logic by hand first.

Why the data has to drift

If you're not simulating some kind of change over time, there's no real reason to have a retraining pipeline at all — you'd just train once and stop. So I generated a synthetic churn dataset (sklearn's make_classification, nothing fancy) and then made a second "new batch" of data with a slight shift applied to it, like customer behavior nudging over time. That's the whole justification for the pipeline existing in the first place.

The part nobody puts in tutorials: it broke

Midway through wiring up Prefect to orchestrate all this, I got a error that looked like this:

Cool, cool, very helpful. Turns out the pipeline had actually finished — trained the model, registered it, made the promotion decision, all correctly — and then crashed while Prefect was cleaning up afterward. A version mismatch between Prefect and one of its dependencies, anyio. Prefect wanted anyio somewhere between 4.4.0 and 5.0.0, and whatever version pip grabbed inside that range had changed something internally that Prefect's code didn't expect.

Pinned it down to anyio==4.4.0 and it went away.

I'm including this because I think it's actually the most "real" part of the whole project. Nobody warns you that half of building these pipelines is just fighting with dependency versions that don't agree with each other. Reading a traceback carefully enough to realize "wait, this isn't my code, my code already finished" is its own skill, and it's one you only get by actually hitting the wall.

What I think this actually proves

Not "I can train a model." Anyone can train a model. What I think this shows instead:

I can build a system that's allowed to say no, not just one that says yes to everything
Every model version is traceable back to the exact run that produced it — no mystery models
Rollback isn't theoretical, I actually tested it
The whole thing runs as one command, not a pile of notebook cells I run in a specific order and hope I remember correctly
I can debug real infrastructure problems, not just tune hyperparameters

What's next

This pipeline is good at deciding which model deserves to be in production. It doesn't actually serve anything yet. So next up: wrapping the promoted model in a real API, with some actual monitoring on it, so I can watch it in the wild instead of just trusting it blindly. That's the other half of this — one system decides what's good enough, the other one keeps an eye on it once it's out there.

Code's here: https://github.com/harshi606/churn-retraining-pipeline

If you've built something similar, especially around the "how do I actually trust an automated promotion" problem, I'd genuinely like to hear how you approached it.

Deploying streamlit apps for free !

harshi606 — Fri, 23 Jul 2021 04:37:17 +0000

Most of the machine learning enthusiasts would be familiar with this framework.

Streamlit is an open-source,free framework using which you can build data webapps in a less amount of time.And you ask me whats so special about it !? Well, it is the fact that you can develop the webapps without any frontend knowledge !!! Sounds cool ! right ?

It has a very user-friendly interface using which you can turn your data scripts into cool apps within minutes . It is all written in python and you just have type these magical words to install streamlit .

pip install streamlit

or if pip isn't working then you can try this

conda install -c conda-forge streamlit

Thats it ! You have installed streamlit. Since this post is dedicated to streamlit deploy, I am gonna restrict myself here and soon, I will make a post on streamlit for beginners.

Now that you have built an app, You can run it only on your local host. Imagine, you are sitting for interviews and you have mentioned about your project in the resume. It will be really cool to attach a working link of your project which can be run anywhere and at anytime without depending on your local host ! Wondering how you can do that !?

We have many free and paid deployment platforms like heroku, where deployment is very easy. You also have github IO where you can deploy your static webpages.

Streamlit sharing is one such platform where you can make your webapps sharable. It is a perfect choice if your project is hosted public in github.

I will list out the steps you need to follow in order to deploy your app.

First, you have to request an invite from the streamlit sharing over here.
I got my invite acceptance after 5 days and hopefully it will be the case for everyone.Check your mail(spam folder too) and once you are accepted, You are ready to deploy.
Now, push your projects into your github repository(You need to be familiar with github !) and dont forget to add requirements.txt file same as you do for heroku.Make sure your repository is public.

4.Visit the streamlit share link here:click
Now you can see the sigin page.

5.Sigin either using your email id or using your github account.I prefer github.

After you sign-in, You can see like this

7.click on new app -> From existing repo.

Now you will be directed to this page.. Here my github is pasted by default. You can even paste the desired url here.
After you click deploy, then you can see a screen like this.It will take some time to deploy.
If the app compiled without errors, It will be shown like this.. I showed the demo with a sample app available.

Try this out with the sample app to get the taste of streamlit by following these steps:

Click new app -> from sample app template.
You will see a message window like this.

Click on Get Authorization and fork sample app.

3.You need to authorize to your github account. After that you can see a window similar to this.

Click deploy and you can see the app compiling and finally running.

Note: You can look and monitor your deployed apps from the dashboard.

We have come to the end.I guess this article went too long and it is really a big one for my first article here !

If you find any mistake or discrepancies do comment it and I'll rectify it straightaway. I am a learner and newbie for both technical articles as well as software world. So drop in a heart if you found this useful . Happy Reading !

DEV Community: harshi606

I built an ML retraining pipeline that's allowed to say no

Deploying streamlit apps for free !

Most of the machine learning enthusiasts would be familiar with this framework.