You Have Built a Machine Learning Model, Now What??

This topic feels too close to home to many of us, it does for me at least.

Almost every tutorial you will find on the Internet will help you build a very simple model to show you that Machine Learning works and how awesome it is. You followed it, finished, proud even. You used it on the few examples that they gave you.

IT WORKS!

Cool, but now what? Should I use my Jupyter Notebook to make predictions?

That doesn't feel right, does it? It only does not, as it is not very useful, like, really, you are not going to use it like that.

One of the first steps that you go through when building a machine learning model is how to preprocess your data.
This process alone raises one of the most important questions that the tutorials are not telling you:

How will I preprocess the incoming inputs (requests from clients)?
If I create another sklearn's StandardScaler(), won't be I just creating a scale based on nothing?

Spoiler: Yes you are.

How can I export to save this thing then?

Now it depends on your stack.

If you are following the most common pipeline which uses Pandas for manipulation, Scikit-learn for preprocessing and Keras or even one of sklearn's models, you can have a sklearn Pipeline().

That way you can divide your pipeline into small steps, one for each preprocessor and then use your model.

This process tends to be the easiest one for beginners, and I highly recommend it. Although, you will probably see that your pipeline can become a huge .pkl/.joblib file if you are using big datasets.

My workflow for this type of machine learning environment is something along the lines of:

## Create Pipeline
clf = Pipeline(steps=[
        ('preprocessor', preprocess), 
        ('classifier', sklearn_wrapper.KerasClassifier(keras_model, args))
])

clf.fit(train, labels)

## Save model as a different file, so keras can save everything correctly

clf.named_steps['classifier'].model.save(f'./model/{MODEL_NAME}.h5')
clf.named_steps['classifier'].model = None

## Dump the pipeline (this file is the one that can be huge)
with open(f"./new_model/{MODEL_NAME}-pipeline.pkl", "wb") as model_file:
    pickle.dump(clf, model_file, protocol = 4)

To use it, just rebuild the pipeline with pickle.load and the model with keras.load_model(), set model back, and it should work fine.

Another alternative, is to use Tensorflow for everything.
You can still have your keras model, but now you will have to preprocess everything inside the tensorflow graph. And for newcomers, I would not go this way first.

With tensorflow, you should be at least familiar with feature_columns, tf.data and tensorflow-transform. As I said, I would not go straight to this route. I rarely use it myself.

Now you have your model saved that will preprocess your data for you on the fly.

Well, how do I use it now?

Glad you asked.
You can use Google Cloud's Machine AI Platform Models or any other model hosting platform of your choice. However with preprocessing and everything you should build a custom prediction routine. Be aware of the prices, request-rate and auto-scaling options.

I personally use Docker + Flask + Gunicorn, deployed under a load balancer of your choice. Why? Simplicity. Docker and Flask make everything easy for me to edit and improve. Flask handles my requests, Docker my infrastructure, gunicorn the workers. There are several tutorials for those. I'll link one from a follow Dev.to writter Docker-Flaks-Gunicorn.

Best Practices for this architecture:

• Save your model on the cloud - I use AWS S3 and sometimes Google's Cloud Storage - download it on your docker CMD command. so every instance will have it on startup.

• Build the model before your first endpoint. Alongside with your Flask(app) instance. This will save you tons of resources, because your are not building it every request.

• Stress test. Check your input/output rate, scale as you see fit.

This is what works for me. Your situation may be different, I hope I have provided you with a good few options to solve your problem.

I will finish this article with the elephant in the room.

What about TFX?

I may not be the best person to talk about it, as I have very little experience with it. However, the fact that most of you have never heard of it proves the point that I want to make.
There's no one using or writing about it, in other words, there's no community. There's the documentation and the source-code, and that's all.
I've had the privilege to talk with one of the TFX's core devs a couple of weeks ago. They are aware that the documentation is lacking and everything is very experimental at this point.
However it has the potential to be my go to technology for machine learning pipelines. It solves most of the problems I have when building machine learning models for production. But not at this stage of development.

I will definitely keep experimenting with it and hopefully change my mind in the near future.

That's it.

---

What are your thoughts on ML Processes and pipelines?
if you are using your machine learning models in production, what is your go to stack?