Understanding the AI/ML Flow From Data to Deployment

Bimochan raj kunwar — Tue, 30 Dec 2025 15:16:57 +0000

The MLOps Philosophy: A Structured Approach
In professional circles, the entire lifecycle of an ML project is often referred to as MLOps (Machine Learning Operations). It's a discipline that streamlines the process from experimentation to production, ensuring reliability and scalability. We can break this down into three distinct, interconnected "zones."

The Data & Training Zone: The "Laboratory" This is where the intelligence is forged. Before a model can be "smart," it needs to learn from vast amounts of information. You correctly identified that a database plays a crucial role here, but it's more than just a storage unit; it's the raw material.

Data Collection & Ingestion: The very first step involves gathering raw data. This could be from traditional SQL or NoSQL databases, cloud storage buckets, real-time streams, or third-party APIs. Think of it as sourcing your ingredients.

Data Preprocessing & Feature Engineering: Raw data is often messy. This crucial phase involves:

Cleaning: Handling missing values, correcting errors, and removing duplicates.

Transformation: Converting data into a format suitable for algorithms (e.g., scaling numbers, encoding categorical variables).

Feature Engineering: This is an art form! It involves creating new, more informative features from existing ones to help the model learn better. An example might be combining day_of_week and hour_of_day to create a time_of_day_category.

Model Training: With clean, well-engineered data, an algorithm (like a neural network, decision tree, or regression model) "studies" these patterns. It adjusts its internal parameters to minimize errors between its predictions and the actual outcomes in the training data.

The Model Artifact: The output of a successful training run isn't a live entity; it's a static file. This file, often called a "model artifact" (e.g., a .pkl for scikit-learn, .h5 for Keras, or .onnx for optimized runtime), encapsulates all the learned intelligence. This is the "brain" ready for deployment.

A visual representation of data flowing through a cleaning and training pipeline, culminating in a model artifact file.

The Serving Zone: The "API Bridge" You've built a brilliant "brain" (your model artifact). Now, how do other applications communicate with it? This is where the API (Application Programming Interface) comes in. The model isn't created in the API; rather, the API acts as a universal translator and gateway to your model.

Model Hosting & Loading: The first step is to load your model artifact into a dedicated server or a cloud-based ML serving platform. Frameworks like FastAPI, Flask, or more specialized tools like TensorFlow Serving or TorchServe are commonly used here.

API Endpoint Creation: The serving layer exposes a specific URL (an "endpoint"), for example, https://api.yourapp.com/predict or https://your-ml-service.cloud.com/sentiment. This is the address other applications will use to send data and receive predictions.

Inference: When an external application sends new data to this API endpoint, the serving layer performs the magic:

It receives the input data (e.g., text, image, numbers).

It preprocesses this incoming data in the same way the training data was processed (crucial for consistency!).

It passes the processed data to the loaded model artifact.

The model makes a prediction (this is called "inference").

The serving layer formats the prediction (often as JSON) and sends it back as the API response.

A diagram illustrating an API gateway receiving a request, forwarding it to a loaded model, and sending back a JSON response.

The Application Zone: The "User Interface" This is where the magic becomes tangible for the end-user. This zone leverages the served ML model within a broader application context.

Backend (The Application Logic): The backend of your application is responsible for orchestrating much more than just the ML model. It handles:

CRUD Operations: Managing user accounts, storing user-specific data, and logging model predictions for auditing or feedback.

Business Logic: Implementing rules specific to your application (e.g., "if the prediction is X, then do Y").

Security: Authentication, authorization, and data encryption.

Integration: Communicating with the ML API, other internal services, and external APIs.

quoted text: The backend basically handles how the model serves its object. It acts as the brain for the entire application, deciding when and how to call the ML API.

Frontend (The User Experience): This is what your users interact with—the mobile app, the web interface, or a desktop application.

It collects user input (e.g., text for sentiment analysis, an image for object detection).

It sends this input to your backend (which, in turn, might call the ML API).

It receives the model's prediction (via the backend) and displays it in a user-friendly manner.

An illustration showing a user interacting with a mobile app (frontend), which communicates with a backend, and that backend then calls the ML API to get a prediction.

Rewritten Summary for Your Notes
For a concise, professional summary to use in discussions or on your resume, consider this:

"The AI/ML flow starts with Data Engineering to prepare raw data. This leads into Model Development, where an algorithm is trained to produce a 'Model Artifact.' This artifact is then Deployed within an API (the Serving Layer) to enable real-time predictions. Finally, the Application Layer (comprising a Backend for business logic and a Frontend for user interaction) consumes this API to deliver an intelligent experience to the end-user, while managing data persistence and security."

Key Takeaway: Separation of Concerns
The most critical refinement to your initial understanding is the separation between the model's training phase and its deployment/serving phase. You train a model once (or periodically), save it, and then load that saved model into an API to serve countless requests. This distinction is vital for scalability, maintainability, and efficient resource utilization.

Understanding this flow is your superpower. While specific programming languages, cloud providers, and machine learning frameworks will change, the fundamental stages of collecting data, training models, serving predictions, and integrating into applications will remain constant.

Google Scholar. for University Scholar Day-1

Bimochan raj kunwar — Mon, 29 Dec 2025 14:39:10 +0000

I’m trying to build a similar Google Scholar for my university. Do you have any ideas?
This is my college project. We are assigned to build a pretty basic search engine using a crawler like Selenium. Then, I thought to myself, “Why stop there? Making a generic project won’t make me shine. So, I started researching.”

What is a search engine, and why is it important? One thing leads to another, and I found out there’s another name for a search engine now. That’s what Perplexity is basically. Then, I heard the CEO of Perplexity in Lex. He talked about how Google’s search engine isn’t its primary source of revenue. It also earns YouTube alone 100 billion dollars annually. Back to search engines, I went off-topic. My aim is also to build a billion-dollar company someday. LOL.

So, search engines let us go back to where vector search naive theory and different algorithms were used previously. Now, Google uses an algorithm with BM25 or BM22, which I need to check on.

I had an idea: how about I build a search engine and answer engine, naming it Sonic? Sonic crawlers will search all the time and rate the webpage. This is an added layer for better ranking. The crawlers must be pretty unbiased, must be.

However, I started building this Sonic search engine. It did get “AI” and “all the help I can get.” Then I realized that if I continued doing this, I have this tendency of overloading my ideas so much that I can’t carry myself and drop it and forget about it. The same thing was going to happen, so I only thought about creating an assignment-worthy project for now and later adding features. This way, I can make it exist and make it better later.

I have this highly distracted yet incredibly curious mind. I believe it’s only me now. I’m learning the basics of algorithms, mathematics, and programming that are necessary to complete this project.

Project Name: University Scholar

The project will be available on GitHub after I finish it. I’ll post it here daily.

Tech Stack:

Frontend: Next.js
Backend: FastAPI (most of the AI stuff happens here)
Data Indexing: Elasticsearch
AI/NLP: I want to add an answering feature as well.
Crawler: Selenium

I’ve completed the frontend using Next.js. I’ll be updating it daily. I’m having issues with Docker and FastAPI.

DEV Community: Bimochan raj kunwar

Understanding the AI/ML Flow From Data to Deployment

Google Scholar. for University Scholar Day-1