Welcome to the next part of my blog series on Azure Machine Learning. Now we are going to jump into one of the most important aspect of the MLOps - Training of Machine Learning Models.
Usually Machine Learning models require high compute power to train on the dataset. It is quite evident that not everyone has access to high processing and storage capabilities. So, keeping in mind of the cost of procuring the infrastructure for the same, its better option to leverage the cloud resources and pay for the usage.
Now, coming to training of Machine Learning models in Azure, we are going to follow simple steps to do so. At the end of the article, you will have created a training environment, trained your ML model and stored the artefacts of the models for future use.
Steps
- Setup the environment
- Creating your ML Script
- Create Training Environment dependency file
- Training the model
Setup the Environment for ML
Follow the previous article to create Workspace, Compute and Data.
Creating your ML Script
Before creating the ML Script, lets first create the directories for source files
import os | |
# Create a folder for the experiment files | |
experiment_folder = 'ml-series-diabetes-exp' | |
os.makedirs(experiment_folder, exist_ok=True) | |
print(experiment_folder, 'folder created' |
The ML Script does the following
- Parsess the Argument passed to the script (Input Data and Hyperparameters)
- Starts the run for the experiment
- Does all the ML Training
- Logs the metrics of the run
- Stores the output model
%%writefile $experiment_folder/diabetes_training.py | |
# Import libraries | |
import os | |
import argparse | |
from azureml.core import Run, Dataset | |
import pandas as pd | |
import numpy as np | |
from sklearn.model_selection import train_test_split | |
from sklearn.linear_model import LogisticRegression, LinearRegression | |
from sklearn.metrics import roc_auc_score | |
from sklearn.metrics import roc_curve | |
from azureml.core import Model | |
from sklearn.linear_model import Ridge | |
from sklearn.metrics import mean_squared_error | |
import joblib | |
import math | |
# Get the script arguments (regularization rate and training dataset ID) | |
parser = argparse.ArgumentParser() | |
parser.add_argument("--input-data", type=str, dest='training_dataset_id', help='training dataset') | |
parser.add_argument("--alpha", type=str, dest='alpha', help='Alpha Value') | |
args = parser.parse_args() | |
alpha = args.alpha | |
inputdata = args.training_dataset_id | |
# Get the experiment run context | |
run = Run.get_context() | |
ws = run.experiment.workspace | |
# get the input dataset by ID | |
dataset = Dataset.get_by_id(ws, id=inputdata) | |
# Get the training dataset | |
print("Loading Data...") | |
x_df = dataset.to_pandas_dataframe().dropna() | |
y_df = x_df.pop("Y") | |
X_train, X_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=66) | |
run.log("alpha_value", alpha) | |
model = Ridge(alpha=alpha) | |
model.fit(X=X_train, y=y_train) | |
y_pred = model.predict(X=X_test) | |
rmse = math.sqrt(mean_squared_error(y_true=y_test, y_pred=y_pred)) | |
run.log("rmse", rmse) | |
model_name = "model_alpha_" + str(alpha) + ".pkl" | |
filename = "outputs/" + model_name | |
joblib.dump(value=model, filename=filename) | |
run.upload_file(name=model_name, path_or_stream=filename) | |
run.complete() | |
Creating Training Environment Dependency File
An Environment is managed and versioned in an Azure Machine Learning Workspace. You can update an existing environment and retrieve a version to reuse. Environments are exclusive to the workspace they are created in and can't be used across different workspaces.
We will use YAML file to list the dependencies of the environment we are going to use
%%writefile environment.yml | |
name: simple_environment | |
dependencies: | |
# The python interpreter version. | |
# Currently Azure ML only supports 3.5.2 and later. | |
- python=3.6.2 | |
- scikit-learn | |
- pandas | |
- pip | |
- pip: | |
- azureml-defaults | |
- azureml-mlflow | |
- azure-ml-api-sdk | |
- scikit-learn | |
- azureml-core | |
- pickle5 | |
- numpy |
Training the Machine learning Model
- Create the environment from the dependency file
- Creating Configuration for Python Script
- Submitting the Script to the Experiment
- Register the model along with the RSME Value
We are going to create 10 different runs for 10 different Alpha values for our Ridge Algorithm to be trained on the Diabetes Dataset. The model with less RMSE will be considered as best model.
A ScriptRunConfig packages together the configuration information needed to submit a run in Azure ML, including the script, compute target, environment, and any distributed job-specific configs.
A model is the result of a Azure Machine learning training Run or some other model training process outside of Azure.Register a model with the provided workspace, we use register method.
from azureml.core import Experiment, ScriptRunConfig, Environment | |
from azureml.widgets import RunDetails | |
# Create a Python environment for the experiment (from a .yml file) | |
env = Environment.from_conda_specification("experiment_env", "environment.yml") | |
# Create a Python environment for the experiment (pip installing requirements.txt) | |
# env = Environment.from_pip_requirements("experiment_env", "requirements.txt", pip_version=None) | |
dataset = Dataset.get_by_name(ws, name='diabetes dataset') | |
alphas = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] | |
for alpha in alphas: | |
# Create a script config | |
script_config = ScriptRunConfig(source_directory=experiment_folder, | |
script='diabetes_training.py', | |
arguments=['--input-data', dataset.as_named_input('diabetes_data'), '--alpha',alpha], | |
compute_target=cpu_cluster, | |
environment=env) | |
# submit the experiment run | |
experiment_name = 'mlseries-train-diabetes' | |
experiment = Experiment(workspace=ws, name=experiment_name) | |
run = experiment.submit(config=script_config) | |
RunDetails(run).show() | |
run.wait_for_completion() | |
model_name = "model_alpha_" + str(alpha) + ".pkl" | |
filename = "outputs/" + model_name | |
run.download_file(filename,filename) | |
# Register the model | |
run.register_model(model_path=filename, model_name='diabetes_model', | |
tags={'Training context':'Script'}, | |
properties={"Alpha": run.get_metrics()['alpha_value'], "rmse" : run.get_metrics()['rmse'] }) | |
# Show the running experiment run in the notebook widget | |
# Block until the experiment run has completed |
We have successfully created and trained our Machine Learning model. In next article, we will see how to deploy our model to production and perform real-time inference.
Top comments (2)
That is just awesome!
Thank you.