DEV Community

Cover image for Training Machine Learning models in AzureML
Vivek0712
Vivek0712

Posted on

4 2

Training Machine Learning models in AzureML

Welcome to the next part of my blog series on Azure Machine Learning. Now we are going to jump into one of the most important aspect of the MLOps - Training of Machine Learning Models.

Usually Machine Learning models require high compute power to train on the dataset. It is quite evident that not everyone has access to high processing and storage capabilities. So, keeping in mind of the cost of procuring the infrastructure for the same, its better option to leverage the cloud resources and pay for the usage.

Now, coming to training of Machine Learning models in Azure, we are going to follow simple steps to do so. At the end of the article, you will have created a training environment, trained your ML model and stored the artefacts of the models for future use.

Steps

  1. Setup the environment
  2. Creating your ML Script
  3. Create Training Environment dependency file
  4. Training the model

Setup the Environment for ML

Follow the previous article to create Workspace, Compute and Data.

Creating your ML Script

Before creating the ML Script, lets first create the directories for source files

import os
# Create a folder for the experiment files
experiment_folder = 'ml-series-diabetes-exp'
os.makedirs(experiment_folder, exist_ok=True)
print(experiment_folder, 'folder created'
view raw sourcefiles.py hosted with ❤ by GitHub

The ML Script does the following

  • Parsess the Argument passed to the script (Input Data and Hyperparameters)
  • Starts the run for the experiment
  • Does all the ML Training
  • Logs the metrics of the run
  • Stores the output model
%%writefile $experiment_folder/diabetes_training.py
# Import libraries
import os
import argparse
from azureml.core import Run, Dataset
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
from azureml.core import Model
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
import joblib
import math
# Get the script arguments (regularization rate and training dataset ID)
parser = argparse.ArgumentParser()
parser.add_argument("--input-data", type=str, dest='training_dataset_id', help='training dataset')
parser.add_argument("--alpha", type=str, dest='alpha', help='Alpha Value')
args = parser.parse_args()
alpha = args.alpha
inputdata = args.training_dataset_id
# Get the experiment run context
run = Run.get_context()
ws = run.experiment.workspace
# get the input dataset by ID
dataset = Dataset.get_by_id(ws, id=inputdata)
# Get the training dataset
print("Loading Data...")
x_df = dataset.to_pandas_dataframe().dropna()
y_df = x_df.pop("Y")
X_train, X_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=66)
run.log("alpha_value", alpha)
model = Ridge(alpha=alpha)
model.fit(X=X_train, y=y_train)
y_pred = model.predict(X=X_test)
rmse = math.sqrt(mean_squared_error(y_true=y_test, y_pred=y_pred))
run.log("rmse", rmse)
model_name = "model_alpha_" + str(alpha) + ".pkl"
filename = "outputs/" + model_name
joblib.dump(value=model, filename=filename)
run.upload_file(name=model_name, path_or_stream=filename)
run.complete()

Creating Training Environment Dependency File

An Environment is managed and versioned in an Azure Machine Learning Workspace. You can update an existing environment and retrieve a version to reuse. Environments are exclusive to the workspace they are created in and can't be used across different workspaces.

We will use YAML file to list the dependencies of the environment we are going to use

%%writefile environment.yml
name: simple_environment
dependencies:
# The python interpreter version.
# Currently Azure ML only supports 3.5.2 and later.
- python=3.6.2
- scikit-learn
- pandas
- pip
- pip:
- azureml-defaults
- azureml-mlflow
- azure-ml-api-sdk
- scikit-learn
- azureml-core
- pickle5
- numpy
view raw environment.yml hosted with ❤ by GitHub

Training the Machine learning Model

  • Create the environment from the dependency file
  • Creating Configuration for Python Script
  • Submitting the Script to the Experiment
  • Register the model along with the RSME Value

We are going to create 10 different runs for 10 different Alpha values for our Ridge Algorithm to be trained on the Diabetes Dataset. The model with less RMSE will be considered as best model.

A ScriptRunConfig packages together the configuration information needed to submit a run in Azure ML, including the script, compute target, environment, and any distributed job-specific configs.

A model is the result of a Azure Machine learning training Run or some other model training process outside of Azure.Register a model with the provided workspace, we use register method.

from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.widgets import RunDetails
# Create a Python environment for the experiment (from a .yml file)
env = Environment.from_conda_specification("experiment_env", "environment.yml")
# Create a Python environment for the experiment (pip installing requirements.txt)
# env = Environment.from_pip_requirements("experiment_env", "requirements.txt", pip_version=None)
dataset = Dataset.get_by_name(ws, name='diabetes dataset')
alphas = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
for alpha in alphas:
# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder,
script='diabetes_training.py',
arguments=['--input-data', dataset.as_named_input('diabetes_data'), '--alpha',alpha],
compute_target=cpu_cluster,
environment=env)
# submit the experiment run
experiment_name = 'mlseries-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion()
model_name = "model_alpha_" + str(alpha) + ".pkl"
filename = "outputs/" + model_name
run.download_file(filename,filename)
# Register the model
run.register_model(model_path=filename, model_name='diabetes_model',
tags={'Training context':'Script'},
properties={"Alpha": run.get_metrics()['alpha_value'], "rmse" : run.get_metrics()['rmse'] })
# Show the running experiment run in the notebook widget
# Block until the experiment run has completed

We have successfully created and trained our Machine Learning model. In next article, we will see how to deploy our model to production and perform real-time inference.

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (2)

Collapse
 
aatmaj profile image
Aatmaj

That is just awesome!

Collapse
 
vivek0712 profile image
Vivek0712

Thank you.