Introduction
Are you interested in machine learning and looking for ways to optimize your workflow? Look no further than Sagemaker, Amazon Web Services' (AWS) fully managed machine learning service. With Sagemaker, one can develop, train, and deploy machine learning models at scale. Impressively, the Sagemaker Pipelines help automate the highly iterative procedure of training, tuning, and deploying machine learning models at scale.
In recent months, I had the opportunity to use Sagemaker Pipelines extensively within my team. So, I decided to start a series of posts to review this service in-depth. In this first post, we are going to break down the basic elements of this service, also called Steps, and providing some real world illustrations for combining these steps elegantly.
Steps Summary
Are you ready to supercharge your machine-learning workflow with Sagemaker pipelines? With 15 different step types at your fingertips, organizing your pipeline has never been easier. Plus, these steps are grouped based on functionality for effortless recall. Keep reading for a summary of the steps below.
1. Data Processing
There are two ways to process data in SageMaker Pipelines:
- Processing: This step offers the flexibility to preprocess data using a custom script or a pre-built container. The resulting output can be utilized as an input to the Training step.
- EMR: This step enables data processing using Amazon Elastic MapReduce (EMR) clusters. EMR offers a managed Hadoop framework that allows processing of massive amounts of data quickly and efficiently.
2. Modeling
Under the Training category, the following steps are included:
Training: This step trains a machine learning model using the data that was preprocessed in the previous Processing step. You can specify the algorithm to be used, along with input/output channels and hyperparameters.
Tuning: This step tunes the hyperparameters of the model generated in the previous Training step. It tries a range of values for each hyperparameter and selects the combination of hyperparameters that yields the best performance.
AutoML: This step automatically selects the optimal machine learning algorithm and hyperparameters for a specific problem. It uses various techniques, such as feature engineering and hyperparameter tuning, to generate the most effective model.
3. Monitoring
This category includes the following steps:
ClarifyCheck: This step examines the input data to detect any instances of bias and fairness problems. It generates a report that can assist in improving the data's quality and guaranteeing that machine learning models are impartial and equitable.
QualityCheck: This step examines the results of the Training step to confirm that the model fulfills predefined quality standards. It can establish the standards based on metrics like accuracy, precision, and recall
4. Deployment
The Deployment category includes the following steps:
- Model: This step creates a model artifact that can be used for inference. The artifact includes the trained model parameters, as well as any additional files or libraries required for inference.
- CreateModel: This step creates an Amazon Sagemaker model using the model artifact generated by the Model step.
- RegisterModel: This step registers the created model with Amazon Sagemaker, which allows you to easily deploy the model to different endpoints.
- Transform: This step creates a batch or real-time endpoint to serve predictions using the registered model.
5. Control Flow
The Control Flow category includes the following steps:
- Condition: This step allows to define conditional logic in the pipeline. It can be used to branch the pipeline based on a specified condition.
- Fail: This step allows to intentionally fail the pipeline if certain conditions are met. It can be used to ensure that the pipeline stops if something unexpected happens.
6. Custom Functionalities
The Custom Functionalities category includes the following steps:
- Callback: This step enables designating a custom script to be executed during the pipeline operation. It can be utilized for executing custom actions, like dispatching notifications or running extra processing steps.
- Lambda: This step allows to execute an AWS Lambda function during pipeline operation. It can be utilized to accomplish various tasks, such as extracting metadata or transforming data.
Example Scenarios
The Sagemaker official documentation contains several great examples on using Sagemaker pipelines. We shed the light on few examples that demonstrate how one can use these steps together, effectively.
These examples show just a few of the many ways Sagemaker Pipeline Steps can be combined to create end-to-end machine learning workflows. With the flexibility and scalability of Sagemaker, the possibilities are endless!
Scenario 1: Create a Sagemaker Pipeline to Automate All the Steps from Data Preparation to Model Deployment
In Fraud Detection for Automobile Claim example, Sagemaker Pipelines is used to facilitate the collaboration within a team of data scientist, machine learning engineer, and ML Ops Engineer.
The architecture below is used to automate all the steps from data preparation to model deployment. (source)
The actual pipeline used in this demo after completion would be as shown below. (source)
The mapping between each block name and type is described as well.
| Step Number | Step Name | Step Type |
|---|---|---|
| 1 | ClaimsDataWranglerProcessingStep | Processing Step |
| 2 | CustomerDataWranglerProcessingStep | Processing Step |
| 3 | CreateDataset | Processing Step |
| 4 | XGBoostTrain | Training Step |
| 5 | ClarifyProcessor | Processing Step |
| 6 | ModelPreDeployment | CreateModel Step |
| 7 | XgboostRegisterModel | Register Model |
| 8 | DeployModel | Processing Step |
Note that this example creates many processing steps using different methods (e.g., sagemaker.processing.Processor, SKLearnProcessor). This is an extremely powerful feature for sagemaker as it enables processing data using either pre-built containers provided by Sagemaker or fully custom docker images. Similarly, this can be used with training step. While this example use pre-built XGBoost container, it is also possible to use either a built-in algorithm, extend existing container, build custom docker image from scratch.
Scenario 2: Orchestrate Jobs to Train and Evaluate Models with Amazon Sagemaker Pipelines
In this abalone age regression problem, Sagemaker Pipelines are used to conditionally approve trained models for deployment or flag an error.
The pipeline for this scenario is shown below (source)
Conclusion
In conclusion, Sagemaker pipelines offer a powerful and flexible solution for building, training, and deploying machine learning models at scale. By automating the end-to-end machine learning process, Sagemaker pipelines can save significant time and effort for data scientists, machine learning engineers, and ML Ops engineers. With a wide range of steps available, including data processing, training, monitoring, deployment, control flow, and custom functionalities, Sagemaker pipelines provide a comprehensive solution for building complex machine learning workflows. The example scenarios provided in this post demonstrate just a few of the many ways in which Sagemaker pipeline steps can be combined to create end-to-end machine learning workflows. Overall, Sagemaker pipelines are an excellent tool for teams looking to collaborate efficiently and deploy machine learning models with ease.
Stay tuned for future updates of this series to speak more about practical and up-to-date tips and tricks of this service, cost-optimization and more. Do you want me to inlcude or review specific part of the service? Let's connect and discuss.
Top comments (0)