AutoML Just Got a Lot More Automated with CD4AutoML and AWS Cloudformation Registry and CLI —…

AutoML Just Got a Lot More Automated with CD4AutoML and AWS Cloudformation Registry and CLI — Developer Preview

Last month I published a tutorial on automating end-to-end AutoML lifecycle with Amazon SageMaker Autopilot and Amazon Step Functions including code sample on GitHub and the architecture.

CD4AutoML with Amazon Cloudformation

Today, I would like to introduce you to my newest project, CD4AutoML via Amazon Cloudformationfor deploying managed end-to-end automated workflows for AutoML. CD4AutoML Cloudformation third-party resource type is now available in public beta — developer preview.

CD4AutoML, AWS Cloudformation and IaC

According to Gartner’s Magic Quadrant for Cloud AI Developer Services report

By 2023, 40% of development teams will be using automated machine learning services to build models that add AI capabilities to their applications, up from 2% in 2019.

This means that development teams will look to leverage DevOps principles to automate processes surrounding the development and deployment of automated machine learning models. Training automated machine learning models is one piece of the entire machine learning landscape. Deploying, monitoring, and maintaining these models make up larger parts of the landscape which might turn out to be overhead for development teams. The question then is, how might we enable development teams to use automated machine learning, remove the burden of maintaining the machine learning systems whilst embracing existing DevOps culture? Enter CD4AutoML via Amazon Cloudformation.

CD4AutoML allows you to train, deploy, and manage end-to-end automated machine workflows on managed cloud infrastructure.

Amazon Cloudformation allows you to define the desired AWS resources along with their configurations and connections in blueprint files called AWS Cloudformation templates. These templates then run within the Amazon Cloudformation console to provision the defined infrastructure.

Using CD4AutoML support for AWS Cloudformation Registry and CLI, you can, with not more than 6 lines of configuration parameters, provision an entire end-to-end AutoML system. This comes with:

1. Fully managed AutoML training infrastructure with retraining capabilities
2. Fully managed model serving infrastructure over REST API.
3. Model monitoring for model drift detection.

This means that we can now benefit from DevOps practices, deploying AutoML workflow Infrastructure as Code, IaC, when using AWS stack.

Integrate CD4AutoML with your AWS Account

As previously mentioned, deploying CD4AutoML workflow in your AWS Account is as simple as 6 configuration parameters in your AWS Cloudformation template. You are also required to have your training data in Amazon S3, CSV format and grant CD4AutoML AWS Account ID read access to your training data. Once all these are set, you are only aws cloudformation deploy command away from your managed AutoML system with CD4AutoML.

A Real Life Use Case

Let’s walk through the process so you see how easy it really is.

Grant Access to Training

{
  "Version": "2008-10-17",
  "Id": "Policy15237839393",
  "Statement": [
    {
      "Sid": "AllowCD4AutoML",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<CD4AutoML-AWS-AccountID>:role/CD4AutoMLWorkflow"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-training-data-bucket/data-path/\*"
    }
  ]
}

Setting up your workflow

Now that you have granted to your Amazon S3 Bucket containing your training data, you can deploy your CD4AutoML workflow as AWS Cloudformation resource. Your AWS Cloudformation template would look similar to the template below:

**Resources** :
**MyDirectMarketingPredictionWorkflow** :
**Type** : CD4AutoML::Workflow::Deploy
**Properties** :
**S3TrainingDataPath** : 's3://my-training-data-bucket/examples/direct-marketing/bank-additional-full.csv'
**TargetColumnName** : 'y'
**NotificationEmail** : '[e-mail]'
**WorkflowName** : 'workflow-v1'
**Schedule** : 14


**Outputs** :
**MyDirectMarketingPredictionWorkflowApi** :
**Value** : !Ref MyDirectMarketingPredictionWorkflow

The resource parameters for CD4AutoML::Workflow::Deploy resource can be found here.

Create Stack

Finally, once we have our templates defined, we can upload them to the AWS CloudFormation console, and hit ‘Create Stack’ and see all our required resources come to life for us. You can also run the AWS Cloudformation command using the AWS CLI command:

aws cloudformation deploy --stack-name <Your\_stack\_name> --template-file <path-to-template>.yaml 

Resources deployed and managed in CD4AutoML AWS Account

The Future with CD4AutoML and Cloudformation Registry and CLI

Using AWS Cloudformation templates with CD4AutoML to deploy end-to-end automated machine learning workflows help you save time, reduce errors, cut back on repetitive work, and removes complexity managing ML systems. As AutoML evolves, adopting IaC will become an integral component to enable development teams continually deploy AutoML models to production with minimal management overhead.

CD4AutoML is in active development and will continue to build features that focus on business outcomes and developer productivity with Automated Machine Learning.

To request access for CD4AutoML Developer Preview, please visit the CD4AutoML Cloudformation resource GitHub project and create an issue.