DEV Community: Anirudh Mehta

Data Acquisition & Exploration: Exploring 5 Key MLOps Questions using AWS SageMaker

Anirudh Mehta — Mon, 21 Aug 2023 05:32:18 +0000

The ’31 Questions that Shape Fortune 500 ML Strategy’ highlighted key questions to assess the maturity of an ML system.

A robust ML platform offers managed solutions to easily address these aspects. In this blog, I will walk through AWS SageMaker's capabilities in addressing these questions.

What?

An MLOps workflow consists of a series of steps from data acquisition and feature engineering to training and deployment. As such, instead of covering all aspects in a single blog, we will focus on key questions surrounding Data Acquisition & Exploration (EDA).

▢ [Automation] Does the existing platform helps the data scientist to quickly analyze, visualize the data and automatically detect common issues
▢ [Automation] Does the existing platform allows integrating and visualizing the relationship between datasets from multiple sources?
▢ [Collaboration] How can multiple data scientists collaborate in real-time on the same dataset?
▢ [Reproducibility] How do you track and manage different versions of acquired datasets
▢ [Governance & Compliance] How do you ensure that the data privacy or security considerations have been addressed during the acquisition

Use Case & Dataset

The questions can be best answered in a context of a use case. For this series, we will consider “Fraud Detection” as a use case with very simple rules:

Any transaction above 500 amount is considered fraud
Any transaction outside the user’s billing address is considered fraud
Any transaction outside the normal hours is considered fraud

The following script generates customer & transaction datasets with occasional fraudulent events.

# Script to generate the transactions dataset
import numpy as np
import pandas as pd
from datetime import datetime, timedelta
import random

np.random.seed(123)

# Define count for the sample dataset
n_customers = 10
n_transactions = 100000

# Define dictionary for sample dataset
states = ['CA', 'NY', 'TX']
cities = ['Los Angeles', 'New York', 'Dallas']
streets = ["Main St", "Oak St", "Pine St", "Maple Ave", "Elm St", "Cedar St"]
zips = [10001, 10002, 90001, 90002, 33101, 33102, 75201, 75202, 60601, 60602]

# Generate customers
customer_df = pd.DataFrame({
    'customer_id': range(n_customers),
    'state': np.random.choice(states, n_customers),
    'city': np.random.choice(cities, n_customers),
    'street': np.random.choice(streets, n_customers),
    'zip': np.random.choice(zips, n_customers)
})
customer_states = dict(zip(customer_df['customer_id'], customer_df["state"]))

# Generate transactions
transaction_df = pd.DataFrame({
    'transaction_id': np.random.choice([random.randint(100000000, 999999999) for i in range(1000)], n_transactions),
    'customer_id': np.random.choice(range(n_customers), n_transactions),
    'amount': [random.uniform(0, 500) if random.random() < 0.9 else random.randint(500, 1000) for i in range(n_transactions)],
    'transaction_time': np.random.choice([datetime(2023, 4, 25, 22, 15, 16) - timedelta(days=random.randint(0, 30), hours=random.randint(0, 12), minutes=random.randint(0, 60)) for i in range(n_transactions)], n_transactions)
})

# Set transaction state to customer state 
transaction_df['transaction_state'] = [customer_states[x] if random.random() < 0.9 else np.random.choice(states) for x in transaction_df['customer_id']]

# Mark transaction as fraud if an outlier
transaction_df['fraud'] = transaction_df.apply(lambda x: random.random() < 0.1 or x['amount'] > 500 or x['transaction_time'].hour < 10 or x['transaction_time'].hour > 22 or x['transaction_state'] != customer_states[x['customer_id']], axis=1)

print(f"Not fraud: {str(transaction_df['fraud'].value_counts()[0])} \nFraud: {str(transaction_df['fraud'].value_counts()[1])}")

customer_df.to_csv("customers.csv", index=False)
transaction_df.to_csv("transactions.csv", index=False)

For real-world data, you can refer Kaggle Credit Card Fraud Dataset.

💡 AWS offers a fully managed service for customized fraud detection — Amazon Fraud Detector.

How?

I have tried to structure the article to be easily readable. However, to truly understand SageMaker’s capabilities, I highly recommend taking a hands-on approach.

For this series, I will be using **SageMaker Studio**, a fully managed ML & MLOps IDE. AWS also offers SageMaker Studio Lab, a free Jupyter-based IDE environment.

💡 AWS SageMaker is offered as part of the free tier for the first 2 months with various sub-limits. I will include the sub-limits where applicable.

Organize with SageMaker Domain

[✓] [Governance & Compliance] How do you ensure that the data privacy or security considerations have been addressed during the acquisition

In an enterprise, multiple models are often developed simultaneously. These models are based on different datasets and algorithms and are often managed by different teams. Effective organization and controlled access are critical for efficient management, as well as ensuring overall data privacy and security.

SageMaker provides the concept of domains to organize ML resources such as notebooks, experiments, and models, and to manage access to them.

Create a domain

Creating a domain in Amazon SageMaker is a quick and straightforward process. The console offers two workflows: Quick Setup (1 min) and Standard Setup (10 min). The latter allows for additional security configurations, such as authentication, encryption, and VPC configuration.

Manage access - User profiles

Amazon SageMaker allows the creation of different profiles based on either custom or predefined personas, such as data scientists, MLOps engineers, or compute.
These profiles enable an organization to effectively manage permissions & govern access across the platform:

Control access to resources such as the SageMaker canvas or a particular bucket.
Control user activities like creating ML jobs or publishing models.

Explore with SageMaker Data Wrangler

**[✓] [Automation] *Does the existing platform helps the data scientist to quickly analyze, visualize the data and automatically detect common issues?*

In any data science exercise, the first step is to make sense of the data and identify correlations, patterns, and outliers. The success of a model depends on the quality of the dataset, making this a crucial step.

SageMaker Data Wrangler simplifies and accelerates this process.

⚠️ The Data Wrangler’s free tier provides only 25 hours of ml.m5.4xlarge instances per month for 2 months. Additionally, there are associated costs for reading and writing to S3.

Importing the data

Data Wrangler supports importing data from various sources, such as Amazon S3, Redshift, Snowflake, and more. For this article, I have already uploaded the customer and transaction datasets generated previously to Amazon S3. I have also granted SageMaker’s user profile access to this bucket.

Automated Analysis

Out of the box, Data Wrangler automatically identifies the data types of various columns within the uploaded data. Additionally, Data Wrangler offers built-in capabilities such as data quality and insights reports.

Let’s run it against our target column - “fraud”, and review the insights it automatically generates.

Dataset Statistics: Statistical summaries of the dataset — feature count, count of valid and invalid records, and feature type distribution. It found 6 features and no duplicate or invalid records.
Target Column Distribution: Understand any imbalances in the dataset.
Feature Summary: Predictive power of individual features. As expected, the features — amount, time, and state play the most important role.
Feature Distribution: Distribution of individual features w.r.t the target label.
Quick Model: Predicts how good a trained model on this dataset might be.
Confusion Matrix: Performance of the quick model to detect fraud or not.

[✓] [Automation] Does the existing platform allows integrating and visualizing the relationship between datasets from multiple sources?

SageMaker Wrangler enables data scientists to quickly join two datasets and visualize them together. In this particular case, we are joining customer data with transaction data. Once joined, data scientists can run the automated analysis of the combined data in a similar manner.

We will explore this further in the next blog on “Data Transformation and Feature Engineering”.

Collaborate with SageMaker Spaces

[✓] [Collaboration] How can multiple data scientists collaborate in real-time on the same dataset?**

SageMaker Spaces enables users within a domain to collaborate and access the same resources, including notebooks, files, experiments, and models. It allows multiple users to access, edit and review the same notebooks in real time within a shared studio application.

Track with SageMaker Lineage & DVC

[✓] [Reproducibility] How do you track and manage different versions of acquired datasets

Version control is a well-known concept in the coding and development world, but it is also essential for data science activities.

DVC (Data Version Control) is a popular open-source tool designed for the same purpose. It allows you to track and manage versions of your datasets, features, and models.

DVC integrates with Git and allows data scientists to store and track references to the data stored in various locations such as Amazon S3, HTTP, or on disk.

# Register S3 remote
dvc remote add -d myremote s3://<bucket>/<optional_key>

# Track file
# This creates ".dvc" file with information necessary for tracking
dvc add data/raw.csv 
git add training.csv.dvc # Version control ".dvc" file like any other git file

# Push data file to S3
dvc push

# Pull data files from S3
dvc pull

# Switch between versions
dvc checkout

[✓] [Governance & Compliance] How do you ensure that the data privacy or security considerations have been addressed during the acquisition (cont..)

DVC enables you to track data set versions. However, we may need to track additional information for governance and understanding the usage:

Where did the raw data originate from?
Who owns or manages the raw data?
What transformations and preprocessing are being applied to the raw data?
What models use a data set?

SageMaker Lineage is capable of providing answers to several of these questions. We will explore this further in the next blog on “Data Transformation and Feature Engineering”.

Here’s a quick example of creating a raw data artifact entity capturing source, origin, and owner information.

# Create a artifact
aws sagemaker create-artifact 
--artifact-name raw-data 
--source SourceUri=s3://my_bucket/training.csv 
--artifact-type raw-data 
--properties owner=anirudh,topic=mlops,orgin=script # Additional details
--tags Key=cost_center,Value=research 

{
    "ArtifactArn": "arn:aws:sagemaker:us-east-1:removed:artifact/24c7ff167309de3b466aab30f95a8810"
}

# Describe a artifact
aws sagemaker decribe-artifact --artifact-arn arn:aws:sagemaker:us-east-1:removed:artifact/24c7ff167309de3b466aab30f95a8810

⚠️ Clean-up

If you have been following along with the hands-on exercises, make sure to clean up to avoid charges.

In summary, AWS SageMaker Wrangler greatly accelerates the complex tasks of data exploration for data scientists.

In the next article, I will explore how SageMaker can assist with Data Transformation and Feature Engineering.

Solving the Image Promotion Challenge Across Multi-Environment with ArgoCD

Anirudh Mehta — Fri, 18 Aug 2023 05:27:48 +0000

When designing cloud environments, it is often recommended to set up multiple accounts. While this approach offers resource independence, isolation, better security, access, and billing boundaries, it also comes with its own set of issues. One such challenge is efficiently promoting and tracking applications between different environments.

The GitOps approach, along with tools like ArgoCD and Kustomize, simplifies tracking and promotion. However, image promotion is often overlooked. Many enterprises adopt a shared image registry, but it soon becomes bloated with many unused versions.

This article explores a recent journey during which we examined the problem of promoting images and the innovative solution that was adopted, all while adhering to the principles of GitOps.

Challenge

Recently, a scenario was presented where a company utilizing the shared ECR registry was considering migrating to separate ECR registries for cost-effectiveness, better governance, and streamlined lifecycle management.

Here is a look at the existing state of infrastructure and pipelines:

Each environment has a dedicated AWS account with its own cluster and ArgoCD installation.
Kustomize is used for managing configuration differences across environments.

├── infra
  │   ├── charts/
  └── overlays
      ├── dev
      │   ├── patch-image.yaml
      └── production
          ├── patch-image.yaml
          └── patch-replicas.yaml

Jenkins is used to continuously build new images in the development environment.

However, none of the tools provided out-of-the-box support for promoting images between ECR registries, leading to the exploration of innovative solutions with some considerations.

Considerations:

Selective Promotion: The company’s application landscape is composed of multiple modules and teams with different timelines. Therefore, it is necessary to support the promotion of images for only selected modules in each release.
Optimized Storage: Environments such as production only need to store promoted image versions, reducing clutter and optimizing resource usage.
Image Tag and Digest Replication: Replicating image tags and digests between ECR registries is critical for security, and traceability.

Potential Solutions

At the outset, two potential solutions were proposed:

ECR Cross Account Replication: AWS’s ECR natively supports replicating images between two accounts. However, as of now, there is no way to filter the images being replicated based on any criteria. Alternatively, AWS recommends event-based design to selectively replicate images based on tag naming conventions. However, since we are not aware of which versions will be promoted, it requires an additional step of retagging before promotion.
Jenkins Promotion Pipeline: A Jenkins pipeline that parses Kustomize Overlays for image tags and programmatical replicates them.

Both options are viable, but they introduce an additional layer of complexity to the promotion process. Additionally, you need to ensure that images are promoted before Kustomize overlays are updated*.*

The Winning Strategy: ArgoCD PreSync Job

In this scenario, the client was already using ArgoCD for continuous deployment of the application changes. Therefore, we decided to also assign ArgoCD the responsibility of delivering images to the target environment cluster.

ArgoCD supports hooks that allow you to run custom scripts before or after a deployment or synchronization process.

1. ECR Repository Permission: Authorize cross-account pull access for Docker images

To enable ArgoCD to pull images from the source ECR, we need to add a resource-based policy to our repository.

// cross-account-ecr-read-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{DESTINATION_ACCOUNT}:root" // Replace with your destination account
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ]
    }
  ]
}

Apply the policy to ECR repositories:

aws ecr set-repository-policy --repository-name example 
--policy-text "file://cross-account-ecr-read-policy.json"

// For multiple repositories:
aws ecr describe-repositories --query "repositories[].[repositoryName]" 
| xargs -I {} aws ecr set-repository-policy --repository-name {} --policy-text "file://cross-account-ecr-read-policy.json"

2. PreSync Hook Job: Copy image between accounts

We use Crane to copy images without changing their tag and digest.
The PreSync Hook job is stored in git along with other application manifests and monitored by ArgoCD. ArgoCD runs the job before the synchronizing changes.
The source account is the Development or DevOps account from which the images will be pulled.
The destination account is the Production or target environment where the image needs to be copied.

// Helm template example
apiVersion: batch/v1
kind: Job
metadata:
  generateName: argo-presync-promote-image-
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      volumes:
        - name: creds
          emptyDir: {}
      initContainers:
        - name: aws-creds
          image: public.ecr.aws/aws-cli/aws-cli
          command:
            - sh
            - -c
            - |
              aws ecr get-login-password > /creds/ecr
          volumeMounts:
            - name: creds
              mountPath: /creds
      containers:
        // For brevity, I have assumed that all Helm values are available on the root.
        - name: promote-image
          image: gcr.io/go-containerregistry/crane:debug
          command:
            - sh
            - -c
            - |
              // Login to both ECR registries
              cat /creds/ecr | crane auth login {{.Values.sourceAccount}}.dkr.ecr.us-east-1.amazonaws.com -u AWS --password-stdin
              cat /creds/ecr | crane auth login {{.Values.destinationAccount}}.dkr.ecr.us-east-1.amazonaws.com -u AWS --password-stdin
              // Copy image from source account to destination account
              crane copy {{.Values.image | replace .Values.destinationAccount .Values.sourceAccount}} {{.Values.image}}
          volumeMounts:
            - name: creds
              mountPath: /creds
      restartPolicy: Never
  backoffLimit: 2

Conclusion

In conclusion, the team was able to promote images on demand by using the pre-sync hook. This made production promotion a single step of updating the Kustomize overlays.

I would love to hear about other options that you have adopted. For instance, an alternative approach could be to use Kubernetes Dynamic Admission Control to intercept and pull missing images on demand.

31 Questions that Shape Fortune 500 ML Strategy

Anirudh Mehta — Thu, 17 Aug 2023 07:00:22 +0000

In May 2021, Khalid Salama, Jarek Kazmierczak, and Donna Schut published a white paper titled “Practitioners Guide to MLOps”. The white paper goes into great depth on the concept of MLOps, its lifecycle, capabilities, and practices. There are hundreds of blogs written on the same topic. As such, my intention with this blog is not to duplicate those definitions but rather to encourage you to question and evaluate your current ML strategy.

I have listed a few critical questions that I often pose to myself and concerned stakeholders on the modernization journey. While ML algorithms & code play a crucial role in success, it’s just a small piece of the large puzzle.

To consistently achieve the same success, there are a vast array of cross-cutting concerns that need to be addressed. Thus, I have grouped the questions under different stages of an ML delivery pipeline. In no way, the questions are targeted for a particular role owning that stage, but are applicable to everyone involved in the process.

Key objectives:

Before diving into the questions, it’s important to understand the evaluation lens through which they are written. If you have additional objectives, you may want to add more questions to the list.

Automation
✓ The system must emphasize automation.
✓ The goal should be to automate all aspects, from data acquisition and processing to training, deployment, and monitoring.

Collaboration
✓ The system should promote collaboration between data scientists, engineers, and the operation team.
✓ It should allow data scientists to effectively share the artifacts and the lineage as created during the model-building process.

Reproducibility
✓ The system should allow for easy replication of the current state and progress.

Governance & Compliance
✓ The system must ensure data privacy, security, and compliance with relevant regulations and policies.

Critical Questions:

Now that we have defined objectives, it’s time to look into the key questions to ask to evaluate the effectiveness of your current AI strategy.

Data Acquisition & Exploration (EDA)
Data is a fundamental building block of any ML system. Data Scientist understands it, identifies and addresses common issues like duplication, missing data, imbalance, outliers, etc. A significant amount of data scientist time goes into this activity of data exploration. Thus, our strategy should focus to support & accelerate these activities and answer the following questions:

▢ [Automation] Does the existing platform helps the data scientist to quickly analyze, visualize the data and automatically detect common issues?
▢ [Automation] Does the existing platform allows integrating and visualizing the relationship between datasets from multiple sources?
▢ [Collaboration] How can multiple data scientists collaborate in real-time on the same dataset?
▢ [Reproducibility] How do you track and manage different versions of acquired datasets?
▢ [Governance & Compliance] How do you ensure that the data privacy or security considerations have been addressed during the acquisition?

Data Transformation & Feature Engineering
After gaining an understanding of the data, the next step is to build and scale the transformations across the dataset. Here are some key questions to consider during this phase:

▢ [Automation] How can the transformation steps be effectively scaled to the entire dataset?
▢ [Automation] How can the transformation steps be applied in real-time to the live data before inference?
▢ [Collaboration] How can a data scientist share and discover the engineered features to avoid effort duplication?
▢ [Reproducibility] How do you track and manage different versions of transformed datasets?
▢ [Reproducibility] Where are the transformation steps and associated code stored?
▢ [Governance & Compliance] How do you track the lineage of data as it moves through transformation stages to ensure reproducibility and audibility?

Experiments, Model Training & Evaluation
Model training is an iterative process where data scientist explores and experiments with different combinations of settings and algorithm to find the best possible model. Here are some key questions to consider during this phase:

▢ [Automation] How can data scientists automatically partition the data for training, validation, and testing purposes?
▢ [Automation] Does the existing platform helps to accelerate the evaluation of multiple standard algorithms and tune hyperparameters
▢ [Collaboration] How can a data scientist share the experiment, configurations & trained models?
▢ [Reproducibility] How can you ensure the reproducibility of the experiment outputs?
▢ [Reproducibility] How do you track and manage different versions of trained models?
▢ [Governance & Compliance] How do you track the model boundaries allowing you to explain the model decisions and detect bias?

Deployment & Serving
In order to realize the business value of a model, it needs to be deployed. Depending on the nature of your business, it may be distributed, deployed in-house, on the cloud, or at the edge. Effective management of the deployment is crucial to ensure uptime and optimal performance. Here are some key questions to consider during this phase:

▢ [Automation] How do you ensure that the deployed models can scale with increasing workloads?
▢ [Automation] How are the new versions rolled out and the process to compare them against the running version? (A/B testing, canary, shadow, etc.)
▢ [Automation] Are there mechanisms to roll back or revert deployments if issues arise?
▢ [Collaboration] How can multiple data scientists understand the impact of their version before releasing it? (A/B testing, canary, shadow, etc.)
▢ [Reproducibility] How do you package your ML models for serving in the cloud or at the edge?
▢ [Governance & Compliance] How do you track the predicted decisions for auditability and accountability?

Model Pipeline, Monitoring & Continuous Improvement:
As we have seen, going from raw data to actionable insights involves complex series of steps. However, by orchestrating, monitoring, and reacting throughout the workflow, we can easily scale, adapt and make the process more efficient. Here are some key questions to consider during this phase:

▢ [Automation] How is the end-to-end process of training and deploying the models managed currently?
▢ [Automation] How can you detect the data or concept drift w.r.t to the historical baseline?
▢ [Automation] How do you determine when a model needs to be retrained or updated?
▢ [Collaboration] What are the agreed metrics to measure the effectiveness of each stage and new deployments?
▢ [Reproducibility] Are there automated pipelines to handle the end-to-end process of retraining and updating models to incorporate feedback and make enhancements?
▢ [Governance & Compliance] How do you ensure data quality, integrity and detect model deviation throughout the process?
▢ [*Governance & Compliance] *How do you budget and plan for the infrastructure requirements for the building of your models?

A MLOps system streamlines & brings structure to your strategy and thus, allowing you to answer these questions. It provides the capability to version control and to track various artifacts through the dataset, feature, metadata, and model repositories.

In the upcoming blogs, I will demonstrate how to implement these MLOps best practices using a simple case study on AWS.