DEV Community: PythicCoder

5 Steps to Training your first Video Classifier in a Flash

PythicCoder — Thu, 13 May 2021 11:51:23 +0000

Learn step by step how to build, train and make predictions on a video classification task with 3 simple tools: Lightning Flash, PyTorch Video, and Kornia.

Video Understanding enables computers to recognize actions, objects, and events in videos. From Retail, Health Care to Agriculture, Video Understanding enables automation of countless industry use cases.

“A case in point: Walmart has developed an AI system that inspects fresh food for signs of defects and spoilage. This system is helping Walmart monitor the temperature and freshness of produce and perishable foods, improve visual inspection at distribution centers, and route perishable food to the nearest store.“ [Source]

Facebook AI Researchrecently released a new library called PyTorchVideo powered by PyTorch Lightning that simplifies video understanding by providing SOTA pre-trained video models, datasets, and video-specific transformers. All the library’s models are highly optimized for inference and support different datasets and transforms.

Lightning Flash is a collection of tasks for fast prototyping, baselining, and fine-tuning scalable Deep Learning models, built on PyTorch Lightning. It allows you to train and finetune models without being overwhelmed by all the details, and then seamlessly override and experiment with Lightning for full flexibility. Learn more about Flash here.

PyTorchLightning/lightning-flash

In its new coming release, PyTorch Lightning Flash will provide a deep integration with PyTorchVideo and its backbones and transforms can be used alongside Kornia ones to provide a seamless preprocessing, training, and fine-tuning experience of SOTA pre-trained video classification models.

Kornia is a differentiable computer vision library for PyTorch that consists of a set of routines and differentiable modules to solve generic computer vision problems.

In this article you will learn how to train a custom video classification model in 5 simple steps using PyTorch Video, Lightning Flash, and Kornia, using the Kinetics dataset.

The Kinetics human action video dataset released by DeepMind is comprised of annotated~10s video clips sourced from YouTube.

Keep reading to train your own model in a flash!

5 Simple Steps for Video Classification

Before we get started all the steps in this tutorial are reproducible and can be run with the free trial of Grid by clicking the Run button below.

All the steps in this tutorial are reproducible and can be run with the free trial of Grid by clicking the Run button above.

The Grid Platform enables you to seamlessly train hundreds machine learning models on the cloud from your laptop without modifying your code in a fully reproducible manner.

Find the full end-to-end code for this tutorial code here and the full reproducible run here on ** Grid **.

Prerequisite Imports

https://gist.github.com/tchaton/f3ceafeb246b46f8c8f7d9e4498760b0

Step 1- Download The Kinetics Dataset

https://gist.github.com/tchaton/a4744780de0f534a613fc738afbac8c2

Step 2- Specify Kornia Transforms to Apply to Video

Flash helps you to place your transform exactly where you want and makes Video Pre and Post Processing Simple as easy as a few lines of initialization code

As videos are memory-heavy objects, data sampling and data augmentation play a key role in the video model training procedure and need to be carefully crafted to reach SOTA results.

Flash optimizes this process as transforms can be provided with simple dictionary mapping between hooks and the transforms. Flash hooks are simple functions that can be overridden to extend Flash’s functionality.

In the code below, we use the post_tensor_transform hook to uniformly select 8 frames from a sampled video clip and resize them to 224 pixels. This is done in parallel within each of the PyTorch DataLoader workers.

We also use the per_batch_transform_on_device hook to enable Kornia normalization to be applied directly to a batch already moved to GPU or TPU .

https://gist.github.com/tchaton/b627d56e6d3da89eb456aede48c3c171

Step 3- Load the Data

Lightning DataModules are shareable, reusable objects that encapsulate all data-related code. Flash enables you to quickly load videos and labels from various formats such as config files or folders into DataModules.

In this example, we use the Flash from_folders helper function to generate a DataModule. To use the from_folders functions videos should be organized into folders for each class as follows:

dir_path//.{mp4, avi}

Code for Initializing a DataModule on using Flash on the Kinetics Dataset

Step 4- Instantiate the VideoClassifier with a pre-trained Model Backbone .

Training a SOTA video classifier from scratch on Kinetics can take around 2 days on 64 GPUS. Thankfully, the Flash Backbones Registry API makes it easy to integrate individual models as well as entire model hubs with Flash.

By default, the PyTorchVideo Model Hub is pre-registered within the Video Classifier Task Backbone Registry.

You can list any of the available backbones and get more details as follows:

https://gist.github.com/tchaton/d2f2b52f77d7ed5a537d686e124c2b60

Once you pick a backbone, you can initialize it with just one line of code:

https://gist.github.com/tchaton/0e83bff7e27e125cc675bcdd31d3bf23

Note: Serializers are optional components that can be passed to Flash Classifiers that enable you to configure how to format the output of the Task Predictions asLogits, Probabilities , or Labels. Read more here.

Step 5- FineTune Model

Once we have a DataModule and VideoClassifier, we can configure the Flash Trainer. Out of the box, the Flash Trainer supports all the Lightning Trainer flags and tricks you know and love including distributed training, checkpointing, logging, mixed precision, SWA, quantization, early stopping, and more.

In addition to enabling you to train models from scratch the Flash Trainer provides quick access to a host of strategies for fine-tuning pre-trained models on new data.

https://gist.github.com/tchaton/685d58357241fe74c656a90798c1e0fe

In this example, we use the NoFreeze Strategy in which the backbone will be entirely trainable from the start. You can even define your own custom FineTuning Strategies .

Conclusion

With these 5 simple steps you can now train your own Video Classification model on any Video Dataset. Once a model is trained, it can easily be saved and reused later for making predictions.

https://gist.github.com/tchaton/6fc6ba4454a6d7aca0b790f8cbf38fe6

One major advantage of using PyTorch Video backbones is that they are optimized for mobile. With Lightning, models can be saved and exported as classic PyTorch Checkpoints or optimized for inference on the edge with ONNX and or TorchScript.

Find the end-to-end tutorial code ** here **.

We thank Tullie Murrell from the PyTorchVideo team for the support and feedback leading to this integration in Flash.

Next Steps

Now that you have train your first model it’s time to experiment. Flash Tasks provide full access to all the Lightning hooks that can be overridden as you iterate towards the state of the art.

Check out the following links to learn more:

About the Authors

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As Head of Developer Advocacy at Grid.ai, he collaborates with the Machine Learning Community, to solve real-world problems with game-changing technologies that are then documented, open-sourced, and shared with the rest of the world.

Thomas Chaton is PyTorch Lightning Research Engineering Manager. Previously, Senior Research Engineer at Fujitsu AI and Comcast-Sky Labs, Thomas is also the creator of TorchPoints3D.

How to Baseline Deep Learning Tasks in a Flash

PythicCoder — Thu, 25 Feb 2021 13:02:02 +0000

This tutorial introduces how to get started building baselines for Deep Learning using PyTorch Lightning Flash

Photo by Dmitry Zvolskiy from Pexels

What is PyTorch Lightning Flash?

PyTorch Lightning Flash is a new library from the creators of PyTorch Lightning to enable quick baselining of state-of-the-art Deep Learning tasks on new datasets in a matter of minutes.

PyTorchLightning/lightning-flash

Consistent with PyTorch Lightning’s goal of getting rid of the boilerplate, Flash aims to make it easy to train, inference, and fine-tune deep learning models.

Flash is built on top of PyTorch Lightning to abstract away the unnecessary boilerplate for common Deep Learning Tasks ideal for:

Data science
Kaggle Competitions
Industrial AI
Applied research

As such, Flash provides seamless support for distributed training and inference of Deep Learning models.

Since Flash is built on top of PyTorch Lightning, as you learn more, you can override your Task code seamlessly with both Lightning and PyTorch to find the right level of abstraction for your scenario.

Motorcycle Photo by Nikolai Ulltang from Pexels

For the remainder of this post, I will walk you through the 5 steps of building Deep Learning applications with an inline Flash Task code example.

Creating your first Deep Learning Baseline with Flash

Photo by Lalesh Aldarwish from Pexels

All code for the following tutorial can be found in the Flash Repo under Notebooks.

I will present five repeatable steps that you will be able to apply to any Flash Task on your own data.

Choose a Deep Learning Task
Load Data
Pick a State of the Art Model
Fine-tune the Task
Predict

Now let’s Get Started!!!

Step 1: Choose A Deep Learning Task

Photo by Pixabay from Pexels

The first step of the applied Deep Learning Process is to choose the task we want to solve. Out of the box, Flash provides support for common deep learning tasks such as Image, Text, Tabular Classification, and more complex scenarios such as Image Embedding, Object Detection, Document Summarization and Text Translation. New tasks are being added all the time.

In this tutorial, we’ll build a Text Classification model using Flash for sentiment analysis on movie reviews. The model will be able to tell us that a review such as “This is the worst movie in the history of cinema.” is negative and that a review such as “This director has done a great job with this movie!” is positive.

To get started, let’s first install Flash and import the required Python libraries for the Text Classification task.

Step 2: Load Data

Now that we have installed flash and loaded our dependencies let’s talk about data. To build our first model, we will be using the IMDB Movie Review dataset stored in a CSV file format. Check out some sample reviews from the dataset.

review, sentiment"I saw this film for the very first time several years ago - and was hooked up in an instant. It is great and much better than J. F. K. cause you always have to think 'Can it happen to me? Can I become a murderer?' You cannot turn of the TV or your VCR without thinking about the plot and the end, which you should'nt miss under any circumstances.", positive"Winchester 73 gets credit from many critics for bringing back the western after WWII. Director Anthony Mann must get a lot of credit for his excellent direction. Jimmy Stewart does an excellent job, but I think Stephen McNalley and John McIntire steal the movie with their portrayal of two bad guys involved in a high stakes poker game with the treasured Winchester 73 going to the winner. This is a good script with several stories going on at the same time. Look for the first appearance of Rock Hudson as Young Bull. Thank God, with in a few years, we would begin to let Indians play themselves in western films. The film is in black and white and was shot in Tucson Arizona. I would not put Winchester 73 in the category of Stagecoach, High Noon or Shane, but it gets an above average recommendation from me.<br /><br />.", positive

The first thing we need to do is download the dataset using the following code.

Once we have downloaded the IMDB dataset, Flash provides a convenient TextClassificationData module that handles the complexity of loading Text Classification data stored in CSV format and converting it into a representation that Deep Learning models need train.

3. Pick a State of the Art Model for Our Task

In the past few years in Natural Language Processing, many of the State of the Art Models are named after Sesame Street Characters. Photo used under CC BY-NC-ND 4.0 license from pixy

Once we have loaded our dataset, we need to pick a model to train. Each Flash Task comes preloaded with support for State of the Art model backbones for you to experiment with instantly.

By default, the TextClassifier task uses the tiny-bert model, enabling strong performance on most text classification tasks. Still, you can use any model from Hugging Face transformers — Text Classification model repository or even bring your own. In Flash, all you need is just one line of code to load a backbone.

4. Fine-tune the Task

Photo by Shae Calkins from Pixy

Now that we have chosen the model and loaded our data, it’s time to train the model on our classification task using the following two lines of code:

Because the Flash Trainer is built on top of PyTorch Lightning, it is seamless to distribute training to multiple GPUs, Cluster Nodes, and even TPUs.

Additionally, you get tons of otherwise difficult to implement features, such as automated model checkpointing and logging integrations with platforms such as Tensorboard, Neptune.ai, and MLFlow built-in without any hassle.

Photo generated by the author.

Tasks come with native support for all standard task metrics. In our case, Flash Trainer will automatically benchmark your model for you on all the standard task metrics for classification, such as Precision, Recall, F1 and more with just one line of code.

Task models can be checkpointed and shared seamlessly using the Flash Trainer as follows.

5. Predict

Photo by Sindre Strøm from Pexels

Once we’ve finished training our model, we can use it to predict on our data. with just one line of code:

Additionally, we can use the Flash Trainer to scale and distribute model inference for production.

For scaling for inference on 32 GPUs, it is as simple as one line of code.

You can even export models to Onnx or Torch Script for edge device inference.

Putting it All Together

How fun was that! The 5 steps above are condensed into the simple code snippet below and can be applied to any Flash deep learning task.

https://medium.com/media/b5f1d203391db937738ea9e4d8e0ef6b/href

Next Steps

Now that you have the tools to get started with building quick Deep Learning Baselines, I can’t wait for you to show us what you can build. If you liked this tutorial, feel free to Clap Below and give us a Star on GitHub.

We are working tirelessly on adding more Flash tasks, so if you have any must-have tasks, comment below or reach out to us on Twitter @pytorchlightnin or in our Slack channel.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As Head of Developer Advocacy at Grid.ai, he collaborates with the Machine Learning Community to solve real-world problems with game-changing technologies that are then documented, open-sourced, and shared with the rest of the world.

Lessons Upon Leaving Microsoft After 7 Transformative Years

PythicCoder — Thu, 14 Jan 2021 09:19:12 +0000

Today is my last day at Microsoft. After almost 7 incredible years at Microsoft it’s time for me to move onto the next step in my professional career (more about that next week). It’s a bittersweet week but before the day is over I want to take this time to reflect and share my experiences.

I joined Microsoft in 2014 after finishing my first degree in Computer Science and History at small liberal arts college in Baltimore Maryland.

At the time, Microsoft was at a crossroads, trailing a distant 3rd in a mobile first world and recovering from alukewarm reception of Windows 8. No one seemed to know which way the winds would blow. Yet, coming from a small liberal arts school, I was excited for the opportunity to learn and prove myself.

The past 7 years have been transformative for both me and Microsoft.

As the company has reinvented itself I had the opportunity to:

Learn cutting edge 1st and 3rd party technologies. From Xbox, Office 365, Windows Phone and Hololens to the intricacies of Azure across countless services (VR, Modern Web, Containers, Machine Learning, Blockchain, IoT, Databases, Automation and more).
Engage with non profits, mentoring students and young professionals from underrepresented high schools to the Ivy Leagues.
Host and be hosted by top technical communities in Israel and abroad.
Build keynote demos and speak at first and third party conferences across the world.
Lead and contribute to business critical engagements with many of the top brands in the world.
Collaborate with cutting edge start ups from the smallest seed companies to larger Series C and D organizations.
Complete a Masters Degree at Israel’s top Natural Language Processing Lab
Publish AI Research at top Academic Conferences.
Beta test and shape core Microsoft products from Windows 10 to Github and Azure ML.
Develop open source solutions and write content consumed by hundreds of thousands if not millions of developers around the world.

I am honored by those I have had the opportunity to work with including some of the most talented engineers and leaders in the world.

As I transition to the next chapter of my life, I want to share my learnings from the past few years as a way of both formalizing them for my future self and paying it forward to any former colleagues. Please keep in mind that these opinions reflect my own experiences and do not in any way represent Microsoft.

Lessons Learned after 7 years at Microsoft

Start your Career with an Uphill Battle.

One of the best things that happened to me at Microsoft was starting on the Technical Evangelism team for Windows Phone in 2014. At this point Windows Phone was a distant third in the phone OS wars and morale was not at its highest.

This provided me with amazing opportunities to learn from my peers and see how some of the smartest engineers I’ve ever met, coped with work not going their way. The need to innovate and try new things in an attempt to get a foothold, provided me with opportunities to work on important engagements that otherwise would have not been entrusted to a college hire. It was here that I was first able to experiment with then cutting edge technologies like progressive applications. My peers who were in more comfortable positions didn’t have such experiences and many became complacent assuming that the winning position is the default position.

While windows phone development was not my passion, being in an uphill battle enabled me to experiment with technologies that I was interested in. I was actively encouraged to work on passion projects which changed the course of my career culminating in me achieving one of my biggest goals of moving to Israel.

Always have a Northern Star and Make your Career Goals Clear

When I started as a college hire, I came in at a relatively low starting level. There are a couple of techniques I learned that helped me to accelerate my career. One of the most useful was the concept of a northern star. A northern star is a document that you should refresh once every few months that outlines your goals, aspirations and how you plan to achieve them.

At Microsoft, I saw many hard working and incredibly talented peers struggle to level up. Many assumed that their hard work and accomplishments would translate directly into a promotion. While this does happen overtime, it is often a slow process, as there are many factors that determine career progression.

To rapidly grow a career, goals shouldn’t be defined by your manager, organization or even company, rather they should be aligned to them. You should make your goals and career expectations clear to your manager so that they can help advocate on your behalf. At the end of the day you are your own best advocate.

If you succeed in properly aligning personal goals with both your manager and your organization, not only does everyone win, but your impact will be much greater than it would have if you simply just checked a bunch of boxes, or overfit other metrics.

If you feel that there is no way to align your team or organizational goals to your northern star document, then it is time to make a change. The fact that you have been defining and measuring your own goals will help you stand out to teams and organizations that are aligned with your northern star in such cases lateral movements are often the key to career growth.

That being said ….

Your Career is much more than a Level

One of my favorite quotes from Bob Berg’s The Go Giver is:

“Your true worth is determined by how much more you give in value than you take in payment.”

One of the worst things to be at a company like Microsoft is to be in a position where you are either over promoted. In my 7 years at Microsoft, I saw that while it is possible to claw your way up a few levels by learning the latest tech buzzwords or playing politics it’s the biggest possible career trap. Overtime, the gap between what you are expected to know and what is possible to reasonably learn grows to a point where it is impossible to be long term successful and respected.

Those who are respected at Microsoft are respected not because they have a fancy level or title but rather because people know that they are competent, fun to work with, and care about those around them.

Which leads me to my next point …

Acknowledge Imposter Syndrome but do not Surrender to It

One of the most surprising aspects of the past years is that I learned that at some level almost everyone I respect has some form of imposter syndrome.

Imposter syndrome is a double edged sword. If you simply dismiss it as being natural, it is easy to become complacent and get stuck in the career trap outlined in the last point. However if you surrender to it, it’s easy to settle and not grow or develop.

While I do not have the cure for imposter syndrome, in my experience the best counter to imposter syndrome is developing a strong mentorship network. Mentors can help you navigate the challenges of imposter syndrome. With strong mentors, acknowledging imposter syndrome can provide opportunities to grow your skills and help grow your confidence over time. At Microsoft I was lucky to have some amazing mentors that helped me at every single career stage.

One piece of advice I have regarding mentors, is that anyone can be a mentor, it doesn’t need to be a formal arrangement. I define a mentor as, “ anyone whose experience can help unblock you, guide you and boost your skills”. One of my favorite approaches to finding mentors is to scope a pet or assigned project with relevant technologies or skills I wanted to learn and then to engage people I wanted to learn from to help me out.

Remember however to pay it forward, we were all juniors once and as you grow in your career make sure you are taking time to help out others when you can. The more authentic you are in your mentor/mentee relationships the more value you will gain from them and provide in return.

Which leads me to my next point …

The best leaders lead through others.

Managers that optimize for their own success end up creating toxic work cultures where as those who learn to bring success to others grow, succeed and are respected. I have been truly lucky to have some amazing managers from whom I’ve learned that true leadership comes from unblocking those who work for you.

My first manager took a big risk hiring from a small liberal arts college, he provided me with some of the most difficult accounts in his portfolio and gave me opportunities to demonstrate leadership as a tech lead on an important keynote. When I finally found the opportunity to move to Israel he helped me fast track a promotion so that when I moved to Israel so I wouldn’t have to take as dramatic a pay cut. My second manager helped me transition to Israel pushed me to truly become a great software engineer and encouraged me to pursue a Masters Degree. My third and fourth managers provided me with the opportunity to help hire and build a global developer relations team and my fifth and final manager empowered me to develop and take my AI and leadership skills to the next level.

In addition to my mentors am indebted to all these amazing people who helped me grow and achieve what I did at Microsoft.

I hope these lessons are valuable to others and I want to thank all those that I had the pleasure of working with for an amazing 7 years.

Looking forward to the future and being able to share with you my next step soon!

Some Memory Highlights of the Past 7 Years

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborated with the Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

No Code Data Enhancement with Azure Synapse Analytics and Azure Auto ML

PythicCoder — Thu, 10 Dec 2020 13:36:56 +0000

TLDR; This post will walk through how to train and evaluate Azure ML AutoML Regression model on your data using Azure Synapse Analytics Spark and SQL pools.

Before we get started let’s make sure we are all on the same page with the core Azure concepts needed to take your data to the next level.

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

What is Azure Synapse Analytics?

Azure Synapse Analytics is an integrated service that accelerates extracting insightful across data warehouses and big data systems.

Azure Synapse ties together traditional relational SQL enterprise data warehousing, unstructured data stores and serverless Apache Spark , to enable limitless pipelines for ETL and ELT operations. Furthermore Synapse Studio provides a unified interface for data monitoring, coding, and security.

Synapse has deep integration with other Azure services such as Power BI, CosmosDB, and AzureML which makes it perfect for wrangling insight out of your data.

What is Azure Machine Learning Auto ML?

Auto ML, is the process of automating the time consuming, iterative tasks of machine learning model development.

Azure Machine Learning (Azure ML) is a cloud-based service for creating and managing machine learning solutions. It’s designed to help data scientists and machine learning engineers to leverage their existing data processing and model development skills & frameworks.

What is Azure Machine Learning

In Azure Machine Learning, Auto ML allows data scientists, analysts, and developers to build ML models with high scale, efficiency, and productivity all while sustaining model quality.

With Auto ML we can transform Synapse Analytics Data into actionable baseline models to enrich datasets at scale without writing a single line of machine learning code.

Regression is used to build models to forecast numeric values such as taxi fares based on learned input features.

In the next section, I will walk you through an end to end example of how to enrich you Synapse data by training and evaluating a model with the NYC Taxi Dataset. Once you get preform these steps you’ll be able to train and run your own Auto ML on models on any tabular dataset of your choosing.

Getting Started

Let’s ingest, train a model and enhance some data to predict taxi fares with the

Step 1: Set Up Azure Machine Learning and Synapse Workspaces

First if you do not have them already we need to create our Azure ML and Synapse workspaces.

Create Azure ML Workspace from the Portal or use the Azure CLI

Step 2: Link Azure Machine Learning Workspace to Synapse Service

Once we have deployed our two workspaces we need to link them. Full steps for linking Azure ML and Synapse Workspaces can be found here

Follow these steps to create a service principle and link the azure ml and machine learning workspaces.

Step 3: Create Dedicated Serverless Apache Spark Spark and SQL Pools

To actually ingest and process our data we need to use pools. Azure Synapse Analytics offers various analytics engines to help you ingest, transform, model, analyze, and serve your data.

For this tutorial since we are using a toy dataset we can use the cheapest pools available for you own data you may want to configure your pools accordingly.

The serverless Apache Spark pools offers open-source big data compute capabilities. This is where the majority of our data processing and Auto ML code will run.

Steps for creating a Dedicated Spark Pool can be found here

A dedicated serverless SQL pool offers T-SQL based compute and storage capabilities. We will use this pool to store the data we want to enhance with our AutoML model.

Steps for creating a dedicated sql pool can be found here

One key advantage of Azure Synapse Analytics is if you configure a time out you only pay for the compute when it’s in use.

Step 4: Upload and run the Spark Taxi Data Notebook to create Spark Database and SQL Test Database

Once we have our serverless Spark and SQL pools up and running we can now ingest our data setup our Spark and SQL tables for training and testing respectively.

Download this Spark Create-Spark-Table-NYCTaxi- Data.ipynb notebook and import it into your workspace.

Once the notebook is uploaded, change the sql_pool_name value to match the name of your sql pool and then select the desired spark pool and run click all.

Step 5: Launch Auto ML Wizard and Train Regression Model with AutoML using NYC Taxi Spark Table.

Once the data is ingested we can use our spark nyc_taxi and Spark Pool to train an AutoML regression model for forecasting taxi fares.

Follow the three steps in wizard below to train your model.

Note: Be sure to set the target column to fareAmount and use the onnx model model compatibility option. For a more in depth explanation of the training steps check out the documentation here

This will kick off your Auto ML Regression training job. It should take about 2hrs to run when it is complete we can then evaluate it on our SQL test table.

Step 6: Enhance SQL Table with Trained Auto ML Model

Once we have the best model we can now evaluate it on our test SQL table using our SQL Pool.

First we need to select the table we want to enhance with the model we just trained.

Then we select our new Auto ML model, map the our input table columns to what the model is expecting and choose or create a table for storing our model locally.

The wizard will generate a T-SQL script that evaluate our model against the test data and outputs the fare predictions.

There you have it all you need to know to train and test you own AutoML models and make them actionable.

Next Steps

Additional Synapse Documentation and Walkthroughs worth checking out can be found below:

Now that you’ve finished the steps above it is time to try them out on you own synapse data. Feel free to post in the comments if you have any questions and to share the cool models you make!

Look forward to seeing what AutoML and Azure Synapse can do for you!!

Acknowledgments

Thanks to Nellie Gustafsson, Yifan Song and Chang Xu from the Azure Synapse product team for their great documentation and support during the writing this post.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with the Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Multi Node Distributed Training with PyTorch Lightning & Azure ML

PythicCoder — Mon, 26 Oct 2020 12:33:42 +0000

TL;DR This post outlines how distribute PyTorch Lightning training on Distributed Clusters with Azure ML

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

Azure ML and PyTorch Lighting

In my last few posts on the subject, I outlined the benefits of both PyTorch Lightning and Azure ML to simplify training deep learning models and logging. Take a look you haven’t yet check it out!

Now that you are familiar with both the benefits of Azure ML and PyTorch lighting let’s talk about how to take PyTorch Lighting to the next level with multi node distributed model training.

Multi Node Distributed Training

Sample traditional distributed training consideration from Azure Docs.

Multi Node Distributed Training is typically the most advanced use case of the Azure Machine Learning service. If you want a sense of why it is traditionally so difficult, take a look at the Azure Docs.

What is distributed training? - Azure Machine Learning

PyTorch Lighting makes distributed training significantly easier by managing all the distributed data batching, hooks, gradient updates and process ranks for us. Take a look at the video by William Falcon here to see how this works.

We only need to make one minor modification to our train script for Azure ML to enable PyTorch lighting to do all the heavy lifting in the following section I will walk through the steps to needed to run a distributed training job on a low priority compute cluster enabling faster training at an order of magnitude cost savings.

Plan and manage costs - Azure Machine Learning

Getting Started

Step 1 — Set up Azure ML Workspace

Create Azure ML Workspace from the Portal or use the Azure CLI

Connect to the workspace with the Azure ML SDK as follows

from azureml.core import Workspace
ws = Workspace.get(name="myworkspace", subscription_id='<azure-subscription-id>', resource_group='myresourcegroup')

Step 2 — Set up Multi GPU Cluster

Create compute clusters - Azure Machine Learning

from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# Choose a name for your GPU cluster
gpu_cluster_name = "gpu cluster"

# Verify that cluster does not exist already
try:
    gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    compute_config = AmlCompute.provisioning_configuration(vm_size='Standard_NC12s_v3',
max_nodes=2)
    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)

gpu_cluster.wait_for_completion(show_output=True)

Step 3 — Configure Environment

To run PyTorch Lighting code on our cluster we need to configure our dependencies we can do that with simple yml file.

channels:
  - conda-forge
dependencies:
  - python=3.6
  - pip:
    - azureml-defaults
    - mlflow
    - azureml-mlflow
    - torch
    - torchvision
    - pytorch-lightning
    - cmake
    - horovod # optional if you want to use a horovod backend

We can then use the AzureML SDK to create an environment from our dependencies file and configure it to run on any Docker base image we want.

**from**  **azureml.core**  **import** Environment

env = Environment.from_conda_specification(environment_name, environment_file)

_# specify a GPU base image_
env.docker.enabled = **True**
env.docker.base_image = (
    "mcr.microsoft.com/azureml/openmpi3.1.2-cuda10.2-cudnn8-ubuntu18.04"
)

Step 4 — Training Script

Create a ScriptRunConfig to specify the training script & arguments, environment, and cluster to run on.

We can use any example train script from the PyTorch Lighting examples or our own experiments.

Once we have our training script we need to make one minor modification by adding the following function that sets all the required environmental variables for distributed communication between the Azure nodes.

https://medium.com/media/290fec1591cf1a84f074aa91af8a010b/href

Then after parsing the input arguments call the above function.

args = parser.parse_args()

# -----------
# configure distributed environment
# -----------

set_environment_variables_for_nccl_backend(single_node=int(args.num_nodes) > 1)

Hopefully in the future this step will be abstracted out for us.

Step 5 — Run Experiment

For Multi Node GPU training , specify the number of GPUs to train on per a node (typically this will correspond to the number of GPUs in your cluster’s SKU), the number of nodes(typically this will correspond to the number of nodes in your cluster) and the accelerator mode and the distributed mode, in this case DistributedDataParallel ("ddp"), which PyTorch Lightning expects as arguments --gpus --num_nodesand --accelerator, respectively. See their Multi-GPU training documentation for more information.

Then set the distributed_job_config to a new MpiConfiguration with equal to one(since PyTorch lighting manages all the distributed training)and a node_count equal to the --num_nodes you provided as input to the train script.

https://medium.com/media/9f896be17f677c2dd2a3681360c2db83/href

We can view the run logs and details in realtime with the following SDK commands.

**from**  **azureml.widgets**  **import** RunDetails

RunDetails(run).show()
run.wait_for_completion(show_output= **True** )

And there you have it with out needing to deal with managing the complexity of distributed batching, Cuda, MPI, logging callbacks, or process ranks, PyTorch lighting scale your training job to as many nodes as nodes as you’d like.

Pictured a complete two node 4 gpu run.

You shouldn’t but if you have any issues let me know in the comments.

Acknowledgements

I want to give a major shout out to Minna Xiao and Alex Deng from the Azure ML team for their support and commitment working towards a better developer experience with Open Source Frameworks such as PyTorch Lighting on Azure.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with the Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Configuring Native Azure ML Logging with PyTorch Lighting

PythicCoder — Tue, 20 Oct 2020 12:17:23 +0000

Combining Azure and Lightning leads to more powerful logging

TL;DR: This post demonstrates how to connect PyTorch Lightning logging to Azure ML natively with ML Flow.

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

Azure ML and PyTorch Lighting

In my last post on the subject, I outlined the benefits of both PyTorch Lightning and Azure ML to simplify training deep learning models. If you haven’t yet check it out!

Training Your First Distributed PyTorch Lightning Model with Azure ML

Once you’ve trained your first distributed PyTorch Lighting model with Azure ML it is time to add logging.

Why do we care about logging?

Logs are critical for troubleshooting and tracking the performance of machine learning models. Since we often train on remote clusters, logs provide a simple mechanism for having a clear understanding of what’s going on at each phase of developing our model.

As opposed to simple print statements, logs are time stamped, can be filtered by severity, and are used by Azure ML to visualize critical metrics such during training, validation, and testing. Logging metrics with Azure ML is alos a perquisite for using the Azure ML HyperDrive Service to help us find optimal model configurations.

Tune hyperparameters for your model - Azure Machine Learning

Logging is a perfect demonstration of how both PyTorch Lighting and Azure ML combine to show simplify your model training just by using lightning we can save ourselves dozens of lines of PyTorch code in our application earning readability in the process.

Logging with PyTorch Lighting

In vanilla PyTorch, keeping track and maintaining logging code can get complicated very quickly.

ML frameworks and services such as Azure ML, Tensor Board, TestTube, Neptune.ai and Comet ML each have their own unique logging APIs. This means that ML engineers often need to maintain multiple log statements at each phase of training, validation and testing.

PyTorch Lighting simplifies this process by providing a unified logging interface that comes with out of the box support with the most popular machine learning logging APIs.

Loggers - PyTorch Lightning 1.0.2 documentation

Multiple Loggers can even be chained together which greatly simplifies your code.

**from** pytorch\_lightning.loggers **import** TensorBoardLogger, TestTubeLogger
logger1 **=** TensorBoardLogger('tb\_logs', name **=**'my\_model')
logger2 **=** TestTubeLogger('tb\_logs', name **=**'my\_model')
trainer **=** Trainer(logger **=** [logger1, logger2])

Once, loggers are provide to a PyTorch Lighting trainer they can be accessed in any lightning_module_function_or_hook outside of __init__.

**class**  **MyModule** (LightningModule):
**def**  **some\_lightning\_module\_function\_or\_hook** (self):
 some\_img **=** fake\_image()
_# Option 1_
 self **.** logger **.** experiment[0] **.** add\_image('generated\_images', some\_img, 0)
_# Option 2_
 self **.** logger[0] **.** experiment **.** add\_image('generated\_images', some\_img, 0)

Azure ML Logging with PyTorch Lighting with ML Flow

Since Azure ML has native integration with ML Flow, we can take advantage of PyTorch Lighting’s ML Flow Logger module to get native metric visualizations across multiple experiment runs and utilize hyperdrive with very minor changes to our training code.

Below I’ll outline the code needed to take advantage of Azure ML Logging with PyTorch lightning.

Step #1 Environment

Add PyTorch Lighting, Azure ML and ML Flow packages to the run environment.

pip 
 - azureml-defaults
 - mlflow
 - azureml-mlflow
 - pytorch-lightning

Step #2 Get Azure ML Run Context and ML Flow Tracking URL

from azureml.core.run import Run

run = Run.get\_context()
mlflow\_url = run.experiment.workspace.get\_mlflow\_tracking\_uri()mlf\_logger =

Step #3 Initialize PyTorch Lighting MLFlow Logger and Link Run.id

MLFlowLogger(experiment\_name=amlexp.name, tracking\_uri=mlflow\_url)
mlf\_logger.\_run\_id = run.id

Step #4 Add logging statements to the PyTorch Lighting the training_step, validation_step, and test_step Hooks

def training\_step(self, batch, batch\_idx):

 # Calculate train loss here 
 self.log("train\_loss", loss)
 # return test loss

def validation\_step(self, batch, batch\_idx):

 # Calculate validation loss here 
 self.log("val\_loss", loss)
 # return test loss

def test\_step(self, batch, batch\_idx):
 # Calculate test loss here 
 self.log("test\_loss", loss)
 # return test loss

Step #5 Add the ML Flow Logger to the PyTorch Lightning Trainer

trainer = pl.Trainer.from\_argparse\_args(args)

trainer.logger = mlf\_logger # enjoy default logging implemented by pl!

And there you have it! Now when you submit your PyTorch Lighting train script you will get real time visualizations and HyperDrive inputs at Train, Validation, and Test time with a fraction of the normal required code.

You shouldn’t but if you have any issues let me know in the comments.

Next Steps

In the next post, I will show you how to configure Multi Node Distributed Training with PyTorch and Azure ML using Low Priority compute instances to minimize training cost by an order of magnitude.

Plan and manage costs - Azure Machine Learning

Acknowledgements

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with the Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Training Your First Distributed PyTorch Lightning Model with Azure ML

PythicCoder — Tue, 13 Oct 2020 12:11:52 +0000

TLDR; This post outlines how to get started training Multi GPU Models with PyTorch Lightning using Azure Machine Learning.

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

What is PyTorch Lightning?

PyTorch Lighting is a lightweight PyTorch wrapper for high-performance AI research. Lightning is designed with four principles that simplify the development and scalability of production PyTorch Models:

Enable maximum flexibility
Abstract away unnecessary boilerplate, but make it accessible when needed.
Systems should be self-contained (ie: optimizers, computation code, etc).
Deep learning code should be organized into 4 distinct categories, Research code (the LightningModule), Engineering code (you delete, and is handled by the Trainer), Non-essential research code (logging, etc… this goes in Callbacks), Data (use PyTorch Dataloaders or organize them into a LightningDataModule).

Once you do this, you can train on multiple-GPUs, TPUs, CPUs and even in 16-bit precision without changing your code which is perfect for taking advantage of distributed cloud computing services such as Azure Machine Learning.

PyTorchLightning/pytorch-lightning

Additionally PyTorch Lighting Bolts provide pre-trained models that can be wrapped and combined to more rapidly prototype research ideas.

PyTorchLightning/pytorch-lightning-bolts

What is Azure Machine Learning?

Azure Machine Learning ( Azure ML ) is a cloud-based service for creating and managing machine learning solutions. It’s designed to help data scientists and machine learning engineers to leverage their existing data processing and model development skills & frameworks.

Azure Machine Learning provides the tools developers and data scientists need for their machine learning workflows, including:

Azure Compute Instances that can be accessed online or linked to remotely with Visual Studio Code.
What is an Azure Machine Learning compute instance? - Azure Machine Learning
Connect to compute instance in Visual Studio Code (preview) - Azure Machine Learning
Out of the box support for Machine Learning libraries such as PyTorch, Tensorflow, ScikitLearn and Keras.

azureml.train.estimator.Estimator class - Azure Machine Learning Python

Code, Data, Model Management

Tutorial: Get started with machine learning - Python - Azure Machine Learning

Scalable Distributed Training and Cheap Low Priority GPU Compute

What is distributed training? - Azure Machine Learning

Auto ML and Hyper Parameter Optimization
Tune hyperparameters for your model - Azure Machine Learning
What is automated ML / AutoML - Azure Machine Learning
Container Registry, Kubernetes Deployment and MLOps Pipelines
MLOps: ML model management - Azure Machine Learning
Deploy ML models to Kubernetes Service - Azure Machine Learning
Interpretability Tools and Data Drift Monitoring
Interpret & explain ML models in Python (preview) - Azure Machine Learning
Analyze and monitor for data drift on datasets (preview) - Azure Machine Learning

You can even use external open source services like MLflow to track metrics and deploy models or Kubeflow to build end-to-end workflow pipelines.

Check out some AzureML best practices examples at

With the advantages of PyTorch Lighting and Azure ML it makes sense to provide an example of how to leverage the best of both worlds.

Getting Started

Step 1 — Set up Azure ML Workspace

Create Azure ML Workspace from the Portal or use the Azure CLI

Connect to the workspace with the Azure ML SDK as follows

from azureml.core import Workspace
ws = Workspace.get(name="myworkspace", subscription\_id='<azure-subscription-id>', resource\_group='myresourcegroup')

Step 2 — Set up Multi GPU Cluster

Create compute clusters - Azure Machine Learning

from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute\_target import ComputeTargetException

# Choose a name for your GPU cluster
gpu\_cluster\_name = "gpu cluster"

# Verify that cluster does not exist already
try:
 gpu\_cluster = ComputeTarget(workspace=ws, name=gpu\_cluster\_name)
 print('Found existing cluster, use it.')
except ComputeTargetException:
 compute\_config = AmlCompute.provisioning\_configuration(vm\_size='Standard\_NC12s\_v3',
max\_nodes=2)
 gpu\_cluster = ComputeTarget.create(ws, gpu\_cluster\_name, compute\_config)

gpu\_cluster.wait\_for\_completion(show\_output=True)

Step 3 — Configure Environment

To run PyTorch Lighting code on our cluster we need to configure our dependencies we can do that with simple yml file.

channels:
 - conda-forge
dependencies:
 - python=3.6
 - pip
 - pip:
 - azureml-defaults
 - torch
 - torchvision
 - pytorch-lightning

We can then use the AzureML SDK to create an environment from our dependencies file and configure it to run on any Docker base image we want.

**from**  **azureml.core**  **import** Environment

env = Environment.from\_conda\_specification(environment\_name, environment\_file)

_# specify a GPU base image_
env.docker.enabled = **True**
env.docker.base\_image = (
 "mcr.microsoft.com/azureml/openmpi3.1.2-cuda10.2-cudnn8-ubuntu18.04"
)

Step 4 — Training Script

Create a ScriptRunConfig to specify the training script & arguments, environment, and cluster to run on.

We can use any example train script from the PyTorch Lighting examples or our own experiments.

Step 5 — Run Experiment

For GPU training on a single node, specify the number of GPUs to train on (typically this will correspond to the number of GPUs in your cluster’s SKU) and the distributed mode, in this case DistributedDataParallel ("ddp"), which PyTorch Lightning expects as arguments --gpus and --distributed_backend, respectively. See their Multi-GPU training documentation for more information.

**import**  **os**
 **from**  **azureml.core**  **import** ScriptRunConfig, Experiment

cluster = ws.compute\_targets[cluster\_name]

src = ScriptRunConfig(
 source\_directory=source\_dir,
 script=script\_name,
 arguments=["--max\_epochs", 25, "--gpus", 2, "--distributed\_backend", "ddp"],
 compute\_target=cluster,
 environment=env,
)

run = Experiment(ws, experiment\_name).submit(src)
run

We can view the run logs and details in realtime with the following SDK commands.

**from**  **azureml.widgets**  **import** RunDetails

RunDetails(run).show()
run.wait\_for\_completion(show\_output= **True** )

Next Steps and Future Post

Now that we’ve set up our first Azure ML PyTorch lighting experiment. Here are some advanced steps to try out we will cover them in more depth in a later post.

1. Link a Custom Dataset from Azure Datastore

This example used the MNIST dataset from PyTorch datasets, if we want to train on our data we would need to integrate with the Azure ML Datastore which is relatively trivial we will show how to do this in a follow up post.

Create Azure Machine Learning datasets to access data - Azure Machine Learning

2. Create a Custom PyTorch Lightning Logger for AML and Optimize with Hyperdrive

In this example all our model logging was stored in the Azure ML driver.log but Azure ML experiments have much more robust logging tools that can directly integrate into PyTorch lightning with very little work. In the next post we will show how to do this and what we gain with HyperDrive.

3. Multi Node Distributed Compute with PyTorch Lightining Horovod Backend

In this example we showed how to leverage all the GPUs on a one Node Cluster in the next post we will show how to distribute across clusters with the PyTorch Lightnings Horovod Backend.

4. Deploy our Model to Production

In this example we showed how to train a distributed PyTorch lighting model in the next post we will show how to deploy the model as an AKS service.

How and where to deploy models - Azure Machine Learning

If you enjoyed this article check out my post on 9 tips for Production Machine Learning and feel free to share it with your friends!

9 Advanced Tips for Production Machine Learning

Acknowledgements

I want to give a major shout out to Minna Xiao from the Azure ML team for her support and commitment working towards a better developer experience with Open Source Frameworks such as PyTorch Lighting on Azure.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with the Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Accelerating Model Training with the ONNX Runtime

PythicCoder — Wed, 20 May 2020 10:28:12 +0000

TDLR; This article introduces the new improvements to the ONNX runtime for accelerated training and outlines the 4 key steps for speeding up training of an existing pyTorch model with the ONNX Runtime (ORT).

What is the ONNX Runtime (ORT)?

ONNX Runtime is a performance-focused inference engine for ONNX (Open Neural Network Exchange) models. ONNX Runtime was designed with a focus on performance and scalability in order to support heavy workloads in high-scale production scenarios. It also has extensibility options for compatibility with emerging hardware developments.

Recently at //Build 2020, Microsoft announced new capabilities to perform optimized training with the ONNX Runtime in addition to inferencing.

Announcing accelerated training with ONNX Runtime-train models up to 45% faster - Open Source Blog

These optimizations led to 45% speed of Microsoft’s own internal Transformer NLP models.

As part of our work to give back to the community Microsoft developed an example repo that demonstrates how to integrate ORT training into the official Nvidia implementation of large BERT model with over 8.3Bn parameters.

microsoft/onnxruntime-training-examples

This implementation can be found with the link above and can be even trained on your data using Azure ML with the example notebook.

However, while this is amazing for getting started it would be nice to understand what is going on under the hood which can be a little overwhelming at first glance.

In the remainder of this article, I will walk through the 4 main modifications that need to be made to pyTorch models for taking advantage of ORT and point you to where in the example code repo you can deep dive to learn more.

I won’t go through every detail of the modification but I will explain the core concepts so that you can get started on your own ORT journey.

Step 1: Set Up ORT Distributed Training Environment

To trains large neural networks often requires distributed compute clusters. In this way we can run a version of our script for each GPU on each VM in our cluster. To properly ensure that our data and our model gradients get updated we need to assign each version of our train script with a:

World Rank — A rank for the process across all the VM Instances
Local Rank — A rank for the script process on a given VM Instance

The ort_supplement provides a setup function that configures the ONNX runtime for distributed training for Open MPI and the Azure Machine Learning Service.

device = ort\_supplement.setup\_onnxruntime\_with\_mpi(args)

The Create ORTTrainer function can be found in the ort_supplement module

Step 2: Create an ORT Trainer Model

Once we have a distributed training environment the next step is to load the pyTorch model into the ORT for training to do this we use the create_ort trainer method from the ort_supplement script.

model = ort\_supplement.create\_ort\_trainer(args, device, model)

The Create ORTTrainer function can be found in the ort_supplement module

The ORT Trainer Model requires a couple of important arguments to implement:

A pyTorch model bundled with a loss function
An optimizer function by default we use the Lamb Optimizer
A model description using the IODescription object to explain the model input and output tensor dimensions. Code for an example of the NVIDIA BERT Description looks as follows

Note the tensor dimensions should be passed as numeric values to get full optimization benefits

Gradient Accumulation Steps — Number of steps to run on a script instance before syncing the gradient
Opset Version — The operation set version for the ONNX runtime. The latest version is 12
A Post Processing Function (Optional) that runs after the ONNX runtime converts the pyTorch model to ONNX that can further be used to support unsupported operations and optimize the ONNX model graph further
Map Optimizer Attributes -maps weight names to a set of optimization parameters.

For more information check out the following resource from the ORT training repo.

microsoft/onnxruntime

Step 3: Call ORT Training Steps to Train Model

Once we’ve initialized our model we need to call run_ort_training_steps to actually step forward with our model, calculate it’s local loss and propagate it’s aggregated gradient.

loss, global\_step = ort\_supplement.run\_ort\_training\_step(args, global\_step, training\_steps, model, batch) # Runs the actual training steps

The run_ort_training steps function can be found in the ort_supplement module

Step 4: Export Trained ONNX Model

Lastly once we have completed all the distributed training iterations of our model. While exporting to ONNX is not mandatory for evaluation doing so enables us to take advantage of ORTs accelerated inferencing. We can export our ORT model to the ONNX format for evaluation by calling the model.save_as_onnx function and providing it with our output destination. This function can also be used to checkpoint our model at each epoch.

 model.save\_as\_onnx(out\_path)

Conclusion

With the four functions above you have the key tools you need to make sense of the full BERT LARGE ONNX training example. To see them in action check out the run_pretraining_ort script below.

microsoft/onnxruntime-training-examples

Next Steps

Now that you are more familiar with how to leverage the ORT SDK take a look at some other really cool ONNX blog posts and examples.

As the Runtime matures we are always looking for more contributors check out our contribution guidelines here. Hope this helps you on your journey to more efficient deep learning.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Visual Brand Detection with Azure Video Indexer

PythicCoder — Fri, 15 May 2020 15:00:03 +0000

TLDR; This post will show how to use the Azure Video Indexer, Computer Vision API and Custom Vision Services to extract key frames and detect custom image tags in indexed videos.

All code for the tutorial can be found in the notebook below. This code can be extended to support almost any image classification or object detection task.

aribornstein/AzureVideoIndexerVisualBrandDetection

The tutorial requires an Azure subscription, however everything can be achieved using the free tier. If you are new to Azure you can get a free subscription here.

Create your Azure free account today | Microsoft Azure

What is Azure Video Indexer?

Azure Video Indexer automatically extracts metadata — such as spoken words, written text, faces, speakers, celebrities, emotions, topics, brands, and scenes from video and audio files. Developers can then access the data within their application or infrastructure, make it more discover-able, and use it to create new over-the-top (OTT) experiences and monetization opportunities

Use the Video Indexer API - Azure Media Services

Often, we wish to extract useful tags from videos content.These tags are often the differentiating factor for having successful engagement on social media services such as Instagram, Facebook, and YouTube

This tutorial will show how to use Azure Video Indexer, Computer Vision API, and Custom Vision service to extract key frames and custom tags. We will use these Azure services to detect custom brand logos in indexed videos.

This code can be extended to support almost any image classification or object detection task.

Step #1 Download A Sample Video with the pyTube API

The first step is to download a sample video to be indexed. We will be downloading an episode of Azure Mythbusters on Azure Machine Learning by my incredible Co-Worker Amy Boyd using the Open Source pyTube API!

Installation:

pyTube can be installed with pip

!pip install pytube3 --upgrade

Code:

from pytube import YouTube
from pathlib import Path

video2Index = YouTube('https://www.youtube.com/watch?v=ijtKxXiS4hE').streams[0].download()

video\_name = Path(video2Index).stem

Step #2 Create An Azure Video Indexer Instance

Navigate to https://www.videoindexer.ai/ and follow the instructions to create an Account

For the next steps, you will need your Video Indexer

Subscription Key
Location
Account Id

These can be found in the account settings page in the Video Indexer Website pictured above. For more information see the documentation below. Feel free to comment below if you get stuck.

Use the Video Indexer API - Azure Media Services

Step #3 Use the Unofficial Video Indexer Python Client to Process our Video and Extract Key Frames

To interact with the Video Indexer API, we will use the unofficial Python client.

Installation:

pip install video-indexer

Code:

Initialize Client:

vi = VideoIndexer(vi\_subscription\_key='SUBSCRIPTION\_KEY',
                  vi\_location='LOCATION',
                  vi\_account\_id='ACCOUNT\_ID')

Upload Video:

video\_id = vi.upload\_to\_video\_indexer(
              input\_filename = video2Index,
              video\_name=video\_name, #must be unique
              video\_language='English')

Get Video Info

info = vi.get\_video\_info(video\_id, video\_language='English')

Extract Key Frame Ids

keyframes = []
for shot in info["videos"][0]["insights"]["shots"]:
    for keyframe in shot["keyFrames"]:
        keyframes.append(keyframe["instances"][0]['thumbnailId'])

Get Keyframe Thumbnails

for keyframe in keyframes:
    img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id,    
                                                  keyframe)

Step #3 Use the Azure Computer Vision API to Extract Popular Brands from Key Frames

Out of the box, Azure Video Indexer uses optical character recognition and audio transcript generated from speech-to-text transcription to detect references to popular brands.

Now, that we have extracted the key frames we are going to leverage the Computer Vision API to extend this functionality to see if there are any known brands in the key frames.

Brand detection - Computer Vision - Azure Cognitive Services

First we will have to create a Computer Vision API key. There is a free tier that can be used for the demo that can be generated with the instructions in the documentation link below. Once done you should get a Computer Vision subscription key and endpoint

Create a Cognitive Services resource in the Azure portal - Azure Cognitive Services

After we have our Azure Computer Vision subscription key and endpoint , we can then use the Client SDK to evaluate our video’s keyframes:

Installation:

pip install --upgrade azure-cognitiveservices-vision-computervision

Code:

Initialize Computer Vision Client

from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials

computervision\_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription\_key))

Send Keyframe To Azure Computer Vision Service to Detect Brands

import time

timeout\_interval, timeout\_time = 5, 10.0
image\_features = ["brands"]

for index, keyframe in enumerate(keyframes):

if index % timeout\_interval == 0:
     print("Trying to prevent exceeding request limit waiting {} seconds".format(timeout\_time))
     time.sleep(timeout\_time)

# Get KeyFrame Image Byte String From Video Indexer
img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id, keyframe)

# Convert Byte Stream to Image Stream
img\_stream = io.BytesIO(img\_str)

# Analyze with Azure Computer Vision
cv\_results = computervision\_client.analyze\_image\_in\_stream(img\_stream, image\_features)

print("Detecting brands in keyframe {}: ".format(keyframe))

if len(cv\_results.brands) == 0:
    print("No brands detected.")

else:
    for brand in cv\_results.brands:

        print("'{}' brand detected with confidence {:.1f}% at location {}, {}, {}, {}".format( brand.name, brand.confidence \* 100, brand.rectangle.x, brand.rectangle.x + brand.rectangle.w, brand.rectangle.y, brand.rectangle.y + brand.rectangle.h))

Azure Computer Vision API — General Brand Detection

Quickstart: Computer Vision client library - Azure Cognitive Services

Step #4 Use the Azure Custom Vision Service to Extract Custom Logos from Keyframes

The Azure Computer Vision API, provides the ability to capture many of the worlds most popular brands, but sometimes a brand may be more obscure. In the last section, we will use the Custom Vision Service, to train a custom logo detector to detect the Azure Developer Relation Mascot Bit in in the keyframes extracted by Video Indexer.

My training set for Custom Bit Detector

This tutorial assumes you know how to train a Custom Vision Service object detection model for brand detection. If not check out the If not, check out the documentation below for a tutorial.

Tutorial: Use custom logo detector to recognize Azure services - Custom Vision - Azure Cognitive Services

Instead of deploying to mobile, however we will use the python client API for the Azure Custom Vision Service. All the information you’ll need can be found in the settings menu of your Custom Vision project.

Settings menu for Custom Vision Service

Installation:

pip install azure-cognitiveservices-vision-customvision

Code:

Initialize Custom Vision Service Client

from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient

prediction\_threshold = .8
prediction\_key = "Custom Vision Service Key"
custom\_endpoint = "Custom Vision Service Endpoint"
project\_id = "Custom Vision Service Model ProjectId"
published\_name = "Custom Vision Service Model Iteration Name"

predictor = CustomVisionPredictionClient(prediction\_key, endpoint=published\_name)

Use Custom Vision Service Model to Predict Key Frames

import time
timeout\_interval, timeout\_time = 5, 10.0

for index, keyframe in enumerate(keyframes):
    if index % timeout\_interval == 0:
       print("Trying to prevent exceeding request limit waiting {} seconds".format(timeout\_time))
       time.sleep(timeout\_time)

    # Get KeyFrame Image Byte String From Video Indexer
    img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id, keyframe)

    # Convert Byte Stream to Image Stream
    img\_stream = io.BytesIO(img\_str)

    # Analyze with Azure Computer Vision
    cv\_results = predictor.detect\_image(project\_id, published\_name, img\_stream)
    predictions = [pred for pred in cv\_results.predictions if pred.probability > prediction\_threshold]
    print("Detecting brands in keyframe {}: ".format(keyframe))

    if len(predictions) == 0:
       print("No custom brands detected.")
    else:
       for brand in predictions:
           print("'{}' brand detected with confidence {:.1f}% at location {}, {}, {}, {}".format( brand.tag\_name, brand.probability \* 100, brand.bounding\_box.left, brand.bounding\_box.top, brand.bounding\_box.width, brand.bounding\_box.height))

Conclusion

And there we have it! I am able to find all the frames that have either Microsoft for or the Cloud Advocacy Bit Logo in my video.

Sample Key Frames with Bit

Next Steps

You now have all you need to extend the Azure Video Indexer Service with your own custom computer vision models. Below is a list of additional resources to take that will help you take your integration with Video Indexer to the next level.

Offline Computer Vision

In a production system, you might see request throttling from a huge number of requests. In this case, the Azure Computer Vision service can be run in an offline container

How to install and run containers - Computer Vision - Azure Cognitive Services

Additionally, the Custom Vision model can be run locally as well.

Tutorial - Deploy Custom Vision classifier to a device using Azure IoT Edge

Video Indexer + Zoom Media

Azure-Samples/media-services-video-indexer

Creating an Automated Video Processing Flow in Azure

Creating an automated video processing flow in Azure

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Evaluating Deep Learning Models in 10 Different Languages (With Examples)

PythicCoder — Tue, 05 May 2020 13:42:18 +0000

ONNX is an open format built to represent machine learning models. ONNX defines a common set of operators — the building blocks of machine learning and deep learning models — and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers. The following post is a compilation of code samples showing how to evaluate Onnx Models in 10 different programming languages.

Next Steps

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

AI April NLP Math Teacher Challenge

PythicCoder — Thu, 30 Apr 2020 10:52:38 +0000

TLDR; Build a model that can perform automatic problem solving, written in natural language on the provided dataset.

Challenge

Build a model that can perform automatic problem solving, written in natural language on the provided dataset.

Example Challenge Problem

Questions

I will be monitoring the github issues of the challenge repo and on twitter feel free to reach out with any questions.

Download Dataset

The math teacher dataset can be downloaded here For more information on the subsets see here.

Submission

Submit a submission by creating a pull request to the submissions folder in the Repo below!

aribornstein/AIApril2020NLPMAthTeacherChallenge

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

Useful Resources

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

Protecting Personal Identifiable Information with Azure AI

PythicCoder — Wed, 22 Apr 2020 18:47:34 +0000

TLDR; The following post will outline both first party and open source techniques for detecting PII with Azure.

What is PII?

Personally Identifiable information (PII), is any data that can be used used to identify a individuals such as names, driver’s license number, SSNs, bank account numbers, passport numbers, email addresses and more. Many regulations from GDPR to HIPPA require strict protection of user privacy.

If you are new to Azure you can get started a free subscription using the link below.

Create your Azure free account today | Microsoft Azure

Detecting PII With Azure Cognitive Search (Preview)

Azure Cognitive Search is a cloud solution that provides developers APIs and tools for adding a rich search experience to their data, content and applications. With cognitive search you can add cognitive skills to apply AI processes during indexing. Doing so can add new information and structures useful for search and other scenarios.

The Azure PII Detection skill (Currently in Preview) extracts personally identifiable information from an input text and gives you the option to mask it from that text in various ways. This skill uses the machine learning models provided by Text Analytics in Cognitive Services.

PII Detection cognitive skill (preview) - Azure Cognitive Search

Detecting PII With Microsoft Presidio

In addition to the first party cognitive search Microsoft also provides an open source PII detection tool for Azure called Presidio which was developed by the Microsoft Commercial Software Engineering team in Israel.

microsoft/presidio

Why use Presidio?

Presidio is open-source, transparent, and scalable. Presidio allows developers and data scientists to customize or add new PII recognizers via API or code to best fit your anonymization needs. Presidio leverages docker and kubernetes for workloads at scale.

Presidio automatically detects Personal-Identifiable Information (PII) in unstructured text, annonymizes it based on one or more anonymization mechanisms, and returns a string with no personal identifiable data. For example:

For each PII entity, presidio returns a confidence score:

Text anonymization in images (beta)

Presidio uses OCR to detect text in images. It further allows the redaction of the text from the original image.

Demo

Check out a public demo to try out on your own data with the link below.

Presidio

Deploying

Installation Steps

Navigate into \deployment from command line.
If You have helm installed, but havn’t run helm init, execute deploy-helm.sh in the command line. It will install tiller (helm server side) on your cluster, and grant it sufficient permissions.
Grant the Kubernetes cluster access to the container registry follow these instructions to grant the AKS cluster access to the ACR.
If you already have helm and tiller configured, or if you installed it in the previous step, execute deploy-presidio.sh in the command line as follows:

deploy-presidio.sh

More information can be found on the github repo and near one click deployment options for Azure are coming soon!

Additional deployment options can be found here.

Next Steps

In this post you learned two of my favorite options for detecting PII in your data with Azure. If you are interested in Azure and AI be sure to check out my other posts and the Azure medium blog.

About the Author

Aaron (Ari) Bornstein is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.