DEV Community: Dmitry Soshnikov

Making an Interactive Cognitive Portrait Exhibit using some Creativity, .NET, Azure Functions and Cognitive Services Magic

Dmitry Soshnikov — Wed, 15 Apr 2020 20:55:56 +0000

As you may know, I am a big fan of Science Art. This January, Moscow ElectroMuseum made an open call for Open Museum 2020 Exhibition. In this post, I will describe the exhibit that I made, and how it was transformed due to quarantine and museum closing.

This post is a part of AI April initiative, where each day of April my colleagues publish new original article related to AI, Machine Learning and Microsoft. Have a look at the Calendar to find other interesting articles that have already been published, and keep checking that page during the month.

In my earlier post, I described Cognitive Portrait technique to produce blended portraits of people from a series of photographs:


Olga, 2019, People Blending	People Circle, 2020, Cognitive Portrait

This January, Moscow ElectroMuseum made a call to all artists to submit their ideas for OpenMuseum exhibition, so I immediately thought of turning Cognitive Portrait idea into something more interactive. What I wanted to create is an interactive stand that will capture photographs of people passing nearby, and transform them into "average" photograph of exhibition visitors.

Because Cognitive Portrait relies on using Cognitive Services to extract face landmarks, the exhibit needs to be Internet-connected. This provides some additional advantages, such as:

Storing produced images and making them available for later expositions
Being able to collect demographic portrait of people at the exhibition, including age distribution, gender, and the amount of time people spend in front of the exhibit. I will not explore this functionality in this post, though.
If you want to chose different cognitive portrait technique, you would only need to change the functionality in the cloud, so you can in fact change exhibits without physically visiting the museum.

The Architecture

From architectural point of view, the exhibit consists of two parts:

Client UWP application, which runs on a computer with the monitor and web cam installed at the museum. UWP application can also be run on Raspberry Pi on Windows IoT Core. Client application does the following:
- Detects a person standing in front of the camera
- When a person stays rather still for a few seconds - it will take the picture and send it to the cloud
- When a result is received from the cloud - show it on the screen / monitor.
Cloud backend, which does the following:
- Receive the picture from the client
- Apply affine transformation to align the picture with predefined eyes coordinates and store the result
- Create the result from a few previous pictures (which already have been aligned) and return it to the client

Below, we will discuss different options for implementing client and cloud parts, and select the best ones.

Virtual Exhibit Bot

When the exhibit has already been developed and demonstrated in the museum for a few weeks, the news came up that the museum will be closed for quarantine for indefinite period of time. The news sparkled the idea to turn the exhibit into the virtual one, so that people can interact with it without leaving their house.

The interaction is done via Telegram chat-bot @PeopleBlenderBot. The bot calls cloud backend via the same REST API as the UWP client application, and the image that it receives back is given as the bot response. Thus, a user is able to send his picture, and get back its people-blended version, which incorporates other users of the bot, together with real people in the gallery.

With @PeopleBlenderBot, the exhibit spans both real and virtual worlds, blending people's faces from their homes and from the exhibition together into one collective image. This new kind of virtual art is a true way to break boundaries and bring people together regardless of their physical location, city or country. You can test the bot here.

UWP Client Application

The main reason why I chose to use Universal Windows Applications platform for client application is because it has face detection functionality out of the box. Also, it is available on Raspberry Pi controller, through the use of Windows 10 IoT Core, however, it turns out to be quite slow, so for my exhibit I used Intel NUC compact computer.

User interface of our application would look like this:

Corresponding XAML layout (slightly simplified) is like this:

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*"/>
        <ColumnDefinition Width="*"/>
    </Grid.ColumnDefinitions>
    <Grid.RowDefinitions>
        <RowDefinition Height="*"/>
        <RowDefinition Height="170"/>
    </Grid.RowDefinitions>
    <Grid x:Name="FacesCanvas" Grid.Row="0" Grid.Column="0">
        <CaptureElement x:Name="ViewFinder" />
        <Rectangle x:Name="FaceRect"/>
        <TextBlock x:Name="Counter" FontSize="60"/>
    </Grid>
    <Grid x:Name="ResultCanvas" Grid.Row="0" Grid.Column="1">
        <Image x:Name="ResultImage" Source="Assets/bgates.jpg"/>
    </Grid>
    <ItemsControl x:Name="FacesLine" Grid.Row="1" Grid.ColumnSpan="2" 
              ItemsSource="{x:Bind Faces,Mode=OneWay}"/>
</Grid>

Here the most important elements are:

ViewFinder to display the live feed from the camera
FaceRect and Counter are elements overlayed on top of ViewFinder to display the rectangle around recognized face, and to provide countdown timer to display 3-2-1 before the picture is taken
ResultImage is the area to display the result received from the cloud
FacesLine is the horizontal line of captured faces of previous visitors displayed at the bottom of the screen. It is declaratively bound to Faces observable collection in our C# code, so to display a new face we just need to add an element to it
All the rest of XAML code is used to lay out the elements properly.

The code to implement face detection is available in this sample. It is a little bit overcomplicated, so I took liberty so simplify it a bit, and will simplify it even further for the sake of clarity.

First, we need to start the camera, and render it's output into the ViewFinder:

MC = new MediaCapture();
var cameras = await DeviceInformation.FindAllAsync(
                              DeviceClass.VideoCapture);
var camera = cameras.First();
var settings = new MediaCaptureInitializationSettings() 
                          { VideoDeviceId = camera.Id };
await MC.InitializeAsync(settings);
ViewFinder.Source = MC;

Then, we create FaceDetectionEffect, which will be responsible for detecting faces. Once the face is detected, it will fire FaceDetectedEvent:

var def = new FaceDetectionEffectDefinition();
def.SynchronousDetectionEnabled = false;
def.DetectionMode = FaceDetectionMode.HighPerformance;
FaceDetector = (FaceDetectionEffect)
     (await MC.AddVideoEffectAsync(def, MediaStreamType.VideoPreview));
FaceDetector.FaceDetected += FaceDetectedEvent;
FaceDetector.DesiredDetectionInterval = TimeSpan.FromMilliseconds(100);
FaceDetector.Enabled = true;
await MC.StartPreviewAsync();

Once face is detected and FaceDetectedEvent is called, we start the countdown timer that will fire every second, and update the Counter textbox to display 3-2-1 message. Once the counter reaches 0, it captures the image from camera to MemoryStream, and calls the web service in the cloud using CallCognitiveFunction:

var ms = new MemoryStream();
await MC.CapturePhotoToStreamAsync(
      ImageEncodingProperties.CreateJpeg(), ms.AsRandomAccessStream());
var cb = await GetCroppedBitmapAsync(ms,DFace.FaceBox);
Faces.Add(cb);
var url = await CallCognitiveFunction(ms);
ResultImage.Source = new BitmapImage(new Uri(url));

We assume that the REST service will take the input image, do all the magic to create the resulting image, store it somewhere in the cloud in the public blob, and return the URL of the image. We then assign this URL to the Source property of ResultImage control, which renders the image on the screen (and UWP runtime is responsible for downloading image from the cloud).

What also happens here, the face is cut out from the picture using GetCroppedBitmapAsync, and added to Faces collection, which makes it automatically appear on the UI.

CallCognitiveFunction does pretty standard call to REST endpoint using HttpClient library:

private async Task<string> CallCognitiveFunction(MemoryStream ms)
{
    ms.Position = 0;
    var resp = await http.PostAsync(function_url,new StreamContent(ms));
    return await resp.Content.ReadAsStringAsync();
}

Azure Function to Create Cognitive Portrait

To do the main job, we will create an Azure Function in Python. Using Azure Functions to manage executable code in the cloud has many benefits:

You do not have to think about the dedicated compute, i.e. how and where the code is executed. That's why Azure Functions are also called Serverless.
In Functions consumption plan, you only pay for the actual function calls, and not for the uptime of the function. The only downside of this plan is that function execution time is limited, and if the call takes a long time to execute - it will time out. In our case, this should not be a problem, as our algorithm is pretty fast.
Function is auto-scaled based on demand, so we don't have to manage scalability explicitly.
Azure Function can be triggered by many different cloud events. In our case, we will fire the function using REST call, but it can also fire as a result of adding blob to storage account, or from new queue message.
Azure Function can be simply integrated with storage in declarative manner.

Just an example: suppose we want to imprint current date on a photograph. We can write a function, specify that it is triggered by a new blob item, and that output of the function should go to another blob. To implement the imprinting, we just need to provide the code for image manipulation, while input and output image will be taken from blob and stored into blob automatically, we will just use them as function parameters.

Azure Functions are so useful that my rule of thumb is to always use Azure Functions whenever you need to do some relatively simple processing in the cloud.

In our case, because Cognitive Portrait algorithm is developed in Python and requires OpenCV, we will create Python Azure Function. In fact, the new version (V2) of Python functions have been significantly improved, so if you have heard something bad about Python implementation of Azure Functions before - forget all about it.

The easiest way to start developing a function is by starting to code locally. The process is well-described in documentation, but let me outline it here. You may also read this tutorial to get familiar with doing some operations from VS Code.

First, you create a function using CLI:

func init coportrait –python
cd coportrait
func new --name pdraw --template "HTTP trigger"

To use this command, you would need to have Azure Functions Core Tools installed.

The function is mainly described by two files: one contains the Python code (in our case, it is __init__.py), and another one - function.json - describes the integrations, i.e. how the function is triggered, and which Azure objects are passed to/from function as input/output parameters.

For our simple function triggered by HTTP request, the function.json would look like this:

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [ "post" ]},
    {
      "type": "http",
      "direction": "out",
      "name": "$return" }]}

Here the name of script file is specified, the input parameter is named req and bound to incoming HTTP trigger, and output parameter is the HTTP response, which is returned as function value ($return). We also specify here that the function supports only POST method - I have removed "get", which was present there initially from the template.

If we look into __init__.py, we have some template code there to start with, which looks like this (in practice, it's a little bit more complicated than this, but never mind):

def main(req:func.HttpRequest) -> func.HttpResponse:
    logging.info('Execution begins…')
    return func.HttpResponse(f"Hello {name}!")

Here, req is the original request. To get the input image, which is encoded as binary JPEG stream, we need the following code:

body = req.get_body()
nparr = np.fromstring(body, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

To work with storage, in our case we will use azure.storage.blob.BlockBlobService object, and not integrations. The reason for this is that we need quite a lot of storage operations, and passing many parameters in and out of the function may be confusing. You may want to see more documentation on working with Azure blob storage from Python.

So we will begin by storing the incoming image into cin blob container:

blob = BlockBlobService(account_name=..., account_key=...) 
sec_p = int((end_date-datetime.datetime.now()).total_seconds())
name =  f"{sec_p:09d}-{time.strftime('%Y%m%d-%H%M%S')}"
blob.create_blob_from_bytes("cin",name,body)

We need to do one trick here with file naming. Because later on we will need to be blending together 10 last photographs, we need a way to retrieve 10 last files without browsing through all blobs in a container. Since blob names are returned in alphabetical order, we need a way for all filenames to be sorted alphabetically in ascending order, last file being the first. The way I solve this problem here is to calcuclate the number of seconds from current time till January 1, 2021, and appending this number with leading zeroes before the filename (which is normal YYYYMMDD-HHMMSS datetime).

Next things we will do is call Face API to extract face landmarks and position the image so that two eyes and middle of the mouth occupy predefined coordinates:

cogface = cf.FaceClient(cognitive_endpoint,
                        CognitiveServicesCredentials(cognitive_key))
res = cogface.face.detect_with_stream(io.BytesIO(body),
                        return_face_landmarks=True)
if res is not None and len(res)>0:
    tr = affine_transform(img,res[0].face_landmarks.as_dict())
    body = cv2.imencode('.jpg',tr)[1]
    blob.create_blob_from_bytes("cmapped",name,body.tobytes())

Function affine_transform is the same as in my other post. Once the image has been transformed, it is stored as JPEG picture into cmapped blob container.

As the last step, we need to prepare the image from the last 10 pictures, store it into the blob, and return the URL. To get last 10 images from blob, we get the iterator for all blobs using list_blobs function, take first 10 elements using islice, and then get the decoded images:

imgs = [ imdecode(blob.get_blob_to_bytes("cmapped",x.name).content) 
         for x in itertools.islice(blob.list_blobs("cmapped"),10) ]
imgs = np.array(imgs).astype(np.float32)

To get the blended picture, we need to average all images along the first axis, which is done with just one numpy call. To make this averaging possible, we convert all values in the code above to np.float32, and after averaging - back to np.uint8:

res = (np.average(imgs,axis=0)).astype(np.uint8)

Finally, storing the image in blob is done in a very similar manner to the code above, encoding the image to JPEG with cv2.imencode, and then calling create_blob_from_bytes:

b = cv2.imencode('.jpg',res)[1]
r = blob.create_blob_from_bytes("out",f"{name}.jpg",b.tobytes())
result_url = f"https://{act}.blob.core.windows.net/out/{name}.jpg"
return func.HttpResponse(result_url)

After putting the function code into __init__.py, we also should not forget to specify all dependencies in requirements.txt:

azure-functions
opencv-python
azure-cognitiveservices-vision-face
azure-storage-blob==1.5.0

Once this is done, we can run the function locally by issuing the command:

func start

This command will start a local web server and print the URL of the function, which we can try to call using Postman to make sure that it works. We need to make sure to use POST method, and to pass the original image as binary body of the request.
We can also specify this URL to function_url variable in our UWP application, and run it on local machine.

To publish Azure Function to the cloud, we need first to create the Python Azure Function through the Azure Portal, or through Azure CLI:

az functionapp create --resource-group PeopleBlenderBot
    --os-type Linux   --consumption-plan-location westeurope
    --runtime python  --runtime-version 3.7
    --functions-version 2
    --name coportrait --storage-account coportraitstore

I recommend doing it through the Azure Portal, because it will automatically create required blob storage, and the whole process is easier for beginners. Once you start looking for ways to automate it - go with Azure CLI.

After the function has been created, deploying it is really easy:

func azure functionapp publish coportrait

After the function has been published, go to Azure Portal, look for the function and copy function URL.

The URL should look similar to this: https://coportrait.azurewebsites.net/api/pdraw?code=geE..e3P==. Assign this link (together with the key) to the function_url variable in your UWP app, start it, and you should be good to go!

Creating the Chat Bot

During social isolation, we need to create a way for people to interact with the exhibit from their homes. The best way to do it is by creating a chat bot using Microsoft Bot Framework.

In the described architecture all the processing happens in the cloud, which makes it possible to create additional user interfaces to the same virtual exhibit. Let's go ahead and create chatbot interface using Microsoft Bot Framework.

The process of creating a bot in C# is described in Microsoft Docs, or in this short tutorial. You can also create the bot in Python, but we will go with .NET as the better documented option.

First of all, we need to install VS Bot Template for Visual Studio, and then create Bot project with the Echo template:

I will call the project PeopleBlenderBot.

Main logic of the bot is located in Bots\EchoBot.cs file, inside the OnMessageActivityAsync function. This function received the incoming message as the Activity object, which contains message Text, as well as its Attachments. We first need to check if the user has attached in image to his message:

if (turnContext.Activity.Attachments?.Count>0)
{
   // do the magic
}
else
{
    await turnContext.SendActivityAsync("Please send picture");
}

If the user has attached an image, it will be available to us as ContentUrl field of Attachment object. Inside the if block, we first get this image as http-stream:

  var http = new HttpClient();
  var resp = await http.GetAsync(Attachments[0].ContentUrl);
  var str = await resp.Content.ReadAsStreamAsync();

Then, we pass this stream to our Azure function, in the same way as we did in the UWP application:

  resp = await http.PostAsync(function_url, new StreamContent(str));
  var url = await resp.Content.ReadAsStringAsync();

Now that we obtained the URL of the image, we need to pass it back to the user as Hero-card attachment:

  var msg = MessageFactory.Attachment(
      (new HeroCard()
      { Images = new CardImage[] { new CardImage(url) } }
      ).ToAttachment());
  await turnContext.SendActivityAsync(msg);

Now you can run the bot locally and test it using Bot Framework Emulator:

Once you make sure the bot works, you can deploy it to Azure as described in Docs using Azure CLI and ARM template. However, you can also do it manually through Azure Portal:

Create new Web App Bot in the portal, selecting Echo Bot as the starting template. This will create the following two objects in your subscription:
- Bot Connector App, which determines the connection between the bot and different channels, such as Telegram
- Bot Web App, where the bot web application itself will run
Download the code of your newly created bot, and copy appsettings.json file from it into the PeopleBlenderBot you are devoping. This file contains App Id and App Password that are required to securely connect to bot connector.
From Visual Studio, right-click on your PeopleBlenderBot project and select Publish. The select the existing Web App created during step 1, and deploy your code there.

Once the bot has been deployed to the cloud, you can make sure that it works by testing it in the web chat:

Making the bot work in Telegram is now a matter of configuring the channel, which is described in detail in the Docs:

Conclusion

I have shared my own experience of creating an interactive exhibit with its logic in the cloud. I encourage you to use this approach if you are doing some work with museums and exhibitions, and if not - just to explore the direction of science art, because it is a lot of fun! You can start by playing with Cognitive Portrait Techniques Repository, and then move on! In my other blog post I discuss AI and Art, and show the examples where AI becomes even more creative. Feel free to get in touch if you want to talk more about the topic, or collaborate!

The bot I have created is currently available in Telegram as @peopleblenderbot and in my local virtual museum, and you can try it our yourself. Remember, that you are not only chatting with the bot, you are enjoying virtual art exhibit that brings people together, and that your photos will be stored in the cloud and sent to other people in blended form. Look forward for more news on when this exhibit will be available in museums!

Credits

In this post, especially for creating drawings, I used some Creative Commons content here, here, here, here. All the work of original authors is greatly appreciated!

Can AI be Creative? Let's Find Out!

Dmitry Soshnikov — Mon, 13 Apr 2020 18:51:29 +0000

Generative Adversarial Network can produce a lot of original paintings much much faster than human painter. But does it make AI creative? Let's discuss the nature of creativity, and try to challenge Artificial Intelligence on this front.

Jump directly to the challenge

This post is a part of AI April initiative, where each day of April my colleagues publish new original article related to AI, Machine Learning and Microsoft. Have a look at the Calendar to find other interesting articles that have already been published, and keep checking that page during the month.

If you were reading some of my earlier blog posts, you know that I like using different AI techniques to produce some Science Art, such as Cognitive Portraits or GAN Paintings like that:


Irina, 2019, People Blending	Summer Landscape, 2020, keragan

A lot of other artists are also using Artificial Intelligence to create their pieces. For example, virtual composer AIVA has been recognized by SACEM French Music Society. In fact, an attempt to use computers to produce music started much earlier with Alan Turing. Lately, on last Microsoft Digital Transformation Summit, our partners at Awara IT combined music and visual arts in the "NeuroKandinsky" performance.

Can AI Create a Piece of Art

To figure this out, we first need to come up with the definition of art. The definition in Wikipedia is the following:

Art is a diverse range of human activities in creating visual, auditory or performing artifacts (artworks), expressing the author's imaginative, conceptual ideas, or technical skill, intended to be appreciated for their beauty or emotional power

For some reason, it explicitly limits artistic activities to humans. However, if we look at the piece of art by itself, without referring to its history, it would sometimes be hard to tell who was the original creator, human being or not. This, it would probably make sense to adopt another definition or art, which would allow us to distinguish art from garbage without looking at the author:

A piece of art is an artifact that is somehow valued by art connoisseurs, for example, by being payed for at auctions.

Which is to say that if an artistic piece is bought for a high price at an auction -- it is definitely art. Remember the case with Edmond de Belamy piece sold for more than $400K at Christie's?

However, this definition does not reflect the inner value or beauty of an artifact, because in the case mentioned above the buyer payed such a high price not for the artwork as such, but rather for the fact of buying the first AI-created art piece. It is the same reason why most of the price for Coca-Cola or Pepsi bottle is not the price of a drink, but of brand. So the actual value lies not in the artwork, but in the story behind it, and this story is created by a human being.

Who Actually Creates Art

So, we have come to realize that it is always a human being who creates a story behind a piece of art, because only human beings have a motivation to be creative. Now let's talk about the process of creating an artwork.

In the case of Cognitive Portrait we used AI to extract coordinates of facial landmarks from people's faces, which helps us to align pictures together in a certain way. How the pictures are aligned is defined by an algorithm written by a human being, and this algorithm helps her/him to achieve certain artistic effect and carry a specific message.


Growing up, 2020, Cognitive Portrait

For example, this picture is constructed from about 50 photos of my daughter, which are grouped together by age intervals, to reflect the process of her growing up. And the following piece, produced from the same set of photos, represents the circular structure of time, when you are caught up in some whirlpool of places and events:


Circle of Faces, 2020, Cognitive Portrait

So, quite clearly, when creating cognitive portraits we used AI as a very powerful tool that helped us with image editing. We could have actually performed the same process in Photoshop manually, aligning all photos, but it would be much more time-consuming and limiting compared to using an algorithm.

The case with Generative Adversarial Networks is more complex to tackle, because it looks like the Neural Network, after having been trained on a dataset of human artworks, has learnt to produce new original art pieces all by itself. In fact, the network learns similar to a human being learning to see or draw: when looking at many images it learns which low-level pixel combinations are used that represent brush strokes, and then how those strokes are typically combined together into larger objects, finally leading to the whole picture composition. In the same way a child learns to see by making sense of light patterns around him, and then painter looks at many earlier artworks to gain some inspiration.

Unlike human being, neural network then randomly combines those pieces of knowledge into a painting, while a human artist would most probably have some goal or idea in mind, which he wants to express using techniques that he has previously seen or tried.

Human beings have a goal, an idea, a motivation that directs his/her work, while artificial intelligence acts randomly.

As an example, look at the two pieces produced by the same network:


-	Countryside, 2020

Two images for us look quite different, somehow we know that image on the left is garbage, and the one on the right is not, because it creates some feelings inside us. However, for a computer those two images are more or less the same, and in fact they were scored similarly by GAN discriminator. So the human being is the one who holds the truth, who knows how to tell the good from the bad.

In order to create the artwork like the one above, the human has to:

Select the dataset of the images to be used for training. Keep in mind that all those pieces were also created by human beings, reflecting our view on what is beautiful, and what is not.
Perform hyperparameter optimization, selecting the parameters for the network which lead to the best results.
Select the best works from hundreds of images generated by the network. Last two steps are something which requires human inspection and human understanding of what to consider art.
Create some story behind the artwork, which at least requires naming it, and then possibly putting it up for sale at an auction, creating artistic value.

Creating artificial art is a joint process, where human artist works together with artificial intelligence tool to produce the artistic impact that he wants.

Actually, the problem of "who is the author of an artwork" has been considered much earlier than AI came about, and it is not specific to AI. For example, Japanese Photographer Tetsuya Kusu used a camera that periodically takes pictures, and collected a gallery of artworks that were taken by the camera itself during his travel through US. This process is in fact very similar to our GAN case: to come up with something beautiful a human being needs to filter out a lot of crap from automatically-generated images. Human being is the one who understands the criteria of beauty, not the camera, and not the artificial neural network.

A Challenge

At the end of this post, I want to challenge you to express your creativity and to prove that AI is in fact only a tool -- but a very useful one, which empowers software developers to be creative in the field of visual arts. I would challenge you to create your own Cognitive Portrait!

To do so:

Go to Cognitive Portrait Repository: http://github.com/CloudAdvocacy/CognitivePortrait
Have a look at the sample code there, which is presented in the form of Jupyter Notebooks.
Fork the repo, and create your own portrait! I recommend following some rules:
- If you are going to change the code in the notebook - please copy the notebook to a new file, so that original code is preserved. This would allow people to see both the original code and your new code.
- If you are uploading your own pictures -- create a subdirectory under images/, so that your pictures do not interfere with others.
- Place the sample image from your script into results folder, so that it is easy to see.
- The front page readme.md contains the gallery of different cognitive portrait techniques, please add your name/resulting image there.
- Do a pull request, and I will include your code into the original repository!
- Finally, post your results in comments here, for everyone to enjoy!

Some ideas for you to explore:

Start simple - just create your own cognitive portrait, by uploading your images into images/your_name, running one of the notebooks which are already there, and then storing the result into results/your_name.jpg.
Once you do that - copy one of the notebooks into a new file and start experimenting! You may try to position eyes along some geometrical curve, or adding some random movement to faces within cognitive portrait, or do some other crazy things related to the coordinates of facial landmarks!
Do not forget to add your creation to the readme.md and do a pull request!

At the end of the month, I will feature the best creations in my blog and social networks, and we will all celebrate together!

Creating Generative Art using GANs on Azure ML

Dmitry Soshnikov — Wed, 08 Apr 2020 13:29:34 +0000

This post is a part of AI April initiative, where each day of April my colleagues publish new original article related to AI, Machine Learning and Microsoft. Have a look at the Calendar to find other interesting articles that have already been published, and keep checking that page during the month.

Deep Learning can look like Magic! I get the most magical feeling when watching neural network doing something creative, for example learning to produce paintings like an artist. Technology behind this is called Generative Adversarial Networks, and in this post we will look at how to train such a network on Azure Machine Learning Service.

If you have seen my previous posts on Azure ML (about using it from VS Code and submitting experiments and hyperparameter optimization), you should know that it is quite convenient to use Azure ML for almost any training tasks. However, all examples up to now have been done using toy MNIST dataset. Today we will focus on real problem: creating artificial paintings like those:


Flowers, 2019, Art of the Artificial keragan trained on WikiArt Flowers	Queen of Chaos, 2019, keragan trained on WikiArt Portraits

Those painting are produced after training the network on paintings from WikiArt. If you want to reproduce the same results, you may need to collect the dataset yourself, for example by using WikiArt Retriever, or borrowing existing collections from WikiArt Dataset or GANGogh Project.

Place images you want to train on somewhere in dataset directory. For training on flowers, here is how some of those images might look like:

We need our neural network model to learn both high-level composition of flower bouquet and a vase, as well as low-level style of painting, with smears of paint and canvas texture.

Generative Adversarial Networks

Those painting were generated using Generative Adversarial Network, or GAN for short. In this example, we will use my simple GAN implementation in Keras called keragan, and I will show some simplified code parts from it.

shwars / keragan

Keras implementation of GANs

GAN consists of two networks:

Generator, which generates images given some input noise vector
Discriminator, whose role is to differentiate between real and "fake" (generated) paintings

Training the GAN involves the following steps:

Getting a bunch of generated and real images:

noise = np.random.normal(0, 1, (batch_size, latent_dim))
gen_imgs = generator.predict(noise)   
imgs = get_batch(batch_size)

Training discriminator to better differentiate between those two. Note, how we provide vector with ones and zeros as expected answer:

d_loss_r = discriminator.train_on_batch(imgs, ones)
d_loss_f = discriminator.train_on_batch(gen_imgs, zeros)
d_loss = np.add(d_loss_r , d_loss_f)*0.5

Training the combined model, in order to improve the generator:

g_loss = combined.train_on_batch(noise, ones)

During this step, discriminator is not trained, because its weights are explicitly frozen during creation of combined model:

discriminator = create_discriminator()
generator = create_generator()
discriminator.compile(loss='binary_crossentropy', optimizer=optimizer, 
                      metrics=['accuracy'])
discriminator.trainable = False

z = keras.models.Input(shape=(latent_dim,))
img = generator(z)
valid = discriminator(img)

combined = keras.models.Model(z, valid) 
combined.compile(loss='binary_crossentropy', optimizer=optimizer)

Discriminator Model

To differentiate between real and fake image, we use traditional Convolutional Neural Network (CNN) architecture. So, for the image of size 64x64, we will have something like this:

discriminator = Sequential()

for x in [16,32,64]: # number of filters on next layer
    discriminator.add(Conv2D(x, (3,3), strides=1, padding="same"))
    discriminator.add(AveragePooling2D())
    discriminator.addBatchNormalization(momentum=0.8))
    discriminator.add(LeakyReLU(alpha=0.2))
    discriminator.add(Dropout(0.3))

discriminator.add(Flatten())
discriminator.add(Dense(1, activation='sigmoid'))

We have 3 convolution layers, which do the following:

Original image of shape 64x64x3 is passed over by 16 filters, resulting in a shape 32x32x16. To decrease the size, we use AveragePooling2D.
Next step converts 32x32x16 tensor into 16x16x32
Finally, after the next convolution layer, we end up with tensor of shape 8x8x64.

On top of this convolutional base, we put simple logistic regression classifier (AKA 1-neuron dense layer).

Generator Model

Generator model is slightly more complicated. First, imagine if we wanted to convert an image to some sort of feature vector of length latent_dim=100. We would use convolutional network model similar to the discriminator above, but final layer would be a dense layer with size 100.

Generator does the opposite - converts vector of size 100 to an image. This involves a process called deconvolution, which is essentially a reversed convolution. Together with UpSampling2D they cause the size of the tensor to increase at each layer:

generator = Sequential()
generator.add(Dense(8 * 8 * 2 * size, activation="relu", 
                                      input_dim=latent_dim))
generator.add(Reshape((8, 8, 2 * size)))

for x in [64;32;16]:
    generator.add(UpSampling2D())
    generator.add(Conv2D(x, kernel_size=(3,3),strides=1,padding="same"))
    generator.add(BatchNormalization(momentum=0.8))
    generator.add(Activation("relu"))

generator.add(Conv2D(3, kernel_size=3, padding="same"))
generator.add(Activation("tanh"))

At the last step, we end up with tensor size 64x64x3, which is exactly the size of the image that we need.
Note that final activation function is tanh, which gives an output in the range of [-1;1] - which means that we need to scale original training images to this interval. All those steps for preparing images is handled by ImageDataset class, and I will not go into detail there.

Training script for Azure ML

Now that we have all pieces for training the GAN together, we are ready to run this code on Azure ML as an experiment! The code I will be showing here is available at GitHub:

CloudAdvocacy / AzureMLStarter

This is some tutorial to get you started with Azure ML Service

There is one important thing to be noted, however: normally, when running an experiment in Azure ML, we want to track some metrics, such as accuracy or loss. We can log those values during training using run.log, as described in my previous post, and see how this metric changes during training on Azure ML Portal.

In our case, instead of numeric metric, we are interested in the visual images that our network generates at each step. Inspecting those images while experiment is running can help us decide whether we want to end our experiment, alter parameters, or continue.

Azure ML supports logging images in addition to numbers, as described here. We can log either images represented as np-arrays, or any plots produced by matplotlib, so what we will do is plotting three sample images on one plot. This plotting will be handled in callbk callback function that gets called by keragan after each training epoch:

def callbk(tr):
    if tr.gan.epoch % 20 == 0:
        res = tr.gan.sample_images(n=3)
        fig,ax = plt.subplots(1,len(res))
        for i,v in enumerate(res):
            ax[i].imshow(v[0])
        run.log_image("Sample",plot=plt)

So, the actual training code will look like this:

gan = keragan.DCGAN(args)
imsrc = keragan.ImageDataset(args)
imsrc.load()
train = keragan.GANTrainer(image_dataset=imsrc,gan=gan,args=args)

train.train(callbk)

Note that keragan supports automatic parsing of many command-line parameters that we can pass to it through args structure, and that is what makes this code so simple.

Starting the Experiment

To submit the experiment to Azure ML, we will use the code similar to the one discussed in the previous post on Azure ML. The code is located inside [submit_gan.ipynb][https://github.com/CloudAdvocacy/AzureMLStarter/blob/master/submit_gan.ipynb], and it starts with familiar steps:

Connecting to the Workspace using ws = Workspace.from_config()
Connecting to the Compute cluster: cluster = ComputeTarget(workspace=ws, name='My Cluster'). Here we need a cluster of GPU-enabled VMs, such as NC6.
Uploading our dataset to the default datastore in the ML Workspace

After that has been done, we can submit the experiment using the following code:

exp = Experiment(workspace=ws, name='KeraGAN')
script_params = {
    '--path': ws.get_default_datastore(),
    '--dataset' : 'faces',
    '--model_path' : './outputs/models',
    '--samples_path' : './outputs/samples',
    '--batch_size' : 32,
    '--size' : 512,
    '--learning_rate': 0.0001,
    '--epochs' : 10000
}
est = TensorFlow(source_directory='.',
    script_params=script_params,
    compute_target=cluster,
    entry_script='train_gan.py',
    use_gpu = True,
    conda_packages=['keras','tensorflow','opencv','tqdm','matplotlib'],
    pip_packages=['git+https://github.com/shwars/keragan@v0.0.1']

run = exp.submit(est)

In our case, we pass model_path=./outputs/models and samples_path=./outputs/samples as parameters, which will cause models and samples generated during training to be written to corresponding directories inside Azure ML experiment. Those files will be available through Azure ML Portal, and can also be downloaded programmatically afterwards (or even during training).

To create the estimator that can run on GPU without problems, we use built-in Tensorflow estimator. It is very similar to generic Estimator, but also provides some out-of-the-box options for distributed training. You can read more about using different estimators in the official documentation.

Another interesting point here is how we install keragan library - directly from GitHub. While we can also install it from PyPI repository, I wanted to demonstrate you that direct installation from GitHub is also supported, and you can even indicate a specific version of the library, tag or commit ID.

After the experiment has been running for some time, we should be able to observe the sample images being generated in the Azure ML Portal:

Running Many Experiments

The first time we run GAN training, we might not get excellent results, for several reasons. First of all, learning rate seems to be an important parameter, and too high learning rate might lead to poor results. Thus, for best results we might need to perform a number of experiments.

Parameters that we might want to vary are the following:

--size determines the size of the picture, which should be power of 2. Small sizes like 64 or 128 allow for fast exprimentation, while large sizes (up to 1024) are good for creating higher quality images. Anything above 1024 will likely not produce good results, because special techniques are required to train large resolutions GANs, such as progressive growing
--learning_rate is surprisingly quite an important parameter, especially with higher resolutions. Smaller learning rate typically gives better results, but training happens very slowly.
--dateset. We might want to upload pictures of different styles into different folders in the Azure ML datastore, and start training multiple experiments simultaneously.

Since we already know how to submit the experiment programmatically, it should be easy to wrap that code into a couple of for-loops to perform some parametric sweep. You may then check manually through Azure ML Portal which experiments are on their way to good results, and terminate all other experiments to save costs. Having a cluster of a few VMs gives you the freedom to start a few experiments at the same time without waiting.

Getting Experiment Results

Once you are happy with results, it makes sense to get the results of the training in the form or model files and sample images. I have mentioned that during the training our training script stored models in outputs/models directory, and sample images - to outputs/samples. You can browse those directories in the Azure ML Portal and download the files that you like manually:

You can also do that programmatically, especially if you want to download all images produced during different epochs. run object that you have obtained during experiment submission gives you access to all files stored as part of that run, and you can download them like this:

run.download_files(prefix='outputs/samples')

This will create the directory outputs/samples inside the current directory, and download all files from remote directory with the same name.

I you have lost the reference to the specific run inside your notebook (it can happen, because most of the experiments are quite long-running), you can always create it by knowing the run id, which you can look up at the portal:

run = Run(experiment=exp,run_id='KeraGAN_1584082108_356cf603')

We can also get the models that were trained. For example, let's download the final generator model, and use it for generating a bunch of random images. We can get all filenames that are associated with the experiment, and filter out only those that represent generator models:

fnames = run.get_file_names()
fnames = filter(lambda x : x.startswith('outputs/models/gen_'),fnames)

They will all look like outputs/models/gen_0.h5, outputs/models/gen_100.h5 and so on. We need to find out the maximum epoch number:

no = max(map(lambda x: int(x[19:x.find('.')]), fnames))
fname = 'outputs/models/gen_{}.h5'.format(no)
fname_wout_path = fname[fname.rfind('/')+1:]
run.download_file(fname)

This will download the file with the highers epoch number to local directory, and also store the name of this file (w/out directory path) into fname_wout_path.

Generating new Images

Once we have obtained the model, we can just need load it in Keras, find out the input size, and give the correctly sized random vector as the input to produce new random painting generated by the network:

model = keras.models.load_model(fname_wout_path)
latent_dim=model.layers[0].input.shape[1].value
res = model.predict(np.random.normal(0,1,(10,latent_dim)))

Output of the generator network is in the range [-1,1], so we need to scale it linearly to the range [0,1] in order to be correctly displayed by matplotlib:

res = (res+1.0)/2
fig,ax = plt.subplots(1,10,figsize=(15,10))
for i in range(10):
    ax[i].imshow(res[i])

Here is the result we will get:

Have a look at some of the best pictures produced during this experiment:


Colourful Spring, 2020	Countryside, 2020

Through the Icy Glass, 2020	Summer Landscape, 2020

Subscribe to @art_of_artificial Instagram - I try to publish new pictures produced by GAN periodically.

Observing The Process of Learning

It is also interesting to look at the process of how GAN network gradually learns. I have explored this notion of learning in my exhibition Art of the Artificial. Here are a couple of videos that show this process:

Food for Thought

In this post, I have described how GAN works, and how to train it using Azure ML. This definitely opens up a lot of room for experimentation, but also a lot of room for thought. During this experiment we have created original artworks, generated by Artificial Intelligence. But can they be considered ART? I will discuss this in one of my next posts...

Acknowledgements

When producing keragan library, I was largely inspired by this article, and also by DCGAN implementation by Maxime Ellerbach, and partly by GANGogh project. A lot of different GAN architectures implemented in Keras are presented here.

Using Azure Machine Learning for Hyperparameter Optimization

Dmitry Soshnikov — Tue, 10 Mar 2020 19:54:20 +0000

Most machine learning models are quite complex, containing a number of so-called hyperparameters, such as layers in a neural network, number of neurons in the hidden layers, or dropout rate. To build the best model, we need to chose the combination of those hyperparameters that works best. This process is typically quite tedious and resource-consuming, but Azure Machine Learning can make it much simpler.

In my previous post about Azure Machine Learning I have described how to start using Azure ML from Visual Studio Code. We will continue to explore the example described there, training simple model to do digit classification on MNIST dataset.

Automation with Azure ML Python SDK

Hyperparameter optimization means that we need to perform large number of experiments with different parameters. We know that Azure ML allows us to accumulate all experiment results (including achieved metrics) in one place, Azure ML Workspace. So basically all we need to do is to submit a lot of experiments with different hyperparameters.

Instead of doing it manually from VS Code, we can do it programmatically through Azure ML Python SDK. All operations, including creating the cluster, configuring the experiment, and getting the results can be done with a few lines of Python code. This code can look a bit complex at first, but once you write (or understand) it once --- you will see how convenient it is to use.

Running the Code

The code that I refer to in this post is available in Azure ML Starter repository. Most of the code that I will describe here is contained inside submit.ipynb notebook. You can execute it in a few ways:

If you have local Python environment installed, you can simply start local instance of Jupyter by running jupyter notebook in the directory with submit.ipynb. In this case, you need to install Azure ML SDK by running pip install azureml-sdk
By uploading it to Notebook section in your Azure ML Portal and running it from there. You will also probably need to create a VM for executing notebooks from Azure ML Workspace, but that can be done from the same web interface quite seamlessly.
By uploading it to Azure Notebooks

If you prefer working with plain Python files, the same code is available in submit.py as well.

Connecting to Workspace and Cluster

The first thing you need to do when using Azure ML Python SDK is to connect to Azure ML Workspace. To do so, you need to provide all required parameters such as subscription id, workspace and resource group name more info on docs:

ws = Workspace(subscription_id,resource_group,workspace_name)

The easiest way to connect is to store all required data inside config.json file, and then instantiate the workspace reference like this:

ws = Workspace.from_config()

You can download config.json file from your Azure Portal, by navigating to the Azure ML Workspace page:

Once we have obtained the workspace reference, we can get the reference to the compute cluster that we want to use:

cluster_name = "AzMLCompute"
cluster = ComputeTarget(workspace=ws, name=cluster_name)

This code assumes that you have already created the cluster manually (as described in the previous post). You can also create the cluster with required parameters programmatically, and the corresponding code is provided in submit.ipynb.

Preparing and Uploading Dataset

In our MNIST training example, we downloaded MNIST dataset from the OpenML repository on the Internet inside the training script. If we want to repeat the experiment many times, it would make sense to store the data somewhere close to the compute --- inside the Azure ML Workspace.

First of all, let's create MNIST dataset as a file on disk in dataset folder. To do that, run [create_dataset.py](https://github.com/CloudAdvocacy/AzureMLStarter/blob/master/create_dataset.py) file, and observe how dataset folder is created, and all data files are stored there.

Each Azure ML Workspace has a default datastore associated with it. To upload our dataset to the default datastore, we need just a couple of lines of code:

ds = ws.get_default_datastore()
ds.upload('./dataset', target_path='mnist_data')

Submitting Experiments Automatically

In this example, we will train the two-layer neural network model in Keras, using train_keras.py training script. This script can take a number of command-line parameters, which allow us to set different values for hyperparameters of our model during training:

--data_folder, that specifies path to the dataset
--batch_size to use (default is 128)
--hidden, size of the hidden layer (default is 100)
--dropout to use after the hidden layer

To submit the experiment with given parameters, we first need to create Estimator object to represent our script:

script_params = {
    '--data_folder': ws.get_default_datastore(),
    '--hidden': 100
}

est = Estimator(source_directory='.',
                script_params=script_params,
                compute_target=cluster,
                entry_script='train_keras.py',
                pip_packages=['keras','tensorflow'])

In this case, specified just one hyperparameter explicitly, but of course we can pass any parameters to the script to train the model with different hyperparameters. Also, note that estimator defines pip (or conda) packages that need to be installed in order to run our script.

Now, to actually execute the experiment, we need to run:

exp = Experiment(workspace=ws, name='Keras-Train')
run = exp.submit(est)

You can then monitor the experiment right inside the notebook by printing our run variable (it is recommended to have azureml.widgets extension installed in Jupyter, if you are running it locally), or by going to the Azure ML Portal:

Hyperparameter Optimization using HyperDrive

Optimization of hyperparameters involves some sort of parametric sweep search, which means that we need to run many experiments with different combinations of hyperparameters and compare the results. This can be done manually using the approach we have just discussed, or it can be automated using the technology called Hyperdrive.

In Hyperdrive, we need to define search space for hyperparameters, and the sampling algorithm, which controls the way hyperparameters are selected from that search space:

param_sampling = RandomParameterSampling({
         '--hidden': choice([50,100,200,300]),
         '--batch_size': choice([64,128]), 
         '--epochs': choice([5,10,50]),
         '--dropout': choice([0.5,0.8,1])})

In our case, the search space is defined by a set of alternatives (choice), while it is also possible to use continuos intervals with different probability distributions (uniform, normal, etc. -- more details here). In addition to Random Sampling, it is also possible to use Grid Sampling (for all possible combinations of parameters) and Bayesian Sampling.

In addition, we can also specify Early Termination Policy. It makes sense if our script reports metrics periodically during the execution -- in this case we can detect that accuracy being achieved by a particular combination of hyperparameters is lower than median accuracy, and terminate the training early:

early_termination_policy = MedianStoppingPolicy()
hd_config = HyperDriveConfig(estimator=est,
  hyperparameter_sampling=param_sampling,
  policy=early_termination_policy,
  primary_metric_name='Accuracy',
  primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,
  max_total_runs=16,
  max_concurrent_runs=4)

Having defined all the parameters for a hyperdrive experiment, we can submit it:

experiment = Experiment(workspace=ws, name='keras-hyperdrive')
hyperdrive_run = experiment.submit(hd_config)

In the Azure ML Portal, hyperparameter optimization is represented by one experiment. To view all results on one graph, select include child runs checkbox:

Chosing the Best Model

We can compare the results and select the best model manually in the portal. In our training script train_keras.py, after training the model, we stored the result into outputs folder:

hist = model.fit(...)
os.makedirs('outputs',exist_ok=True)
model.save('outputs/mnist_model.hdf5')

Having done this, we can now locate the best experiment in the Azure ML Portal and get the corresponding .hdf5 file to be used in inference:

Alternatively, we can use Azure ML Model Management to register the model, which would allow us to keep better track of it, and use it during Azure ML Deployment. We can programmatically find the best model and register it using the following code:

best_run = hyperdrive_run.get_best_run_by_primary_metric()
best_run_metrics = best_run.get_metrics()
print('Best accuracy: {}'.format(best_run_metrics['Accuracy']))
best_run.register_model(model_name='mnist_keras', 
                        model_path='outputs/mnist_model.hdf5')

Conclusion

We have learnt how to submit Azure ML Experiments programmatically via Python SDK, and how to perform Hyperparameter optimization using Hyperdrive. While it takes a while to get used to this process, you will soon realize that Azure ML simplifies the process of model tuning, comparing to doing it "by hand" on a Data Science Virtual Machine.

References

The Best Way to Start With Azure Machine Learning

Dmitry Soshnikov — Mon, 27 Jan 2020 19:24:11 +0000

I know many data scientists, including myself, who do most of their work on a GPU-enabled machine, either locally or in the cloud, through Jupyter Notebooks or some Python IDE. During my two years as AI/ML software engineer that is exactly what I was doing, preparing data on one machine without GPU, and then using GPU VM in the cloud to do the training.

On the other hand, you have probably heard of Azure Machine Learning - a special platform service for doing ML. However, if you start looking at some getting started tutorials, you will have the impression that using Azure ML creates a lot of unnecessary overhead, and the process is not ideal. For example, the training script in the example above is created as a text file in one Jupyter cell, without code completion, or any convenient ways of executing it locally or debugging. This extra overhead was the reason we did not use it as much in our projects.

However, I recently found out that there is a Visual Studio Code Extension for Azure ML. With this extension, you can develop your training code right in the VS Code, run it locally, and then submit the same code to be trained on a cluster with just a few clicks of a button. By doing so, you achieve several important benefits:

You can spend most of the time locally on your machine, and use powerful GPU resources only for training. The training cluster can be automatically resized on demand, and by setting the min amount of machines to 0 you can spin up the VM on demand.
You keep all results of your training, including metrics and created models, in one central location - no need to keep the record of your accuracy for each experiment manually.
If several people work on the same project - they can use the same cluster (all experiments will be queued), and they can view each other's experiment result. For example, you can use Azure ML in a classroom environment, and instead of giving each student an individual GPU machine you can create one cluster that will serve everyone, and foster the competition between students on model accuracy.
If you need to perform many trainings, for example for hyperparameter optimization - all that can be done with just few commands, no need to run series of experiments manually.

I hope you are convinced to try Azure ML yourself! Here is the best way to start:

Install Visual Studio Code, Azure Sign In and Azure ML Extensions
Clone the repository https://github.com/CloudAdvocacy/AzureMLStarter - it contains some sample code to train the model to recognize MNIST digits. You can then open the cloned repository in VS Code.
Read on!

Azure ML Workspace and Portal

Everything in Azure ML is organized around a Workspace. It is a central point where you submit your experiments, store your data and resulting models. There is also a special Azure ML Portal that provides web interface for your workspace, and from there you can perform a lot of operations, monitor your experiments and metrics, and so on.

You can either create a workspace through Azure Portal web interface (see step-by-step instructions), or using Azure CLI (instructions):

az extension add -n azure-cli-ml
az group create -n myazml -l northeurope
az ml workspace create -w myworkspace -g myazml

Workspace contains some Compute resources. Once you have a training script, you can submit experiment to the workspace, and specify compute target - it will make sure the experiment runs there, and stores all the results of the experiment in the workspace for future reference.

MNIST Training Script

In our example, we will show how to solve very traditional problem of handwritten digit recognition using MNIST dataset. In the same manner you will be able to run any other training scripts yourself.

Our sample repository contains simple MNIST training script train_local.py. This script downloads MNIST dataset from OpenML, and then uses SKLearn LogisticRegression to train the model and print the resulting accuracy:

mnist = fetch_openml('mnist_784')
mnist['target'] = np.array([int(x) for x in mnist['target']])

shuffle_index = np.random.permutation(len(mist['data']))
X, y = mnist['data'][shuffle_index], mnist['target'][shuffle_index]

X_train, X_test, y_train, y_test = 
  train_test_split(X, y, test_size = 0.3, random_state = 42)

lr = LogisticRegression()
lr.fit(X_train, y_train)
y_hat = lr.predict(X_test)
acc = np.average(np.int32(y_hat == y_test))

print('Overall accuracy:', acc)

Of course, we are using Logistic Regression just as illustration, not implying that it is a good way to solve the problem...

Running the Script in Azure ML

You can just run this script locally and see the result. If we chose to use Azure ML, however, it will give us two major benefits:

Scheduling and running training on a centralized compute resource, which is typically more powerful than a local computer. Azure ML will take care of packaging our script into a docker container with appropriate configuration
Logging the results of training into a centralized location inside Azure ML Workspace. To do so, we need to add the following lines of code to our script, to record the metrics:

from azureml.core.run import Run
...
try:    
    run = Run.get_submitted_run()
    run.log('accuracy', acc)
except:
    pass

Modified version of our script is called train_universal.py (it is just a bit more complicated than the code presented above), and it can be run both locally (without Azure ML), and on remote compute resource.

To run it on Azure ML from VS Code, follow those steps:

Make sure your Azure Extension is connected to your cloud account. Select Azure icon in the left menu. If you are not connected, you will see a notification on the right bottom offering you to connect (see picture). Click on it, and sign in through browser. You can also press Ctrl-Shift-P to bring up command palette, and type in Azure Sign In.
After that, you should be able to see your workspace in the MACHINE LEARNING section of Azure bar:

Here you should see different objects inside your workspace: compute resources, experiments, etc.
Go back to the list of files, and right-click on train_universal.py and select Azure ML: Run as experiment in Azure.
Confirm your Azure subscription and you workspace, and then select Create new experiment:
Create new Compute and compute configuration:
- Compute defines a computing resource which is used for training/inference. You can use your local machine, or any cloud resources. In our case, we will use AmlCompute cluster. Please create a scalable cluster of STANDARD_DS3_v2 machines, with min=0 and max=4 nodes. You can do that either from VS Code interface, or from ML Portal.
- Compute Configuration defines the options for containers which are created to perform training on a remote resource. In particular, it specifies all libraries that should be installed. In our case, select SkLearn, and confirm the list of libraries.
You will then see a window with JSON description of the next experiment. You can edit the information there, for example change the experiment or cluster name, and tweak some parameters. When you are ready, click on Submit Experiment:
After the experiment has been successfully submitted in VS Code, you will get the link to the Azure ML Portal page with experiment progress and results.

You can also find your experiment from Experiments tab in Azure ML Portal, or from Azure Machine Learning bar in VS Code:
If you want to run the experiment again after adjusting some parameters in your code, this process would be much faster and easier. When you right-click your training file, you will see a new menu option Repeat last run - just select it, and the experiment will be submitted right away.

You will then see the metric results from all runs on Azure ML Portal, as in the screenshot above.

Now you know that submitting runs to Azure ML is not complicated, and you get some goodies (like storing all statistics from your runs, models, etc.) for free.

You may have noticed that in our case the time it takes for the script to run on the cluster is more than running locally - it may even take several minutes. Of course, there is some overhead in packaging the script and all environment in a container, and sending it to the cloud. If the cluster is set to automatically scale down to 0 nodes - there might be some additional overhead due to VM startup, and all that is noticeable when you have a small sample script that otherwise takes a few seconds to execute. However, in real life scenarios, when training takes tens of minutes and sometimes much more - this overhead becomes barely important, especially given the speed improvements you can expect to get from the cluster.

What's Next?

Now that you know how to submit any scripts for execution on a remote cluster, you can start taking advantage of Azure ML in your daily work. It will allow you to develop scripts on a normal PC, and then schedule it for execution on GPU VM or cluster automatically, keeping all the results in one place.

However, there are more advantages from using Azure ML than just those two. Azure ML can also be used for data storage and dataset handling - making it super-easy for different training scripts to access the same data. Also, you can submit experiments automatically through the API, varying the parameters - and thus performing some hyperparameter optimization. Moreover, there is a specific technology built into Azure ML called Hyperdrive, which does more clever hyperparameter search. I will talk more about those features and technologies in my next posts.

Useful Resources

You may find the following courses from Microsoft Learn useful, in case you want to know more:

8 Reasons You Absolutely Need to Use Azure Notebooks

Dmitry Soshnikov — Tue, 19 Nov 2019 16:46:36 +0000

If you are a data scientist or machine learning software engineer as I am, you probably write most of your code in Jupyter Notebooks. For those of you who are not -- Jupyter is a great system that allows you to combine markdown-based text and executable code in one web-based and web-editable document called notebook.

The way most data scientists work is to install their own copy of Python dev environment (such as Anaconda or even better Miniconda) on their computer, start Jupyter server and then edit/run code on your own PC/Mac. In more demanding situations, your dev environment can be hosted on some high-performance compute server, and accessed through the web. However, using cloud resources as notebook dev environment sounds like even better alternative. This is exactly what Azure Notebooks are --- public Jupyter server hosted in the Azure cloud, which you can use from anywhere through the browser to write your code.

There are many advantages of using Azure Notebooks instead of local Jupyter installation, and I will try to cover them here.

1. Start Coding Immediately

Whether you want to learn Python, or do some experimentation with F# --- you typically need to install your development environment, be that Visual Studio or Anaconda. This requires some time and disk space, and while it will probably pay off if you are into serious development --- spending time on environment setup is not something you want to do just to try a piece of code, or if you drop by to a party at your friend's house and want to show them your latest data analysis result. In those cases, you can just log into online Notebook environment and start coding right away in any of the supported languages: Python 2 and 3, R or F#.

2. Access to your Code from Anywhere

Coming back to the point of visiting your friend's house --- you probably want to be able to have access to your code immediately, without the need to carry the USB stick or download it from OneDrive. Azure Notebooks allow you to keep all your projects online. Notebooks are organized into Projects, which are similar to GitHub repositories, but without version control, and any project can be made either private, or public.

3. Easily Share Code

Azure Notebooks is a great way to share the code with other people. Each project has a unique link, and if you want to share it -- just use this link (also making sure the project is marked as public). With the link other people will be able to:

view your code
clone it into their own copy of your project, and start playing and editing notebooks online

Because, unlike in Google Colab, you share on per-project basis, you can share several notebooks through one link, and you can also include data, README, and other useful information into the project.

If you need to configure your Python environment in some way or install specific packages - you can also do it through config files, or including pip install commands into the notebook. Installing packages on F# notebooks is done through Paket manager, and is documented here.

4. Write Documented Code / Data-Driven Journalism

Notebook is a great way to add thorough instructions to your code, or to add executable code to your text. There are plenty of scenarious where it can be useful:

Writing instructions or explaining some concepts that are related to algorithms. For example, if you want to explain what affine transformation is, you can provide the explanation first (which can also include formulae, because Azure Notebooks support LaTeX), and then include some executable examples of applying affine transformation to a sample picture. Readers would not only be able to see how the code works, but they will be able to modify the code in place and further play with it
Writing a text with some arguments supported by data, for example, in data-driven journalism or computational journalism. You can write an article in the form of an Azure Notebook, which will automatically gather data from public sources, produce some live graphs, and compute resulting figures that will be used to drive reader to the conclusion.

5. Make Presentations

One great feature that differentiates Azure Notebooks from all other similar services is the pre-installed RISE extension, which allows you to make presentations. You can mark cells as separate slides, or as continuations to previous slide - and then in presentation mode it will look like animation.

While you will not be able to make fancy presentations in terms of design, but for many cases content and simplicity matter. Azure Notebooks are especially great for academic-style presentations, especially because you can use LaTeX formulae inside any text.

6. Clone-and-Run any GitHub Repo

If you are the owner of a GitHub-hosted Python project, you probably want to give people the simple opportunity to try your project out. One of the easiest way to do so is to provide a set of notebooks, which you will then be able to clone immediately in Azure Notebooks. Azure Notebooks support direct cloning from any GitHub repository - so all you need to do is place a piece of code into the Readme.md file for your repo:

<a href="https://notebooks.azure.com/import/gh/<git_user>/<repo>">
<img src="https://notebooks.azure.com/launch.png" /></a>

7. Run the Code on Different Compute

In most of the data science tasks, you first spend some time writing your code and making it work on small-scale data, and then you run it using more powerful compute options. For example, if you need GPU for neural network training, it may be wise to start development on non-GPU machine, and then switch to GPU once your code is ready.

Azure Notebooks allow you to do it seamlessly. When starting/opening a library, you can chose to run it on Free compute, or you can select any compatible virtual machine from an Azure subscription associated to your account (by compatible I mean Data Science Virtual Machine under Ubuntu). So, in most of my tasks, I will start with free compute option, develop most of my code, and then switch to the VM. Azure Notebooks will make sure that the same project environment (including notebooks and other project files) is transferred (or, to be more precise, mounted) to the target VM and used there seamlessly.

I have to mention that the free compute option that you are getting is also quite good, with 4 Gb of memory and 1 Gb of disk space.

8. Teach

I personally teach a couple of courses at University, and I find Azure Notebooks to be extremely useful in teaching. I also have to give a lot of presentations, labs, workshops as a Cloud Advocate. Here's how you can use Notebooks:

Give lectures using presentation feature. I find Azure Notebooks slides far easier to maintain and write, because you do not need to focus on design, but on content, which stays in markdown format. Often it is much easier to manipulate plain text, and inserting mathematical formulae in TeX is far easier than using Word Equations. On the down side, inserting diagrams and pictures is more painful, so do not use Notebooks for marketing presentations.
Write Textbook with Samples If you write slides in Notebooks, you can also include some additional text which is not included into the slides, but which elaborates more on the topic. Thus the same notebook can be used both as slides and as a textbook. Moreover, such a textbook will include executable examples, which act as demos, or starting point for student's own work.
Give Labs/Exams. Prepare all the materials and initial code for your lab/exam into one Azure Notebooks project, and then share it immediately with all students through one link. Your students will then clone the code, and start working on it. To collect results, you can get their individual project links from students (if the work is not precisely time-bound), or ask them to send you or upload .ipynb-file somewhere (if you want to make sure that student finished working on his code on time).

I also use more advanced things like Azure Functions to collect the results from labs, but that is another story, which I will probably also share one day...

Some Drawbacks

While Azure Notebooks is a great tool, there are some things that you need to keep in mind when using them:

Network access is somehow limited. Because with Azure Notebooks you are getting some free compute, it would not be very wise to let you do whatever you want on the internet, like sending spam. For this reason, network access inside Azure Notebooks is limited to a certain set of resources, that include all Azure resources, GitHub, Kaggle, OneDrive, and some more. So, if you want your notebook to get some data from the outside, you may want to place the data on GitHub/OneDrive, or upload it manually into the project directory of a notebook through the web interface.
Inserting Images/Diagrams into the text. Because you enter all the text as plain text, you lack the simple PowerPoint/Word features of copy-pasting images and drawing diagrams. So, if you want to insert a picture or a diagram, you need to export it to JPEG/PNG, upload somewhere on the internet (I use GitHub most of the time), and insert it into the text using Markdown syntax. So, if you are creating a marketing/motivation presentation -- use PowerPoint, and for scientific/academic presentation or a university course -- use notebooks.
GPU Access is not included into Free Compute options for Azure Notebooks at this time.

Examples

A good way to learn about specific features of Azure Notebooks, like installing packages, getting external data and drawing a graph, is to look at the sample notebooks. A great collection of Jupyter samples can be found here, and remember -- you can run any Jypyter notebook in Azure by uploading .ipynb file into a project.

Conclusion

Azure Notebooks are a great tool which can be useful in many scenarious that I have tried to cover here. If you have some more use-cases - feel free to share them in the comments, I would be very interested to know!

While while there are some other options for running notebooks in the cloud, including Google Colab and Binder, the comparison shows that Azure Notebooks is a great tool that has the most useful features.

I hope you will enjoy using Azure Notebooks in your everyday life as much as I do!

P.S. Some more useful documentation may be found here: http://aka.ms/aznb