DEV Community: Stratiteq

Puffins Detection with Azure Custom Vision and Python

Goran Vuksic — Thu, 06 Aug 2020 10:08:00 +0000

In our recent project "Dragonfly Mini" here at Stratiteq we used different technologies and programming languages in order to build an autonomous vehicle powered by NVIDIA Jetson Nano, equipped with a drone, that collects and processes data and uploads it to the Microsoft's Azure cloud. Following this project we will publish series of blogposts here at Dev.to explaining how to achieve specific things and in this first blog post I will show how easy it is to train a model for custom detection with Azure Custom Vision and how you can use this model via Python.

Puffin is, as you probably know, a small bird with brightly coloured beak and Iceland is home to the Atlantic puffins. One of use cases we demonstrated in "Dragonfly Mini" project was inspired by protection of endangered species by use of AI, therefore we trained a custom model that is capable of detecting puffins. If you want to build a model that can recognise different birds you can use: Wah C., Branson S., Welinder P., Perona P., Belongie S. "The Caltech-UCSD Birds-200-2011 Dataset" Computation & Neural Systems Technical Report, CNS-TR-2011-001. You can also find many other datasets on Kaggle for this use. Data preparation is manual work and takes some time, but more effort you invest in preparing and tagging your images the results will be better.

It is really easy to create object detection project with Azure Custom Vision, and I explained this with step-by-step walkthrough in one of the previous blogposts "Where's Chewie? Object detection with Azure Custom Vision". Log into the Custom Vision, create new project for object detection, add and tag images, train your project and publish the trained iteration. Before you publish the iteration make sure you make few tests with different images. Using only Caltech dataset will not be sufficient to make good recognition so try to add more images from other datasets or from other Internet sources.

We can use this model from Python in less than 20 lines of code. First let's install OpenCV and Custom Vision service SDK for Python by running following commands:

pip install opencv-python
pip install azure-cognitiveservices-vision-customvision
pip install msrest

OpenCV will be used to get image from the camera, we'll save this and resulting image with it, and we'll also use it to draw bounding boxes for objection detection results.

Create a new Python script and import packages you just installed.

import cv2
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import ApiKeyCredentials

Define the camera and set properties for width and height, keep in mind you should keep aspect ratio, in my case it will be 640 by 480 pixels.

camera = cv2.VideoCapture(0)
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

OpenCV's VideoCapture can use input from a camera, but it can also use a video file as input which is a really useful feature.

Next thing we need to do is to define credentials which we will use for our predictor.

credentials = ApiKeyCredentials(in_headers={"Prediction-key": "<PREDICTION_KEY>"})
predictor = CustomVisionPredictionClient("<ENDPOINT_URL>", credentials)

Prediction key can be found in the Custom Vision interface by clicking on "Prediction URL" button. In the window that pops up you will see it below URL.

Endpoint URL can be found if the project settings, in top bar click the settings and you will find endpoint URL (use it without Resource ID). In settings you will also find the project ID which will be in the following format: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.

We will take an image from camera and we will save it as "capture.png" with following two lines of code.

ret, image = camera.read()
cv2.imwrite('capture.png', image)

The image we saved we will use for detection and you will have to use the project ID and name of the published iteration. If you are interested in more details how results look like you can print them out.

with open("capture.png", mode="rb") as captured_image:
    results = predictor.detect_image("<PROJECT_ID>", "<ITERATION_NAME>", captured_image)

We are now able to loop through our results and we will take predictions that have probability over 50%. With these predictions we will draw bounding boxes on the image and store it as new image "result.png". For the bounding boxes we make simple calculation based on the image size, we set bounding box color and border thickness.

for prediction in results.predictions:
    if prediction.probability > 0.5:
        bbox = prediction.bounding_box
        result_image = cv2.rectangle(image, (int(bbox.left * 640), int(bbox.top * 480)), (int((bbox.left + bbox.width) * 640), int((bbox.top + bbox.height) * 480)), (0, 255, 0), 3)
        cv2.imwrite('result.png', result_image)

In the end we release the camera we have used.

camera.release()

When you execute the code, camera will turn on, take an image, Azure Custom Vision should be invoked via URL to get the results and results will be saved as new image. Here is quick test I made by using fridge magnet with puffins.

If you do not get really good results at first, you can retrain your model by adding and tagging additional images.

Full code we used in this tutorial is here:

import cv2
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import ApiKeyCredentials

camera = cv2.VideoCapture(0)
camera.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

credentials = ApiKeyCredentials(in_headers={"Prediction-key": "<PREDICTION_KEY>"})
predictor = CustomVisionPredictionClient("<ENDPOINT_URL>", credentials)

ret, image = camera.read()
cv2.imwrite('capture.png', image)

with open("capture.png", mode="rb") as captured_image:
    results = predictor.detect_image("<PROJECT_ID>", "<ITERATION_NAME>", captured_image)

for prediction in results.predictions:
    if prediction.probability > 0.5:
        bbox = prediction.bounding_box
        result_image = cv2.rectangle(image, (int(bbox.left * 640), int(bbox.top * 480)), (int((bbox.left + bbox.width) * 640), int((bbox.top + bbox.height) * 480)), (0, 255, 0), 3)
        cv2.imwrite('result.png', result_image)

camera.release()

I hope you learned how simple it is to utilise Azure Custom Vision via Python, and as you were able to see it was done in less than 20 lines of code.

Thanks for reading!

Detecting social distancing with use of Azure Custom Vision

Cecilia Rundberg — Thu, 02 Jul 2020 06:56:47 +0000

I am a student at University of Gothenburg studying Cognitive Science. The last couple of weeks I have spent as a summer intern at Stratiteq and have been working on an AI project about how to detect social distancing with use of drone surveillance.

Nowadays social distancing is important in our society and it sure would be useful to develop autonomous tools for helping us out with keeping the safe distance. In this post I will explain how I trained a custom model and used it for calculating distance between people.

The model was trained with Azure Custom Vision, an AI service which allows easy customisation and training of custom models. It was then tested via a custom-made web page and with JavaScript code. The drone used for this project was DJI Mavic Mini.

The first step is to make sure that the model we are creating in Custom Vision is trained to be able to detect a person and to distinguish one from other appearing objects. To do this we need enough data that can be trained on and to get hold of this we used Aerial Semantic Segmentation Drone Dataset found on Kaggle together with a few test pictures we took with the drone at our Stratiteq after-work event. In total I used 100 pictures for the training.

When uploading a picture, the so far untrained model will point out what it thinks is an object and you will then need to correctly tag it, in this case we will point out and tag all the people in each picture with "person". After doing this the model can be trained and tested. For each iteration you will get a performance measure consisting of precision, recall and mAP, standing for:

Precision – the fraction of relevant instances among the retrieved instances
Recall – the fraction of the total amount of relevant instances that were retrieved
mAP – overall object detector performance across all tags

A good starting point for creating the custom-made web page is to use Microsoft’s Quickstart: Analyze a remote image using the REST API and JavaScript in Computer Vision. We can easily modify this quick start example for our own calculations.

Beside defining the subscription key and the endpoint URL we need to change the quick start example to use Custom Vision endpoint. These can be seen in the the Custom Vision dashboard under "Prediction URL".

var uriBase = endpoint + "customvision/v3.0/Prediction/…";

We also need to set the custom header “Prediction-Key” for our request.

xhrObj.setRequestHeader("Prediction-Key","…");

Custom Vision will analyze the pictures we send and provide with result data out of our created model. For our testing purposes we uploaded the pictures to the Azure Blob Storage.

In order to calculate the distance in code from the result data, we will use the prediction values for the detected people. With each of the x, y, height and width values we get we will calculate the center of the object bounding boxes.

var x0 = x[i] + width[i] / 2;
var y0 = y[i] + height[i] / 2;

Applying the Pythagorean theorem gives us the distance between two centers, in our case that gives us the distance between two persons.

var distanceInPixels = Math.sqrt((x0 - x1)**2 + (y0 - y1)**2);

The calculation is currently made in pixels and we would like to have it in meters. When taking the test pictures with the drone we made measurements and markings on the ground to be able to tell the actual area size. Before we tested the pictures, we cropped them to these markings. We also knew the flight height of the drone.

The calculations were visualised on our web page by drawing a bounding box for each of the detected person, and by drawing lines between them. The line between the persons will be green if the distance is 2 meters or more and red if the distance is less than 2 meters.

var canvas = document.getElementById('resultImage');
var context = canvas.getContext('2d');
context.beginPath();
context.moveTo(x0, y0);
context.lineTo(x1, y1);
if (distanceInPixels < meterToPixel) {
    context.strokeStyle = 'Red';
} else {                        
    context.strokeStyle = 'LightGreen';
}
context.lineWidth = 4;
context.stroke();

In the following animated GIF you can see the test results from 16 pictures we used in the testing.

Except in the Covid-19 situation this kind of autonomous distance calculation can be useful in other areas. Different kind of working places such as the ones dealing with hazardous materials for example, could have use of it. That would of course require different kinds of improvements of this simple model. It would need to be able to detect different kinds of objects and not only people which would require additional data to train on.

Looking at this workflow and the result you can see that Microsoft Azure Cognitive Services provides an easy way to develop custom applications powered by Artificial Intelligence.

Thank you for reading, I hope this post gave you an idea how to build custom models and how to empower your applications with AI!

Managed Identity - How it works behind the scenes

Mattias Vest — Tue, 30 Jun 2020 12:41:28 +0000

Managed Identity is an awesome feature in Azure which allows your Azure applications and services to communicate securely without handling or maintaining any credentials to do so. It is a very simple service to use and work with. But how can a Virtual Machine or App Service identify itself and be allowed access to other services? That is what we will be looking into in this post. First, let's quickly go over why we should be using Managed Identity and what it really is.

Using Managed Identity means that there is no risk of accidentally committing secrets into git, no secrets that are shared over email etc. Added to that, the task of maintaining and regenerating credentials is all handled by Azure behind the scenes. The lifetime of the Managed Identity is the same as the resource, so if the resource is removed, so is the Identity, meaning there is no additional clean up necessary. It also doesn't matter what language you are writing your code in, the requirement is that your application is running on an Azure Service which supports Managed Identity. All these benefits really means that we let Azure manage the maintenance and security parts for us, which they are happy to do, and we as developers can focus on what it is that we do best.

Now that we know why we should be using Managed Identity, what is it really? In order to explain it, let us first take a look at the context of it all. Managed Identity is really a feature inside Azure Active Directory (AAD), and it's place inside the AAD is shown below in the picture. Note that this picture only shows a very limited part of the Identity part of the AAD, but it helps to put it into context.

Simply put, Managed Identity is a Service Principal, much similar to an App Registration, but instead maintained completely by Azure behind the scenes. Meaning it has the same basic functionality, but less customizable. What happens when we enable Managed Identity on a service is that a Service Principal is created in our AAD, which we then give access to other services, in the same way we would give any User Principal or Service Principal access, as shown below.

Once we have enabled Managed Identity on our service, a Function App in this case, we can give it access to another resource, for instance a Key Vault like below.

Now that the service has MI enabled, you can retrieve an access token to use towards different resources. How to retrieve this token is usually based on what language you are using and what tools are available. As a .NET developer for instance, I would use the AppAuthentication NuGet provided by Microsoft, and these lines would pretty much be enough.



using Microsoft.Azure.Services.AppAuthentication;
...
...
var resource = "https://database.windows.net/";
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var accessToken = await azureServiceTokenProvider.GetAccessTokenAsync(resource);

In the example above, resource is the resource that you are trying to access. To find all the services supporting Managed Identity, and what resources to specify, look at the official documentation provided by Microsoft here as support for Managed Identity is frequently being added.

This is kind of all you need to know in order to use Managed Identity in a good and effective way. However, the scope of this blog post is to look behind the scenes of it all in order to understand it.

What is happening on the resources is different depending on the type of resource. Let us look at what is happening for App Services (I.e. Web, API and Function Apps) and Virtual Machines.

App Services

On the App Services, when you enable Managed Identity, a local endpoint is set up, and this endpoint and a secret is added to the App Service environmental variables. You can check this by going into the App Service using Kudu and looking at the environment variables, there are now 4 new variables.

The endpoint is only reachable from this app service, and the port used and the secret is regenerated and updated regularly, multiple times a day. With these settings, it is now possible to retrieve a token that is valid for the resource we are trying to access. If we use the console inside Kudu, we can simply make a HTTP request using CURL to the endpoint provided, attaching a X-IDENTITY-HEADER to the request, and specifying the resource we want to access.



curl -H "X-IDENTITY-HEADER: {secret}" "{endpoint}?resource={resource}&api-version=2019-08-01"

In this case, for my App Service to retrieve a token which is valid for an Azure Managed Database, the request could look like this:



curl -H "X-IDENTITY-HEADER: 4B739BBFDF5D43E48BAE2E79515314B2" "http://127.0.0.1:41570/MSI/token/?resource=https://database.windows.net/&api-version=2019-08-01"

Virtual Machines

For Virtual Machines, The idea is the same, but the details differ slightly. Azure VMs have access to an endpoint called Azure Instance Metadata service (IMDS). This is also an endpoint that is only accessible from the VM in question, where it is possible to retrieve a bunch of metadata about this particular VM. When enabling Managed Identity on this VM, a Service Principal is again created in the AAD, and the Client Id and Certificate of that Service Principal is added to this IMDS. So when the Virtual Machine wants to retrieve an access token using Managed Identity, it makes a HTTP call to the IMDS endpoint (which is always at http://169.254.169.254/) instead.



curl -H "Metadata:true" "http://169.254.169.254/metadata/identity/oauth2/token??resource=https://database.windows.net/&api-version=2018-02-01"

A more detailed picture as to how the entire flow works is shown below.

In conclusion, Managed Identity works very similar as a regular App Registration, it simply adds the Secret/Certificate behind the scenes where it is only available to the service itself.

Simply put, Managed Identity is a Service Principal which Azure have abstracted to the point where it is really simple to use and nothing to manage, so that you as a developer can focus on the important things and bring value to the business.

Cryptography with Practical Examples in .Net Core

Paul Ulvinius — Mon, 08 Jun 2020 09:57:19 +0000

This week at Stratiteq, on our weekly tech talk we spoke about cryptography and concepts behind it with practical examples in .Net Core.

Cryptography (from Ancient Greek, cruptos = "hidden", graphein = "to write") is the study of techniques for preventing third parties from reading or manipulating private messages. Cryptography has been around for a long time and it was mostly used for military purposes. Technology behind it did not change much over time and "boom" happened in second part of 20th century, so nowadays we can find cryptography everywhere:

Secure network communication (TLS/SSL)
Authentication and digital signatures
Contactless
Block chain technology
Bank / credit cards
Hard disk encryption
GSM
E-mail, FTP, biometry, etc...

Cryptography is not just used in the technologies listed above, we can see many industry leaders talking about it, we see it disrupting industries and we can for sure say it is in some way changing the world around us.

Even it sounds complicated, it more or less boils down into three simple concepts. We'll not go into math and science behind it, but we are going to explain the concepts and go through examples which you can try by yourself.

Those three main concepts are:

Hashing
Encryption
Signatures

Hashing

Hashing can be simply explained as translation from input text into resulting text of fixed size by use of the algorithm.

As you can see in above examples with just a slight change of input text, result will be completely different.

Let's take a look at an example in .Net Core with use of SHA-256 algorithm. For following examples we'll have to use System.Security.Cryptography namespace, you can find more info about it here.



using System.Security.Cryptography;

In following function we are hashing texts to demonstrate how hashing works, and how results change if we just make a slight change to the text.



        internal static void CryptographicHashFunctionDemo()
        {
            Console.WriteLine("***** Cryptographic hash demo *****");

            foreach (var message in new[] {
                "Fox",
                "The red fox jumps over the blue dog",
                "The red fox jumps ouer the blue dog",
                "The red fox jumps oevr the blue dog",
                "The red fox jumps oer the blue dog"})
            {
                Console.WriteLine($"{ message } => { CryptographicHash.ComputeHash(message) }");
            }

            Console.Write(Environment.NewLine);
        }

If we take a closer look at the ComputeHash we can see texts are encrypted with SHA-256 algorithm.



        internal static string ComputeHash(string message)
        {
            using var sha256 = SHA256.Create();
            var hashedBytes = sha256.ComputeHash(Encoding.UTF8.GetBytes(message));
            return hashedBytes.ToHex();
        }

By running the code, you'll get following results.

Encryption

Encryption is the process of sending information to a recipient that can't be interpreted unless having access to a key. Encryption always consist of two parts, an algorithm and a key.

Encryption can be symmetric and asymmetric. In symmetric encryption same key is used for encryption and decryption, while in asymmetric different keys are in use: public key and private key.

Symmetric encryption can be seen in the following example. We create a key, encrypt message using the key and decrypt message using the same key.



        internal static void SymmetricEncryptionDemo()
        {
            Console.WriteLine("***** Symmetric encryption demo *****");

            var unencryptedMessage = "To be or not to be, that is the question, whether tis nobler in the...";
            Console.WriteLine("Unencrypted message: " + unencryptedMessage);

            // 1. Create a key (shared key between sender and reciever).
            byte[] key, iv;
            using (Aes aesAlg = Aes.Create())
            {
                key = aesAlg.Key;
                iv = aesAlg.IV;
            }

            // 2. Sender: Encrypt message using key
            var encryptedMessage = SymmetricEncryption.Encrypt(unencryptedMessage, key, iv);
            Console.WriteLine("Sending encrypted message: " + encryptedMessage.ToHex());

            // 3. Receiver: Decrypt message using same key
            var decryptedMessage = SymmetricEncryption.Decrypt(encryptedMessage, key, iv);
            Console.WriteLine("Recieved and decrypted message: " + decryptedMessage);

            Console.Write(Environment.NewLine);
        }

In SymmetricEncryption class we have Encrypt and Decrypt defined as follows. You can notice we are using AES algorithm.



        internal static byte[] Encrypt(string message, byte[] key, byte[] iv)
        {
            using var aesAlg = Aes.Create();
            aesAlg.Key = key;
            aesAlg.IV = iv;

            var encryptor = aesAlg.CreateEncryptor(aesAlg.Key, aesAlg.IV);

            using var ms = new MemoryStream();
            using var cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write);
            using (var sw = new StreamWriter(cs))
            { 
                sw.Write(message); // Write all data to the stream.
            }
            return ms.ToArray();
        }

        internal static string Decrypt(byte[] cipherText, byte[] key, byte[] iv)
        {
            using var aesAlg = Aes.Create();
            aesAlg.Key = key;
            aesAlg.IV = iv;

            var decryptor = aesAlg.CreateDecryptor(aesAlg.Key, aesAlg.IV);

            using var ms = new MemoryStream(cipherText);
            using var cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Read);
            using var sr = new StreamReader(cs);
            return sr.ReadToEnd();
        }

If you run the code, you'll get following result.

Asymmetric encryption has few common algorithms that are in the use:

RSA; by Ron Rivest, Adi Shamir and Len Adleman; from 1977
EC (Elliptic Curve); Neal Koblitz and Victor S. Miller; from 1985

These algorithms can be used for encryption and decryption, as well as digital signatures.

In another example we can see asymmetric encryption in action using the RSA algorithm.



        internal static void AsymmetricEncryptionDemo()
        {
            Console.WriteLine("***** Asymmetric encryption demo *****");

            var unencryptedMessage = "To be or not to be, that is the question, whether tis nobler in the...";
            Console.WriteLine("Unencrypted message: " + unencryptedMessage);

            // 1. Create a public / private key pair.
            RSAParameters privateAndPublicKeys, publicKeyOnly;
            using (var rsaAlg = RSA.Create())
            {
                privateAndPublicKeys = rsaAlg.ExportParameters(includePrivateParameters: true);
                publicKeyOnly = rsaAlg.ExportParameters(includePrivateParameters: false);
            }

            // 2. Sender: Encrypt message using public key
            var encryptedMessage = AsymmetricEncryption.Encrypt(unencryptedMessage, publicKeyOnly);
            Console.WriteLine("Sending encrypted message: " + encryptedMessage.ToHex());

            // 3. Receiver: Decrypt message using private key
            var decryptedMessage = AsymmetricEncryption.Decrypt(encryptedMessage, privateAndPublicKeys);
            Console.WriteLine("Recieved and decrypted message: " + decryptedMessage);

            Console.Write(Environment.NewLine);
        }

As you can see, in the AsymmetricEncryptionDemo function we create a public and a private key pair, we encrypt the message using the public key and we decrypt message using the private key.

Encrypt and Decrypt are defined as follows and we are using RSA algorithm.



        internal static byte[] Encrypt(string message, RSAParameters rsaParameters)
        {
            using var rsaAlg = RSA.Create(rsaParameters);
            return rsaAlg.Encrypt(Encoding.UTF8.GetBytes(message), RSAEncryptionPadding.Pkcs1);
        }

        internal static string Decrypt(byte[] cipherText, RSAParameters rsaParameters)
        {
            using var rsaAlg = RSA.Create(rsaParameters);
            var decryptedMessage = rsaAlg.Decrypt(cipherText, RSAEncryptionPadding.Pkcs1);
            return Encoding.UTF8.GetString(decryptedMessage);
        }

Result in the console looks as follows.

Digital signatures

Digital signatures are built on top of asymmetric cryptography. They are used to send information to recipients who can verify that the information was sent from a trusted sender, using a public key.

Let's take a look at the example of digital signature:



        internal static void MessageSignatureDemo()
        {
            Console.WriteLine("***** Message signature demo *****");

            var message = "To be or not to be, that is the question, whether tis nobler in the...";
            Console.WriteLine("Message to be verified: " + message);

            // 1. Create a public / private key pair.
            RSAParameters privateAndPublicKeys, publicKeyOnly;
            using (var rsaAlg = RSA.Create())
            {
                privateAndPublicKeys = rsaAlg.ExportParameters(includePrivateParameters: true);
                publicKeyOnly = rsaAlg.ExportParameters(includePrivateParameters: false);
            }

            // 2. Sender: Sign message using private key
            var signature = AsymmetricEncryption.Sign(message, privateAndPublicKeys);
            Console.WriteLine("Message signature: " + signature.ToHex());

            // 3. Receiver: Verify message authenticity using public key
            var isTampered = AsymmetricEncryption.Verify(message, signature, publicKeyOnly);
            Console.WriteLine("Message is untampered: " + isTampered.ToString());

            Console.Write(Environment.NewLine);
        }

In the MessageSignatureDemo function we create a public and a private key pair, we sign the message with the private key and we verify the message for authenticity using public key.

Sign and Verify works in the following way, we're using the RSA algorithm.



        internal static byte[] Sign(string message, RSAParameters rsaParameters)
        {
            using var rsaAlg = RSA.Create(rsaParameters);
            return rsaAlg.SignData(Encoding.UTF8.GetBytes(message), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
        }

        internal static bool Verify(string message, byte[] signature, RSAParameters rsaParameters)
        {
            using var rsaAlg = RSA.Create(rsaParameters);
            return rsaAlg.VerifyData(Encoding.UTF8.GetBytes(message), signature, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
        }

Console will display following result for running digital signature example.

Project and all code from this blog post is available at the GitHub here. Thanks for reading!

What does the Computer Vision see? Analyse a local image with JavaScript

Goran Vuksic — Mon, 25 May 2020 07:00:18 +0000

Every week here at Stratiteq we have tech talks called "Brown bag". Idea behind it is to grab your lunch (brown) bag and join a session where we watch presentation about different tech topics, and discuss it afterwards. Last week our session was about Azure Computer Vision.

Computer Vision is an AI service that analyses content in images. In documentation you can find several examples how to use it from different programming languages, in this post you'll also see one example that is not in official documentation and that is: how to analyse a local image with Javascript.

In order to set up Computer Vision you should log in to the Azure Portal, click "Create a resource", select "AI + Machine learning" and "Computer Vision".

Define resource name, select subscription, location, pricing tier and resource group, and create the resource. In resource overview click on "Keys and Endpoint" in order to see keys and endpoint needed to access the Cognitive Service API. This values you'll need later in code we'll write.

Sketch of HTML page we will create is visible on the image below. We'll use camera and show feed on the page, take screenshot of camera every 5 seconds, analyse that screenshot with Computer Vision and display description under it.

For setup of our page we'll use following HTML code, please note JQuery is included in page head.

<!DOCTYPE html>
<html>
<head>
    <title>Brown Bag - Computer Vision</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
</head>
<body>
    <h2>What does AI see?</h2>
    <table class="mainTable">
        <tr>
            <td>
                <video id="video" width="640" height="480" autoplay></video>
            </td>
            <td>
                <canvas id="canvas" width="640" height="480"></canvas>
                <br />
                <h3 id="AIresponse"></h3>
            </td>
        </tr>
    </table>
</body>
</html>

We'll use simple CSS style to align content on top of our table cells and set colour of result heading.

table td, table td * {
    vertical-align: top;
}
h3 {
    color: #990000;
}

Inside of document.ready function we'll define our elements, check for camera availability and start camera feed.

$(document).ready(function () {

    var video = document.getElementById("video");
    var canvas = document.getElementById("canvas");
    var context = canvas.getContext("2d");

    if(navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
        navigator.mediaDevices.getUserMedia({ video: true }).then(function(stream) {
            video.srcObject = stream;
            video.play();
        });
    }

});

You can check compatibility of mediaDevices on following link: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/mediaDevices

Every 5 second we'll take a screenshot of our camera feed and we'll send blob of it to the Computer Vision API.

window.setInterval(function() {
    context.drawImage(video, 0, 0, 640, 480);

    fetch(canvas.toDataURL("image/png"))
        .then(res => res.blob())
        .then(blob => processImage(blob));
}, 5000);

Result processing is done in processImage function where you need to enter your subscription key and endpoint in order to make it work. Those values are available in the Azure Computer Vision overview as mentioned earlier.

function processImage(blobImage) {
    var subscriptionKey = "COMPUTER_VISION_SUBSCRIPTION_KEY";
    var endpoint = "COMPUTER_VISION_ENDPOINT";
    var uriBase = endpoint + "vision/v3.0/analyze";

    var params = {
        "visualFeatures": "Categories,Description,Color",
        "details": "",
        "language": "en",
    };

    $.ajax({
        url: uriBase + "?" + $.param(params),
        beforeSend: function(xhrObj){
            xhrObj.setRequestHeader("Content-Type","application/octet-stream");
            xhrObj.setRequestHeader("Ocp-Apim-Subscription-Key", subscriptionKey);
        },
        type: "POST",
        cache: false,
        processData: false,
        data: blobImage
    })
        .done(function(data) {

    });
}

Result we receive from the Computer Vision API is JSON, we'll take description from it and add it to the header 3 element named "AIresponse".

document.getElementById('AIresponse').innerHTML = data.description.captions[0].text;

We did few tests with it, Computer Vision describes images really well, if you mess around with you could also get few funny results as we did:

Thanks for reading, you can find full code on the GitHub: https://github.com/gvuksic/BrownBagComputerVision

Where's Chewie? Object detection with Azure Custom Vision

Goran Vuksic — Sun, 03 May 2020 21:33:16 +0000

"Where's Waldo?" game is very well known by everyone, in this post I'll explain how to make such game with Azure Custom Vision and LEGO minifigures. This was my weekend project, feel free to recreate it and while learning how Azure Custom Vision works you can also entertain your child to help you out with it.

I'll be using Chewbacca LEGO minifigure (nicknamed Chewie) that AI should detect in image among other minifigures.

Board where Chewbacca will be hiding can be seen on the following image, it consist of 49 different LEGO minifigures.

In order to follow next steps you should have an Azure account, if you don't have one you can create it here.

When you sign in with your Azure account into Custom Vision (www.customvision.ai) you'll see a page that shows your projects.

Clicking on "New project" will ask you to enter projects name, you can add description and you need to select resource.

If you are creating a new resource, you should define resource name, subscription, resource group, what kind of resource is that (CognitiveServices), location and pricing tier.

For new resource group you should just define name and location.

After you fill this in you can select type and domain for your new project.

Image classification is used when you want to classify the whole image, which object is represented the most in the image. Object detection is used to find location of content in the image and this is what we need for this project.

For domain we'll use General domain which is explained by Microsoft as "Optimised for a broad range of object detection tasks. If none of the other domains are appropriate, or you are unsure of which domain to choose, select the Generic domain". Feel free to read more about the domains here.

Once project is created you'll be able to see project page that is currently empty and gives you option to upload images to train your model.

I took several images of the LEGO Chewbacca minifigure from different angles. I also made a little trick, as you can see on the images below, I've placed minifigure on top of the LEGO catalogue and I switched pages for each image. That way model will be able to distinguish better what is minifigure and what is background in the image.

When images are uploaded you can manually tag them. UX is really nice here and you'll notice that tagging tool will suggest a frame around objects it detects. For first tag you'll need to enter a name, afterwards you can select it for other images. Adjusting frame to fit the minifigure is super easy.

We prepared the images and we can now start the training, in this case quick training. Advanced training should be used on advanced and challenging datasets.

When training is finished you'll see information about it:

Precision: this number will tell you: if a tag is predicted by your model, how likely is that to be right
Recall: this number will tell you: out of the tags which should be predicted correctly, what percentage did your model correctly find
mAP: mean average precision will tell you: the overall object detector performance across all the tags

Quick test I made with the image of the whole board showed me that some objects were identified as target minifigure but with really low precision (20%). Dataset I made was really small number of images, so I was not surprised and I expected to see something like this.

Predictions allows you to see images that were used for prediction and here you can go into images and correctly tag them.

I fixed predictions several times and added much more images in my dataset. Once everything is in place this is quick and simple to do. After few iterations of my model I tried with new image (that Custom Vision model haven't seen yet) and it successfully found the Chewie with accuracy 40.8%!

Accuracy of the model can be improved further with adding more images to the data set, by using advanced training, etc. In my case I was happy with the result since this is what I aimed for: "Where's Waldo"-like game.

Thanks for reading!