DEV Community: Cassie Breviu

Inference machine learning models in the browser with JavaScript and ONNX Runtime Web

Cassie Breviu — Fri, 26 Nov 2021 17:03:16 +0000

In this video tutorial we will go over how to do client side inferencing in the browser with ONNX Runtime web. Below is a video on how to understand and use a QuickStart template to start building out a static web app with an open source computer vision model. Additionally, you can find a written step-by-step tutorial in the onnxruntime.ai docs here. Let's learn a bit more about the library, ONNX Runtime (ORT), which allows us to inference in many different languages.

What is ORT and ORT-Web?

ONNX Runtime (ORT) is a library to optimize and accelerate machine learning inferencing. It has cross-platform support so you can train a model in Python and deploy with C#, Java, JavaScript, Python and more. Check out all the support platforms, architectures, and APIs here.

ONNX Runtime Web (ORT-Web) enables JavaScript developers to run and deploy machine learning models client-side. With ORT-Web you have the option to use a backend of either WebGL for GPU processing or WebAssembly WASM for CPU processing. If you want to do JavaScript server side inferencing with node checkout the onnxruntime-node library.

Video tutorial:

Written tutorial:
Check out the written tutorial here: ONNX Runtime Web Docs tutorial

Resources

Start using the template now by going to GitHub NextJS ORT-Web Template.
ONNX Runtime Web Docs tutorial
ONNX Runtime docs
ONNX Runtime GitHub
Deploy with Azure Static Web Apps

From No Data to Computer Vision Machine Learning Model in 10 Minutes or less!

Cassie Breviu — Sat, 18 Jul 2020 00:44:28 +0000

First we will scrape a dataset using a NodeJS console app and Azure Cognitive Service Bing Search to create an image dataset in minutes. Then use the Azure Cognitive Service Custom Vision SDK to automagically create the Custom Vision project, upload and tag your images, then train and test the model using the Python Custom Vision SDK in Jupyter notebooks.

Prerequisites

1. Create Azure Resources

Create Bing Search Api Azure Resource

Go to portal.azure.com
Select "Create Resource"
Select "AI + Machine Learning"
Select "Bing Search"
Complete the required fields
Select "Create"
When the deployment succeeds you will get a notification in the top right corner.
Select "Go to resource"
Select "Keys" from the left hand navigation for the resource.
We will come back to the Keys when we are ready to call it from our App.

Create Custom Vision Resource in Azure

Go to portal.azure.com
Select "Create Resource"
Search "Custom Vision"
Select "Create"
Complete the required fields
Select "Create"
When the deployment succeeds you will get a notification in the top right corner.
Select "Go to resource"

2. Clone the Repo

git clone https://github.com/cassieview/ImageScraperAndCustomVisionMagic.git

3. Get the Data with Bing Search and the NodeJS Console App

We are now ready to run the app and scrape our dataset together using the Bing Search API.

Update the key at the top of the script to the key from the Bing Search Resource you created

const subscriptionKey = "add your key here";

Update the searchTermList in the script to scrape the type of images you need for the custom vision model

var searchTermList = ["cardinal", "bluejay"];

To run the app open the VS Code terminal (CTRL+SHIRT+) and use the following command

node index.js

Here is the full index.js script file:

"use strict";

var https = require("https");
var fs = require("fs");
var download = require("image-downloader");
// Replace the subscriptionKey string value with your valid subscription key.
const subscriptionKey = "add your key here";
// Verify the endpoint URI.  At this writing, only one endpoint is used for Bing
// search APIs.  In the future, regional endpoints may be available.  If you
// encounter unexpected authorization errors, double-check this host against
// the endpoint for your Bing Search instance in your Azure dashboard.
const host = "api.cognitive.microsoft.com";
const path = "/bing/v7.0/images/search";
//filter by license
const filter = "&qft=+filterui:license-L2_L3_L4&FORM=IRFLTR";

var searchAndSaveImages = (search) => {
  console.log("Searching images for: " + search);
  //set global to current search term

  let request_params = {
    method: "GET",
    hostname: host,
    path: path + "?q=" + encodeURIComponent(`${search}`) + filter,
    headers: {
      "Ocp-Apim-Subscription-Key": subscriptionKey,
    },
  };
  var req = https.request(request_params, (response) => {
    console.log("statusCode:", response.statusCode);
    let body = "";
    response.on("data", (d) => {
      body += d;
    });
    response.on("end", () => {
      let imageResults = JSON.parse(body);
      console.log(`Image result count: ${imageResults.value.length}`);
      if (imageResults.value.length > 0) {
        //create directory folder for current search term
        let rootDir = `./birds`;
        let searchDir = `${rootDir}/${search}`;
        let testDir = `${searchDir}/test`;
        let trainDir = `${searchDir}/train`;

        if (!fs.existsSync(rootDir)) {
          fs.mkdirSync(rootDir);
        }
        if (!fs.existsSync(searchDir)) {
          fs.mkdirSync(searchDir);
          fs.mkdirSync(testDir);
          fs.mkdirSync(trainDir);
        }
        //create count so we can split the results between test and train folder
        let count = 0;

        //save image results
        imageResults.value.forEach((imageResult) => {
          let destDir = count % 3 == 0 ? testDir : trainDir;
          let options = {
            url: imageResult.contentUrl,
            dest: `./${destDir}/${imageResult.imageId}.png`,
          };

          download
            .image(options)
            .then(({ filename, image }) => {
              console.log("File saved to", filename);
            })
            .catch((err) => {
              console.error(err);
            });
          count++;
        });
      } else {
        console.log("Couldn't find image results!");
      }
    });
    response.on("error", function (e) {
      console.log("Error: " + e.message);
    });
  });
  req.end();
};

let main = () => {
  //bird species search term list
  var searchTermList = ["cardinal", "bluejay"];
  //loop search terms
  searchTermList.forEach((term) => {
    searchAndSaveImages(term);
  });
};

//check to make sure the subscription key was updated and kick off main func
if (subscriptionKey.length === 32) {
  main();
} else {
  console.log("Invalid Bing Search API subscription key!");
  console.log("Please paste yours into the source code.");
}

Now we have a dataset split out by tag name with a train and test folder for each.

4. Create the Custom Vision Model

Lets run the Python Jupyter notebook to create a project, tags, upload data, train and test the model. NOTE: The Custom Vision SDK is available in the following languages: Python, .Net, Node, Go and Java.

Open Anaconda Prompt, navigate to the notebook directory that you cloned from the GitHub and run the jupyter notebook cmd.
Open the CustomVisionSdkUpload.ipynb notebook
Run the following command to import the package.

!pip install azure-cognitiveservices-vision-customvision

Update the endpoint and key with the values from the resource you created in Azure.

ENDPOINT = "https://<resource-name>.cognitiveservices.azure.com/"

# Replace with a valid key
training_key = "<training-key>"
publish_iteration_name = "classifyBirdModel"
prediction_resource_id ="/subscriptions/<sub-id>/resourceGroups/<resource-group-name>/providers/Microsoft.CognitiveServices/accounts/<resource-name-Prediction>"
trainer = CustomVisionTrainingClient(training_key, endpoint=ENDPOINT)

# Create a new project
print ("Creating project...")

# Create the project based on if you need a default or compact model. Only the compact models can be exported, default is deployed with custom vision only.
#project = trainer.create_project("Bird Classification")
project = trainer.create_project(name="Bird Classification Compact", domain_id="General (compact)", classification_type="Multiclass", target_export_platforms="Basic platforms")
print("Project created!")

Go to customvision.ai if you would like to validate the project was created in the UI.

Next create a tag list from the file names. You will need to update the directory for your project.

#Create Tag List from folders in bird directory
import os
os.chdir('./bird_photos/train')
tags = [name for name in os.listdir('.') if os.path.isdir(name)]
print(tags)

We will create three methods that we will loop thru for each tag. First one creates the tag in custom vision.

def createTag(tag):
    result = trainer.create_tag(project.id, tag)
    print(f'{tag} create with id: {result}')
    return result.id

Next it takes the tag and returned tag id to create a image upload list.

def createImageList(tag, tag_id):
    #set directory to current tag
    base_image_url = f"./{tag}/"
    photo_name_list = os.listdir(base_image_url)
    image_list = []
    for file_name in photo_name_list:
        with open(base_image_url+file_name, "rb") as image_contents:
            image_list.append(ImageFileCreateEntry(name=base_image_url+file_name, contents=image_contents.read(), tag_ids=[tag_id]))
    return image_list

Then we take the created image list and upload it to Custom Vision

def uploadImageList(image_list):  
    upload_result = trainer.create_images_from_files(project.id, images=image_list)
    if not upload_result.is_batch_successful:
        print("Image batch upload failed.")
        for image in upload_result.images:
            print("Image status: ", image.status)
        exit(-1)

Now that we have created our methods lets loop thru each tag and upload in batches. This could take a little while depending on the amount of images and tags you are using.

#get images names from directory
for tag in tags:
    tag_id = createTag(tag)
    print(f"tag creation done with tag id {tag_id}")
    image_list = createImageList(tag, tag_id)
    print("image_list created with length " + str(len(image_list)))

    #break list into lists of 25 and upload in batches
    for i in range(0, len(image_list), 25):
        batch = image_list[i:i + 25]
        print(f'Upload started for batch {i} total items {len(batch)} for tag {tag}...')
        uploadImageList(batch)
        print(f"Batch {i} Image upload completed. Total uploaded {len(batch)} for tag {tag}")

Train the model

import time

print ("Training...")
iteration = trainer.train_project(project.id)
while (iteration.status != "Completed"):
    iteration = trainer.get_iteration(project.id, iteration.id)
    print ("Training status: " + iteration.status)
    time.sleep(5)

Publish the endpoint

# The iteration is now trained. Publish it to the project endpoint
trainer.publish_iteration(project.id, iteration.id, publish_iteration_name, prediction_resource_id)
print ("Done!")

Now we can test the endpoint. You will need to update the key and endpoint for your prediction endpoint that was created.

from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient
from msrest.authentication import ApiKeyCredentials

prediction_key = "<prediction-key>"
prediction_endpoint ="https://<resource-name-prediction>.cognitiveservices.azure.com/"
# Now there is a trained endpoint that can be used to make a prediction
predictor = CustomVisionPredictionClient(prediction_key, endpoint=prediction_endpoint)

Update the directory to the test/validation image

#get currect directory
os.getcwd()
#move back
os.chdir("../")
val_image_url = f"./test/American Crow/img.jpg"

Test the endpoint and print the results

with open(val_image_url, "rb") as image_contents:
    results = predictor.classify_image(project.id, publish_iteration_name, image_contents.read())
        # Display the results.
    for prediction in results.predictions:
        print("\t" + prediction.tag_name +
              ": {0:.2f}%".format(prediction.probability * 100))

... and DONE! We scraped a image data set with Bing Search and created a model with Azure Custom Vision SDK!

Checkout other Cognitive Services and Azure Machine Learning to learn more about building AI/ML on Azure!

Get Started with Power Apps Component Framework (PCF)

Cassie Breviu — Wed, 08 Jul 2020 14:48:36 +0000

#PowerfulDevs Conference

RSVP HERE BY JULY 15:
https://aka.ms/powerfuldevsconf

The Powerful DEVs Conference is the first virtual conference of its kind. We will showcase how developers can leverage the Power Platform to build applications faster and with far less effort. Connect with industry-recognized ProDev influencers, Microsoft Cloud Advocates, trusted and diverse community leaders, and members of the Power Platform Team.

BOOKMARK THIS FOR RESOURCES AND DISCUSSION WITH SPEAKERS

#powerfuldevs Conference: Join us on July 15th Online!

JennyMEvents for Microsoft Azure ・ Jul 10 '20

#powerfuldevs #powerplatform #prodev #programming

Revisit this page during the event to engage in live (and post-event) discussions on those topics with both speakers and community. The speakers will be here for a live Q&A for at least 30 minutes immediately after their session concludes. After the event, come back to find additional slides, videos, and resources for this session.

About This Session:

July 15, 2020: 14:00 PDT - 14:25 PDT

Learn how the Power Apps Component Framework helps you create reusable components that can be integrated in your Power Apps. The component framework empowers developers and app makers to build code components when the out-of-the-box components don't fit an app maker's needs. We will go step-by-step to create a code component using familiar technologies like TypeScript, node, html, and css.

About the Speakers:

Cassie is a Cloud Developer Advocate @ Microsoft and .Net Software Engineer. Follow @cassieview on Twitter.

Make a Custom Pipe Desk Mount 📐💻

Cassie Breviu — Tue, 19 May 2020 16:15:53 +0000

Here is a quick write up on how to create this desk setup:

Parts List for pipes

4 - 24" 3/4 pipes
1 - 24" 1 1/4 pipe
2 - 3/4 elbow connector
1 - 1 1/4 Tee connector
2 - 1 1/4 to 3/4 reducing connectors
2 - 3/4 base
1 - 1 1/4 base
12 - 1/4 bolts

Mount List and other desk tech

Build the pipe mount

Measure your desk and see what size will work best for you. My desk was 30 by 60 so 24 inch pipes worked perfect!
Assemble pipes and make sure they are level.
Wash and sand the pipes before painting. I used dish soap and a sponge in the driveway.
Get spray paint that works for metal
Let dry over night. Then measure to center on your desk. Dont mount directly on the back. This is a heavy mount!
Bolt to desk. I used 2" bolts but depending on the thickness of your desk you may need different ones.

Have fun building and feel free to post questions below. I will try to update with more detail later but I wanted to get a parts list out there for people that are asking! 😊

Get Free 3D Models in MS Paint 3D then Add it to BabylonJS Project

Cassie Breviu — Sun, 17 May 2020 03:36:21 +0000

Open Paint 3D
Click 3D Library
Search for the model you want
Select the model to add it to the canvas
Menu -> Save As -> 3D Model
BOOM you have a free GLB 3D model to use in your game!

Add it to BabylonJS

Copy GLB model file to the assets file in your project
Import the model into the scene:

  SceneLoader.ImportMesh("", "/assets/", "grapes.babylon", scene, function (
    grapes
  ) {
    // Set the target of the camera to the first imported mesh
    camera.target = grapes[0];
  });

Happy game building!

Test localhost on Oculus Quest with Remote Debugging in Firefox

Cassie Breviu — Sun, 17 May 2020 03:15:10 +0000

Here are some quick steps to help with debugging for WebXR. These things are LIFE CHANGING for WebXR development! Note: the steps are specific to Windows OS.

Install Firefox and Android Debug Bridge (adb)

Install Firefox on both the Oculus and your PC
Install Android Debug Bridge (adb)

Add Android Debug Bridge (adb) to Path

Type path in the windows search and hit enter on Edit the system environment variables
Click Environment Variables
Select Path and click Edit
Select New
Paste the location of adb.exe. The location should be something like this C:\Users\USERNAME\AppData\Local\Android\sdk\platform-tools
Restart your computer

Run the project locally and connect to remote debugging

Open your WebXR Project (Starter Template for BabylonJS Here)
Run the project npm start
Open Firefox on PC
Connect to Oculus Quest to PC with USB
Click the menu on the right of the Firefox browser
Then select Web Developer -> Remote Debugging
Click Connect on device

Reverse the Ports to make `localhost` available

Open cmd or windows terminal
List connected devices: adb devices
Set the reverse device id: adb reverse -s <ID from device attached above>
Reverse the port and update to the port you are using: adb reverse tcp:8080 tcp:8080
If it gets disconnected (sometimes it does) just rerun the last cmd to reconnect.
On the Oculus headset go to localhost:8080 (update 8080 to your port)

That's it! You are now running localhost on your VR headset and the debug output on your PC.

ONNX: No, it's not a Pokemon! Deploy your ONNX model with C# and Azure Functions

Cassie Breviu — Fri, 17 Apr 2020 14:50:15 +0000

Ok you got a ML model working in Jupyter notebook, now what? Lets deploy it! There are many ways to operationalize your model. In this tutorial we are going to be using the ONNX model format and runtime. ONNX gives you the ability to use the same model and application code across different platforms. This means I can create this model in Python with SciKit Learn and use the resulting model in C#! Say whaaat? Yes, that is right. Save it to ONNX format then run it and do the inferencing in C# with the onnxruntime!

We are going to be using a model created with Python and SciKit Learn from this blog post to classify wine quality based on the description from a wine magazine. We are going to take that model, update it to use a pipeline and export it to an ONNX format.

We want to use ONNX format is because this is what will allow us to deploy it to many different platforms. There are other benefits (such as performance) to using ONNX. Learn more about that here. Since I ♥ C# we are going to use C# and the onnxruntime nuget library available for dotnet. However, if you prefer to use a different language many are supported. Checkout all the supported libraries and languages here

Prerequisites

NOTE: If you prefer to create the model locally I recommend downloading Anaconda. However, this tutorial is written as if you are using Azure ML.

What is Open Neural Network Exchange (ONNX)?

ONNX is an open/common file format to enable you to use models with a variety of frameworks, tools, runtimes, and compilers. Once the model is exported to the ONNX format then you can use the ONNX Runtime: a cross-platform, high performance scoring engine for ML models. This provides framework interoperability and helps to maximize the reach of hardware optimization.

Create the Model with Azure Machine Learning

We have a model from the previous blog post to classify wine quality that we will use as the example model. See the note below if you have your own model you would like to use. Additionally, you should already have an Azure Machine Learning (AML) Studio created to generate the model. If not, follow these steps to create the workspace.

NOTE: To use your own model visit the ONNX GitHub tutorials to see how to convert different frameworks and tools.

1. Create a Cloud Compute Instance in the Azure ML Resource

Navigate to your Azure ML Resource (https://ml.azure.com)
Click on the nav "Compute"
Click "New"
Enter a name for the resource
Select the machine size
Click "Create"

2. Clone the Git repo with the Jupyter Notebook

Open JupyterLab for the compute instance in AML
Click the Terminal to open a terminal tab
Clone the GitHub repo
The onnx-csharp-serverless folder will appear. Navigate to the Jupyter Notebook onnx-csharp-serverless/notebook/wine-nlp-onnx.ipynb.

git clone https://github.com/cassieview/onnx-csharp-serverless.git

3. Install the package

ONNX has different packages for python conversion to the ONNX format. Since we used SciKit Learn we will use the skl2onnx package to export our trained model

In the terminal install the package with the following command

pip install skl2onnx

4. Run the code

The notebook provided goes over how to create a basic bag-of-words style NLP model. Run each cell until you get to the Export the Model step near the bottom of the notebook.

The first cell in the export block of the notebook is importing the ONNX packages.

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import StringTensorType

The next cell is running convert_sklearn and sending in the the following values:

The first parameter is a trained classifier or pipeline. In our example we are using a pipeline with 2 steps to vectorize and classify the text.
The second parameter is a string name value of your choice.
Lastly setting the initial_types This is a list with each list item a tuple of a variable name and a type.

model_onnx = convert_sklearn(pipeline,
                             "quality",
                             initial_types=[("input", StringTensorType([None, 1]))])

Then we will simply export the model.

with open("pipeline_quality.onnx", "wb") as f:
    f.write(model_onnx.SerializeToString())

5. Save Model to Azure Storage

Now that we have exported the model into ONNX format lets save it to Azure Storage. Follow these steps to create a storage account and upload the model created.

Deploy Model with Azure Functions

Azure Functions is the serverless Azure Resource we are going to use to deploy our model.

1. Install VS Code Azure Function extension:

Install the Azure Functions extension. You can use the Azure Functions extension to create and test functions and deploy them to Azure.
In Visual Studio Code, open Extensions and search for Azure functions, or select this link
Select Install to install the extension for Visual Studio Code:

2. Use VS Code to create Azure function

Hit CTRL-SHIFT-P to open the command pallet
Type create function and select the create function option
From the popup select Create new project and create a folder for the project
When prompted for a language select C#. Note that you have many language options with functions!
Next select a template. We want an HttpTrigger and give it a name
Next it will prompt you for a namespace. I used Wine.Function but feel free to name it as you wish.
Access Rights are next, select Function
Select open in current window
You should be prompted in the bottom right corner to restore packages. If not you can always open the terminal and run dotnet restore to restore nuget packages.

3. Run the project to validate its working

Hit F5 to run project and test that its working
Once the function is up there will be a localhost endpoint displayed in the terminal output of VS Code. Paste that into a browser with a query string to test that it is working. The endpoint will look something like this:

  http://localhost:7071/api/wine?name=test

The result in the browser should look like this Hello, test. This HTTP triggered function executed successfully.
Stop the run.

4. Install the Nuget Packages

Sweet, we now have an Azure Function ready to go. Lets install the nuget package we need to inference with our exported model in C#.

Open the terminal and run the below commands to install the ONNX package

dotnet add package Microsoft.ML.OnnxRuntime --version 1.2.0
dotnet add package System.Numerics.Tensors --version 0.1.0

Add the Azure Storage package with the below command

dotnet add package Azure.Storage.Blobs

Import the libraries at the top of the C# class.

using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Microsoft.ML.OnnxRuntime;
using System.Numerics.Tensors;

5. Update the Code

Copy and paste the below code into the class created. Read through the comments to understand what is happening at each step in the code.

        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log, ExecutionContext context)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string review = req.Query["review"];

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            review = review ?? data?.review;

            // Get path to model to create inference session.
            var modelPath = GetFileAndPathFromStorage(context, "model327", "pipeline_quality.onnx");
            var inputTensor = new DenseTensor<string>(new string[] { review }, new int[] { 1, 1 });

            // Create input data for session.
            var input = new List<NamedOnnxValue> { NamedOnnxValue.CreateFromTensor<string>("input", inputTensor) };

            // Create an InferenceSession from the Model Path.
            var session = new InferenceSession(modelPath);

            // Run session and send input data in to get inference output. The run is returned as an object but it is a list. Call ToList then get the Last item. Then use the AsEnumerable extension method to return the Value result as an Enumerable of NamedOnnxValue.
            var output = session.Run(input).ToList().Last().AsEnumerable<NamedOnnxValue>();

            // From the Enumerable output create the inferenceResult by getting the First value and using the AsDictionary extension method of the NamedOnnxValue.
            var inferenceResult = output.First().AsDictionary<string, float>();

            // Return the inference result as json.
            return new JsonResult(inferenceResult);
        }

Add the GetFileAndPathFromStorage helper method to the class below the Run method. This method will download the model from the storage account we created and uploaded the model to.

 internal static string GetFileAndPathFromStorage(ExecutionContext context, string containerName, string fileName)
        {
            //Check if file already exists
            var filePath = System.IO.Path.Combine(context.FunctionDirectory, fileName);
            var fileExists = File.Exists(filePath);

            if (!fileExists)
            {
                //Get model from Azure Blob Storage.
                var connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
                var blobServiceClient = new BlobServiceClient(connectionString);
                var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
                var blobClient = containerClient.GetBlobClient(fileName);

                // Download the blob's contents and save it to a file
                BlobDownloadInfo blobDownloadInfo = blobClient.Download();
                using (FileStream downloadFileStream = File.OpenWrite(filePath))
                {
                    blobDownloadInfo.Content.CopyTo(downloadFileStream);
                    downloadFileStream.Close();
                }

            }

            return filePath;
        }

6. Update the storage parameters

Update the Storage Account connection in the local.settings.json to connect to your storage account. You will find the connection string in the created resource in Azure under Access Keys.
Update the containerName and fileName to the names you used.

7. Test the endpoint

Its time to test the function locally to make sure everything is working correctly.

Hit F5 to run project and test that its working
This time we are going to use an actual wine review from Wine Enthusiast. Rather than doing this through the browser we are going to use Postman.
Grab the endpoint from the terminal in vs code and paste it into a new tab in Postman.
Change the GET dropdown to POST
Select the Body tab
Select the raw radiobutton
check the text dropdown to JSON
Paste the body into the text editor

{
  "review": "From the famous Oakville site, this aged wine spends three years in 100% new French oak, one in neutral oak and an additional year in bottle. Though it has had time to evolve, it has years to go to unfurl its core of eucalyptus, mint and cedar. It shows an unmistakable crispness of red fruit, orange peel and stone, all honed by a grippy, generous palate."
}

Hit send and you should see inference results.

8. Deploy the Endpoint to Azure

WOOHOO! We have created our model, the C# Azure Function, and tested it locally with Postman. Lets deploy it! The VS Code Azure Functions extension makes deploying to Azure quite simple. Follow these steps to publish the function from VS Code.

Once the app is deployed we need update some application settings in the Function App.

Navigate to the Function App in Azure Portal
Select the name of the Function App you created
Select Configuration
Click New Application setting and add the storage connection string name and value.
Click Edit on WEBSITE_RUN_FROM_PACKAGE and change the value to 0. This allows us to write files from our function. (You may have to redeploy your function from VS Code after making this change.)
Save the changes

Play the Somm Wine Game here

I created a game from these models called Somm Wine Game. Its game to see how well you can describe your wine like a sommelier! It uses these machine learning models to predict the price, points, and variety based on a description. The better your description is the more accurate the predictions!

Dont let the learning stop here! Check out these additional resources for topics covered in this tutorial.

ONNX Docs
Onnx C# API Docs
Scikit learn pipeline onnx
ONNX Runtime Github
Blog post on how models were built

Cheers to you!

Warm Up by the Fire with Bash (Fire Ascii Art)

Cassie Breviu — Wed, 15 Jan 2020 19:07:59 +0000

Burrrr, its cold outside! Lets add a fire theme to our terminal to warm up!

Prerequisites for Windows

Setup Steps

Install libaa sudo apt-get install libaa-bin
Test command aafire
Add it to windows terminal:

Profile json to add it to terminal

    {
        "acrylicOpacity" : 0.5,
        "closeOnExit" : true,
        "colorScheme" : "Fireplace",
        "commandline" : "wsl.exe -d Ubuntu aafire",
        "cursorColor" : "#FFFFFF",
        "cursorShape" : "bar",
        "fontFace" : "Consolas",
        "fontSize" : 20,
        "guid" : "{6f9994f0-4403-5e85-9eec-98e5da3839xx}",
        "historySize" : 9001,
        "icon" : "ms-appx:///ProfileIcons/{9acb9455-ca41-5af7-950f-6bca1bc972xx}.png",
        "backgroundImage" : "",
        "backgroundImageOpacity" : 0.75,
        "backgroundImageStrechMode" : "fill",
        "name" : "Ubuntu Fireplace",
        "padding" : "0, 0, 0, 0",
        "snapOnInput" : true,
        "startingDirectory" : "%USERPROFILE%",
        "useAcrylic" : false
    }

Theme fireplace json to give it fire colors

    {
      "background": "#002B36",
      "black": "#073642",
      "blue": "#268BD2",
      "brightBlack": "#002B36",
      "brightBlue": "#839496",
      "brightCyan": "#93A1A1",
      "brightGreen": "#586E75",
      "brightPurple": "#6C71C4",
      "brightRed": "#CB4B16",
      "brightWhite": "#FFFF00",
      "brightYellow": "#657B83",
      "cyan": "#2AA198",
      "foreground": "#FF8000",
      "green": "#859900",
      "name": "Fireplace",
      "purple": "#D33682",
      "red": "#DC322F",
      "white": "#EEE8D5",
      "yellow": "#B58900"
    }

That's 🔥!

Now you can warm up by the terminal fireplace until summer returns. 😊

Easily add Anaconda Prompt to Windows Terminal to make life better

Cassie Breviu — Thu, 05 Dec 2019 23:53:07 +0000

I love the Windows Terminal and if you haven't tried it out yet I highly recommend it. Its super customizable which is absolutely lovely. Here are the steps to add Anaconda Prompt to Windows Terminal.

Prerequisites

Add Anaconda Prompt

Open Windows Terminal and click the arrow and then settings. This will open the settings.json file to customize the terminal.
Duplicate the cmd settings by copying and pasting them below or highlight CTRL + D if using visual studio to duplicate the selected text. (I ♥ shortcuts)
Update the guid to something unique
Update the name to Anaconda
Next update the commandline to the cmd for Anaconda Prompt. The easiest way to do this is to grab the target path from the Anaconda Prompt shortcut.
- Search for Anaconda Prompt in the windows search bar
- Right click and open file location
- Right click on the Anaconda Prompt shortcut
- Click properties
- Grab the target path
- Delete the extra text to get the path (shown below)
- If you prefer you can replace the "C:\Users\xxx" to "%USERPROFILE%" The new terminal settings.json should look something like this:
Optional: Update the icon to the Anaconda navigator icon
- It will be the same root path as above up to the anaconda3
- Append \\Menu\\anaconda-navigator.ico to the end.
- End result should look something like this: "icon": "%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\Menu\\anaconda-navigator.ico"
Save the changes to the settings.json file
Open Windows Terminal
Click the down arrow and select Anaconda
BOOM! Anaconda in Windows Terminal.
Listen to Sir-Mix-A-Lot and rejoice in your awesomeness

🤖 Grab Your Wine. It's Time to Demystify ML and NLP 🍷

Cassie Breviu — Tue, 23 Jul 2019 13:09:24 +0000

In this tutorial you will learn how to use ML to predict wine price, points and variety from the text description.

We will process the wine description text with the library SciKit Learn to create a Bag-of-Words Logistic Regression Multi-Classification model. If that didn't make sense then you are in the right place! If it did make sense continue reading because wine.

Configure your ML dev environment

There are two options: cloud or local.

Azure Cloud Setup - Use Azure Machine Learning Workspace with the integrated cloud Notebook VMs. Workspace gives you a LOT of functionality and I would highly recommend this.

Create a free Azure account if you dont already have one.
Follow the Create Workspace tutorial through the "Create a notebook VM" step
Click "Jupyter Lab" from the Notebook VM navigation
Then run the below command in the terminal to clone the notebook from GitHub.

#CTRL+SHIFT+V to paste
curl https://raw.githubusercontent.com/cassieview/intro-nlp-wine-reviews/master/winereview-nlp.ipynb --output winereview-nlp.ipynb

Local Setup

Download Jupyter notebooks
Download Anaconda
Run Jupyter notebook

Once you are set with one of the above environment configurations its time to start building!

Import packages and data

If you get an error "No module named" install it with the command !pip install wordcloud. Replace wordcloud with the module name in the error message.

#This package need to be installed first
!pip install wordcloud

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import math
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_curve
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import CountVectorizer
from joblib import dump, load

We need data! I used a dataset I found on Kaggle. Kaggle is an online community of data scientists. Download the dataset from the github repo or kaggle.

Import the data as a Pandas DataFrame

#File path to the csv file
csv_file = "https://raw.githubusercontent.com/cassieview/intro-nlp-wine-reviews/master/dataset/winemag-review.csv"

# Read csv file into dataframe
df = pd.read_csv(csv_file)

# Print first 5 rows in the dataframe
df.head()

Visualize the data

Once we have the data then its time to analyze it and do some Feature Selection and Engineering. We will visualize our data using Seaborn. This will allow us to see if there is a strong correlation between different data points and help us answer questions about our data. Since our initial question was around predicting price, points or variety from the description we already know that our Feature will be the description and our Label will be price, pointsor variety. Features are the data we use to make predictions and Labels are what we are predicting. Each label will be a separate model so there will be three models in total if you choose to build all three predictive models.

For fun, let's ask some questions about the data and answer them by graphing it with Seaborn.

Is there a correlation between price and points?

sns.barplot(x = 'points', y = 'price', data = df)

sns.boxplot(x = 'points', y = 'price', data = df)

Does one wine critic give higher ratings than the others?

sns.catplot(x = 'points', y = 'taster_name', data = df)

Let's look at a WordCloud of the description Text

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt
text = df.description.values
wordcloud = WordCloud(
    width = 3000,
    height = 2000,
    background_color = 'black',
    stopwords = STOPWORDS).generate(str(text))
fig = plt.figure(
    figsize = (40, 30),
    facecolor = 'k',
    edgecolor = 'k')
plt.imshow(wordcloud, interpolation = 'bilinear')
plt.axis('off')
plt.tight_layout(pad=0)
plt.show()

I like to think of this WordCloud as a cheatsheet of discriptive words to use when tasting wine to make yourself sound like a wine expert 😃

What other questions could you ask and answer by graphing this data?

Create Calculated Columns for Labels

This is going to be multi-classification for the price points or grape variety of the wines reviewed by the wine critics. Right now our points and price are number features. This needs to be updated to a text feature category, to do this we will create a couple functions to generate calculated columns based on the values in the points and price columns to use as are our labels.

Create quality column from points values to classes of bad, ok, good, and great. Below is a function to return string quality based on the points value.

def getQuality(points):
    if(points <= 85):
        return 'bad'
    elif(points<=90 ):
        return 'ok'
    elif(points<=95):
        return 'good'
    elif(points<=100):
        return 'great'
    else:
        return 'If this gets hit, we did something wrong!'

Next apply the function to the points column of the dataframe and add a new column named quality.

df['quality'] = df['points'].apply(getQuality)

Visualize the new column against the price column like we did above.

sns.catplot(x = 'quality', y = 'price', data = df)

sns.barplot(x = 'quality', y = 'price', data = df)

Create priceRange column from price column of 1-30, 31-50, 51-100, Above 100 and 0 for columns with NaN. Below is a function to return string priceRange based on price value.

def getPriceRange(price):
    if(price <= 30):
        return '1-30'
    elif(price<=50):
        return '31-50'
    elif(price<=100): 
        return '51-100'
    elif(math.isnan(price)):
        return '0'
    else:
        return 'Above 100'

Apply the function to the points column of the dataframe and add a new column named priceRange.

df['priceRange'] = df['price'].apply(getPriceRange)

Print totals for each priceRange assigned to see how the labels are distributed

df.groupby(df['priceRange']).size()
Output: priceRange
        0             8996
        1-30         73455
        31-50        27746
        51-100       16408
        Above 100     3366
        dtype: int64

We now have our labels for models to predict quality, priceRange and grape variety.

Process description text with the library SciKit Learn to create a Bag-of-Words using the CountVectorizer functionality.

The docs do a great job of explaining the CountVectorizer. I recommend reading through them to get a full understanding of whats going on, however I will go over some of the basics here.

At a high level the CountVectorizer is taking the text of the description, removing stop words (such as “the”, “a”, “an”, “in”), creating a tokenization of the words and then creating a vector of numbers that represents the description. The text description is now represented as numbers with only the words we care about and can be processed by the computer to train a model. Remember the computer understand numbers and words can be represented as numbers so the computer can "understand".

This is an example of how the words become numbers. We will go over this in more detail with an example from the dataset as well.

Before we jump into the CountVectorizer code and functionality. I want to list out some terms and point out that CountVectorizer does not do the Lemmatization or Stemming for you.

StopWords: A stopword can be a word with meaning in a specific language. For example, in the English language, words such as "a," "and," "is," and "the" are left out of the full-text index since they are known to be useless to a search. A stopword can also be a token that does not have linguistic meaning.
N-Gram: A feature extraction scheme for text data: any sequence of N words turns into a feature value.
Lemmatization: converts multiple related words to a single canonical form ("fruity", "fruitiness" and "fruits" would all become "fruit")
Stemming: Similar to Lemmatization but a bit more aggressive and can leave words fragmented.

Let's take a look at how to use the CountVectorizer.

These are all the properties that you can set within the CountVectorizer. Many of them are defaulted or if set override other parts of the CountVectorizer. We are going to leave most of the defaults and then play with changing some of them to get better results for our model.

CountVectorizer( 
  input='content', encoding='utf-8', decode_error='strict', 
  strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, 
  stop_words=None, token_pattern='(?u)\b\w\w+\b', ngram_range=(1, 1), 
  analyzer='word', max_df=1.0, min_df=1, max_features=None, 
  vocabulary=None, binary=False, dtype=<class 'numpy.int64'>
)

Create the function to get the vector and vectorizer from the description feature.

There are different CountVectorizer configurations commented out so that we can play with different configs and see how it changes our result. Additionally this will help us look at one description and pick apart what is actually happening in the CountVectorizer.

def get_vector_feature_matrix(description):
    vectorizer = CountVectorizer(lowercase=True, stop_words="english", max_features=5)
    #vectorizer = CountVectorizer(lowercase=True, stop_words="english")
    #vectorizer = CountVectorizer(lowercase=True, stop_words="english",ngram_range=(1, 2), max_features=20)

    #vectorizer = CountVectorizer(lowercase=True, stop_words="english", tokenizer=stemming_tokenizer) 
    vector = vectorizer.fit_transform(np.array(description))
    return vector, vectorizer

For the first run we are going to have the below config. What this is saying is that we want to convert the text to lowercase, remove the english stopwords and we only want 5 words as feature tokens.

vectorizer = CountVectorizer(lowercase=True, stop_words="english", max_features=5)

#Optional: remove any rows with NaN values. 
#df = df.dropna()

Next let's call our function and pass in the description column from the dataframe.

This returns the vector and the vectorizer. The vectorizer is what we apply to our text to create the number vector representation of our text so that the machine learning model can learn.

vector, vectorizer = get_vector_feature_matrix(df['description'])

If we print the vectorizer we can see the current default parameters for it.

print(vectorizer)
Output: CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=5, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words='english',
        strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)

Examine our variables and data to understand whats happening here.

print(vectorizer.get_feature_names())
Output: ['aromas', 'flavors', 'fruit', 'palate', 'wine']

Here we are getting the features of the vectorizer. Because we told the CountVectorizer to have a max_feature = 5 it will build a vocabulary that only considers the top feature words ordered by term frequency across the corpus. This means that our description vectors would only include these words when they are tokenized, all the other words would be ignored.

Print out our first description and first vector to see this represented.

print(vector.toarray()[0])
Output: [1 0 1 1 0]

df['description'].iloc[0]
Output: "_Aromas_ include tropical _fruit_, broom, brimstone and dried herb. The _palate_ isn't overly expressive, offering unripened apple, citrus and dried sage alongside brisk acidity."

The vector array ([1 0 1 1 0]) that represents the vectorization features (['aromas', 'flavors', 'fruit', 'palate', 'wine']) in first description in the corpus. 1 indicates its present and 0 indicates not present in the order of the vectorization features.

Play around with different indexes of the vector and description. You will notice that there isn't lemmatization so words like fruity and fruits are being ignored since only fruit is included in the vector and we didn't lemmatize the description to transform them into their root word.

Train the Model

Update the function so that the second vectorizer configuration is being used.

def get_vector_feature_matrix(description):
    #vectorizer = CountVectorizer(lowercase=True, stop_words="english", max_features=5)
    vectorizer = CountVectorizer(lowercase=True, stop_words="english", max_features=1000)
    #vectorizer = CountVectorizer(lowercase=True, stop_words="english",ngram_range=(1, 2), max_features=1000)
    #vectorizer = CountVectorizer(lowercase=True, stop_words="english", tokenizer=stemming_tokenizer) 
    vector = vectorizer.fit_transform(np.array(description))
    return vector, vectorizer

And call the function to update the vectorizer

vector, vectorizer = get_vector_feature_matrix(df['description'])

Now create our feature matrix. If you get a MemoryError here reduce the max_features in the CountVectorizer.

features = vector.todense()

We have three different labels for three different models. Let's assign the label variable next and use the quality label first.

label = df['quality'] 
#label = df['priceRange']
#label = df['variety']

We have the features and label variables created. Next we need to split the data to train and test.

We are going to use 80% to train and 20% to test. This will allow us to get an accuracy estimation from the training to see how the model is performing.

X, y = features, label
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

Train the model using a LogisticRegression algorithm and print the accuracy.

lr = LogisticRegression(multi_class='ovr',solver='lbfgs')
model = lr.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
print ("Accuracy is {}".format(accuracy))
Output: "Accuracy is 0.7404885554914407"

This is an ok accuracy but I am sure it can be improved! For this tutorial we are going to call it "good enough" which is a decision that needs to be made with every model you ever build!

Test the model

When you select a candidate model it should always be tested on unseen data. If a model is overfitted to its data it will perform really will on its own data and poorly on new data. This is why its very important to test on unseen data.

I grabbed this review from the wine mag site. Its a 95 point and $60 bottle of wine review.

test = "This comes from the producer's coolest estate near the town of Freestone. White pepper jumps from the glass alongside accents of lavender, rose and spice. Compelling in every way, it offers juicy raspberry fruit that's focused, pure and undeniably delicious."


x = vectorizer.transform(np.array([test]))
proba = model.predict_proba(x)
classes = model.classes_
resultdf = pd.DataFrame(data=proba, columns=classes)
resultdf

This is a correct prediction! 🎉

Another way to look at the result is to transpose, sort and then print the head resulting in a list of the top 5 predictions.

topPrediction = resultdf.T.sort_values(by=[0], ascending = [False])
topPrediction.head()

Other things to try

Change the label and run again for the price bucket prediction or grape variety.
Try to use different algorithms to see if you can get a better result
Add additional features to the description text to improve accuracy. There was a strong correlation between price and points. Maybe adding those would improve the accuracy score?
Add lemmatization to the text to improve score using the NLTK
Try doing a text classification on a different dataset.

Data science is a trial and error process. I am sure there are ways to improve this model and accuracy. Play around and see if you can get a better result!

Build a Web VR game with JavaScript using BabylonJS

Cassie Breviu — Thu, 27 Jun 2019 17:25:02 +0000

In this tutorial I will show you step by step how to build a web VR game with BabylonJS

Prerequisites

-nodejs
-vs code

What is BabylonJS and CannonJS

BabylonJS is a complete JavaScript framework for building 3D games and experiences with HTML5, WebGL, WebVR and Web Audio.

CannonJS is a physics engine, written in JavaScript. And what is a physics engine you might ask? Well its "software that provides an approximate simulation of certain physical systems, such as rigid body dynamics (including collision detection), soft body dynamics, and fluid dynamics, of use in the domains of computer graphics, video games and film."

First we need to get the base starter project using babylonjs, webpack, and typescript

Steps to Run Starter Project and Git Repo Link

Clone the repo git clone https://github.com/cassieview/babylonjs-webpack-typescript-starter-project.git cd babylonjs-webpack-typescript-starter-project
Install packages npm install
Build Project npm run build
Run the script to test the project npm start
Open in VS Code code .

Lets talk about the starter project

Simple index.html template.

<!DOCTYPE html>
<html>

    <head>
        <style>
            html,
            body {
                overflow: hidden;
                width: 100%;
                height: 100%;
                margin: 0;
                padding: 0;
                text-align: center;
            }

            #renderCanvas {
                width: 100%;
                height: 100%;
                touch-action: none;
            }
        </style>
    </head>

    <body>
        <canvas id="renderCanvas"></canvas>
        <script src="dist/index.js"></script>
    </body>

</html>

The index.ts typescript file

The index.ts file is the typescript file that creates the main scene. It is typescript that is transpiled to javascript in the dist folder.

The script source for the game is found in the dist folder. Webpack is an open-source JavaScript module bundler it generates static assets representing those modules. This is what is loaded from the dist folder. WebPack compiles the script down to one source and that is used to serve the game script.

First thing we import the packages needed from BabylonJS to create our game scene. Create the canvas variable and use vanilla javascript to grab the renderCanvas canvas tag from the html body section. Then we create the engine and pass in the BabylonJS engine.


import { Engine, Scene, HemisphericLight, Vector3, MeshBuilder, Mesh } from "babylonjs";
var canvas: any = document.getElementById("renderCanvas");
var engine: Engine = new Engine(canvas, true);

Next we have the create scene function. Here we define the scene, pass in the engine. The we create a camera. The camera is the point of view of the game player. We are using the universal camera.

Next we add a simple sphere mesh to our scene and set the basic properties. The vr helper adds the vr button to the bottom right of the screen so that a user can enter the game in vr. This does create issues when viewing the game in the browser and testing. For testing I recommend commenting out that line of code. Then when you want to test with your vr headset, uncomment it to enter the vr game.

TIP: You can easily test changes as you make them by running npm run build then open the path to the index.html file in the browser C:/Code/babylonjs-webpack-typescript-starter-project/index.html. This is a static site so you dont actually have to run it with npm start. Simply run the build and refresh the browser path to the index.html.

function createScene(): Scene {
    // Create scene
    var scene: Scene = new Scene(engine);

    // Create camera
    var camera = new BABYLON.UniversalCamera("UniversalCamera", new BABYLON.Vector3(0, 0, -10), scene);

    // Create sphere
    var sphere1: Mesh = MeshBuilder.CreateSphere("sphere", { diameter: 1 }, scene);
    sphere1.position.y = 5;
    sphere1.material = new BABYLON.StandardMaterial("sphere material", scene)

    // Enable VR
    var vrHelper = scene.createDefaultVRExperience();
    vrHelper.enableInteractions();

    return scene;
}

var scene: Scene = createScene();

engine.runRenderLoop(() => {
    scene.render();
});

Start building the game

Now you should have a basic understanding of whats in the starter project and what babylonjs is doing for us. Next we want to add gravity so we need the Cannonjs library mentioned above.

import { Engine, Scene, ArcRotateCamera, HemisphericLight, Vector3, MeshBuilder, Mesh, CannonJSPlugin } from "babylonjs";

Copy and paste this code block under the scene variable. Here we are adding the ground mesh and giving it a physicsImpostor so that the sphere will fall and land on the ground.

var gravityVector = new BABYLON.Vector3(0, -1, 0);
    scene.enablePhysics(gravityVector, new CannonJSPlugin);

    var light = new HemisphericLight("light",Vector3.Zero(),scene);

    // Parameters : name, position, scene
    var camera = new BABYLON.UniversalCamera("UniversalCamera", new BABYLON.Vector3(0, 0, -10), scene);
    camera.checkCollisions = true;
    camera.applyGravity = true;
    // Targets the camera to a particular position. In this case the scene origin
    camera.setTarget(BABYLON.Vector3.Zero());

    // Attach the camera to the canvas
    camera.attachControl(canvas, true);

    // Create Ground
    var ground = BABYLON.Mesh.CreatePlane("ground", 25.0, scene);
    ground.position = new BABYLON.Vector3(0, -10, 0);
    ground.rotation = new BABYLON.Vector3(Math.PI / 2, 0, 0);

    ground.material = new BABYLON.StandardMaterial("groundMat", scene);
    ground.material.backFaceCulling = false;
    ground.receiveShadows = true;
    ground.physicsImpostor = new BABYLON.PhysicsImpostor(ground, BABYLON.PhysicsImpostor.BoxImpostor, { mass: 0, friction: 1, restitution: 0 }, scene);

Add physics, shadow and light to sphere:

import { Engine, Scene, ArcRotateCamera, HemisphericLight, Vector3, MeshBuilder, Mesh, CannonJSPlugin, ShadowGenerator, DirectionalLight } from "babylonjs";



// Create sphere
    var sphereLight = new DirectionalLight("dir02", new Vector3(0.2, -1, 0), scene);
    sphereLight.position = new Vector3(0, 80, 0);

    var sphere1: Mesh = MeshBuilder.CreateSphere("sphere", { diameter: 1 }, scene);
    sphere1.position.y = 5;
    sphere1.material = new BABYLON.StandardMaterial("sphere material", scene)
    sphere1.physicsImpostor = new BABYLON.PhysicsImpostor(sphere1, BABYLON.PhysicsImpostor.SphereImpostor, { mass: 1 }, scene);
    var shadowGenerator = new ShadowGenerator(2048, sphereLight);
    shadowGenerator.addShadowCaster(sphere1);

Now we have a sphere falling onto the ground panel we created. How exciting.

Project Architecture

We have a lot to add to this game still and although we could add it all in one giant function. That is not best practice for a variety of reasons. Lets add a sphere.ts file and move our sphere logic to it.


import { Scene, Vector3, MeshBuilder, Mesh, ShadowGenerator, DirectionalLight } from "babylonjs";


export function addSphere(scene: Scene) {

    // Create sphere
    var sphereLight = new DirectionalLight("dir02", new Vector3(0.2, -1, 0), scene);
    sphereLight.position = new Vector3(0, 80, 0);

    var sphere: Mesh = MeshBuilder.CreateSphere("sphere", { diameter: 1 }, scene);
    sphere.position.y = 5;
    sphere.material = new BABYLON.StandardMaterial("sphere material", scene)
    sphere.physicsImpostor = new BABYLON.PhysicsImpostor(sphere, BABYLON.PhysicsImpostor.SphereImpostor, { mass: 1 }, scene);
    var shadowGenerator = new ShadowGenerator(2048, sphereLight);
    shadowGenerator.addShadowCaster(sphere);
}

Then go back to index.ts and import the file we created and call the addSphere function where the addSphere logic used to be.


line 2: import { addSphere } from "./sphere";
line 35:  addSphere(scene);

Now would be a good time to npm run build and refresh your browser to see that you completed the logic move successfully.

Add the start button

Ok like any good game you need a start button, to ya know, start the game.

Import the gui library so that we can use the 3d button and panel.

import * as GUI from  "babylonjs-gui";

Add the startGameButton function below the createScene function. Move the addSphere function call to the button.onPointerUpObservable event. This event is used to trigger events on click.


var startGameButton = function (panel) {
    var button = new GUI.Button3D();
    panel.addControl(button);
    button.onPointerUpObservable.add(function () {
        addSphere(scene);
    });
    var text1 = new GUI.TextBlock();
    text1.text = "Start Game";
    text1.color = "white";
    text1.fontSize = 24;
    button.content = text1;
}

Update the createScene function to add the button to the scene. This will go on line 35 where addSphere previously was.


    // Create the 3D UI manager
    var manager = new GUI.GUI3DManager(scene);
    // Create a horizontal stack panel
    var panel = new GUI.StackPanel3D();
    panel.margin = 0.02;
    manager.addControl(panel);
    startGameButton(panel);

This would be a good time to npm run build and test the changes you made. When you click the button the sphere should drop from the sky on to the ground.

Make sphere disappear on click

To do this we are going to the sphere.ts file and add the ActionManager to the sphere so when we click on the sphere it disappears. Add the below logic start at line 17. Additionally you will need to update the import at the top of the file to include ActionManager and ExecuteCodeAction.

import { Scene, Vector3, MeshBuilder, Mesh, ShadowGenerator, DirectionalLight, ActionManager, ExecuteCodeAction } from "babylonjs";


    sphere.actionManager = new ActionManager(scene);

    //add click event to sphere
    sphere.actionManager.registerAction(new 
    ExecuteCodeAction(ActionManager.OnPickUpTrigger, function () {

        scene.removeMesh(sphere);

    }));

Add loop to add multiple spheres when you click start

Add the below code in the sphere.ts above the addSphere function. This will add 10 spheres when we click the button instead of one. Update the addSphere function to var addSphere = function (scene: Scene) { since we will no longer call it from the index.ts file directly.

export function addSpheres(scene: Scene) {
    for (let index = 0; index < 10; index++) {
        addSphere(scene);
    }
}

Update the index.ts file to import the addSpheres function and call that instead of addSphere.

line 3: import { addSpheres } from "./sphere";
line 54: addSpheres(scene);

Then update the sphere position in the sphere.ts file so it doesn't create 10 spheres all in the same spot. Delete sphere.position.y = 5; and add

    line 17: sphere.position = new Vector3(Math.random() * 20 - 10, 10, Math.random() * 10 - 5);

Add particle animation to sphere to mimic an explosion

The sphere disappearing is cool but lets make it more dramatic by adding a particleSystem that will spray particles in a cartoon looking explosion type of way.

Add a new file called particles.ts and paste in the following code:

import { AbstractMesh, Texture, ParticleSystem, Scene, Vector3, Color4, Animation } from "babylonjs";
import { AdvancedDynamicTexture } from "babylonjs-gui";


let advancedTexture: AdvancedDynamicTexture;

export function addParticlesToMesh(mesh: AbstractMesh, scene: Scene): ParticleSystem {
    // Fountain object
    //var fountain = Mesh.CreateBox("foutain", 1.0, scene);

    var particleSystem = new ParticleSystem("particles", 2000, scene);

    //Texture of each particle
    particleSystem.particleTexture = new Texture("textures/flare.png", scene);

    // Where the particles come from
    particleSystem.emitter = mesh; // the starting object, the emitter
    particleSystem.minEmitBox = new Vector3(-1, 0, 0); // Starting all from
    particleSystem.maxEmitBox = new Vector3(1, 0, 0); // To...

    // Colors of all particles
    particleSystem.color1 = new Color4(0.7, 0.8, 1.0, 1.0);
    particleSystem.color2 = new Color4(0.2, 0.5, 1.0, 1.0);
    particleSystem.colorDead = new Color4(0, 0, 0.2, 0.0);

    // Size of each particle (random between...
    particleSystem.minSize = 0.1;
    particleSystem.maxSize = 0.5;

    // Life time of each particle (random between...
    particleSystem.minLifeTime = 0.3;
    particleSystem.maxLifeTime = 1.5;

    // Emission rate
    particleSystem.emitRate = 1500;

    // Blend mode : BLENDMODE_ONEONE, or BLENDMODE_STANDARD
    particleSystem.blendMode = ParticleSystem.BLENDMODE_ONEONE;

    // Set the gravity of all particles
    particleSystem.gravity = new Vector3(0, -9.81, 0);

    // Direction of each particle after it has been emitted
    particleSystem.direction1 = new Vector3(-7, 8, 3);
    particleSystem.direction2 = new Vector3(7, 8, -3);

    // Angular speed, in radians
    particleSystem.minAngularSpeed = 0;
    particleSystem.maxAngularSpeed = Math.PI;

    // Speed
    particleSystem.minEmitPower = 1;
    particleSystem.maxEmitPower = 3;
    particleSystem.updateSpeed = 0.005;

    // Start the particle system
    particleSystem.start();

    // Fountain's animation
    var keys = [];
    var animation = new Animation("animation", "rotation.x", 30, Animation.ANIMATIONTYPE_FLOAT,
        Animation.ANIMATIONLOOPMODE_CYCLE);
    // At the animation key 0, the value of scaling is "1"
    keys.push({
        frame: 0,
        value: 0
    });

    // At the animation key 50, the value of scaling is "0.2"
    keys.push({
        frame: 50,
        value: Math.PI
    });

    // At the animation key 100, the value of scaling is "1"
    keys.push({
        frame: 100,
        value: 0
    });

    // Launch animation
    animation.setKeys(keys);
    mesh.animations.push(animation);
    scene.beginAnimation(mesh, 0, 100, true);

    return particleSystem;
}

export function removeParticlesFromMesh(particleSystem: ParticleSystem): any {
    particleSystem.stop();
}

Import the particles.ts script into the spheres.ts script.

import { addParticlesToMesh, removeParticlesFromMesh } from "./particles";

Update the sphere on click event and add the sleep function in. This will add the particles to the sphere when it its clicked, wait 250 milliseconds and then stop adding particles. If you didnt stop the particles there would just be particles appearly everywhere long after the sphere was removed from the scene.

    sphere.actionManager.registerAction(new 
        ExecuteCodeAction(ActionManager.OnPickUpTrigger, function () {
        var particleSystem = addParticlesToMesh(sphere, scene);
        scene.removeMesh(sphere);
        sleep(250).then(() => {
            removeParticlesFromMesh(particleSystem);
        })

    }));

    const sleep = (milliseconds) => {
        return new Promise(resolve => setTimeout(resolve, milliseconds))
    }

Add score.ts because every game needs a way to keep score

Create the score.ts script and paste in the code below.

import { AdvancedDynamicTexture, Rectangle, Control, TextBlock } from 'babylonjs-gui';

let advancedTexture: AdvancedDynamicTexture;
let scoreText: TextBlock = new TextBlock();
let score = 0;
function init(): void {
    if (!advancedTexture) {
        advancedTexture = AdvancedDynamicTexture.CreateFullscreenUI("ui1");
    }
}

export function addLabelToScene(): void {

    if (!advancedTexture) {
        init();
    }
    let label = new Rectangle("score");
    label.background = "black";
    label.height = "30px";
    label.alpha = 0.5;
    label.width = "100px";
    label.cornerRadius = 20;
    label.thickness = 1;
    label.linkOffsetY = 30;
    label.top = "10%";
    label.zIndex = 5;
    label.verticalAlignment = Control.HORIZONTAL_ALIGNMENT_CENTER;
    advancedTexture.addControl(label);


    scoreText.text = "score: 0"
    scoreText.color = "white";
    label.addControl(scoreText);
}
export function incrementScore(): void{
    score++;
    scoreText.text = "score: " + score.toString();

}


export function updateScore(newScore: number): void{
    score = newScore;
    scoreText.text = "score: " + score.toString();

}

Then import the script in the index.ts script.

import { addLabelToScene, updateScore } from "./score";

In the index.ts file we want to add the function call addLabelToScene(panel) after we add the button startGameButton(panel); and we want to reset the score when the startGameButton is clicked.

var startGameButton = function (panel) {
    var button = new GUI.Button3D();
    panel.addControl(button);
    button.onPointerUpObservable.add(function () {
        //reset score
        updateScore(0);
        addSpheres(scene);
    });
    var text1 = new GUI.TextBlock();
    text1.text = "Start Game";
    text1.color = "white";
    text1.fontSize = 24;
    button.content = text1;
}

In the sphere.ts we need to import { incrementScore } from "./score"; from the score.ts script and then add the incrementScore(); after removeParticlesFromMesh(particleSystem); to increase the score when a sphere is clicked.

Remove `PhysicsImpostor` from ground mesh so balls fall through the ground instead of sitting on top.

We dont want people to be able to shoot the balls on the ground so we need to delete the PhysicsImpostor from the ground mesh.

ground.physicsImpostor = new BABYLON.PhysicsImpostor(ground, BABYLON.PhysicsImpostor.BoxImpostor, { mass: 0, friction: 0, restitution: 0 }, scene);

Last we will add some material to our spheres

Import the babylon materials to the sphere.ts script.

import {StandardMaterial, Texture, Color3} from "babylonjs-materials";

Then add the material to the sphere mesh with the following code


    // Material
    var materialAmiga = new StandardMaterial("amiga", scene);
    materialAmiga.diffuseTexture = new Texture("textures/amiga.jpg", scene);
    materialAmiga.emissiveColor = new Color3(0.5, 0.5, 0.5);
    sphere.material = materialAmiga;

Ok lets npm run build and see if it works!

More cool things you could add

texture
backgrounds
custom meshes
sound effects
the sky is the limit!

Deploy site to azure storage as a static site so all your friends can play too

Check out the docs on how to host this site for cheap on Azure

Full git repo of end of project

Shout out to the awesome docs and playground on Babylonjs and the awesome developers that did the hard work to create a sweet library to enable us to build games!

Happy game building!

How to Get a Dataset and Create a ML Model to Classify Yoga Poses

Cassie Breviu — Tue, 25 Jun 2019 18:25:52 +0000

This tutorial will go over how to use Azure Cognitive Services to automatically get an image dataset and train a Custom Vision machine learning model. The github repo has a simple nodejs console app that uses Bing Image Search to scrape images to a local directory based on search term(s). Then use the images scraped to create a Custom Vision model with Azure Cognitive Services.

Prerequisites

First create an Azure account (if you dont have one already)

Click here to create your account

Next we will create a Bing Search Api Resource needed to get our dataset

Go to portal.azure.com
Select "Create Resource"
Select "AI + Machine Learning"
Select "Bing Search v7"
Complete the required fields
Select "Create"
When the deployment succeeds you will get a notification in the top right corner.
Select "Go to resource"
Select "Keys" from the left hand navigation for the resource.
Copy the Subscription Key and paste it into the script.

Cognitive Service Bing Search Docs

Image Scraper using Bing Search Cognitive Service

We are going to use the image search function of this service however you can use this same endpoint to search web, news, image, video and entity. To run the Image Scraper Script with the Bing Search Cognitive Service resource we created in Azure follow the below steps.

Clone the repo, install packages, and open in VS Code

git clone https://github.com/cassieview/Use-ML-To-Classify-Yoga-Poses.git
cd Use-ML-To-Classify-Yoga-Poses
npm install
code .

Add the key into the app.js script javascript const subscriptionKey = 'addyourkeyhere'
Run the script node app.js

When the script completes you will have a new folder named poses with subfolders for each yoga pose specified in the search term collection. Within each pose folder there will be a test and train folder. The data is split at aproximately 70% to train and 30% to test.

Now that we have a dataset lets create our Custom Vision API Resource to train a model that will recognize yoga poses

Go to portal.azure.com
Select "Create Resource"
Search "Custom Vision"
Select "Create"
Complete the required fields
Select "Create"
When the deployment succeeds you will get a notification in the top right corner.
Select "Go to resource"

Create a Custom Vision Project and Train Model

Select "Quick Start"
Select "Custom Vision Portal"
Select 'Sign in"
Select "New Project"
- Enter a Name and Resource Group
- Project Types: Classificaion
- Classification Types: MultiClass General
- Domains: General
Click Create project
Now we need to add and tag images
- Select "Add Images" and navigate to the train folder for the scraped images
- Ctrl + A to select all the images in the folder
- Select "Open"
- Add tag for photos to indicate the yoga pose
- Select "Upload photos"
- Repeat these steps for each yoga pose since each yoga pose is a class in our classification model.
- Review each class to see if the images are correct. THere are some images that are returned that are either incorrect or not good examples. If you dont have good data you wont get good results. Review and update the data so that the classification data for each pose is a good, clear dataset.
Once all the data is uploaded and tagged select "Train"
They API will provide some stats on how your model is performing. Use this is an indication of how its preforming but not and absolute truth. Test your model with the data from the scarped images in the test folder. Its important to test your model on unseen data.
Steps to upload a test image and test your model.
- Select "Quick Test" from the upper right corner
- Browse to local files and select a image from the test folder
- The predication will appear on the right side of the window
Once you have tested your model and feel you have a good dataset. Its time to operationalize the selected model. This is super easy with the Azure endpoints. No need to deploy anything. In the next steps I will show you how to grab the URLs to integrate your model into your project.

Get the Prediction Url

Select "Performace" from the top navigation in the Custom Vision portal
Select "Prediction Url"
Copy the Url to either use an image url or image file
Additionally by selecting the settings icon in the right corner will bring you to the keys and urls to call the endpoint.

Consume the API in an App

Once the model is trained and at a satisfactory accuracy its ready to be used in your App

Go to the Resource Group that contains the Custom Vision Resource
There should be a resource named "YourCustomVisionResourceName_Prediction - Quick start"
The Select "Api Reference" from the list of helpful links
This will open a page with docs on how to call the api to get predictions from the model
If you scoll all the way to the bottom there are code samples in multiple languages to get you started

I love to test APIs with Postman. Here are steps so use postman

Get the predction url as described above
Select "Post" from the dropdown and paste in the url
Go to the settings and get the Prediction Key
In the headers add the Prediction-key as the key and the actual key as the value
Auth should be set to no auth
Select Body > Raw > Json
Paste in the following Json for the body of the request

{ "Url": "UrlForImageGoesHere" }

Send Request

This same logic can be applied to all sorts of image classification problems. If you have quesitons or comments please leave them below. Thanks!

Helpful Doc Links

Quickstart: Search for images with the Bing Image Search SDK for Node.js
Quickstart: Create an image classification project with the Custom Vision Node.js SDK
Cognitive Service Bing Search Docs
Cognitive Service Custom Vision Docs
Create a Node.js app in Azure App Service on Linux

DEV Community: Cassie Breviu

Inference machine learning models in the browser with JavaScript and ONNX Runtime Web

What is ORT and ORT-Web?

Resources

From No Data to Computer Vision Machine Learning Model in 10 Minutes or less!

Prerequisites

1. Create Azure Resources

Create Bing Search Api Azure Resource

Create Custom Vision Resource in Azure

2. Clone the Repo

3. Get the Data with Bing Search and the NodeJS Console App

4. Create the Custom Vision Model

Get Started with Power Apps Component Framework (PCF)

#PowerfulDevs Conference

#powerfuldevs Conference: Join us on July 15th Online!

JennyMEvents for Microsoft Azure ・ Jul 10 '20

About This Session:

About the Speakers:

Make a Custom Pipe Desk Mount 📐💻

Parts List for pipes

Mount List and other desk tech

Build the pipe mount

Get Free 3D Models in MS Paint 3D then Add it to BabylonJS Project

Add it to BabylonJS

Test localhost on Oculus Quest with Remote Debugging in Firefox

Install Firefox and Android Debug Bridge (adb)

Add Android Debug Bridge (adb) to Path

Run the project locally and connect to remote debugging

Reverse the Ports to make localhost available

ONNX: No, it's not a Pokemon! Deploy your ONNX model with C# and Azure Functions

Prerequisites

What is Open Neural Network Exchange (ONNX)?

Create the Model with Azure Machine Learning

1. Create a Cloud Compute Instance in the Azure ML Resource

2. Clone the Git repo with the Jupyter Notebook

3. Install the package

4. Run the code

5. Save Model to Azure Storage

Deploy Model with Azure Functions

1. Install VS Code Azure Function extension:

2. Use VS Code to create Azure function

3. Run the project to validate its working

4. Install the Nuget Packages

5. Update the Code

6. Update the storage parameters

7. Test the endpoint

8. Deploy the Endpoint to Azure

Play the Somm Wine Game here

Cheers to you!

Warm Up by the Fire with Bash (Fire Ascii Art)

Prerequisites for Windows

Setup Steps

Easily add Anaconda Prompt to Windows Terminal to make life better

Prerequisites

Add Anaconda Prompt

🤖 Grab Your Wine. It's Time to Demystify ML and NLP 🍷

Configure your ML dev environment

There are two options: cloud or local.

Import packages and data

Visualize the data

Create Calculated Columns for Labels

Process description text with the library SciKit Learn to create a Bag-of-Words using the CountVectorizer functionality.

Let's take a look at how to use the CountVectorizer.

Train the Model

Test the model

This is a correct prediction! 🎉

Other things to try

Other helpful links

Build a Web VR game with JavaScript using BabylonJS

In this tutorial I will show you step by step how to build a web VR game with BabylonJS

Prerequisites

What is BabylonJS and CannonJS

First we need to get the base starter project using babylonjs, webpack, and typescript

Steps to Run Starter Project and Git Repo Link

Lets talk about the starter project

Simple index.html template.

The index.ts typescript file

Start building the game

Project Architecture

Add the start button

Reverse the Ports to make `localhost` available

Remove `PhysicsImpostor` from ground mesh so balls fall through the ground instead of sitting on top.