DEV Community: Sammy Deprez

Azure Machine Learning – The Responsible Road

Sammy Deprez — Wed, 20 May 2020 06:26:18 +0000

Artificial Intelligence can change people's lives. It can help people see again, and it can help us to get faster diagnosed for deceases. It is helping me right now to write a blog post that reads easier and with fewer spelling mistakes. Companies use AI to predict their sales, to optimize their production. AI assists us wholly or partially in driving a car. These are just some examples of what it can do.

But ... Yes, there is always a but to every story. Data can be dangerous if used or interpreted in the wrong way. A very famous paradox is the 'Simpson Paradox.' In this paradox, it shows that data can tell the truth, but it is not the full truth.

Now imagine you are working for the Department of Corrections, and you need to build a model that predicts what the risk is for recidivism. Based on that, they could decide whether to grant people parole or not.

Or you work for a bank, and you build an algorithm that calculates if you will be able to pay back a loan or not. Based on the result, the bank will give you credit or not.

These projects can work great, but what if you cannot explain why Ms. Emma Smith cannot get a loan. Or why is Mr. Kalifa Abara is a higher risk for recidivism? And should you, as a data scientist, have access to all these very confidential data. (*)

Last year's Build (2019) was all about AI for all skill levels. With many new features within the cognitive services and automated ML.\
Then we had Ignite (2019), where the focus was on bringing AI into production with MLOps. By converting the techniques of software development cycles into the machine learning industry.

Today Build 2020 has started, and Microsoft put their focus now on "Responsible ML." Within this topic, there are three different pillars.

Understand

"Why did Mr. Carl Jones not get a loan?"\
It has happened many times customers leaves a bank office devastated, because they did not get a loan, and the banker could not give a reason.

For this, Microsoft is working on three projects, namely:

Interpret-Text: a library that incorporates explainers for text-based machine learning models and visualizes the results with a built-in dashboard

FairLearn: a python package that implements a variety of algorithms that mitigate unfairness in supervised machine learning

DiCE (Diverse Counterfactual Explanations): a library that generates "what-if" explanations for a model output

Protect

How do we make the difference in our datasets between general information and private information, and how can you protect that part of your data.

Or what if you don't want your data scientist even to see the data. Or how can we protect data while it is getting processed?

To fix these problems, Microsoft revealed some other great tools:

Differential Privacy Whitenoise (Core & System): Differential Privacy validator and runtime + tools and services for DP processing of tabular and relational data

Seal SDK: An easy-to-use and powerful homomorphic encryption library, or in other words how can you do mathematical calculations on encrypted data

Open Enclave: New Azure VMs, where data can stay encrypted while getting processed, by making use of specialized hardware.

Control

How do we explain that a new iteration of training your Machine Learning model gives suddenly different results?

Azure Machine Learning Audit Trail enables you to automatically track your experiments and datasets that correspond to your registered ML Model.

Stay tuned for more details.

The above items are a lot of excellent new features that will get built-in within AML; some soon, some might still take some time.

Myself and three other Microsoft AI MVP's were asked by Microsoft to have a look at it. And on Thursday (21th of May 2020) we will kick-off with a live stream on Twitch (hosted by Henk Boelman and Tess Ferrandez)

If you want to hear more details about these topics, then join us on Twitch:

http://twitch.com/microsoftdeveloper

Each four of us will go more in-depth into one of the features.

Eve Pardi (@EvePardi) will talk about Interpret-Text. You can read her intro blog [here]
Willem Meints (@Willem_Meints) is going to discuss FairLearn. You can find his intro blog [here]
Alicia Moniz (@AliciaMoniz) looked into Differential Privacy. Her intro blog can be found [here]

And me myself, I will take you into the magical world of Confidential ML with Microsoft Seal and OpenEnclave.

Stay tuned!

Initially posted on: https://www.datafish.eu/article/azure-machine-learning-the-responsible-road/

*: The names used in this article are random, not based on any real person.

Integrating Qna Maker Into The Power Virtual Agent

Sammy Deprez — Wed, 22 Jan 2020 15:48:53 +0000

In another post I introduced the Power Virtual Agent. Which is a nice tool for business users to build simple chat bots. One big feature that it is missing is standard integration with Qna Maker.But since we can call Power Automate Flows, there is a work around. Let’s have a look how.

Initially posted on datafish.eu

Data Gathering For Image Recognition

Sammy Deprez — Tue, 07 Jan 2020 22:24:46 +0000

The other time I was reading an interesting blog from Henk Boelman (a Microsoft Advocate). He was describing how you can build an image classifier with Azure Machine Learning Studio. He used a dataset that he had downloaded from Kaggle. But what if you cant find on Kaggle what you need. This blog post will discuss how you can make use of the Bing Image Search to generate your own dataset.

So first of all some explanation. "Bing Image Search API" is part of the Cognitive Services on Azure and it gives you the possibility to search for images just like how you would do on bing.com/images. The API has a search function that allows you to search for a specific keyword but also filter the results based on size (height/width), file size, license, color, ...

More info and free easy try out can be found on the product page of Microsoft.\
https://azure.microsoft.com/en-us/services/cognitive-services/bing-image-search-api/

Since we are talking about Machine Learning all the code that will be discussed will be in Python. That way you can just copy it in your Jupiter/Azure Notebook.

Step 1: Install/Import necessary packages

The Bing Image Search has its own Python SDK, what is very handy. But if you want you can also make use of the classic HTTP request methods.

!pip install azure-cognitiveservices-search-imagesearch

from azure.cognitiveservices.search.imagesearch import ImageSearchClient
from msrest.authentication import CognitiveServicesCredentials
import pandas as pd

Step 2: Configure subscription key + endpoint

Subscription Key + Endpoint can be found in your Azure Portal after you create a Cognitive Service resource. You don't need to look for a specific Bing Search Cognitive Services. Since a while Bing Search and many more cognitive services have been implemented in 1 resource. Which means 1 endpoint and 1 key to use them all.

I also added the search term here

subscription_key = "[YourKeyHere]"

subscription_endpoint = "[YourEndpointHere]"
search_term = "apples"

Step 3: Configure Client

So in this step we create 2 new objects. One is the 'credentials' which contains the subscription key that we just configured. And the other is the client that we will use to make the call. Last mentioned also needs the endpoint.

credentials = CognitiveServicesCredentials(subscription_key)

client = ImageSearchClient(endpoint=subscription_endpoint, credentials=credentials)

Step 4: Search and gather the results

Searching for items is very easy. Just by making use of the client object and the search function that has a bunch of variables that you can configure (like color, size, ...) In this case we only configure what we are search for and how many results we want to get back.

Search for images

image_results = client.images.search(query=search_term,count=150)

Step 5: Convert to dataframe

The result we receive from the API is a list of objects. This might we handy if we are in an c# application or so. But in this case we want to work with clear data. So with below functionality we convert the data to a dataframe. You will notice we get quite some information back. Some information is still stored in an object like "_type", but in this case we don't need this data. So we will leave it like this.

convert to dataframe

df = pd.DataFrame([x.as_dict() for x in image_results.value])

df.head()

	_type	accent_color	content_size	content_url	date_published	encoding_format	height	host_page_display_url	host_page_url	image_id	image_insights_token	insights_metadata	name	thumbnail	thumbnail_url	web_search_url	width
0	ImageObject	9D2E39	224643 B	https://upload.wikimedia.org/wikipedia/commons...	2019-09-14T23:40:00.0000000Z	jpeg	1200	https://en.wikipedia.org/wiki/Apple	https://en.wikipedia.org/wiki/Apple	6AEAF1C3894ED7D563916634549ED00175C970D3	ccid_pfp3ysAm*mid_6AEAF1C3894ED7D563916634549E...	{}	Apple -- Wikipedia	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse3.mm.bing.net/th?id=OIP.pfp3ysAmXA6...	https://www.bing.com/images/search?view=detail...	1200
1	ImageObject	4F300B	161831 B	https://upload.wikimedia.org/wikipedia/commons...	2019-12-21T20:48:00.0000000Z	jpeg	1083	https://en.wikipedia.org/wiki/Honeycrisp	https://en.wikipedia.org/wiki/Honeycrisp	4531DA369849BECBA2980492B1F021A9EF15BB99	ccid_wPWDNJmR*mid_4531DA369849BECBA2980492B1F0...	{}	Honeycrisp -- Wikipedia	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse1.mm.bing.net/th?id=OIP.wPWDNJmRmNP...	https://www.bing.com/images/search?view=detail...	1200
2	ImageObject	C47807	172706 B	https://www.tasteofhome.com/wp-content/uploads...	2018-08-15T00:12:00.0000000Z	jpeg	1200	https://www.tasteofhome.com/collection/new-typ...	https://www.tasteofhome.com/collection/new-typ...	EC1EBFE716FD1033C0E0983931FEC881E8A1177D	ccid_7jSg14s4*mid_EC1EBFE716FD1033C0E0983931FE...	{}	15 New Types of Apples You Should Be Buying	...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse2.mm.bing.net/th?id=OIP.7jSg14s46gp...	https://www.bing.com/images/search?view=detail...
3	ImageObject	7E0F24	4462721 B	http://www.michiganapples.com/portals/0/MAC%20...	2019-11-15T14:57:00.0000000Z	jpeg	3738	www.michiganapples.com/About/Varieties	http://www.michiganapples.com/About/Varieties	753A53E3C6A184B4C55A074E275F120D20D4914C	ccid_3X+TRdGR*mid_753A53E3C6A184B4C55A074E275F...	{}	Michigan Apple Varieties	Michigan Apple Comm...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse2.mm.bing.net/th?id=OIP.3X-TRdGReOQ...	https://www.bing.com/images/search?view=detail...
4	ImageObject	4F1E0F	419131 B	http://www.flinchbaughsorchard.com/wp-content/...	2019-11-01T04:51:00.0000000Z	jpeg	1691	www.flinchbaughsorchard.com/apple-varieties	http://www.flinchbaughsorchard.com/apple-varie...	7480E3C7DDE4776D891D003001872D954786935C	ccid_GBWUKwPQ*mid_7480E3C7DDE4776D891D00300187...	{}	Apple Varieties	Flinchbaugh's Orchard & Farm...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse1.mm.bing.net/th?id=OIP.GBWUKwPQyNo...	https://www.bing.com/images/search?view=detail...

Step 6: Generate new filenames

Since we get data back from many different websites, there is a chance that some filenames might be totally the same. Plus some files might not have an extension within the URI, because of routing or image generation. To fix this we add a function that guesses the MimeType and based on that looks up the extension that belongs to it. This in combination with a GUID that is generated we are sure we have a unique filename.

import mimetypes

import uuid

def getFileName(contentUrl):
mt = mimetypes.guess_type(contentUrl)
if mt[0] != None :
ext = mimetypes.guess_extension(mt[0])
return str(uuid.uuid1()) + ext
else:
return ""

By applying above funtion to each row we can add an extra column to the dataframe that contains the newly generated fileName. NOTE: you might have noticed that the function can return an emtpy string. This is only the case when it can't figure out the MimeType. Those results we filter out.

df['fileName'] = df.apply(lambda x: getFileName(x.content_url), axis=1)

df = df[df['fileName'] != ""]

df.head()

	_type	accent_color	content_size	content_url	date_published	encoding_format	height	host_page_display_url	host_page_url	image_id	image_insights_token	insights_metadata	name	thumbnail	thumbnail_url	web_search_url	width	fileName
0	ImageObject	9D2E39	224643 B	https://upload.wikimedia.org/wikipedia/commons...	2019-09-14T23:40:00.0000000Z	jpeg	1200	https://en.wikipedia.org/wiki/Apple	https://en.wikipedia.org/wiki/Apple	6AEAF1C3894ED7D563916634549ED00175C970D3	ccid_pfp3ysAm*mid_6AEAF1C3894ED7D563916634549E...	{}	Apple -- Wikipedia	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse3.mm.bing.net/th?id=OIP.pfp3ysAmXA6...	https://www.bing.com/images/search?view=detail...	1200	11c6ddea-3197-11ea-87d5-000d3aaa7d6e.jpe
1	ImageObject	4F300B	161831 B	https://upload.wikimedia.org/wikipedia/commons...	2019-12-21T20:48:00.0000000Z	jpeg	1083	https://en.wikipedia.org/wiki/Honeycrisp	https://en.wikipedia.org/wiki/Honeycrisp	4531DA369849BECBA2980492B1F021A9EF15BB99	ccid_wPWDNJmR*mid_4531DA369849BECBA2980492B1F0...	{}	Honeycrisp -- Wikipedia	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse1.mm.bing.net/th?id=OIP.wPWDNJmRmNP...	https://www.bing.com/images/search?view=detail...	1200	11c6e2ea-3197-11ea-87d5-000d3aaa7d6e.jpe
2	ImageObject	C47807	172706 B	https://www.tasteofhome.com/wp-content/uploads...	2018-08-15T00:12:00.0000000Z	jpeg	1200	https://www.tasteofhome.com/collection/new-typ...	https://www.tasteofhome.com/collection/new-typ...	EC1EBFE716FD1033C0E0983931FEC881E8A1177D	ccid_7jSg14s4*mid_EC1EBFE716FD1033C0E0983931FE...	{}	15 New Types of Apples You Should Be Buying	...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse2.mm.bing.net/th?id=OIP.7jSg14s46gp...	https://www.bing.com/images/search?view=detail...	1200
3	ImageObject	7E0F24	4462721 B	http://www.michiganapples.com/portals/0/MAC%20...	2019-11-15T14:57:00.0000000Z	jpeg	3738	www.michiganapples.com/About/Varieties	http://www.michiganapples.com/About/Varieties	753A53E3C6A184B4C55A074E275F120D20D4914C	ccid_3X+TRdGR*mid_753A53E3C6A184B4C55A074E275F...	{}	Michigan Apple Varieties	Michigan Apple Comm...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse2.mm.bing.net/th?id=OIP.3X-TRdGReOQ...	https://www.bing.com/images/search?view=detail...	3738
4	ImageObject	4F1E0F	419131 B	http://www.flinchbaughsorchard.com/wp-content/...	2019-11-01T04:51:00.0000000Z	jpeg	1691	www.flinchbaughsorchard.com/apple-varieties	http://www.flinchbaughsorchard.com/apple-varie...	7480E3C7DDE4776D891D003001872D954786935C	ccid_GBWUKwPQ*mid_7480E3C7DDE4776D891D00300187...	{}	Apple Varieties	Flinchbaugh's Orchard & Farm...	{'_type': 'ImageObject', 'width': 474, 'height...	https://tse1.mm.bing.net/th?id=OIP.GBWUKwPQyNo...	https://www.bing.com/images/search?view=detail...	1800

Step 7: Create folder to save the images

This is a folder on your local or cloud environment.

dirName = "image_gathering"

os.mkdir(dirName)

Requirement already satisfied: wget in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (3.2)
Directory image_gathering/apples already exists. Folder will be cleared

Step 8: Download images

First we create a function that tries to download the file to the new destination with the new filename. Bing search might be outdated, so there are possibilities that certain downloads fails (Ex. NotFound, NotAuthenticated, ...) Therefore the function returns True or False depending if download was succesful.

import wget

def downloadFile(dirName, contentUrl, fileName):
try:
wget.download(contentUrl, dirName + "/" + fileName)
return True
except:
return False

By applying this function do every row, every result will be downloaded and we add an extra column that keeps track if it was successfull or not.

df['fileDownloaded'] = df.apply(lambda x: downloadFile(dirName, x.content_url, x.fileName), axis=1)

THE END

Its not that hard and quite fast to gather your own data based on results from Bing Search. Depending on what you want to realize you will need to run this script multiple times if you want images for different search terms. Or you can convert this script into a function that accepts an array of search strings.

Extra -- Move data to datastore

In case you are using Azure Machine Learning Studio, its a good idea to move your images to a datastore. That way other data scientists can also make use of it. Below code uploads the folder of images to the same folder on your datastore.

import azureml.core

from azureml.core import Workspace, Datastore

ws = Workspace.from_config()

datastore_name = 'workspaceblobstore'
datastore = Datastore.get(ws, datastore_name)
datastore.upload(dirName, dirName)

The full Jupyter Notebook can be found on my GitHub

https://github.com/sammydeprez/PythonExamples/blob/master/BingImageSearch/BingImageSearch.ipynb

Enjoy gathering new data!

Originally posted on https://www.datafish.eu/article/data-gathering-for-image-recognition/

Alterations - A hidden gem in QnaMaker

Sammy Deprez — Mon, 16 Dec 2019 22:22:07 +0000

Qna Maker is a great and easy tool. It can help you to build, in a matter of minutes, a mini chatbot. A chatbot that will replace your boring FAQ page.

Since the last releases, a lot of cool features have been added.

Active Learning -- a built in function to let your users help you with suggesting the correct answer to their question.
Multi Turn Questions -- in some cases you will need to ask the user multiple questions to get to the correct answers. Ex. 'I am looking for a school.' Then you might ask the user first: 'What school are you looking for?' with some suggestions like 'Music School', 'Primary School', ... With multi turn this is a possibility without any extra coding in your bot.
Publish bot -- before this option you had to built your own chatbot with the SDK that calls the Qna Maker service. Now you can create a new chatbot within the Qna Maker Portal.

But there is one feature that was there from the beginning. But it is hidden within the Qna Maker API.

Alterations - less training with synonyms

Alterations can be compared to synonyms. Its a list of words that mean the same thing. For example the word 'gift' can be swapped by the word 'present'. Or you might even use it for abbreviations. 'GDPR' for example is a widely used term, but some people might call it AVG (Dutch term)

But why would I call this a hidden gem? First of all I am not a native English speaker and I don't live in an English speaking country. So I built bots in other languages than English. Why is this an important fact? Well the Qna Maker has been pre-trained with a lot of synonyms in English and some other big languages. And you guessed well, its not done in the language I need (not at the moment at least).

When you add a list of well chosen alterations, you will make your QnA Maker smarter in a short time. Since you need to add less alternative questions in your knowledge base.

Enough talking. I said it's a hidden gem. Because you can do this within the Qna Maker portal. You need to use the API for it.

How its done!

Lets start by open your favorite REST API Client (Ex. Postman)

So the API has 2 function for the alterations.

The first one is getting a list of all the alterations that have been stored. The second one is to override your current list.

Get current alterations

Call the REST API

GET {Endpoint}/qnamaker/v4.0/alterations

Endpoint is for example https://westus.api.cognitive.microsoft.com

With in the headers of your request the Ocp-Apim-Subscription-Key which is equal to the Key you can find in the Azure Portal.

In Postman it would look like this:

If you have never used alterations then the first time you execute this, you will receive an empty array:

{"wordAlterations": []}

Add alterations

Now let us add some alterations. To do this we need to call the REST API

PUT {Endpoint}/qnamaker/v4.0/alterations

At this point Endpoint is the same as to get the alterations. And we also need to add the Ocp-Apim-Subscription-Key to the header.

On top of that we need to add some alterations too. So in the body of our request we can add something like this:

{

"wordAlterations": [

{

"alterations": ["auto","wagen"]

{

"alterations": ["gebruiker","medewerker"]

{

"alterations": ["wachtwoord","paswoord"]

}

]

}

So the structure exists out of the 'wordAlterations' object that is an array of different 'alterations'. An 'alterations' object again is an array of a list of words that are synonyms.

Now read this carefully.

This function replaces all your alterations that were already stored in your Qna Maker service
Alterations are added for all the knowledge bases within your QnA Maker service.

Conclusion

Sadly enough alterations are not built in into the Qna Maker portal. But do not forget them while you are building a knowledge base. They can really help you to improve the quality of your knowledge base and will save you time in training it. So less training with synonyms.

More detailed information about the alteration function can be found here:

https://docs.microsoft.com/en-us/rest/api/cognitiveservices/qnamaker/alterations

Original posted at: https://datafish.eu/article/alterations-a-hidden-gem-in-qnamaker/