Helping Santa with the power of Azure AI

This article is part of the Festive Tech Calendar initiative by Gregor Suttie and Richard Hooper. You'll find other helpful articles and tutorials published daily by community members and experts there, so make sure to check it out every day.

In this post, I take last year's Xamarin Santa Talk Challenge as the base to improve a mobile application that relies on Azure Cognitive Services – Text Analytics and Azure Functions to determine if someone’s getting a gift.

As you can see, this application analyzes a letter (text) and determines its sentiment, i.e. a value between 0 and 1 that states if the content is positive (the score is closer to 1) or negative (score is closer to 0). An Azure Function connects the mobile app with the AI services to send the message and receive a response in return.

We start with the initial code shared for the challenge and then perform the following changes in order to add another AI capability: Computer Vision to automatically read the letter (using OCR, Optical Character Recognition).

*Phase 1 - Creating an Azure resource *

Step 1. Open the Azure Portal and create a Computer Vision resource.

Alternatively, you can also create a Cognitive Services resource.

Step 2. From the Keys and Endpoint section, copy the Key 1 and Endpoint.

Phase 2 - Modifying the Functions project

Step 1. Open the SantaTalk.Functions project and add the Microsoft.Azure.CognitiveServices.Vision.ComputerVision Nuget package.

Step 2. As indicated here with the Text Analytics service, In the local.settings.json file, add two variables: ComputerVisionAPIKey and ComputerVisionAPIEndpoint with the corresponding values obtained from Step 2 in the previous phase.

{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "APIKey": "YOUR KEY WILL GO HERE",
    "APIEndpoint": "https://westus2.api.cognitive.microsoft.com/",
    "ComputerVisionAPIKey": "YOUR COMPUTER VISION KEY WILL GO HERE",
    "ComputerVisionAPIEndpoint": "https://aiinadaycognitive.cognitiveservices.azure.com/"
  }
}

Step 3. Add a new class to the project: ScanSanta, with the following code which provides a ScanSanta function that receives an image (the letter) and uses the RecognizeTextInStreamAsync method from the ComputerVisionClient class in order to detect the text from the letter. The auxiliary, private GetTextAsync method is used to poll for results by calling the GetTextOperationResultAsync method (also from ComputerVisionClient).

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using System.Text;

namespace SantaTalk.Functions
{
    public static class ScanSanta
    {
        static ComputerVisionClient visionClient;
        private const int numberOfCharsInOperationId = 36;

        static ScanSanta()
        {
            var keys = new ApiKeyServiceClientCredentials(Environment.GetEnvironmentVariable("ComputerVisionAPIKey"));

            visionClient = new ComputerVisionClient(keys) { Endpoint = Environment.GetEnvironmentVariable("ComputerVisionAPIEndpoint") };
        }

        [FunctionName("ScanSanta")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] Stream image,
            ILogger log)
        {
            var mode = TextRecognitionMode.Handwritten;
            var text = string.Empty;

            try
            {
                var result = await visionClient.RecognizeTextInStreamAsync(image, mode);
                text = await GetTextAsync(result.OperationLocation);
            }
            catch (Exception ex)
            {
                log.LogError(ex.ToString());

                return new StatusCodeResult(StatusCodes.Status500InternalServerError);
            }

            return new OkObjectResult(text);
        }

        private static async Task<string> GetTextAsync(string operationLocation)
        {
            var operationId = operationLocation.Substring(
                operationLocation.Length - numberOfCharsInOperationId);

            var result = await visionClient.GetTextOperationResultAsync(operationId);

            int i = 0;
            int maxRetries = 10;

            while ((result.Status == TextOperationStatusCodes.Running ||
                    result.Status == TextOperationStatusCodes.NotStarted) && i++ < maxRetries)
            {
                await Task.Delay(1000);
                result = await visionClient.GetTextOperationResultAsync(operationId);
            }

            var sb = new StringBuilder();

            foreach (var line in result.RecognitionResult.Lines)
            {
                foreach (var word in line.Words)
                {
                    sb.Append(word.Text);
                    sb.Append(" ");
                }

                sb.Append("\r\n");
            }

            return sb.ToString();
        }
    }
}

Phase 3 - Modifying the mobile app

Step 1. Add the Xam.Plugin.Media Nuget and Xamarin.FFImageLoading.Transformations packages in all the projects of the solution

Now you have to configure the Android project. Everything that is mentioned from Steps 2-6 happens in the SantaTalk.Android project:

Step 2. Add a MainApplication class. The code goes as follows, and is used to interact with the current activity:

using System;

using Android.App;
using Android.Runtime;

using Plugin.CurrentActivity;

namespace SantaTalk.Droid
{
#if DEBUG
    [Application(Debuggable = true)]
#else
    [Application(Debuggable = false)]
#endif
    public class MainApplication : Application
    {
        public MainApplication(IntPtr handle, JniHandleOwnership transer)
            : base(handle, transer)
        {
        }

        public override void OnCreate()
        {
            base.OnCreate();
            CrossCurrentActivity.Current.Init(this);
        }
    }
}

Step 3. Add the following code to the AndroidManifest.xml inside the tags:

<provider android:name="android.support.v4.content.FileProvider" 
          android:authorities="${applicationId}.fileprovider" 
          android:exported="false" 
          android:grantUriPermissions="true">

      <meta-data android:name="android.support.FILE_PROVIDER_PATHS" 
                     android:resource="@xml/file_paths"></meta-data>
</provider>

Step 4. Add the following permissions to the same file (use the GUI):

• Camera
• Read External Storage
• Write External Storage

Step 5. Create an xml folder under Resources, there, create a file_paths.xml file with the following content:

<?xml version="1.0" encoding="utf-8"?>
<paths xmlns:android="http://schemas.android.com/apk/res/android">
  <external-files-path name="my_images" path="Pictures" />
  <external-files-path name="my_movies" path="Movies" />
</paths>

Step 6. Add the camera and photo images to the drawable folders (under Resources). You can download them from here

Now it's turn to set up the SantaTalk.iOS project (Steps 7 and 8)

Step 7. Add the following keys and strings to the Info.plist file:

  <key>NSCameraUsageDescription</key>
  <string>This app needs access to the camera to take photos.</string>
  <key>NSPhotoLibraryUsageDescription</key>
  <string>This app needs access to photos.</string>
  <key>NSMicrophoneUsageDescription</key>
  <string>This app needs access to microphone.</string>
  <key>NSPhotoLibraryAddUsageDescription</key>
  <string>This app needs access to the photo gallery.</string>

Step 8. Add the camera and photo images to the Resources folder. You can download them from https://github.com/icebeam7/santa-talk/tree/master/src/SantaTalk.iOS/Resources. Include tthe 2x and 3x versions.

The final steps modify the SantaTalk project.

Step 10. Inside the Services folder, create a Base folder that contains a new class BaseService that interacts with the two Functions (WriteSanta and ScanSanta). It is based on the LetterDeliveryService that already exists in the project.

using System;
using System.Net.Http;

using Xamarin.Essentials;

namespace SantaTalk.Services.Base
{
    public class BaseService
    {
        //string santaUrl = "{REPLACE WITH YOUR FUNCTION URL}/api/";

        protected string santaUrl = "http://localhost:7071/api/";
        protected static HttpClient httpClient = new HttpClient();

        public BaseService()
        {
            // if we're on the Android emulator, running functions locally, need to swap out the function url
            if (santaUrl.Contains("localhost") && DeviceInfo.DeviceType == DeviceType.Virtual && DeviceInfo.Platform == DevicePlatform.Android)
            {
                santaUrl = santaUrl.Replace("localhost", "10.5.132.243");
            }

            httpClient.BaseAddress = new Uri(santaUrl);
        }
    }
}

Step 11. Now, actually, modify the LetterDeliveryService. A lot of its code has already been implemented in the BaseService, so it is easier to simply replace it with the following code. It does exactly the same it was doing before (call the WriteSanta Function), but now inheriting the functionality from BaseService:

using System;
using System.Net.Http;
using System.Threading.Tasks;

using Newtonsoft.Json;

using SantaTalk.Models;
using SantaTalk.Services.Base;

namespace SantaTalk
{
    public class LetterDeliveryService : BaseService
    {
        public async Task<SantaResults> WriteLetterToSanta(SantaLetter letter)
        {
            SantaResults results = null;
            try
            {
                var letterJson = JsonConvert.SerializeObject(letter);

                var httpResponse = await httpClient.PostAsync("WriteSanta", new StringContent(letterJson));

                results = JsonConvert.DeserializeObject<SantaResults>(await httpResponse.Content.ReadAsStringAsync());
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine(ex);
            }

            if (results == null)
                results = new SantaResults { SentimentScore = -1 };

            return results;
        }
    }
}

Step 12. In the Services folder, create a LetterScanService class, that is similar to the previous one and will call the ScanSanta Function service

using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;

using SantaTalk.Services.Base;

namespace SantaTalk
{
    public class LetterScanService : BaseService
    {
        public async Task<string> ScanLetterForSanta(Stream image)
        {
            string results = string.Empty;
            try
            {
                var httpResponse = await httpClient.PostAsync("ScanSanta", new StreamContent(image));

                results = await httpResponse.Content.ReadAsStringAsync();
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine(ex);
            }

            if (string.IsNullOrWhiteSpace(results))
                results = "The letter is illegible";

            return results;
        }
    }
}

Step 13. In the same folder, create a PhotoService class that uses the MediaPlugin package that was installed from the very beginning in order to allow the user to either select a photo from the device or take a new one with the camera:

using System;
using System.Threading.Tasks;

using Plugin.Media;
using Plugin.Media.Abstractions;

using Plugin.Permissions;
using Plugin.Permissions.Abstractions;

namespace SantaTalk
{
    public class PhotoService
    {
        private PermissionStatus cameraOK;
        private PermissionStatus storageOK;

        private async Task Init()
        {
            await CrossMedia.Current.Initialize();

            cameraOK = await CrossPermissions.Current.CheckPermissionStatusAsync(Permission.Camera);
            storageOK = await CrossPermissions.Current.CheckPermissionStatusAsync(Permission.Storage);

            if (cameraOK != PermissionStatus.Granted || storageOK != PermissionStatus.Granted)
            {
                var status = await CrossPermissions.Current.RequestPermissionsAsync(new[] { Permission.Camera, Permission.Storage });
                cameraOK = status[Permission.Camera];
                storageOK = status[Permission.Storage];
            }
        }

        public async Task<MediaFile> TakePhoto()
        {
            await Init();

            MediaFile file = null;

            if (cameraOK == PermissionStatus.Granted
                && storageOK == PermissionStatus.Granted
                && CrossMedia.Current.IsCameraAvailable
                && CrossMedia.Current.IsTakePhotoSupported)
            {
                var options = new StoreCameraMediaOptions()
                {
                    Directory = "SantaTalk",
                    Name = $"{Guid.NewGuid()}.jpg",
                    SaveToAlbum = true
                };

                file = await CrossMedia.Current.TakePhotoAsync(options);
            }

            return file;
        }

        public async Task<MediaFile> ChoosePhoto()
        {
            MediaFile file = null;

            if (CrossMedia.Current.IsPickPhotoSupported)
                file = await CrossMedia.Current.PickPhotoAsync();

            return file;
        }
    }
}

Step 14. Open the MainPageViewModel class (under ViewModels folder) and...

A. Add the System.Threading.Tasks namespace:

using System.Threading.Tasks;

B. Create an ScanLetterCommand ICommand:

public ICommand ScanLetterCommand { get; }

C. Create the ScanLetterForSanta method:

private async Task ScanLetterForSanta(bool useCamera)
{
    var photoService = new PhotoService();

    var photo = useCamera ? await photoService.TakePhoto() : await photoService.ChoosePhoto();

    var scanService = new LetterScanService();
    var scannedLetter = await scanService.ScanLetterForSanta(photo.GetStream());

    LetterText = scannedLetter;
}

D. In the constructor, create an instance of ScanLetterCommand:

ScanLetterCommand = new Command<bool>(async (useCamera) =>
{
    await ScanLetterForSanta(useCamera);
});

Step 15. Finally, in the MainPage.xaml file, add the following code:

A. The FFImageLoading.Forms namespace (top section):

    xmlns:ffimageloading="clr-namespace:FFImageLoading.Forms;assembly=FFImageLoading.Forms"

B. A couple of boolean resources (below the BindingContext):

    <ContentPage.Resources>
        <x:Boolean x:Key="FalseValue">False</x:Boolean>
        <x:Boolean x:Key="TrueValue">True</x:Boolean>
    </ContentPage.Resources>

C. Replace the Label control that contains the text "WRITE YOUR LETTER TO SANTA" with the following controls that allows the user to use the camera or select an image from the device to provide a picture of the letter:

<StackLayout Orientation="Horizontal" Spacing="8">
    <Label Text="WRITE YOUR LETTER TO SANTA" FontSize="12" VerticalOptions="Center"/>

    <ffimageloading:CachedImage Source="camera.png" Aspect="AspectFill" DownsampleToViewSize="True">
        <ffimageloading:CachedImage.GestureRecognizers>
            <TapGestureRecognizer Command="{Binding ScanLetterCommand}" CommandParameter="{StaticResource TrueValue}" />
        </ffimageloading:CachedImage.GestureRecognizers>
    </ffimageloading:CachedImage>

    <ffimageloading:CachedImage Source="photo.png" Aspect="AspectFill" DownsampleToViewSize="True">
        <ffimageloading:CachedImage.GestureRecognizers>
            <TapGestureRecognizer Command="{Binding ScanLetterCommand}" CommandParameter="{StaticResource FalseValue}" />
        </ffimageloading:CachedImage.GestureRecognizers>
    </ffimageloading:CachedImage>
</StackLayout>

Compile the app and execute it (the Functions must also be running locally, or you can publish them to Azure). If everything goes as planned, this is the expected output:

Full code is available on my GitHub repository

As you can see, you can now select a photo of a letter and the app will scan it using AI! The text will be entered, and by clicking on the button, the sentiment will be calculated!

Azure Cognitive Services is amazing! If you want to learn more, this is a good starting point.

I hope that this entry was interesting and useful for you. I invite you to visit my blog for more technical posts about Xamarin, Azure, and the .NET ecosystem. I write in Spanish language =)

Thanks for your time, and enjoy the rest of the Festive Tech Calendar publications!

See you next time,
Luis