Dialog Flow, Google Home & an Internet Connected Swimming Pool - A Deep Dive Into Building a Pool Bot

In this series we have covered arduino hardware & sensors, connecting a swimming pool to the internet, saving events to Azure Table Storage and building a PWA using Vue.

Now let's talk about interfacing the pool with google assistant!

Dialog Flow

If you want to build Google Assistant integrations, you could use Actions on Google directly, or you could check out Dialog Flow.

Google has made it simple to write conversational experiences and handle the machine learning for you. We'll just define some intents and utterances.

Intents

What is the user trying to accomplish? What is their intent? For my case, I want to know the temperature in the pool. So I created an Intent called "Water Temperature".

Utterances

Next we need to defined a few training phrases. These are the things we think users will say to trigger our intent.

Responses

When a user triggers this Intent by matching an utterance, we need to send a response. Dialog Flow allows us to hard code responses, but we want real time data from the pool bot! For that we'll enable Fulfilment. We can still define a hard coded list, this will be useful if our backend service is down or being slow.

Fulfillment

Switching over to the fulfillment tab, We'll add some webhook settings. You can dive into the docs for the full details, but basically Dialog Flow is going to POST a json object to us, and we will need to send back the response shape it's expecting.

Azure Function Webhook

Earlier in this series we talked about Azure Functions, now we'll add one more endpoint to the backend services.

Google offers Dialog Flow client libraries for several different languages. We'll grab the C# Nuget:

nuget install Google.Cloud.Dialogflow.V2

This gives us the request and response objects we'll need to deal with. Let's build a function that takes in the Dialog Flow WebhookRequest, a reference to the pool bot CloudTable data and we'll end up returning a Dialog Flow WebhookResponse object.

using Google.Cloud.Dialogflow.V2;
using Microsoft.WindowsAzure.Storage.Table;
using PoolBot.Data.Storage;

public static class AiFulfillment
{
    public static async Task<WebhookResponse> HandleQuery(CloudTable dataTable, WebhookRequest request)
    {
        // Get latest data from cloud storage
        var sensorData = await TableStorageRepo.GetLatestSensorData(dataTable);

        // Build friendly messages to send back to Dialog Flow
        var textMsg = $"Pool temperature is {sensorData.IntakeTemp:#}°";
        var speech = $"The pool temperature is {sensorData.IntakeTemp:#}°";
        var solar = sensorData.ReturnTemp - sensorData.IntakeTemp;
        if(solar > 1)
        {
            speech += " and the solar panels are on";
        }
        var response = new WebhookResponse
        {
            FulfillmentText = speech,
            FulfillmentMessages =
            {                  
                new Intent.Types.Message
                {
                    SimpleResponses = new Intent.Types.Message.Types.SimpleResponses
                    {
                        SimpleResponses_ =
                        {
                            new Intent.Types.Message.Types.SimpleResponse
                            {
                                DisplayText = textMsg,
                                TextToSpeech = speech
                            }
                        }
                    }
                },
                new Intent.Types.Message
                {
                    BasicCard = new Intent.Types.Message.Types.BasicCard
                    {
                        Title =  $"{sensorData.IntakeTemp:#}°",
                        Subtitle =  solar > 1 ? $"+{solar:#.#}°" : "",
                        Image = new Intent.Types.Message.Types.Image
                        {
                            ImageUri = "https://poolbot.azurewebsites.net/img/water.gif"
                        }
                    }
                }
            }
        };

        return response;
    }


}

This function makes two messages and stuffs them in a WebhookResponse object. Why are we setting two objects into FulfillmentMessages? This allows us to do different things on different devices! If the user is talking to a Google Home speaker, they won't have a screen and will hear the SimpleResponse TextToSpeech read to them. If they have a Google Hub with a screen (or are using Assistant on their phone) Dialog Flow will use the BasicCard message.

Note About Google Object Serialization! For some dumb reason Google's object don't serialize correctly. If you just sent back their WebhookResponse object, you'd get an error from Dialog Flow. You need to .ToString() the response before sending it back, but keep in mind it also needs to be a json object, not a string of json. So this may look hacky, but you have to ToString, then Deserialize the string back into a object (the correct object this time!). Then we can send the response:

[FunctionName("AI_Fulfillment")]
public static async Task<HttpResponseMessage> Fulfillment(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "AI/Fulfillment")] HttpRequestMessage req,
    [Table("Data", Connection = "AzureWebJobsStorage")] CloudTable dataTable,
    ILogger log)
{
    // Read the request into a Dialog Flow Request object
    var q = await req.Content.ReadAsAsync<WebhookRequest>();
    // Build a response to the incoming query
    var result = await AiFulfillment.HandleQuery(dataTable, q);
    // Use ToString to build the correct JSON needed for the response
    // Deserialize the JSON string back into an object and send it
    return req.CreateResponse(Newtonsoft.Json.JsonConvert.DeserializeObject(result.ToString()));
}

Here is Demo Video of this running on my Google Hub.