arpitsrm

Posted on Nov 8, 2020 • Edited on Nov 29, 2020

Voice Over Mule Connected App

#mulesofthackathon #alexa #anypointplatform #monitoring

Category: IoT and MuleSoft

Current Status: COMPLETED

Alexa Skill Demo

Code Walk Through and Testing in Simulator

Overview

Voice Over Mule is an alexa skill which will serve as Anypoint Platform Monitoring Voice Assistant, and will perform the below tasks on user's Voice commands

Find Status of an app deployed to Cloudhub
Start, Stop or restart apps deployed to Cloudhub
Add alerts for apps

I will not be going into details of Alexa Skill Building for now. The Main focus will be on how the complete integrations were designed, and the flow designs of different components associated in the IOT.

What Led to the Idea

Last year, I was involved in developing lot of Voice Apps & chat bots using Alexa, Google dialogue flow, and Samsung Bixby.

One thing about Amazon Alexa and Google flow backend is that they interact completely using REST API and JSON payload.

Now, Amazon recommends using their Lambda functions as a backend to the skill development, since

it is their service
it provides a serverless architecture.

But to write code in lambda, you will have to extensively use their SDK either Nodejs, Java or Python.

This year, when I entered the world of mule 😊 I decided to do POC to check if I can replace Lambda backend with Mule, and It worked! It was so much simple with Mule and dataweave, that I thought of extending the use case, and creating a Voice assistant using Mulesoft for Mulesoft!

Pre-requisites

Amazon Developer account (Required to check out the skill and test it in Alexa Simulator). You can find a test account credentials below under Testing section
Anypoint Platform account with admin privileges. Trial account will work as well.
Sample apps deployed to Cloudhub in the above account (Required to test the Skill’s functionality)
Alexa Companion App installed in your phone. Available both in android and iPhone.
Good To Have: Alexa Device: preferably one having a screen
And an extrovert nature to soon become friends with The Anypoint Platform Monitoring Assistant 😊

Voice User Interface Designs

You can refer the below links to access the VUI. I’ve used trial account for this, which has limitations on the number of flow blocks, so I’ve split them into parts
Please use the below link to access the VUI

App Link: https://app.fabble.io/
Credentials: Please provide me an email address to send the credentials
VUI Flows
- Find App Status (Part 1) & Welcome conversation: VOM_FindAppStatus_1
- Find App Status (Part 2): VOM_FindAppStatus_2
- App Operations Conversation: VOM_AppOperations
Few Screenshots are shown below

IOT Components Flow Design – Application Integration

The IOT Components are separated into two major components:

Alexa Eco System
Mulesoft EcoSystem

Alexa Eco-System

This will consist of

Alexa device/Alexa Test Simulator
Phone having Amazon Alexa App installed
Voice Over Mule Alexa Skill (developed using alexa developer console: https://developer.amazon.com/alexa/console/ask)

Mulesoft Eco-System

This will consist of all the required Mule Applications deployed on cloudhub

voice-over-mule-account-link
- used to link Anypoint platform account with Amazon Developer account(Alexa)
vom-alexa-request-router
- Receives user requests from AVS (Alexa Voice Service)
- Decodes the JWT token created during account linking
- Routes the request to voice-over-mule-alexa-exp by splitting the requests into end points based on use cases
voice-over-mule-alexa-exp
- Responsible for sending AVS compatible json responses back to AVS
- Interacts with voice-over-mule-anypoint-exp to get the required data on the apps
voice-over-mule-anypoint-exp
voice-over-mule-anypoint-prc
voice-over-mule-anypoint-sys
The above three together interacts with the Anypoint Platform API to get
- App Statistics
- Start, Stop, Restart apps
- Create Alerts I will be explaining the functionality of the above apps below

Use Cases Covered for Hackthon

Link Anypoint Platform User account to Amazon developer account
Find Status of apps deployed on Cloudhub
Start, stop or restart apps
Create a new Alert
Fallback Dialogues (Voice Error Handling 😊)

Lets start with some Low level designs

Account Linking Process

API used -> voice-over-mule-account-link
Flow diagram

Account Linking Overview

In order for alexa to monitor your apps on cloudhub in realtime and to provide you a personalized app experience, I have used the process of account link, which follow the below Sequence Flow
EndPoints
- /account-link/web -> This is the entry point for account linking flow. Alexa Companion app makes a GET request and a Login Page gets rendered
  - Login Page:
  - The authorization code is coded as a jwt which will have the below payload encoded
- /account-link/token -> This flow will be called by Alexa for - -
  - exchanging token with the code we generated in the above step
  - Regenerating the token using refresh the token.
  - Both the conditions have been handled in the implementation
Once the jwt token is returned back to alexa, the alexa companion app will redirect to a screen displaying if the linking was successful or not as show below
On successful linking, all the subsequent requests from the Voice over Mule Alexa skill will have an access token associated in the payload. Specifically at payload.context.user.token !

vom-alexa-request-router

Alexa Service sends all the requests, like for launch, finding app status etc., to a single end Point. This will obviously lead to several if-else (choice routers / dw expressions). To overcome this shortcoming, I have created this api, whose sole function is to
- inspect the request payload coming from Alexa service
- Decode the access token and extract the payload like userid, email, username, accessToken
- Route it to voice-over-mule-alexa-exp.
- Flow Design

voice-over-mule-alexa-exp

This API has all the use cases exposed as End Points.
The Functionality of this Layer is to interact with the anypoint exp and get the required response from the Anypoint Platform Rest API’s.
It is also responsible for transforming the response in a format acceptable/recognised by AVS.

Brief Description on the end points:

/alexa/app-operation -> This is used to start, stop or restart an app. For example, User can say the below to invoke this resource via request router api:
- Start an app
- Restart my app under “abc” environment [Where abc is one of your custom environments
- Stop my app under abc organization in Sandbox environment
/alexa/app-status ->
- This is used to find status of an app. For example, User can say the below to invoke this resource via request router api:
  - Find status of an app
  - Find status of my app under “abc” environment [Where abc is one of your custom environments]
  - Find my app status under abc organization in Sandbox environment
/alexa/welcome ->
- Called when user invokes the skill using the Skill Invocation Name i.e. Voice Over Mule:
- Called only after user’s Anypoint platform account has been successfully linked to Amazon account
- We also get a master map Master Map which has the mappings between organization, environments and Apps
- The response set is as below. I am showcasing the response for one of the use case here, just to show the json format accepted by AVS
/alexa/create-alert ->
- This will create a new alert bound to the api deployed on Cloudhub.

<![endif]-->

And Finally the Anypoint EXP, PRC and SYS Layers
To be Updated Soon
TODO

Steps to Test the Alexa Skill

Since it takes around a month time to get the Skill certified by Amazon, I will not be able to make it live on alexa store. The skill isnt live yet but thankfully Amazon has provided option to add Beta users to the skill to enable distributed testing. So I will require some email id’s that I will be adding as beta users for the Skill

Testing the Skill Using Alexa Skill Simulator

I’ve created a sample amazon developer account and added that test user as a beta user for the skill. Any tester can use the below credentials to test out the skill. Please provide me the email address to send across the credentials

Steps

Login to https://developer.amazon.com/alexa/console/ask using the above credentials
Since the user is added as a beta user, you will be able to see the skill under Alexa Console as below
Go to Test
Make sure that the skill testing is enabled. What you see below is the Alexa Test Simulator, which developers use to test the skills in case they do not have an actual Amazon device like echo dot, show 5 etc.
Now starts your first interaction with the skill
- Click on the check box showing Device Log. This will help you see the request sent by AVS and the response returned from the Mulesoft Eco System.
- You type open voice over mule, or you can press the mic button and speak the same text as well
- From then onwards, you can continue the conversation with Alexa. I’ve tried my best to make the conversations realistic
- Also, You will be able to see the device display. I’ve included few UI in places where it was required, like choosing an app from the list

Testing from any amazon Alexa enabled devices

Amazon alexa enabled devices are associated to your amazon account.
If you have an alexa enabled device, and you would like to test out the skill, you will have to provide me your email address associated with your amazon account, and I will add you as a Beta User.
Once added, you will be able to access the Voice over mule skill the same way you access other alexa skills i.e. The skill will behave as being Live for the beta Users

Testing from Amazon Alexa Companion App

<![endif]-->

Amazon provides a companion app for android and iPhones. You can get them from respective store. Here’s the Link for android app https://play.google.com/store/apps/details?id=com.amazon.dee.app&hl=en_IN&gl=US

You can login to the app using the test credentials I’ve provided above and you will be able to test out the skill from there as well

Testing the Account Linking Flow

If a user’s anypoint platform account is not connected to the amazon acount, Alexa will state that to the user. Alexa will also ask the user to go to the Companion app and Link the account

Please follow the below screenshot

On Clicking the Link Account, you will be redirected to the account-link endpoint that will load the static page. You’ll have to enter your anypoint platform credentials, under which you want to monitor your apps.
Once completed, you will see a page stating that the app is successfully linked. The screenshot for success account linking

Wrong Creds Error Handling - Implemented
Now you can proceed interacting with the skill again. This time alexa will not ask you to link your account and your conversations will begin.

Code can be found out at the below link: