DEV Community: Avicus Delacroix

Let's talk with chatGPT with your voice

Avicus Delacroix — Fri, 28 Apr 2023 06:31:22 +0000

Hello! This is Avi from Voximplant.

Last time I was talking about AutoML solutions and about how we created a virtual assistant called Avatar at Voximplant.

Since then, I received several messages about artificial intelligence in general and about virtual assistants. I noticed that lately, people became obsessed with chatGPT.

I don't think I need to explain what ChatGPT is. But for those who haven't heard of it: it's a conversational robot based on machine learning like Voximplant Avatar, but it is a different kind of a robot. Avatar is a goal-oriented robot which is made for companies to assist their customers. ChatGPT is a chit-chat robot that learns from users and provides a realistic conversation.

At Voximplant, we provide not only our virtual assistant, but a lot of communication services, such as calls, conferences, contact centers, interactive menus for hotlines, etc. And, of course, with all that, we have speech synthesis and recognition. So, I decided to integrate chatGPT into Voximplant, give it a voice and teach it to recognize human speech.

So, I'm going to show you how to build an application that allows talking with chatGPT with your voice.

Create the application

First, you need to register an account and obtain an API key for chatGPT. This can be easily done at http://platform.openai.com/.

After you get your API key, you need to create an application at Voximplant. This is also an easy task. Read the guide on how to create an app in the Voximplant documentation.

In the Voximplant application, you need to create a scenario where all the code will be present; buy a real or a testing phone number and bind it to your application if you want to call chatGPT from your phone and talk with it; and, of course, you will need a routing rule to launch the scenario (leave all the routing rule settings as default).

Write the scenario

In the scenario, we need to connect to chatGPT via an API request, use speech synthesis to pronounce what chatGPT says, and use the speech recognition module to convert your words to text and send them back as an utterance.

Let’s start with requires and the API request.

All we need to require is the Voximplant's ASR module. Then, we prepare the API url and the API key received in the very first step. For OpenAI, we will also need an array of messages.

I will make the API request to chatGPT each time the ASR module recognizes the utterance. Let's see what it will look like.

require(Modules.ASR)

const openaiURL = 'https://api.openai.com/v1/chat/completions'
const openaiApiKey = 'your_api_key'
var messages = []

async function requestCompletion() {
   return Net.httpRequestAsync(openaiURL, {
       headers: [
           "Content-Type: application/json",
           "Authorization: Bearer " + openaiApiKey
       ],
       method: 'POST',
       postData: JSON.stringify({
           "model": "gpt-3.5-turbo",
           "messages": messages
           // you can configure the length of the answer
           // by sending the max_tokens parameter, e.g.:
           // "max_tokens": 150
       })
   })
}

As you can see, I commented the "max_tokens" parameter, which is sent to chatGPT. By default, chatGPT responds with very long messages, which are okay for IM, but can be quite annoying in oral communication. With this parameter, you can adjust the length of incoming messages (utterances) from chatGPT.

After that, we need to process the inbound call in the scenario. Let's create a couple of necessary variables and integrate our logic into the CallAlerting event.

We need to create an automatic speech recognition instance to process its events. At this step, you can choose the ASR provider. Voximplant provides a huge list of them from multiple vendors for multiple languages. You are free to choose any.

Also, you need to choose a voice for your robot. The same as for ASR vendors, we have a huge list of voices, and at this part, you may try a lot of voices to find your favorite.

After that, we need to process the ASR result each time you stop talking to chatGPT. Let's push your utterance to the messages array and then call the requestCompletion() function that we created in the previous step to send a request to OpenAI.

If the API request is successful, we pronounce the chatGPT utterance via the media player and push the robot's utterance to the messages array. If the request is unsuccessful, we just ask the user to repeat it.

Let's see how it works.

var call, player, asr;
const defaultVoice = VoiceList.Google.en_US_Neural2_C;

VoxEngine.addEventListener(AppEvents.CallAlerting, (e) => {
   call = e.call
   asr = VoxEngine.createASR({
       profile: ASRProfileList.Google.en_US,
       singleUtterance: true
   })
   asr.addEventListener(ASREvents.Result, async (e) => {
       messages.push({ "role": "user", "content": e.text })
       var res = await requestCompletion()

       if (res.code == 200) {
           let jsData = JSON.parse(res.text)
           player = VoxEngine.createTTSPlayer(jsData.choices[0].message.content,
               {
                   language: defaultVoice,
                   progressivePlayback: true
               })
           player.sendMediaTo(call)
           player.addMarker(-300)
           messages.push({ role: "assistant", content: jsData.choices[0].message.content })
       } else {
           Logger.write(res.code + " : " + res.text)
           player = VoxEngine.createTTSPlayer('Sorry, something went wrong, can you repeat please?',
               {
                   language: defaultVoice,
                   progressivePlayback: true
               })
           player.sendMediaTo(call)
           player.addMarker(-300)
       }
       player.addEventListener(PlayerEvents.PlaybackMarkerReached, (ev) => {
           player.removeEventListener(PlayerEvents.PlaybackMarkerReached)
           call.sendMediaTo(asr)
       })
   })
})

As you can notice, I added the Logger.write() function to add the error message from chatGPT to the application logs for later analysis. You can add more logging to understand, for example, how long the API request completion takes.

Let’s extend the ASREvents.Result event processing with logging.

After that, we process the call's Connected and Disconnected events. When the call is established, let's pronounce a greeting and send the call's data to the voice recognition module. And when the call is disconnected, let's terminate the scenario.

After these preparations are done, we can answer the inbound call. The complete event handler will look like this:

VoxEngine.addEventListener(AppEvents.CallAlerting, (e) => {
   call = e.call
   asr = VoxEngine.createASR({
       profile: ASRProfileList.Google.en_US,
       singleUtterance: true
   })
   asr.addEventListener(ASREvents.Result, async (e) => {
       messages.push({ "role": "user", "content": e.text })
       Logger.write("Sending data to the OpenAI endpoint")
       let ts1 = Date.now();
       var res = await requestCompletion()
       let ts2 = Date.now();
       Logger.write("Request complete in " + (ts2 - ts1) + " ms")

       if (res.code == 200) {
           let jsData = JSON.parse(res.text)
           player = VoxEngine.createTTSPlayer(jsData.choices[0].message.content,
               {
                   language: defaultVoice,
                   progressivePlayback: true
               })
           player.sendMediaTo(call)
           player.addMarker(-300)
           messages.push({ role: "assistant", content: jsData.choices[0].message.content })
       } else {
           Logger.write(res.code + " : " + res.text)
           player = VoxEngine.createTTSPlayer('Sorry, something went wrong, can you repeat please?',
               {
                   language: defaultVoice,
                   progressivePlayback: true
               })
           player.sendMediaTo(call)
           player.addMarker(-300)
       }
       player.addEventListener(PlayerEvents.PlaybackMarkerReached, (ev) => {
           player.removeEventListener(PlayerEvents.PlaybackMarkerReached)
           call.sendMediaTo(asr)
       })
   })
   call.addEventListener(CallEvents.Connected, (e) => {
       player = VoxEngine.createTTSPlayer('Hi, ChatGPT bot is at your service, how may I help you?',
           {
               language: defaultVoice
           })
       player.sendMediaTo(call)
       player.addMarker(-300)
       player.addEventListener(PlayerEvents.PlaybackMarkerReached, (ev) => {           
           player.removeEventListener(PlayerEvents.PlaybackMarkerReached)
           call.sendMediaTo(asr)
       })
   })
   call.addEventListener(CallEvents.Disconnected, (e) => {
       VoxEngine.terminate()
   })
   call.answer()
})

Now, let's test our application. Call the phone number you have bound to your application, wait for the greeting, and start talking with your robot. Or use the debug softphone in the control panel to call the scenario.

You can find the complete scenario with the comments in the corresponding article in the Voximplant documentation. Feel free to modify the scenario at your wish, process more events, change voices and utterances' length, and much more. It has been tested and is ready to use.

I hope you enjoyed this article. Until the next time!

How we created an AutoML solution in Voximplant

Avicus Delacroix — Tue, 21 Feb 2023 11:53:30 +0000

Hello everyone, this is Avi from Voximplant.

In this article, I’m going to discuss the history of machine learning and natural language processing, tell you about what AutoML is, and explain how Voximplant made this technology available for everyone with its AutoML component.

A brief history of NLP technology

Let’s start with a short historical overview of natural language processing: how it evolved and why we at Voximplant decided that now is the best time to enter the competitive market with our AutoML technology.

The NLP history can be divided into three phases. We’re going to take an in-depth look at each of these stages below:

Stage one

The first phase is applying classic machine learning methods to natural language processing. This approach implied the vectorization method, meaning that we needed to transform text to a representation that ML models could accept as input.

However, the main disadvantage of such an approach was that the words “good” and “perfect” were as far apart as the words “good” and “tree.” Thus, if you were to paraphrase a sentence with synonyms, you would get a completely different ML vector. In addition, the vectors themselves turned out to be huge, and it was almost impossible to provide them as input to a neural network architecture. However, this was merely the first stage of NLP.

Stage two

The second phase in the development of NLP is word embeddings. This is a technique that allowed each word to have a compact vector representation, and similar words had similar vectors. Each word is represented by a real-valued vector with either tens or hundreds of dimensions.

Such representations made it easier to process by neural engines, and NLP solutions became noticeably smarter. The algorithms started to understand the basic language structures and semantics based on huge unlabeled inputs, and later we could train the results for a specific task.

The processing became much faster because the basic language knowledge was already hardwired into the model, and the data for each task decreased significantly. For the NLP industry, this created a new kind of job: research centers trained fundamental models on huge arrays of data, and other companies used them as templates to train for more specific tasks.

Stage three

The third phase is connected to the creation of the transformer architecture. This architecture didn’t have recurrent network problems and had a much higher level of parallelism, which meant that a much higher level of data could be processed simultaneously, making the whole model much bigger.

In 2018, Google released BERT, a transformer that de facto became the template for solving complex NLP tasks. With some modifications, this model becomes dominant in NLP. Moreover, nowadays, the standard process of solving an NLP problem is to take a pre-trained BERT and educate it for more specific tasks.

What is AutoML?

Despite the decent number of ML solutions by large companies such as Nvidia, Amazon, and Google, many retail companies still can’t use them. Why is that? Even if we don’t consider the problems of working with data, one needs a team of highly qualified specialists to support these solutions — this can be quite expensive.

For many companies that don’t have strong technical expertise teams, it’s difficult to discern whether the efforts are worth it and whether having an ML team will pay off in the long run. Additionally, pilot projects are quite expensive and take time to deploy. For all these reasons, many companies feel discouraged from using this technology.

This is where AutoML comes to the rescue. AutoML is an attempt to automate the work of an ML team. In the ideal state, it allows people without machine learning education to create ML-based solutions and put them into production. A company uploads its data into the system, and at the output, it receives a ready-to-use scalable service integrated into the company’s infrastructure at some level. The company needs only to work with data and analytics, leaving all the engineering work under the hood. Sounds interesting, right?

Of course, there are already quite a few players on the market, such as Google Cloud, AWS, and h2o.ai. Their solutions solve some problems in the NLP sphere, but the entry threshold for such solutions is relatively high, and they aim at software engineers instead of technical specialists. This looks like a part of a constructor but not a complete box solution.

Why Voximplant decided to create an AutoML engine

Voximplant is a cloud platform for creating different communication services, such as chats and chatbots, calls and conference services, contact centers, and IVRs. We also have telephony and a lot of vendors for voice recognition and synthesis. However, one thing that was missing from our toolkit was a state-of-the-art NLU engine.

Here's how we went about it. First, we added a 3rd-party solution into our system, but in practice, this solution did not have enough flexibility for real-world scenarios, and this solution could not use the full power of our platform. That’s why we decided to create our own solution.

How we created an AutoML solution in Voximplant

There are two types of conversational bots: chit-chat bots and goal-oriented ones. Chit-chat is designed only for chats, like Replica. These are only useful in answering the client with the phrase that makes the most sense. Goal-oriented bots are designed to find out what kind of action a user wants to perform and perform it. Examples of goal-oriented bots are Siri or Alexa and some smart IVRs in contact centers. Since we are releasing a B2B product, we’re more interested in goal-oriented bots.

So, what components did we need to create such a robot? Let’s take a look at the typical flow. A user pronounces some phrase, and then it is processed by a voice recognition module, which converts voice into text. From this text, first of all, we want to extract user intent, which means the phrase's normalized meaning.

During the creation of our bot, we came up with a list of possible intents so our bot could match the user’s phrase with one of these intents. Also, we want to extract all the available data from the user’s speech. For example, if a user wants to book a table at a restaurant, we need to extract the exact timing for the reservation. All these tasks are to convert the natural language into a machine-readable one.

All the extracted information needs to be passed into the dialog state manager — a component that decides how to respond to the normalized request. There are many approaches to this task, both based on ML and scripting, but in general, we view this component as some kind of script written for each case. After that, the only thing left to do is synthesize speech to answer the user.

Hence, we need the following components:

Intent classifier, which learns on user input
Component for extracting and normalizing user data
Dialog state manager to make decisions
API for managing

The intent classifier needs to have a front end that accepts user data and stores it in the back end. After that, we need to start training the user model, store the resulting artifact somewhere and send it to the inference server that processes user requests in real-time.

While it doesn’t look too complicated, the system has several limitations:

The service should be responsive and not have any delays during a phone call
The service quality should be comparable with SoTA
The service should be resilient to the influx of users
The service should have a competitive price

What was also important, we needed to launch the project with a small team of 4 people at the beginning, and we had six months to get the MVP. Also, we wanted to start receiving feedback from the market as soon as possible to understand whether our product really solves customers’ problems or not.

Therefore, we decided to launch our AutoML service in several smaller iterations.

Iteration 1

In the first iteration, we didn’t focus on scaling problems. We wanted to make a quick prototype and receive feedback from the market. We decided to construct a base project, which we could enhance with different technologies in the future, and started with the intent classifier. By making only the intent classifier, we could cover a very large number of tasks, such as automatic comment classification, NPS surveys, and smart IVR.

Next, we automated the training process for RoBerta Large for a project team that can assemble solutions because this approach gives the highest quality. Inference service prices were intentionally avoided, not to get into too much detail on the one hand and to frighten customers with poor quality on the other.

We took Bootstrap to make a simple control panel for data management, made a backend via FastAPI, which trains the tasks, and used MLflow ModelRegistly to store the results. We also used it for storing metrics for debugging. For the inference server, we deployed several hardware nodes with GPU and Nvidia Triton installed.

Our telemetry needs were met by MLflow out of the box, as well as the UI tools for its analysis. We trained each user's model on both the cross-validation dataset as well as the full dataset for final model training. As a result, we were able to identify situations in which our hyperparameter fitting heuristics failed and adjust the formula manually to avoid problems.

More than that, MLflow helped us solve the problem of delivering models to the inference server. The ModelRegistry service it provides makes it possible to track metrics and store model artifacts in a versioned way.

The Nvidia Triton inference service pulled the models into memory. The only task left for us was writing a small Python solution that examined the model registry for new models and sending them to Triton.

As a result, in just a few months, we got a ready-to-use product that could be deployed fast and solve customers' cases without any intervention from ML engineers.

But although our solution had decent quality and low latency, it was completely non-scalable. The main problem was that each client model always required at least 2 GB of video memory, even if the client made a “hello world”. If we get a large number of new clients, our solution will not withstand the load.

Iteration 2

The goal of the 2nd iteration was to make our solution scalable. To achieve it, we needed a way to share the GPU resources with our clients in an efficient way.

What are possible solutions? We can conduct an autoscaling of GPU nodes, but it has drawbacks — not only is it expensive, but it also has high latency during high loads, and the hardware utilization is very low.

Besides autoscaling, we can add smart rotation logic for GPU nodes, load them on demand, and unload them when not in use. When working with voice, however, you should keep in mind that Nvidia Triton loads models within seconds, not milliseconds. This will lead to a large delay and not a natural conversation. This also doesn’t solve the cost problem. Another variant is switching to the previous generation models and moving from GPU to CPU, but the quality we got did not satisfy us. So, we decided to dig deeper.

The thing we want to keep in mind is that we do not allocate the GPU memory for each client exclusively. A better idea would be if some parts of the ML model were the same for everyone. This variant simplifies scaling a lot because, in this case, we scale GPU resources, not for client applications (most of them are hello worlds).

One curious fact we learned from our small team's experience with ML is that the upper layers of transformers change a lot with fine-tuning, while the lower layers hardly change. This seems reasonable because the lower layers are in charge of basic language rules and semantics, and the upper layers are in charge of high-level representations and cases.

In fact, you can suspend the lower layers while fine-tuning a model for a particular case without losing quality. And the embeddings and subword layer can take about half of the whole model, and if we suspend the first N layers, then up to 90% of the entire model can be left unchanged.

This solves our problem: 95% of our model will be suspended and only used to add new features and custom models, and we will be fine-tuning the last few layers of users' data.

In this case, we solve the scaling problem as follows: we have a pool of GPU machines, which we scale when the number of requests to the system increases, which is quite easy, and at the same time, we infer the user part of the system on CPUs with model rotation in RAM. We can quickly deploy a user model on the CPU in less than half a second.

The final variant looks like this: when a user request arrives, it gets to a GPU machine with a backbone, which vectorizes it using the transformer, and then from the 22nd layer of RoBerta, the embeddings array transfers to a CPU machine, where the user’s model is either already cached in RAM, or if not, it is lifted from the local disk cache in another 50-100 milliseconds. Hence we are dealing with a stateless system. We don’t need to think about which pods have which user models — we can send a request to any pod, and it will work correctly. Even though sending the request to the pod when the model is already cached in RAM is better, this problem can be easily solved at the infrastructure level without changing the application logic.

Thus, we solved the main problem: scaling without sacrificing quality. Thanks to this, we didn’t have to worry about peak loads on the system, and in addition, we got a very clear billing model, calculated by the price of a request.

Iteration 3

The final touches we needed to feel that our system was complete were hyperparameter selection and multi-language support for our product. That’s what we turn our attention to in the 3rd iteration.

Taking into account that our system is primarily designed for people who are not ML engineers and don’t have a proper education in this field, we left only one hyperparameter available: the number of training loops. This problem could be solved in two ways. The first one is to perform a two-step training. However, the main issue with this solution is that it makes the training more complicated, and second, we can’t control the dataset size. It was difficult to predict how such a system would behave in a few-shot mode, but most likely, it won’t be good. Therefore, we decided to continue researching the dependence of dataset parameters and the optimal number of training loops.

Here is where the Massive dataset by Meta Research came to aid. With this multilingual classification dataset, we were able to cut thousands of datasets of different sizes and data distribution and began to research the optimal number of training loops and dataset parameters. Specifically, the number of examples in a dataset, the number of classes, and the average and variance of the example’s distribution by class. To our surprise, we found that on different slices of this multi-dimensional distribution, we saw one two-dimensional space, and the dependence was mapped with fairly good quality. Moreover, we got almost the same inverse dependence.

After that, we only needed to use the usual regression to get the general formula form from the obtained data(of course, not forgetting the minimum and the maximum number of epochs to get rid of any artifacts).

And the last problem for us to solve was related to multi-language support. Due to Voximplant’s working geography, we needed support in Russian, English, Spanish, and Portuguese. We wanted to know if it would be a nice idea to use a multilingual model as a backbone for all languages or if each language needs its own model. Luckily for us, the guys at Meta had already researched this problem and proved the excellent work of the multilingual XLM-Roberta. So we left a separate Russian backbone because it was already noticeably better than the XLM-Roberta’s. After all, we already had a huge amount of raw data; for other languages, we took XLM-Roberta as a backbone.

Outcome

In the end, after circa nine months of hard work, we made a full-fledged tool for creating flexible communication scenarios based on SoTA NLU and serverless JavaScript, deeply integrated with Voximplant features such as chats and telephony.

Voximplant Avatar: An AI chat and voice robot for you

Avicus Delacroix — Mon, 28 Mar 2022 16:49:21 +0000

We encounter voice or chat robots every day. Some of them call to notify us about goods shipments, some of them irritate us with ads. Many such robots greet us on websites asking if we need any help.

The most significant disadvantage of such robots is that they can say or perceive only previously scripted information. This is the main reason why such robots irritate us, because it's impossible to have an open dialogue with them. So we usually prefer to speak to a human instead.

Here is where Artificial Intelligence (AI) comes in handy. AI took great steps forward over the last few years and is still evolving. There are plenty of AI companions which can freely talk like real human beings. Such companions learn during communication, understand many phrases, and even try to answer the same question differently.

So, what if we combine such an AI companion and a voice/text robot? Doing this, we can create an assistant which can maintain a smooth conversation with customers. Teach this assistant everything that it may need, add a realistic speech synthesis engine — that’s what you get with Voximplant Avatar.

How avatar works

Avatar is a Voximplant service that uses AI and NLP (native language processing) for voice and text communication with customers. You can teach your avatar all necessary information, such as your cafe's working hours and delivery options, so it can answer customers' questions. You can also integrate it with your CRM so your avatar can help your customers book tables.

Additionally, you can supply your avatar with realistic speech synthesis and recognition engines, making it a realistic interlocutor. Modern speech engines sound great and AI and NLP make conversations more natural. Add telephony or chat features, and you receive a perfect assistant for your hotline, contact center, or website.

How to create an avatar

Let's create an avatar that can perform everything explained above. For this, you will need a Voximplant account. Log in to the control panel, find the Avatars section, and click Create. Give your avatar a name, choose a language and time zone, and create it!

At this step, you already have a smart avatar that can analyze human speech and understand the speaker's intentions. Now we can teach our avatar all the information it may need.

How to teach an avatar

To teach your avatar, open it, go to the Intents section and click Add.

To understand what a customer wants, an avatar needs to find intents in a customer's speech. But a customer can ask the same question differently, thus why your avatar needs training.

Open the intent you created and go to the Training phrases section. We need to write several phrases on how a customer can ask the question.

What are your opening hours?
Do you work tomorrow?
How late are you open until tonight?

Then go to the Answers section and write some variants of your answers.

When you save the intent, you will see a yellow Training required button at the top of the screen.

Click Train and wait, this will take a moment. AI will analyze all possible variants of this question and will be ready to answer in a real case.

You can add as many intents as you want. Let's teach our avatar intents surrounding delivery options and booking a table.

Next write a JavaScript scenario on the Dialogue scenario tab. I created a simple scenario where the avatar greets a customer and searches for intents to learn about working hours and delivery options in the customer's speech and answers.

addState({
    name: 'start',
    onEnter:async(event)=> {
        // greet a customer when it connects to the dialogue
        return Response({utterance: 'Pineapple garden, how can I help you?', listen: true})
    },
    onUtterance:async(event)=>{
        // search for intents in a customer's speech
        if (event.intent === 'openHours' || event.intent === 'delivery' || event.intent === 'reservation') {
            // answer a customer's intent and keep listening
            return Response({utterance: event.response, listen: true});
        } else {
            // if an intent is not clear, ask to rephrase
            return Response({utterance: 'Sorry, I didn\'t catch that. I can help you with open hours, deliveries, and reservations', listen: true});
        }
    }
});

// set the entry point
setStartState('start');

You can find more information on how to write avatar scenarios in the Voximplant documentation.

Now let's test our avatar. Click the Debug button at the top right corner to run the scenario.

Ask your avatar any questions to see if it processes incoming intents. The avatar should recognize your intent and answer your question correctly. Let's try!

Voila! Your avatar recognizes intents perfectly and gives correct answers. Now it's time to teach the avatar to do something more serious than just answering the questions, like booking tables.

How avatar can book tables

First, teach your avatar a booking intent the same way we did previously. Upon recognizing this intent, the avatar needs to collect all the necessary information for booking, primarily time, date, and the number of people.

A customer might provide necessary information during their intention to book a table; for example, they could say, "I need to book a table for two", so we already know the number of people. That is why we need to collect the information during intent recognition.

Let's create a booking object in the scenario:

let reservationForm = {
    slotTime: null,
    slotPeopleCount: null,
    uncertainUtterancesCount: 0
};

Now we need to collect all the necessary information to fill out the form. Check what is missing and ask the customer questions. If the customer could not provide a certain answer, increase the uncertainUtterancesCount counter, to avoid looping.

When all the information is collected, we need to summarize it to the customer and ask for confirmation. If the customer confirms, we record the booking object and now can send it to your CRM or backend via an API request.

I also added several exit points to the scenario to avoid looping. For example, if the avatar did not understand the client three times, or the customer states that the avatar cannot help them, or says goodbye, the scenario will end. Take a look at my final scenario:

let reservationForm = {
    slotTime: null,
    slotPeopleCount: null,
    uncertainUtterancesCountweirdUtterancesInRow: 0
};

addState({
    name: 'start',
    onEnter:async(event)=> {
        // if this is the first time in this state, greet the client. if it is not, ask what we can help with
        if (event.visitsCounter === 1) {
            return Response({utterance: 'Pineapple garden, how can I help you?', listen: true})
        } else {
            return Response({utterance: 'Can I help you with somehting else?', listen: true})
        }
    },
    onUtterance:async(event)=>{
        // search for intents
        if (event.intent === 'openHours' || event.intent === 'delivery') {
            // answer the intent and go to the 'start' state
            return Response({utterance: event.response, nextState: 'start'});
        } else if (event.intent === 'reservation') {
            // the client may already give you some information
            if (event.entities.systemTime) {
                reservationForm.slotTime = event.entities.systemTime[0].value;
            }
             if (event.entities.systemNumber) {
                reservationForm.slotPeopleCount = event.entities.systemNumber[0].value;
            }
            return Response({utterance: 'Sure!', nextState: 'reservation'});
        } else if (event.intent === 'no') {
            // if the client says "no thanks", say goodbye and end the conversation
            return Response({utterance: 'Ok! Hope I\'ve helped. See you!', nextState: 'final'});
        } else if (event.intent === 'yes') {
            // if the client says that there's a question, ask and listen
            return Response({utterance: 'Sure, so what\'s your question?', listen: true});
        } else {
            // if the client's intent is not clear, ask for missing information, but not more than 3 times
            if (event.utteranceCounter < 3) {
                return Response({utterance: 'Sorry, I didn\'t catch that. I can help you with open hours, deliveries, and reservations', listen: true});
            } else {
                return Response({utterance: 'I\'m so sorry, but I couldn\'t understand you. Bye!', nextState: 'final'});
            }
        }
    }
});

addState({
    name: 'reservation',
    onEnter:async(event)=> {
        if (reservationForm.uncertainUtterancesCount > 2 ) {
            // if the client cannot answer avatar's questions, or if your avatar does not understand the client, stop asking anything
            reservationForm.uncertainUtterancesCountweirdUtterancesInRow = 0;
            return Response({utterance: 'Sorry I couldn\'t understand you', nextState: 'start'});
        } else if (reservationForm.slotTime && reservationForm.slotPeopleCount) {
            // if the information is given, confirm it
            return Response({nextState: 'reservationConfirm'})
        } else if (!reservationForm.slotTime && !reservationForm.slotPeopleCount) {
            // if something is wrong, ask about it
            return Response({utterance: 'For how many people and which date would you like a reservation?', listen: true})
        } else if (!reservationForm.slotPeopleCount) {
            return Response({utterance: 'And for how many people do you need a table?', listen: true})
        } else {
            return Response({utterance: 'And for which date?', listen: true})
        }
    },
    onUtterance:async(event)=>{
        // check if you have all the necessary information
        if (event.entities.systemTime || event.entities.systemNumber) {
            if (event.entities.systemTime) {
                reservationForm.slotTime = event.entities.systemTime[0].value;
            }
            if (event.entities.systemNumber) {
                reservationForm.uncertainUtterancesCount = 0;
                reservationForm.slotPeopleCount = event.entities.systemNumber[0].value;
            }
            reservationForm.uncertainUtterancesCount = 0;
            return Response({nextState: 'reservation'});
        } else {
            reservationForm.uncertainUtterancesCountweirdUtterancesInRow += 1;
        }

        if (event.intent === 'openHours' || event.intent === 'delivery') {
            // if during filling the form any other intent is found, answer it and continue filling the form
            return Response({utterance: event.response, nextState: 'reservation'});
        } else {
            // continue filling the form
            return Response({nextState: 'reservation'});
        }
    }
});

addState({
    name: 'reservationConfirm',
    onEnter:async(event)=> {
        // convert to human-readable date
        const months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
        const monthStr = months[parseInt(reservationForm.slotTime.substring(5, 7), 10) - 1];
        const day = parseInt(reservationForm.slotTime.substring(8, 10), 10);
        const hour = parseInt(reservationForm.slotTime.substring(11, 13), 10);
        const minute = reservationForm.slotTime.substring(14, 16);
        return Response({utterance: `So you want to book a table for ${reservationForm.slotPeopleCount} people at ${day} ${monthStr} ${hour}:${minute}`, listen: true});
    },
    onUtterance:async(event)=>{
        if (event.intent === 'yes') {
            return Response({utterance: 'Awesome! We will be waiting for you', nextState: 'start'});
        } else if (event.intent === 'no') {
            reservationForm.slotTime = null;
            reservationForm.slotPeopleCount = null;
            reservationForm.uncertainUtterancesCount = 0;
            return Response({utterance: 'I see, sorry.', nextState: 'start'});
        } else {
            if (event.utteranceCounter < 3) {
                return Response({utterance: 'I\'m sorry, so do you want to make a reservation?', listen: true});
            } else {
                return Response({utterance: 'Sorry, I can\'t help you. Hopefully I will be able to assist you next time. Bye', nextState: 'final'});
            }
        }
    }
});

addState({
    name: 'final',
    onEnter:async(event)=> {
        return Response({isFinal: true, needRedirectionToOperator: false, reservation:reservationForm})
    }
});

// set the entry point
setStartState('start');

Now we have an avatar that can tell customers about hours of operation, delivery options, and also book a table and integrate with your CRM or backend.

How to integrate telephony and chat

The last thing your avatar needs is telephony or chat service to be able to communicate with your customer. To integrate it with telephony, go to the Integration tab of your avatar and copy the integration script. Then create an application on the Voximplant platform and paste the integration code into the application scenario.

Set up the speech synthesis and recognition modules by choosing the language and a suitable voice, rent or connect a phone number, set up a routing rule, and your avatar is ready to answer your customers' calls!

You can also connect your avatar to a text chat, for example, on your website. Avatars work perfectly with both voice and text channels. You can test how an avatar communicates in a chat via this simple demo.

Result

We created a simple avatar that can communicate with customers, answer their questions, and book tables via API requests. This is a simple case, but you can extend the logic and make your avatar an indispensable assistant for your hotline or contact center.

Voximplant's speech synthesis and recognition modules allow you to choose a very realistic voice or even integrate 3rd party voices, so your customers won't be able to tell if it is a real person or a robot. And constantly evolving AI and NLP will make your avatar better every day!

Register at the Voximplant platform and create your own avatar today! Everyone who tests in and leaves a review will get a guaranteed prize.

How KFC employs thousands of candidates with good conversion

Avicus Delacroix — Mon, 31 May 2021 13:27:43 +0000

Hi, this is Avi from Voximplant. Last time I showed you an example of How to create a DIY cross-platform Clubhouse, and today I have something more interesting to show to you — a real case of the employment system in KFC restaurants.

In Russia alone, KFC opens more than 100 restaurants every year (imagine worldwide numbers), and all these restaurants need staff. Working in fast-food restaurants is popular among young people; many seek this kind of work temporarily. And all of them send their CVs to KFC.

Because of this KFC gets more than 15,000 applications every month and needs to process them. But imagine how many HR employees they would need to call all the applicants? Factor in that most applicants are underage or that they don’t need the job anymore and it looks like a waste of time and money spent on HR.

This is why KFC asked Voximplant for help. And the solution was obvious: we created a robot that called all the applicants and asked them a simple question, "Do you want to work for KFC?". If they say yes, the robot asks for some additional information, about their age/documents/desired work hours and more, and passes the information to an HR manager who is ready to employ them.

The results were astounding. The robot processes the applications 10 times faster than a human, filters 37% of candidates as irrelevant, and HR managers increase their conversion by 11%.

Today I want to talk to you about how to create such a robot for your own purposes.

Actually, it's not as hard as it may seem. Let's create a simple robot like the one KFC uses, right now.

The task will be simple: call all the candidates from a call list, greet them, ask them if they want to work for us, recognize their answer, and connect to a call-center specialist depending on their answer.

Let's log into the Voximplant platform, create an application, and start with a scenario. First, we need to import speech recognition and speech synthesis modules, declare necessary constants and start writing our logic.

require(Modules.ASR);
require(Modules.Player);

const callerid = '7xxxxxxxxxx';
const operator = '7xxxxxxxxxx';
let data;
let callCandidate, callOperator;
let asr;

We can pass all the necessary input to the scenario in the VoxEngine.customData field. For our example, let's use the following JSON format: { "phone" : "7xxxxxxxxxx", "name": "John" }.

At scenario start, we parse the JSON from customData and initiate a call from callerid to data.phone and connect the event handlers:

VoxEngine.addEventListener( AppEvents.Started, async e => {
 data = JSON.parse( VoxEngine.customData() );
  // Start a call from callerid to data.phone
 callCandidate = VoxEngine.callPSTN( data.phone, callerid );

 // Add event listeners
 callCandidate.addEventListener( CallEvents.Connected, callCandidate_connected_handler );
 callCandidate.addEventListener( CallEvents.Disconnected, callCandidate_disconnected_handler );
 callCandidate.addEventListener( CallEvents.Failed, callCandidate_failed_handler );
});

When our candidate answers the call, we create a Player to synthesize speech, set some text (use your imagination), and play it to our candidate. We can integrate 3rd-party voices, so let's use one from Yandex:

function callCandidate_connected_handler( e ) {
 // Create a text-to-speech player
 const text = `Здравствуйте ${data.name}. Хотите работать?`;
 const language = VoiceList.Yandex.Neural.ru_RU_alena;
 const player = VoxEngine.createTTSPlayer( text, language );

 // Start playback and connect an event listener after
 player.sendMediaTo( callCandidate );
 player.addEventListener( PlayerEvents.PlaybackFinished, playbackFinishedHandler );
};

When the playback finishes, we need to process the candidate's answer. For our example, let's integrate a voice recognition profile from Tinkoff:

function playbackFinishedHandler() {
 // Create an ASR for speech recognition
 const asrProfile = ASRProfileList.Tinkoff.ru_RU;
 asr = VoxEngine.createASR( { profile: asrProfile } );

 // Wait for the human’s response and add an event listener
 // which will trigger after speech recognition
 callCandidate.sendMediaTo( asr );
 asr.addEventListener( ASREvents.Result, asrResultHandler );
}

Now the candidate's answer is recorded and transcribed to text. We can process the result as we need. As a simple way, if the candidate's answer has the keyword "no" in it, let's just record it to our data object and hang up (you may want another logic).

If the candidate's answer has the word "yes", let's also record this to our data object's field and connect it to an operator:

async function asrResultHandler( answer ) {
 // Disable event listener
 asr.removeEventListener( ASREvents.Result );

 // The answer is written into text field
 // we can proceed the result as we want
 if( answer.text.includes( 'no' ) ) {

   // Update the data object
   data.wantsToWork = 'no';

   // Hang up ( Triggers the callCandidate_disconnected_handler )
   callCandidate.hangup();
 }
 else if ( answer.text.includes( 'yes' ) ) {
   data.wantsToWork = 'yes';

   // Call the operator function
   startOperatorCall();
 }
 else {
   callCandidate.hangup();
 }
}

function startOperatorCall() {
 const callOperator = VoxEngine.callPSTN( operator, callerid );

 // Unite two calls into one
 VoxEngine.sendMediaBetween( callCandidate, callOperator );

 // Connect event handler
 callOperator.addEventListener( CallEvents.Connected, () => {
   Logger.write( 'Подключились к оператору' );
 });
 callOperator.addEventListener( CallEvents.Disconnected, async () => {
   callCandidate.hangup();
 });
 callOperator.addEventListener( CallEvents.Failed, async () => {
   callCandidate.hangup();
 });
}

All that we need to process now is the session's termination. For example, if the call attempt was successful, we send the result object (with candidates' answers) via an HTTP request, and terminate our session. If the call attempt was not successful, just terminate the session.

// The call was successful
async function callCandidate_disconnected_handler( e ) {
 const url = 'https://servicename';

 // Send the data object via a POST http request to specific url
 await Net.httpRequestAsync( url, { method: 'POST', postData: JSON.stringify(data) } );

 // Terminate session
 VoxEngine.terminate();
};

// The call was not successful
function callCandidate_failed_handler( e ) {
 VoxEngine.terminate();
};

Voila. Now we have a primitive robot who accepts a call list, calls every candidate, and asks them if they want to work for us. Then it writes their answer in text form to the database and connects to an HR manager if the candidate wants to work.

This was a simple example. Next you can extend your robot's logic, create complex dialogs and algorithms, connect your candidates to different departments depending on their answer, or collect any additional information you want. It has no limits, only your imagination!

I hope this explanation was useful, and I hope you all find a great team and develop a productive business!

DIY Clubhouse: for iOS, Android, and even Unity

Avicus Delacroix — Tue, 09 Mar 2021 10:00:03 +0000

The pandemic changed us all. Working and studying remotely, we craved live communications and seeing our friends and colleagues in person. Maybe that's why a new social network, Clubhouse, has gathered more than 6 million people in less than a year.

Clubhouse is an all-new type of social networking. No texting at all, communicate only by voice. All the communication happens in so-called "rooms", which can be open for everyone or private. The room's owner (moderator) decides who can talk, and the listeners can "raise a hand" to indicate that they want to ask a question.

However, Clubhouse has its limitations. For now, it's only available for iOS users, and to join, you need to receive an invitation from an existing user. But what if you are an Android user?

You can wait until they release an Android or desktop version, or you can make your own Clubhouse, with ~~blackjack~~ better security, voice recording, AI, and support for any platform. And Voximplant will help you.

In this article, you’ll see how to create a Clubhouse-like audio room with Voximplant's Web SDK, so clients can use a common browser as a client.

Please note that Voximplant supports multiple SDKs, from Web, iOS, and Android to React Native, Flutter, and even Unity. Imagine, you can embed an audio (or video) chat into your game design

Set up VoxEngine

VoxEngine is a cloud platform to enable serverless communications. Let's create a room. Log into your account (or create one) and create a new application. Then create a user for authorization and a new scenario.

First import the conference module. Then implement user disconnecting:

require(Modules.Conference)
let owner = null;
const onEndpointDisconnected = (event)=>{
   const members = conference.getList();
   if(members.length === 1){
       VoxEngine.terminate();
       return;
   }
}

Then implement permissions check:

const checkPermissions = ({call,headers}) =>{
   return new Promise((resolve)=>{
       setTimeout(()=>{resolve(true)},500);
   });
}

Implement room creation and adding a new user:

let logURL = ''; // for debug reason
let conference = null;
VoxEngine.addEventListener(AppEvents.Started, event => {
   logURL = event.logURL;
   conference = VoxEngine.createConference({hd_audio:true});
})
VoxEngine.addEventListener(AppEvents.CallAlerting, async event => {
   const permissions = await checkPermissions(event);
   if(permissions) {
       event.call.addEventListener(CallEvents.Disconnected, onEndpointDisconnected);
       event.call.answer();
       conference.add({
           call: event.call,
           displayName: event.headers['X-Name'],
           mode: "FORWARD",
           direction: "BOTH",
           scheme: event.scheme
       });
       if(conference.getList().length === 1){
           owner = conference.getList()[0].id();
           conference.getList()[0].getCall().sendMessage('owner');
       }
       conference.get(owner).getCall()
       .sendMessage(conference.getList().length);)
   } else {
       event.call.hangup({'X-Reason':'DENIED'});
   }
});

The first part is done. Now let's create a client.

Set up the client

As I decided to make an example with Web SDK, my client will be an HTML-page. I'll hide the control buttons until the SDK initialization.

<!DOCTYPE html>
<html lang="en">
<head>
   <style>
       .hidden {
           display: none !important;
       }
   </style>
   <meta charset="UTF-8">
   <title>The demo</title>
</head>
<body>
<div id="btns" class="hidden">
   <p id="myname">Avi</p>
   <button id="viewer">Join as listener</button>
   <button id="speaker">Join as speaker</button>
   <button id="leave" disabled>Leave</button>
</div>
<div id="audio"></div>
<h3>Current speakers <span id="countSpeakers">0</span></h3>
<div id="endpoints"></div>
</body>
<script src="*****"></script>

An interesting part. Let's initialize our SDK in the tag and show the control buttons if the initialization is successful. <div class="highlight"><pre class="highlight javascript"><code><script> // sdk init const sdk = VoxImplant.getInstance(); let user = 'user*****'; const init = async () => { await sdk.init({ showDebugInfo: true, serverIp: 'url*****' }); await sdk.connect(); await sdk.login(`${user}@app**.acc**.voximplant.com`, 'pass*****'); } init().then(() => { document.getElementById('btns').classList.remove('hidden'); }); </code></pre></div> Create some necessary constants. I will get the user name from an HTML-tag with id=”myname”. Then let's define when each button is displayed: <div class="highlight"><pre class="highlight javascript"><code>let currentCall; let currentRole; let countSpeakers = 0; const confNumber = 'Test room'; const speakerBtn = document.getElementById('speaker'); const viewerBtn = document.getElementById('viewer'); const leaveBtn = document.getElementById('leave'); document.getElementById('myname').innerText = myName; let setRole = (role) => { currentRole = role; if(role === 'speaker') { speakerBtn.disabled = true; leaveBtn.disabled = false; viewerBtn.disabled = false; } if(role === 'viewer') { speakerBtn.disabled = false; leaveBtn.disabled = false; viewerBtn.disabled = true; } if(role === 'start') { speakerBtn.disabled = false; viewerBtn.disabled = false; leaveBtn.disabled = true; } } </code></pre></div> Handle <a href="https://voximplant.com/docs/references/websdk/voximplant/call#hangup">leaving the conversation</a>: <div class="highlight"><pre class="highlight javascript"><code>let endCall = () => { if(currentCall && currentCall.state() !== 'ENDED') { document.getElementById('endpoints').innerText = ''; currentCall.hangup(); setRole('start'); } } </code></pre></div> Then <a href="https://voximplant.com/docs/references/websdk/voximplant/endpointevents#remotemediaadded">adding</a> and <a href="https://voximplant.com/docs/references/websdk/voximplant/endpointevents#remotemediaremoved">removing a user</a> to/from a conversation: <div class="highlight"><pre class="highlight javascript"><code>let onEndpointAdded = (e) => { console.warn('Endpoint added', e.endpoint.id); const nameTable = document.getElementById('endpoints'); let p = document.createElement('p'); p.id = e.endpoint.id; p.innerText = `Name: ${e.endpoint.displayName}, id: ${e.endpoint.id}`; nameTable.append(p); document.getElementById('countSpeakers').innerText = countSpeakers + 1; e.endpoint.addEventListener(VoxImplant.EndpointEvents.RemoteMediaAdded, (ev)=> { console.warn('RemoteMediaAdded', ev.mediaRenderer); const nodeCall = document.getElementById('audio'); ev.mediaRenderer.render(nodeCall); }) e.endpoint.addEventListener(VoxImplant.EndpointEvents.RemoteMediaRemoved, (ev)=>{ console.warn(`Endpoint ${e.endpoint.id} media removed ${ev.mediaRenderer}`); }) // ENDPOINT REMOVED e.endpoint.addEventListener(VoxImplant.EndpointEvents.Removed, (ev)=>{ console.warn(`Endpoint ${e.endpoint.id} removed`); let removeP = document.getElementById(e.endpoint.id); nameTable.removeChild(removeP); }) } </code></pre></div> Handle the Leave button and some necessary events: <div class="highlight"><pre class="highlight javascript"><code>const setCall = () => { leaveBtn.onclick = endCall; currentCall.addEventListener(VoxImplant.CallEvents.EndpointAdded, onEndpointAdded); currentCall.addEventListener(VoxImplant.CallEvents.MessageReceived, (e) => { console.warn('MessageReceived', e.text); }); //handle connection currentCall.addEventListener(VoxImplant.CallEvents.Connected, () => { console.warn(`Call connected successfully`); }); //other call event listeners currentCall.addEventListener(VoxImplant.CallEvents.Disconnected, () => { console.warn(`Call disconnected`); endCall(); }); currentCall.addEventListener(VoxImplant.CallEvents.Failed, (e) => { console.warn(`Call failed`); endCall(); }); } </code></pre></div> And handle the role buttons (Speaker or Listener): <div class="highlight"><pre class="highlight javascript"><code>speakerBtn.onclick = async () => { document.getElementById('endpoints').innerText = ''; if(currentCall) { document.getElementById('endpoints').innerText = ''; await currentCall.hangup(); } setTimeout(() => { setRole('speaker'); currentCall = sdk.callConference({ number: confNumber, extraHeaders: {'X-Name': myName} }); setCall(); }, 300) } viewerBtn.onclick = async () => { if(currentCall) { document.getElementById('endpoints').innerText = ''; await currentCall.hangup(); } setTimeout(() => { setRole('viewer'); currentCall = sdk.joinAsViewer(confNumber); setCall(); }, 300) } </script> </html> </code></pre></div> Done. Now we have a working room with two roles (Speaker and Listener) simple button controls and a list of active speakers. All we need to do to make it a complete app is to make a decent interface and add some user/room management with databases and search. And do not forget that the functionality of the Voximplant engine goes far beyond this: you can schedule a voice room to record if you're unable to join due to a scheduling conflict, you can add messages if you need, you can even create interactive voice menus with speech synthesis and recognition, which can be lots of fun.