RASA: a Newbie's Journey

Part2: Is Anybody Out There?

EDIT: Guys, sorry, I have abandoned this project, and RASA changed a lot since I've last seen it. Be back with a brand new project, and new insights hopefully not in almost two years. ^^'

BUT IF YOU ARE INTERESTED IN THE LESS TECHNICAL PART OF CHATBOT MAKING:

In the last part you started your journey as the mini coder hero you are.

You installed the RASA stack and enough dependencies to last...
a few weeks... since this is computer science.

Congrats!

Now wipe away the sweat from your brows, we are going in for the next round.

TL;DR

• Before you do anything, decide what problem you want to solve with your chatbot.
• You need to know early on who will interact with the bot.
• You can make structured data out of the wild things your users say by
  • identifying what they want to do (intent classification)
  • grabbing the information you need from them (entity extraction)

Choose a project

If you have't already taken the time to think about what purpose your chatbot will have, here are a few approaches:

Check out tutorials, and follow them to the letter (you know my opinion on those).

Look up existing bots for inspiration. There are sites like the aptly named chatbots.org, and the more hip chatbottle.co. A lot of the bots are either old, or not around anymore, but they show just how many purposes chatbots can fulfill.

Consider the communication problems around you. See where people need to run the same searches over and over and go through the same motions while they are trying to accomplish a goal or get information. Where would people actually benefit from being able to just tell what they want to accomplish?

Even if you choose a project that is way too hard to make, you can break it down, start with the easiest features and build from there. That will help you to get better at organizing your workflow. Or at least you will be better at gauging the difficulty of projects.

Understand the user

Creating a chatbot is like designing any user experience. First you need to consider the ones who will interact with it. A good way to do that is to imagine a prototypical user in need of the bot you are creating. Lets say you are making a calendar bot (which I actually am, if you took the time to check my project), a personal assistant, if you will.

Look at what is the role of your user. What duties they perform, what situations they face, what framework they are acting in. There is a great difference between:

• A mum, who wants an easy and fast way to organize the housework for each family member.
• A forgetful programmer who wants to track her water consumption for the day.

Then think about what would help these people.

• The mum might want to see the tasks of the day laid out at once and send individual tasks directly to the inboxes of her husband and children, maybe assign XP-s for each task.
• The coder friend will want to set periodical reminders during her work hours, and may need a gentle ease out from the flow she is (hopefully) in.

Finally, probably the most important part, the motivation of your users. Why do they want what they do?

• Our mum tends to forget what exactly needs to be done, unless she jots down the tasks as soon as they emerge, she also wants teach her family that keeping things orderly around them will free up mum's time to have family adventures.
• Our programmer is so immersed in her work that she get so dehydrated, she has regular headaches, but she already uses so many tools to organize her work (the company's version control system, kanban, Slack etc.), she doesn't want a new one.

Write that down in a story format. Keep it short and sweet, only write down what is necessary to evoke thoughts on how your app should look like.
In the end you will get something along the lines of:

• As a lawyer on a flexible contract I want to track my work schedule and log my activities, so my manager is able to track my billable hours.
• Mr. Apples, as an English teacher would love to organize classwork by sharing agendas with the children, and revise them as the semester progresses.
• I, Snuffles, as the puppy of a depressed writer want to time my snuggling sessions to the dry spells of my human.

When you start doing this you will probably realize that many different people are out there to use what you are creating. Group the ones with similar roles and treat them as one person (show the world that stereotypes are sometimes handy) then create stories for each of them.

After all this, list all the features that come into mind. Choose the ones that will give the most value to most of your users. Then choose the ones that you can make first.

Or just choose the story that resonates with you the most, and start with whatever feature would be the most important for that character. This is enough for a beginner project, but don't tell anyone I told you that...

Now sit up and stretch a bit, because you will do some work!

.
.
.

On your brain... because now you can learn how machines understand the world!

(Oh, I love old memes...)

Really understand the user

When your chatbot grows big and strong, first he (she, it... mine's a he) is going to see jumbles of characters people send him.

People that want to accomplish something and expect a useful answer in return. And their patience is thin, and they are prone to be afraid of machines...

To help your bot in navigating an environment like this, the easiest way to manage interactions is to give him a definite purpose, and focus only on the tasks to fulfill that purpose, which luckily you already did when you decided on our project.

It will be best that you state upfront what the intended use of our bot is (but that is a topic for a later time).

Other than that, the type of things people say when they accomplish goals with the help of others are dictated by processes and customs. and your Your bot will usually need to deal with requests, and pleasantries, like please and thank you, are regular elements of conversations.

Finally, grammar structures a text, so some word combinations are more probable than others, for example "You are" is often followed by and adjective, or "I want to" will most likely be followed by a verb.

User intent

Once you know what features (or we can call them abilities) the bot should have, you need to think about how the user will prompt those features with messages written in their own words. In short, you will need a way to analyze the text to get the intent of the user. That is why this process is called intent classification.

For that, you need to give a set of examples, where you identify the intents, so one of the intent classifiers that RASA support (later on that) can find similarities between the examples, so it will be able to predict later for sentences without labels what is most likely said. This is what Machine Learning folk call supervised learning.

Lets see how that works. First the sentences are split into their smallest meaningful units (often character sets divided by whitespaces, some tools will recognize punctuation marks, the possessive 's or the n't in don't as an independent unit). This is called tokenization, and the resulting elements are called tokens.

Then it tries to find the closest thing the intent classifier can to the meaning of the identified tokens. It does that by taking a huge set of words that have been categorized, based on how close they are to all other words in the set - via a process that right now I can only identify as Machine Learning Magic, or MLM for short. Then it will find a place for the words you are feeding to it in that huge set.

Finally it takes the label you gave for the whole sentence, and slaps that on the series of what it now can identify as words, and lumps the now labeled series of words with the other series of that kind, creating as much of a coherent group as it can. And if a new, unlabeled series of words is sufficiently similar to the examples you have given and sufficiently different for everything else, then it will be labeled the same way.

Entities, the juice of it all

But identifying intents and creating responses for each will solve very few and highly specific problems, like looking for the opening hours of a place. That is why you will create other applications, or communicate with existing ones. And you will need to feed data to those applications. Data that you want to get from your user.

Luckily users are usually happy to give that information embedded in their messages. So we will pretty much do the same thing as with the intents, categorize what type of information can be found in user messages, but with one big exception. We will need to indicate where that information can be located in the sentences we get.

And those sentences will go through the same process as with intent classification, with the bonus of being able to predict, based on how many and what kind of words were identified in the message before of after a given word, if they are part of some useful information, and what type of information will it be. These bits, that will be used by the connected apps of the chatbot are called entities (because they usually have names like person, address, currency etc.) and the process is called entity recognition.

WHEW

Congrats again, if you have read this far. I feel like I need to reward you for that. Lets say you have unlocked an achievement:

⭐Smartypants⭐

Take a cookie:

See you in part 3, where you actually, actually will do some work.