Getting your user's country and name in a Whatsapp bot using Fastapi and FaunaDB - Part 3

We're back! So a bit recap of the last part: we now have our Fauna database set up, as well as our API with our greeting endpoint and the functionality to search for a user via their phone number. But before we continue, let me tell you I lied to you: I told you we would fetch the user's country from the phone number, but we did not. So let's do that right away!

Getting a country from a phone number

For this, we'll use the great phonenumbers library, which is the Python fork of Google's libphonenumber. Install it like so:

pip install phonenumbers

Now, let's go back to our greet.py file:

from fastapi import APIRouter, HTTPException, Form
import phonenumbers
from phonenumbers import geocoder # For geographical data

from src.models import User

user_greeting = APIRouter()

def phone_to_country(number: str) -> str:
    parsed_number = phonenumbers.parse(number)

    if not phonenumbers.is_possible_number(parsed_number):
        raise HTTPException(status_code=400, detail="Invalid phone number")

    return geocoder.country_name_for_number(parsed_number, "en")

@user_greeting.post("/greeting")
def greet_user(UserIdentifier: str = Form(...)):
    """
    User greet endpoint
    :param: UserIdentifier: user's phone number from Twilio
    """

    user = User.get_by_phone(phone_number=UserIdentifier)

# Greeting the user since already exists
    if user is not None:
        return {
            "actions": [
                {"remember": {"name": user.name, "country": user.country}},
                {"say": "Hi there {name}! Welcome back, how are things in {c}?".format(name=user.name, c=user.country)},
            ]
        }
    country = phone_to_country(UserIdentifier)

    return {
        "actions": [
            {
                "say": "Hello there! Looks like you are writing from {c}".format(c=country)
            },
            {"redirect": "task://can-have-name"},
        ]
    }

See that, besides importing phonenumbers we also import geocoder, this adds the ability to obtain the country (among other geographical information) from a phone number; it is a separated import because it adds a large amount of metadata.

Next, we write a little helper function to parse a phone number and throw a Bad Request error in case the number it's not valid. This would only happen if this endpoint it's hit outside of Twilio, but let's add it anyway. Then, it's just a matter of using that helper function in case the user its not in the database.

Cool, with that out of the way, let's do the rest of the API.

Saving users to FaunaDB

Ok, so the first thing we need its a method to store a user in Fauna. Lets quickly add one to our User class:

from typing import Optional, Any

from pydantic import BaseModel

from faunadb import query as q
from faunadb.objects import Ref

from src.core import session

class User(BaseModel):
  _collection_name: str = 'users'
  ref: Optional[Ref]
  ts: Optional[str]
  phone_number: str
  name: str
  country: str


  class Config:
    arbitrary_types_allowed = True

  def __init__(self, **data: Any):
    super().__init__(**data)

  def save(self):
    """
    Saves the document in the collection,
    the attributes are serialized utilizing Pydantic's dict method, which
    traverses trough the child class attributes
    :return: An instance of the newly saved object.
    """
    attributes = self.dict()
    result = session().query(
        q.create(q.collection(self._collection_name), {"data": attributes})
    )

    return self.__class__(ref=result["ref"], ts=result["ts"], **result["data"])
 # The code from the previous post

Thanks to Pydantic, this is pretty straightforward:

• First, I added an extra attribute (_collection_name) to get the collection's name; it matches the name we set on the Fauna dashboard
• We get the data we want to save as a dict, we use the BaseModel's dict instance method; which does exactly this.
• Next is the saving process itself:
  • We then use the session object to create a query
  • We pass a collection object using the collection name attribute
  • Finally, we pass the attributes dict as a second argument. Remember we said that non-standard Fauna objects need to be inside a data dict? Here we do exactly that.
  • Finally, we create a new User object using the class's __class__ magic method; which it's basically like calling the constructor. The result object is nothing more than the object returned by Fauna's saving operation. I know that this may not be the best approach (perhaps we should just set the new attributes instead of returning a new instance), but hey that's what I did

Parsing collect actions

Now, let's finally go to the reason for this entire series: asking the user for their name.

Asking the user if they want to give us their name

Remember the first post, when we defined our schema? We said that our endpoint's route would be can-have-name, the task's name would be the same, the action's name would be ask-for-name and the answer from the action would be can_have_name. Recall that action and tasks are two different things: a task is a series of one or more actions that a bot performs, while the action its the specific thing the bot will do in that task; in this case, asking a question.

Now, the code:

import json
# The other imports

# The code we added at the beginning

@user_greeting.post("/can-have-name")
def can_have_name(Memory: str = Form(...)):
    """
    Asks the user if he/she wants to give us their name
    :param: Memory: JSON Stringified object from Twilio
    """
    memory = json.loads(Memory)

    answer = memory["twilio"]["collected_data"]["ask-for-name"]["answers"][
        "can_have_name"
    ]["answer"]
    if answer == "Yes":
        return {"actions": [{"redirect": "task://store-user"}]}
    return {
        "actions": [
            {
                "say": "😭"
            }
        ]
    }

Recall that our collect question had a type of Twilio.YES_NO, that means that Autopilot's natural language processing engine has done it's magic to convert whatever the user wrote into a Yes or No answer (those two are literal). So we just need to parse that response.

Here, you need to take into account the thing that had me hitting the keyboard with my head a couple of days: Autopilot sends all requests with an encoding of application/x-www- form-urlencoded and the Memory object it's in JSON, but, because of the encoding, this JSON is not parsed; it is just a plain string. Because of that, we set the Memory parameter to be of type str and we then use Python's standard library json.loads to parse the Memory string into a dict. Now, what's that intricate dictionary right there? Well, the Memory object contains other things besides parsed responses, the parsed responses are saved under collected_data. After that key: the format is action_name.answers.question_name.answer, as we said above, our action's name is ask-for-name and the question is can_have_name.

So after that confusing prop drilling, we have our answer. If the user said yes we just redirect to the task tasked (no pun intended) to get the name; if not well we just can do whatever we want. In this case let's just return an emoji, why? Because we are using Whatsapp and we can!

Finally getting the name

For the last endpoint, let's get the user's name. Recall that we
wrote in our schema that the endpoint's route is store-user:

# Rest of greet.py up here

@user_greeting.post("/store-user")
def store_user(UserIdentifier: str = Form(...), Memory: str = Form(...)):
    """
    Stores a user in the database, fields stored are: country, name, and phone number
    :param: UserIdentifier: Phone number from Twilio
    :param: Memory: JSON Stringified object from Twilio
    """
    memory = json.loads(Memory)
    name: str = memory["twilio"]["collected_data"]["collect-name"]["answers"][
        "first_name"
    ]["answer"]

    country = phone_to_country(UserIdentifier)
    # This needs error handling right here ;)
    new_user = User(
        name=name.capitalize(), phone_number=UserIdentifier, country=country
    ).save()

    return {
        "actions": [
            {"remember": {"name": new_user.name, "country": new_user.country}},
            {
                "say": "Hi, {name} from {c}!".format(
                    name=new_user.name, c=country
                )
            },
            {
                "say": "This is a WIP ok bye"
            }
        ]
    }

Besides the Memory object, we also need the UserIdentifier; see how FastAPI allows us to pass as a parameter just the fields we need from the request.

The next couple of lines are the same as with the last endpoint, the only thing that changes is the keys for the memory dict: in this case, the name of the action is collect-name and the name of the question is first_name. If you check our schema, you'll realize that the type of question is Twilio.FIRST_NAME; was does this mean? You guessed it! Autopilot's natural language processing engine will try to parse the answer as a first name.

Now, the saving process:

• First, we parse the country just like we did in the /greeting endpoint
• Then, we pass instantiate a User object with the parameters we have: name (we sanitize the input a bit via capitalizing the string), country, and the phone; we also call the save method right away to have a full instance
• Then, like in the first endpoint, we tell the bot to remember the name and the country and return whatever action we want

Testing all out

So now comes the part we've waited for: testing this bad boy out. For that, we first need to deploy our API to get an endpoint to put in our schema. There are several ways to do this, for example, you could deploy this on Heroku, Digital Ocean, or just write the code directly on Glitch to get a live URL (which is what I did for this series); whatever you choose is up to you! Just remember to set the DEBUG flag to false and to fill the FAUNA_SERVER_KEY env variable with a server key.

After the API is live, let's go to the schema file and replace all the occurrences of our_super_yet_to_be_endpoint with the live URL of our API. Now is the time to create the bot, go create a Twilio account if you already haven't; after that, copy the SID and the auth token you'll find in the dashboard.

There are two ways of creating an Autopilot bot: the first one is through the console and the second one is using the CLI. Since we already have the schema for the bot ready, let's use the CLI; you need either yarn or npm for that, I'll use the former:

yarn global add twilio-cli
twilio plugins:install @dabblelab/plugin-autopilot

After that's installed, to a twilio login in your terminal and paste the SID and auth key when prompted.

Now, set the terminal on the path where the finished schema is stored and do:

twilio autopilot:create -s=./schema.json

That will create a bot named almighty-bot with the account you used to log in. Now, for the final part, we need to request a phone number from Twilio and enable WhatsApp integration. The process for doing that is out of the scope of this post, so check here for enabling WhatsApp and connecting that number to our bot.

Texting the bot

Now, let's add our bot to our contact list a send a message!

We can greet the bot again:

Finally, let's go to the Fauna dashboard and erase our user; just to say no and see what happens:

Great! We see that our bot reacts correctly to "yes" and "no".

"But wait a minute", I hear you say "Didn't you tell me that a session lasts for 24 hours? Why did the conversation reset the second time I greeted the bot". Well, that was a half-truth actually: a session lasts for 24 hours if we include a listen action at the end or if we redirect to another action. If you check the schema and the actions we returned from our endpoints, you'll realize that we did not return that action: we just drove the users to a pre-defined path, so to speak. So, if the returned action neither a redirection nor a listen action, the session will finish.

Are we done?

Of course not! You probably already thought about this: this does not seem too secure, does it? How do I know that the requests really come from Twilio? If you did, you're right, this is completely insecure: everyone could figure our API out and make requests to it. So let's change that.

Twilio has a package for Python that has, among other things, a class to validate that incoming requests come from Twilio; let's install it:

pip install twilio

Now, how do we implement this for every request? Well, if you are like me and already have worked with Python in the past, you probably are thinking of a decorator. And, if you're like me, you went directly to implement said decorator just to have it fail miserably when deployed live.

You see, Starlette (and FastAPI, by extension), don't quite work like this. Like I've mentioned a million times before, they are asynchronous; thus, various properties of the request object (like the body or the form) are implemented as asynchronous streams: if you consume them in, say, a decorator, they will complete and will not be available to the next handler in the chain. So, in mortal words: accessing the async parts of the request in any other part that it's not its final destination will make that data unavailable in that final destination; therefore an error will occur.

So, if not a decorator, how do we implement this? Well, we need to make a custom router. What's this? A router is a class whose methods get called before the request (or the response) hits the handler function. Because of this, we have, in a router, complete access to the request (or the response). Let's write the code:

from typing import Callable

from fastapi import Request, Response, HTTPException
from fastapi.routing import APIRoute
from starlette.datastructures import FormData
from twilio.request_validator import RequestValidator

from src.core import config

class AutopilotRequest(Request):
    """
    This class serves two purposes. First one, know that Starlette (framework on which Fastapi is built upon)
    is an ASGI framework. That means that parts of the request (like the body) are async. So, if we await those streams in middleware they will be consumed and will not be available to the final route.
    For that, this class consumes the steam (in this case the form) does what it needs to do with the data,
    and creates a new FormData object to pass to the final route.
    """

    async def form(self):
        if not config.DEBUG:
            return await self.__handle_non_dev_env()
    return super().form()

    async def __handle_non_dev_env(self):
        """
        In production or staging, validate that the request comes from Twilio
        """
        validator = RequestValidator(config.TWILIO_AUTH_TOKEN)

        params = await super().form()
        x_twilio_signature = super().headers.get("X-Twilio-Signature", "no-header")
        is_valid = validator.validate(str(super().url), params, x_twilio_signature)
        if not is_valid:
            raise HTTPException(status_code=403)
        return FormData(dict(params.items()))

class AutopilotRoute(APIRoute):
    """
    The custom route to route requests through our AutopilotRequest object
    """

    def get_route_handler(self) -> Callable:
        original_route_handler = super().get_route_handler()

        async def custom_route_handler(request: Request) -> Response:
            new_request = AutopilotRequest(request.scope, request.receive)
            return await original_route_handler(new_request)

        return custom_route_handler

Ok, what's going on here? First, for a custom router, we need two things: the custom router and the custom request object. For this, we subclass ApiRoute and Request (from FastApi) respectively.

Now, for the request:

• We are only interested in the form property since that is where Twilio data is
• We check if we're on debug mode if we are then we just return the original form
• Now, in the __handle_non_dev_env is where the magic happens
  • We create a RequestValidator object with the TWILIO_AUTH_TOKEN (remember that?) property that is already in our configuration object
  • We consume the form stream, see that wee need to await it. We need to do this because it contains data the validator needs
  • Then, we extract the header from the request. The signature is in the X-Twilio-Signature header
  • Now, we use the RequestValidator's validate method to perform the validation
  • Next, we need to check if that method returned True, if not we raise a 403 Forbidden error
  • Finally, if the request did indeed come for Twilio, then we need to construct a new form since the old one was consumed; for this we just do: FormData(dict(params.items())) which will return a new FormData with the original data from the request

Now, the router is simpler: it just needs to convert the incoming request to an AutopilotRequest and pass it to the original handler. That way, when the handler calls the form method under the hood, the validation will occur.

How do we use this? Easily, let's go to our greet.py file:

# routes/greet.py

# Original imports
from .custom_router import AutopilotRoute

user_greeting = APIRouter()
user_greeting.route_class = AutopilotRoute # <- here

That's it, now just remember to fetch your Twilio auth token and set the TWILIO_AUTH_TOKEN with it.

Bonus

Now you're thinking: "Wait a minute, do I need to use my phone to debug this? Doesn't this like, cost money?" Yes! You're also right, texting a number provided by Twilio costs money. For Autopilot, they provide a simulator; which is great for debugging. But, unfortunately, the simulator does not send the UserIdentifier property; since is not a phone.

But fear not! I have you covered as well. Remember that FAKE_NUMBER property? Let's use that now, back our AutopilotRoute:

# The imports

class AutopilotRequest(Request):
    # The huge docstring

    async def form(self):
        if not config.DEBUG:
            return await self.__handle_non_dev_env()
        return await self.__handle_dev_env()

    async def __handle_dev_env(self):
        """
        Here we just inject a fake number for testing, this so we can test from
        the Twilio Autopilot Simulator through an SSH tunnel.
        """
        form = await super().form()
        new_form = FormData(dict(form.items()), UserIdentifier=config.FAKE_NUMBER)
        return new_form

    # __handle_non_dev_env goes here
# Our router goes here

Now, we have added a __handle_dev_env method that gets executed when DEBUGis set to True. And whats does it do? Well, it takes whatever the request was and injects the UserIdentifier property as whatever number we specified in FAKE_NUMBER. Now behold!

So, for debugging, just set DEBUG to True and set FAKE_NUMBER to a valid phone number. Pro-tip: you can use this with something like Ngrok to debug your API locally.

Wrapping up

That's all folks! I hope you liked the series. I certainly did learn a lot while doing this. Before finishing up let me state that this is all very basic, it could be improved in so many ways:

• Needs error handling
• While the Twilio default types do a good job in recognizing common language, the bot still needs to be trained using samples from real conversations
• Needs better automation for deployment
• So much more!

If you want to check out how I solved this things (and also check a slightly more complex bot) you can check out the original bot I was developing before writing this post.

PS: Oh, also, here's the code for this post, did you think I would forget? ;)