This pandemic has taken a huge toll on my mental and emotional health. In order to keep me occupied and brighten up the lives of those around me, I started on yet another Python project — this time, a WhatsApp bot that sends me random cat pictures, trending memes, the best cooking recipes, and of course, the latest world news and COVID19 statistics.
The full project can be found on my GitHub repository, and my webhook is live on https://zeyu2001.pythonanywhere.com/bot/.
Prerequisites
We will be using Python, the Django web framework, ngrok and Twilio to create this chatbot. I will show you how to install the required packages, but you need to have Python (3.6 or newer) and a smartphone with an active phone number and WhatsApp installed.
Following Python best practices, we will create a virtual environment for our project, and install the required packages.
First, create the project directory.
$ mkdir whatsapp-bot
$ cd whatsapp-bot
Now, create a virtual environment and install the required packages.
For macOS and Unix systems:
$ python3 -m venv whatsapp-bot-venv
$ source whatsapp-bot-venv/bin/activate
(whatsapp-bot-venv) $ pip install twilio django requests
For Windows:
$ python3 -m venv whatsapp-bot-venv
$ whatsapp-bot-venv\Scripts\activate
(whatsapp-bot-venv) $ pip install twilio django requests
Configuring Twilio
You will need a free Twilio account, which allows you to use a Twilio number as your WhatsApp bot. A free account comes with a trial balance that will be enough to send and receive messages for weeks to come. If you wish to continue using your bot after your trial balance is up, you can top up your account.
You won’t be able to use your own number unless you obtain permission from WhatsApp, but the Twilio number would be good enough for this project. You will need to set up your Twilio sandbox here by sending a WhatsApp message to the Twilio number. This has to be done once and only once.
Setting Up Your Webhook
Twilio uses what is called a webhook to communicate with our application. Our chatbot application would need to define an endpoint to be configured as this webhook so that Twilio can communicate with our application.
Django is a web framework that allows us to do just that. Although the Django vs. Flask debate can go on for eternity, I chose to use Django simply because I have just started using it a few weeks ago and I wanted to get used to using it. You can use Flask to achieve the same thing, but the code would be different.
First, navigate to your whatsapp-bot directory and establish a Django project.
(whatsapp-bot-venv) $ django-admin startproject bot
This will auto-generate some files for your project skeleton:
bot/
manage.py
bot/
__init__.py
settings.py
urls.py
asgi.py
wsgi.py
Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.
(whatsapp-bot-venv) $ python manage.py startapp bot_app
This will create the following:
bot_app/
__init__.py
admin.py
apps.py
migrations/
__init__.py
models.py
tests.py
views.py
For the sake of this chatbot alone, we won’t need most of these files. They will only be relevant if you decide to expand your project into a full website.
What we need to do is to define a webhook for Twilio. Your views.py file processes HTTP requests and responses for your web application. Twilio will send a POST request to your specified URL, which will map to a view function, which will return a response to Twilio.
from twilio.twiml.messaging_response import MessagingResponse
from django.views.decorators.csrf import csrf_exempt
@csrf_exempt
def index(request):
if request.method == 'POST':
# retrieve incoming message from POST request in lowercase
incoming_msg = request.POST['Body'].lower()
# create Twilio XML response
resp = MessagingResponse()
msg = resp.message()
This creates an index view, which will process the Twilio POST requests. We retrieve the message sent by the user to the chatbot and turn it into lowercase so that we do not need to worry about whether the user capitalizes his message.
Twilio expects a TwiML (an XML-based language) response from our webhook. MessagingResponse() creates a response object for this purpose.
resp = MessagingResponse()
msg = resp.message()
msg.body('My Response')
msg.media('https://example.com/path/image.jpg')
Doing this would create a response consisting of both text and media. Note that the media has to be in the form of a URL, and must be publicly accessible.
from twilio.twiml.messaging_response import MessagingResponse
from django.views.decorators.csrf import csrf_exempt
from django.http import HttpResponse
@csrf_exempt
def index(request):
if request.method == 'POST':
# retrieve incoming message from POST request in lowercase
incoming_msg = request.POST['Body'].lower()
# create Twilio XML response
resp = MessagingResponse()
msg = resp.message()
if incoming_msg == 'hello':
response = "*Hi! I am the Quarantine Bot*"
msg.body(response)
return HttpResponse(str(resp))
With this knowledge, we can now return a HttpResponse that tells Twilio to send the message “Hi! I am the Quarantine Bot” back to the user. The asterisks (*) are for text formatting — WhatsApp will bold our message.
This won’t work unless we link it to a URL. In the bot_app directory, create a file urls.py. Include the following code:
from django.urls import path
from . import views
urlpatterns = [
path('', views.index, name='index'),
]
Now, we need the root URLconf to point to our bot_app/urls.py. In bot/urls.py, add the following code:
from django.contrib import admin
from django.urls import include, path
urlpatterns = [
path('bot/', include('bot_app.urls')),
path('admin/', admin.site.urls),
]
The include() function allows referencing other URLconfs. Whenever Django encounters include(), it chops off whatever part of the URL matched up to that point and sends the remaining string to the included URLconf for further processing.
When Twilio sends a POST request to bot/, it will reference bot_app.urls, which references views.index, where the request will be processed.
Testing It Works
Make sure you are in the directory with manage.py, and run
(whatsapp-bot-venv) $ python manage.py runserver
You should see the port that your Django application is running on. In this screenshot, it is port 8000. But this is still running from your computer. To make this service reachable from the Internet we need to use ngrok.
Open a second terminal window, and run
$ ngrok http 8000
The lines beginning with forwarding tell you the public URL ngrok uses to redirect requests to your computer. In this screenshot, https://c21d2af6.ngrok.io is redirecting requests to my computer on port 8000. Copy this URL, and go back to your Twilio console.
Paste the URL into the “When a message comes in” field. Set the request method to HTTP post.
If you want to use my app, use https://zeyu2001.pythonanywhere.com/bot/ instead for the “When a message comes in” field.
In your settings.py, you also need to add your ngrok URL as one of the ALLOWED_HOSTS.
Now you can start sending messages to the chatbot from your smartphone that you connected to the sandbox at the start of this tutorial. Try sending ‘hello’.
Adding Third-Party APIs
In order to accomplish most of our features, we have to use third-party APIs. For instance, I used Dog CEO’s Dog API to get a random dog image every time the user sends the word ‘dog’.
import requests
...
elif incoming_msg == 'dog':
# return a dog pic
r = requests.get('https://dog.ceo/api/breeds/image/random')
data = r.json()
msg.media(data['message'])
...
requests.get(url) sends a GET request to the specified URL and returns the response. Since the response is in JSON, we can use r.json() to convert it into a Python dictionary. We then use msg.media() to add the dog picture to the response.
My Final Code
My final chatbot includes the following features:
- Random cat image
- Random dog image
- Random motivational quote
- Fetching recipes from Allrecipes
- Latest world news from various sources
- Latest COVID19 statistics for each country
- Trending memes from r/memes subreddit
| from twilio.twiml.messaging_response import MessagingResponse | |
| from django.views.decorators.csrf import csrf_exempt | |
| from django.http import HttpResponse | |
| import requests | |
| import datetime | |
| import emoji | |
| import random | |
| import json | |
| @csrf_exempt | |
| def index(request): | |
| if request.method == 'POST': | |
| # retrieve incoming message from POST request in lowercase | |
| incoming_msg = request.POST['Body'].lower() | |
| # create Twilio XML response | |
| resp = MessagingResponse() | |
| msg = resp.message() | |
| responded = False | |
| if incoming_msg == 'hello': | |
| response = emoji.emojize(""" | |
| *Hi! I am the Quarantine Bot* :wave: | |
| Let's be friends :wink: | |
| You can give me the following commands: | |
| :black_small_square: *'quote':* Hear an inspirational quote to start your day! :rocket: | |
| :black_small_square: *'cat'*: Who doesn't love cat pictures? :cat: | |
| :black_small_square: *'dog'*: Don't worry, we have dogs too! :dog: | |
| :black_small_square: *'meme'*: The top memes of today, fresh from r/memes. :hankey: | |
| :black_small_square: *'news'*: Latest news from around the world. :newspaper: | |
| :black_small_square: *'recipe'*: Searches Allrecipes.com for the best recommended recipes. :fork_and_knife: | |
| :black_small_square: *'recipe <query>'*: Searches Allrecipes.com for the best recipes based on your query. :mag: | |
| :black_small_square: *'get recipe'*: Run this after the 'recipe' or 'recipe <query>' command to fetch your recipes! :stew: | |
| :black_small_square: *'statistics <country>'*: Show the latest COVID19 statistics for each country. :earth_americas: | |
| :black_small_square: *'statistics <prefix>'*: Show the latest COVID19 statistics for all countries starting with that prefix. :globe_with_meridians: | |
| """, use_aliases=True) | |
| msg.body(response) | |
| responded = True | |
| elif incoming_msg == 'quote': | |
| # returns a quote | |
| r = requests.get('https://api.quotable.io/random') | |
| if r.status_code == 200: | |
| data = r.json() | |
| quote = f'{data["content"]} ({data["author"]})' | |
| else: | |
| quote = 'I could not retrieve a quote at this time, sorry.' | |
| msg.body(quote) | |
| responded = True | |
| elif incoming_msg == 'cat': | |
| # return a cat pic | |
| msg.media('https://cataas.com/cat') | |
| responded = True | |
| elif incoming_msg == 'dog': | |
| # return a dog pic | |
| r = requests.get('https://dog.ceo/api/breeds/image/random') | |
| data = r.json() | |
| msg.media(data['message']) | |
| responded = True | |
| elif incoming_msg.startswith('recipe'): | |
| # search for recipe based on user input (if empty, return featured recipes) | |
| search_text = incoming_msg.replace('recipe', '') | |
| search_text = search_text.strip() | |
| data = json.dumps({'searchText': search_text}) | |
| result = '' | |
| # updates the Apify task input with user's search query | |
| r = requests.put('https://api.apify.com/v2/actor-tasks/o7PTf4BDcHhQbG7a2/input?token=qTt3H59g5qoWzesLWXeBKhsXu&ui=1', data = data, headers={"content-type": "application/json"}) | |
| if r.status_code != 200: | |
| result = 'Sorry, I cannot search for recipes at this time.' | |
| # runs task to scrape Allrecipes for the top 5 search results | |
| r = requests.post('https://api.apify.com/v2/actor-tasks/o7PTf4BDcHhQbG7a2/runs?token=qTt3H59g5qoWzesLWXeBKhsXu&ui=1') | |
| if r.status_code != 201: | |
| result = 'Sorry, I cannot search Allrecipes.com at this time.' | |
| if not result: | |
| result = emoji.emojize("I am searching Allrecipes.com for the best {} recipes. :fork_and_knife:".format(search_text), | |
| use_aliases = True) | |
| result += "\nPlease wait for a few moments before typing 'get recipe' to get your recipes!" | |
| msg.body(result) | |
| responded = True | |
| elif incoming_msg == 'get recipe': | |
| # get the last run details | |
| r = requests.get('https://api.apify.com/v2/actor-tasks/o7PTf4BDcHhQbG7a2/runs/last?token=qTt3H59g5qoWzesLWXeBKhsXu') | |
| if r.status_code == 200: | |
| data = r.json() | |
| # check if last run has succeeded or is still running | |
| if data['data']['status'] == "RUNNING": | |
| result = 'Sorry, your previous query is still running.' | |
| result += "\nPlease wait for a few moments before typing 'get recipe' to get your recipes!" | |
| elif data['data']['status'] == "SUCCEEDED": | |
| # get the last run dataset items | |
| r = requests.get('https://api.apify.com/v2/actor-tasks/o7PTf4BDcHhQbG7a2/runs/last/dataset/items?token=qTt3H59g5qoWzesLWXeBKhsXu') | |
| data = r.json() | |
| if data: | |
| result = '' | |
| for recipe_data in data: | |
| url = recipe_data['url'] | |
| name = recipe_data['name'] | |
| rating = recipe_data['rating'] | |
| rating_count = recipe_data['ratingcount'] | |
| prep = recipe_data['prep'] | |
| cook = recipe_data['cook'] | |
| ready_in = recipe_data['ready in'] | |
| calories = recipe_data['calories'] | |
| result += """ | |
| *{}* | |
| _{} calories_ | |
| Rating: {:.2f} ({} ratings) | |
| Prep: {} | |
| Cook: {} | |
| Ready in: {} | |
| Recipe: {} | |
| """.format(name, calories, float(rating), rating_count, prep, cook, ready_in, url) | |
| else: | |
| result = 'Sorry, I could not find any results for {}'.format(search_text) | |
| else: | |
| result = 'Sorry, your previous search query has failed. Please try searching again.' | |
| else: | |
| result = 'I cannot retrieve recipes at this time. Sorry!' | |
| msg.body(result) | |
| responded = True | |
| elif incoming_msg == 'news': | |
| r = requests.get('https://newsapi.org/v2/top-headlines?sources=bbc-news,the-washington-post,the-wall-street-journal,cnn,fox-news,cnbc,abc-news,business-insider-uk,google-news-uk,independent&apiKey=3ff5909978da49b68997fd2a1e21fae8') | |
| if r.status_code == 200: | |
| data = r.json() | |
| articles = data['articles'][:5] | |
| result = '' | |
| for article in articles: | |
| title = article['title'] | |
| url = article['url'] | |
| if 'Z' in article['publishedAt']: | |
| published_at = datetime.datetime.strptime(article['publishedAt'][:19], "%Y-%m-%dT%H:%M:%S") | |
| else: | |
| published_at = datetime.datetime.strptime(article['publishedAt'], "%Y-%m-%dT%H:%M:%S%z") | |
| result += """ | |
| *{}* | |
| Read more: {} | |
| _Published at {:02}/{:02}/{:02} {:02}:{:02}:{:02} UTC_ | |
| """.format( | |
| title, | |
| url, | |
| published_at.day, | |
| published_at.month, | |
| published_at.year, | |
| published_at.hour, | |
| published_at.minute, | |
| published_at.second | |
| ) | |
| else: | |
| result = 'I cannot fetch news at this time. Sorry!' | |
| msg.body(result) | |
| responded = True | |
| elif incoming_msg.startswith('statistics'): | |
| # runs task to aggregate data from Apify Covid-19 public actors | |
| requests.post('https://api.apify.com/v2/actor-tasks/5MjRnMQJNMQ8TybLD/run-sync?token=qTt3H59g5qoWzesLWXeBKhsXu&ui=1') | |
| # get the last run dataset items | |
| r = requests.get('https://api.apify.com/v2/actor-tasks/5MjRnMQJNMQ8TybLD/runs/last/dataset/items?token=qTt3H59g5qoWzesLWXeBKhsXu') | |
| if r.status_code == 200: | |
| data = r.json() | |
| country = incoming_msg.replace('statistics', '') | |
| country = country.strip() | |
| country_data = list(filter(lambda x: x['country'].lower().startswith(country), data)) | |
| if country_data: | |
| result = '' | |
| for i in range(len(country_data)): | |
| data_dict = country_data[i] | |
| last_updated = datetime.datetime.strptime(data_dict.get('lastUpdatedApify', None), "%Y-%m-%dT%H:%M:%S.%fZ") | |
| result += """ | |
| *Statistics for country {}* | |
| Infected: {} | |
| Tested: {} | |
| Recovered: {} | |
| Deceased: {} | |
| Last updated: {:02}/{:02}/{:02} {:02}:{:02}:{:03} UTC | |
| """.format( | |
| data_dict['country'], | |
| data_dict.get('infected', 'NA'), | |
| data_dict.get('tested', 'NA'), | |
| data_dict.get('recovered', 'NA'), | |
| data_dict.get('deceased', 'NA'), | |
| last_updated.day, | |
| last_updated.month, | |
| last_updated.year, | |
| last_updated.hour, | |
| last_updated.minute, | |
| last_updated.second | |
| ) | |
| else: | |
| result = "Country not found. Sorry!" | |
| else: | |
| result = "I cannot retrieve statistics at this time. Sorry!" | |
| msg.body(result) | |
| responded = True | |
| elif incoming_msg.startswith('meme'): | |
| r = requests.get('https://www.reddit.com/r/memes/top.json?limit=20?t=day', headers = {'User-agent': 'your bot 0.1'}) | |
| if r.status_code == 200: | |
| data = r.json() | |
| memes = data['data']['children'] | |
| random_meme = random.choice(memes) | |
| meme_data = random_meme['data'] | |
| title = meme_data['title'] | |
| image = meme_data['url'] | |
| msg.body(title) | |
| msg.media(image) | |
| else: | |
| msg.body('Sorry, I cannot retrieve memes at this time.') | |
| responded = True | |
| if not responded: | |
| msg.body("Sorry, I don't understand. Send 'hello' for a list of commands.") | |
| return HttpResponse(str(resp)) |
Deploying Your Webhook
Once you’re done coding, you probably want to deploy your webhook somewhere so that it runs 24/7. PythonAnywhere provides a free Django-friendly hosting service.
In order to deploy your project on PythonAnywhere, upload your project to GitHub and follow the documentation to set up your web app.
Once deployed, update your Twilio sandbox configuration so that the webhook is set to your new URL (such as https://zeyu2001.pythonanywhere.com/bot/).
Thanks for Reading!
Now you know how to set up your very own WhatsApp chatbot with Twilio. If you have any questions, please feel free to comment on this post.
Top comments (22)
This is a really nice post! I tried to implement my own version of it, but have run into errors while sending requests to Twilio. The logs say that my ngrok URL is returning a 404. I've done exactly what you have and thus this is very annoying.
Do you have any ideas as to what the issue may be?
Hi there, thanks for reading :)
Can you check whether the URL you set in the Twilio sandbox configuration includes the /bot/ at the end? In our
urls.py, we have configured the application on the /bot/ URL. So if the nrgok URL is c21d2af6.ngrok.io, you would have to use c21d2af6.ngrok.io/bot/.Alternatively, you could change this by editing this line in
urls.pyto
Let me know if this helps!
Hi, thanks for the quick response.
It still doesn't work, unfortunately. The URLs change every time the server is started, yes? I tried to play around with that and added /bot/ at the end, but it still doesn't seem to work for me.
/bot/ is added only in the Twilio config, yes? Or does that go in the python file too. It throws an error rn, but I just want to be sure.
Yup, the ngrok URLs change every time you start ngrok, and /bot/ is added only in the Twilio config.
Have you added the ngrok URL to your
ALLOWED_HOSTSinsettings.py?Is Python raising any errors or is it just a 404? If it's just a 404, you might want to check your
urls.pyfiles in both the bot and the bot_app folder.The source code is available at my GitHub, so you can check against that.
Perfect, I'll compare my code against your github repo and then let you know if something's up.
Thanks!
Hi Zhang Zeyu,
Congrats! Great tutorial, I am learning a lot. I have an error:
"requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.apify', port=443): Max retries exceeded with url:..."
I think it is related to a restriction about apify.com.
Please could you tell how to solve this? Could you suggest a tutorial to start with Apify?
Thanks, nice job! Greetings from Ecuador!
hey great post,I just want to know what these part of codes do:
1>data['data']['status'] == "SUCCEEDED"
what value does ['data'] and ['status'] contains.
2>memes = data['data']['children']:
same for this one also what values do ['children'] contain.
Thanks Zeyu, my bot is working! I did it!
Congrats! Sorry for the late reply, but to answer your previous question, the old Python requests library has an unintuitive error message. There was actually an issue raised on the GitHub repo of the requests library about this (check out github.com/psf/requests/issues/1198).
The error says "Max retries exceeded" when it has nothing to do with retrying. What happens is that the requests library "wraps" around urllib3's HTTPConnectionPool function, and the original exception message raised by urllib3 was shown as part of the error raised by the requests library.
Indeed this can be very confusing for end-users who don't know the source code of the requests library.
This seems to be an old issue and has since been fixed, so I suggest updating your requests library to the newest version for more meaningful error messages.
(In short, it's not an issue with Apify. Rather, it's an unintuitive error message telling you that your connection was refused - this could happen for any number of reasons, including a typo in your URL. Cheers!)
Thansk Zeyu. I fixed my error creating my own urls in apify. You are right because the links worked well in isolation. Indeed, the errors are really hard to debug because the message is not clear.
Have a nice weekend!
Tatiana
Nice post! :)
isn't twilio too expensive for whatsapp API?
Thank you! Perhaps, but I think it's good enough for a fun personal side project without having to register for a WhatsApp business account :)
Hey! Nice post! It's very complete
I have only one question: Is it normal to get a screen like in the pictures that I am linking with this comment? It's because I don´t know if it is normal to have the server running correctly and getting no response in my smartphone when I type "hello" but in the ngrok I got an "OK" and in the console I got "POST / HTTP/1.1" 200 16351
I let the links from my screenshots showing this. Thank you
dev-to-uploads.s3.amazonaws.com/i/...
dev-to-uploads.s3.amazonaws.com/i/...
Hey there! It looks like your server is running correctly, since it is receiving POST requests from Twilio and responding with a 200 OK status code. Let's narrow down the problem here. Perhaps your server is responding as intended to Twilio, but the
HttpResponseis not in the correct format accepted by Twilio? Make sure you are using the TwilioMessagingResponseobject correctly. Can you also check your Twilio debugger (twilio.com/console/debugger/) for any error messages? I'm fairly certain the error has to do with how Twilio interprets your server's response, so you might find some useful information in the Twilio debugger. Keep me posted!Nice post! It is really informative!
Thank you!
Great post!
I have been trying to try building a Whatsapp Bot.
Will this bot work on a group chat?
So I found a meme API, but I want to implement it on your code. How to do it?