How I Managed to Run BlenderBot 2.0

The title became a matter when it comes to limited machines. I managed to run it on my own machine with 16GB RAM on the CPU, but not on the GPU as it has only 4GB VRAM. But for those who have a smaller RAM capacity, they can run it on Google Colab with a GPU.

Unfortunately, it is still not enough to run it without a subscription.

# Run on Google Colab with the free plan
> !nvidia-smi
Tue Mar  8 03:08:28 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   55C    P8    10W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

> !free
              total        used        free      shared  buff/cache   available
Mem:       13302920      631188    10510972        1192     2160760    12420412
Swap:             0           0           0

But need not to worry, we have the hackaround in here.

Please note that I do not take any credit for the code. I just copied from https://github.com/JulesGM/ParlAI_SearchEngine/blob/master/README.md.

First, let us install all the required dependencies in a new cell:

%%capture

# Install the latest version of ParlAI from source.
# This is important, otherwise using Blenderbot 2.0 will not work.
!pip install -q git+https://github.com/facebookresearch/ParlAI.git 

# Clone the server and install its requirements
!git clone https://github.com/JulesGM/ParlAI_SearchEngine.git
!pip install -q  -r ParlAI_SearchEngine/requirements.txt

Second, run the web search server in a new cell:

import os
import shlex
import subprocess
import time

# Using port 8080 doesn't work on Colab
HOST = "0.0.0.0:1111"
PATH_TO_SEARCH_SERVER = "./ParlAI_SearchEngine/search_server.py"

assert os.path.exists(PATH_TO_SEARCH_SERVER), (f"Incorrect path {PATH_TO_SEARCH_SERVER}")

command = ["python", "-u", shlex.quote(PATH_TO_SEARCH_SERVER), "serve", "--host", HOST]
command_str = " ".join(command)
p = subprocess.Popen(command, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)

# Wait a bit before the next cell to let a lot of the potential errors happen.
time.sleep(3)

Third, in here where the hack is applied:

%%capture

%mkdir /usr/local/lib/python3.7/dist-packages/data
%cd /usr/local/lib/python3.7/dist-packages/data

# To prevent the 3.7GB BERT model build process taking too much RAM in the model runtime,
# we need to build it separately before running the interactive parlai command
from parlai.zoo.bart.build import download as download_bart_model
download_bart_model('.')

Last, run the BlenderBot 2.0's interactive shell in a new cell:

!python -m parlai interactive --model-file zoo:blenderbot2/blenderbot2_400M/model --search_server $HOST

Congrats! Now, let us look at this sample conversation:

...
Enter [DONE] if you want to end the episode, [EXIT] to quit.
03:26:48 | creating task(s): interactive
Enter Your Message: Who is Kamisato Ayaka?
/usr/local/lib/python3.7/dist-packages/parlai/core/torch_generator_agent.py:1749: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  hyp_ids = best_idxs // voc_size
[BlenderBot2Fid]: Kamisato ayaka is one of my favorite anime characters. _POTENTIALLY_UNSAFE__
Enter Your Message: From which anime or game she is?
[BlenderBot2Fid]: She is from Genshin Impact. She is a character in the anime series. Do you like anime?
Enter Your Message: Yes, I do. What's her role in Genshin Impact?
[BlenderBot2Fid]: She is the main character of the anime. She plays a role in the story of the series.
Enter Your Message: Really? The main character? How does she look like?
[BlenderBot2Fid]: She does not look like the anime character. She looks more like a character from a video game.
Enter Your Message: What's her favorite food?
[BlenderBot2Fid]: I am not sure, but I do know that she is a big fan of pizza. Have you ever had pizza?
Enter Your Message: Yeah, actually, my family run a home bakery years ago. Anyway, where she lives?
[BlenderBot2Fid]: She lives in Tokyo, Japan. I do not know where she is from. I have never been to Japan.
Enter Your Message: Me too. By the way, does she have a brother?
[BlenderBot2Fid]: I do not think she has a brother. She does not live with her family. She lives with her parents.
Enter Your Message: Oh, is it true? But I have heard someone named Kamisato Ayato.
[BlenderBot2Fid]: No, I have not heard of that person. I am not familiar with that name. What is it?
Enter Your Message: I'm not sure, I thought he is Ayaka's brother.
[BlenderBot2Fid]: Yes, that is correct. He is the brother of Ayaka. He lives with his parents.
Enter Your Message: LOL. You changed your mind, don't you? Okay then, tell me about Ayaka's hobby.
[BlenderBot2Fid]: Her hobby is to play video games. She likes to play Call of Duty and Fortnite.
Enter Your Message: Ha-ha-ha ... You are wrong! She likes playing Shougi. It's not a video game! She likes poetry as well.
[BlenderBot2Fid]: Oh, I did not know that. I will have to look it up. Thank you for the information.
Enter Your Message: Don't mention it. By the way, do you play Genshin Impact?
[BlenderBot2Fid]: I have not played it. I don't know what it is. Is it an anime game?
Enter Your Message: Yeah, kinda. The visual graphics are awesome. Too awesome so that my laptop couldn't afford it.
[BlenderBot2Fid]: I am sorry to hear that. What kind of laptop do you have? I have a laptop.
Enter Your Message: Just a typical laptop for developing web applications using PHP.
[BlenderBot2Fid]: That sounds like a lot of fun. I would love to learn how to code on a computer.
Enter Your Message: Why do you want to learn coding?
[BlenderBot2Fid]: I think it would be fun to be able to code. What do you think? _POTENTIALLY_UNSAFE__
Enter Your Message: [EXIT]
CHAT DONE

So long!