DEV Community: Twilio

Create AI-generated art via SMS with Replicate in Python

Lizzie Siegle — Thu, 21 Sep 2023 21:22:37 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

It's fun to get creative! Read on to learn how to create AI-generated images via SMS using Replicate and Twilio Programmable Messaging.

Do you prefer learning via video more? Check out this TikTok summarizing this tutorial.

Prerequisites

A Twilio account - sign up for a free Twilio account here
A Twilio phone number with SMS capabilities - learn how to buy a Twilio Phone Number here
Replicate account to use the Stable Diffusion text-to-image model – make a Replicate account here
Python installed - download Python here
ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access.

⚠️ ngrok is needed for the development version of the application because your computer is likely behind a router or firewall, so it isn’t directly reachable on the Internet. You can also choose to automate ngrok as shown in this article.

Replicate

Replicate offers a cloud API and tooling so you can more easily run machine learning models, abstracting away some lower-level AI concepts and handling infrastructure so you can focus more on your own applications. You can run open-source models that others have published, or package and publish your own, either publicly or privately.

Configuration

Since you will be installing some Python packages for this project, you need to make a new project directory and a virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir replicate-prompt-to-image-sms 
cd replicate-prompt-to-image-sms 
python3 -m venv venv 
source venv/bin/activate 
pip install replicate flask twilio

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir replicate-prompt-to-image-sms  
cd replicate-prompt-to-image-sms  
python -m venv venv 
venv\Scripts\activate 
pip install replicate flask twilio

Grab your default Replicate API Token or create a new one here. Run this command in the terminal in your current folder:

export REPLICATE_API_TOKEN={replace with your api token}

Now it's time to write some code to receive a prompt via SMS and return an AI-generated image!

Turn a Prompt into an AI-generated image via SMS with Replicate

Make a file called app.py and place the following import statements at the top.

import replicate
from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse

Next, make a Flask app so your app can receive the inbound text message to your Twilio phone number. That string is then passed to Replicate 's stable diffusion model to generate photo-realistic images given that prompt.

app = Flask(__name__)
@app.route('/sms', methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body'].lower().strip()
    output = replicate.run(
        "stability-ai/stable-diffusion:27b93a2413e7f36cd83da926f3656280b2931564ff050bf9575f1fdf9bcd7478",
        input={"prompt": inb_msg}
        )
    res_prompt = "prompt was " + inb_msg
    msg = resp.message(res_prompt)
    img = output[0]
    msg.media(img)
    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

On the command line, run python app.py to start the Flask app. Now it's time to set up a Twilio phone number so you can text it a prompt and get an AI-generated image back!

Configure a Twilio Number for the SMS Chatbot

You should see the screen above. Grab that ngrok Forwarding URL to configure your Twilio number: select your Twilio number under Active Numbers in your Twilio console, scroll to the Messaging section, and then modify the phone number’s routing by pasting the ngrok URL with the /sms path in the textbox corresponding to when A Message Comes In as shown below:

Click Save and now your Twilio phone number is configured so that it maps to your web application server running locally on your machine and your application can run. Text a prompt of a picture you'd like to generate to your Twilio number and you should get an AI-generated image of whatever you sent over SMS!

Instead of ngrok, you could alternatively use another tunneling service like Tailscale–my teammate Rishab reviewed this post, and kept getting a 403 error with ngrok. The solution was to switch port numbers with ngrok–it seems that Tailscale has more privileges to the network settings compared to ngrok. You can view the complete code on GitHub here.

What's Next for Replicate and Twilio Programmable Messaging?

There is so much fun for developers to have around building with LLMs! Replicate offers so many models to play with like sdxl, a text-to-image generative AI model that creates beautiful 1024x1024 images, or you can fine-tune your own. I can't wait to see what you build with AI–let me know online what you're working on!

Build an AI Personal Trainer with LangChain Agents and SendGrid

Lizzie Siegle — Tue, 19 Sep 2023 19:53:32 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

Can AI help me run a faster marathon?

Problem: I play a lot of tennis and run, bike, and walk as my forms of transit, so I have never trained for a half marathon or 10k. However, I KNOW I must train to run a marathon. As a developer, I turned to code, wondering how AI could help me run a faster marathon.

Solution: Read on to learn how to build a personal marathon training plan generator Streamlit-hosted app from personal workout data using the Strava API, LangChain agents, chains, few-shot prompt templating, PromptTemplates, OpenAI, and SendGrid. This can also be applied to personal training, nutrition, and more!

In the Streamlit page above, a user inputs their Strava API token so the app can access their workout data, the date they want to begin training, and the marathon (or race) day. When they click Submit, a custom training plan is generated.

You can test the app out here using a Strava API token generated by following the directions under Strava Activity Webhook Authentication in this tutorial on querying data with LangChain agents.

Do you prefer learning via video more? Check out this TikTok relating to this tutorial--massive thank you to Twilio Community Engagement Manager Sandra Mendez for spearheading this video.

Understanding LangChain Agents

This meaty app uses many features of LangChain, such as agents: agents use LLMs to decide what actions should be taken and they have access to tools like web search or calculators which can be packaged into a logical loop of operations.

For more background on LangChain, you can read this blog post here.

Prerequisites

A free SendGrid account
Python installed - download Python here
An email address to test out this project
OpenAI Account – make an OpenAI Account here
A Strava account - sign up for a Strava account if you don't have one already
A Strava app and API key - follow directions here under Setup the Strava API
A Strava Auth token - follow directions here under Strava Activity Webhook Authentication You need a Strava app and API key–following the directions above will provide you with the necessary Strava Client Secret in order to generate a Strava Access Token to make a request to access your Strava data. Different Strava API app values include:
Client ID: Your application ID
Client Secret: Your client secret (please keep this confidential)
Authorization/Access token: Your Strava API app's authorization token will need to be regenerated every six hours (should be confidential) Next, after making an OpenAI account if you don't have one already, you can get an OpenAI API Key here by clicking on + Create new secret key. Then, grab a SendGrid API key here.

The Python application will need to have access to these keys, so in your root directory (called strava-langchain) make a .env file to store the API keys safely. The application we create will be able to import this key as an environment variable soon.

In the .env file add the following lines of text, making sure to replace <YOUR-OPENAI-API-KEY> and <YOUR-SENDGRID-KEY> with your actual keys. STRAVA_CLIENT_SECRET should have been used above to generate a Strava Access Token:

OPENAI_API_KEY= <YOUR-OPENAI-KEY>
SENDGRID_API_KEY = <YOUR-SENDGRID-KEY>
STRAVA_CLIENT_SECRET= <YOUR-STRAVA-CLIENT-SECRET>

Your Strava Access Token will be placed into a Streamlit form. Streamlit is an open-source Python app framework that helps developers quickly create web apps for data science and machine learning.

In the root directory run the following commands to install the project's required libraries after making a virtual environment:

python3 -m venv venv 
source venv/bin/activate

Make a requirements.txt file containing the following:

chromadb==0.4.5
google-search-results==2.4.2
langchain==0.0.261
numpy==1.25.0
openai==0.27.2
pandas==2.0.3
python-dotenv==1.0.0
PyYAML==6.0
reportlab==4.0.4
Requests==2.31.0
sendgrid==6.10.0
streamlit==1.25.0
tabulate==0.9.0
urllib3==2.0.4

Install them on the command line by running pip install -r requirements.txt.

Then run mkdir data_files to make a folder called data_files to store the activity data.

We will import these in our code soon.

Few-shot prompting Examples

There are a few ways one could go about making an AI personal marathon trainer, workout trainer, or recommendation system. My college classmate-turned Redis AI engineer Sam Partee suggested using few-shot prompting, in which a few explicit examples (or "shots") guide the Large Language Model (LLM) to respond in a certain way.

Create a file named stapp.py and include some examples of marathon training plans. I used OpenAI to create these plans and then formatted them a bit because most marathon plans I found online were not copy-and-pasteable.

examples = [
  {
    "training_start_date" : "July 19",
    "marathon_date" : "November 8",
    "pd_run_output": "The average distance is 3.71 miles and the maximum distance is 9.52 miles. The average moving time is not available and the maximum moving time is not available. Other running statistics include total elevation gain, with an average of 47.6 and a maximum of 200.6.",
    "num_workouts": "112",
    "plan": 
"""
Week 1: 
Day 1, {training_start_date}: Cross-training - Swim
Day 2: Easy run - 4 miles at an easy pace
Day 3: Cross-training - Elliptical 
Day 4: Medium run - 5 miles at a medium pace 
...
{marathon_date}: Marathon Day!
Note: The above plan assumes that the student is currently able to comfortably run 4 miles at an easy pace. Adjustments may need
to be made if the student is not yet at this level. Additionally, the plan includes cross-training and rest days to
ensure proper recovery and prevent injury.
"""
  }
]

The complete code for that file is here on GitHub. Now let's make some custom tools to help LangChain suggest personalized workouts.

Make Custom Tools to assist LangChain Agents

Using agents allows LLMs access to tools with which LLMs can search the web, do math, run code, and more. Though LLMs are powerful, they still have trouble with certain tasks–that's why you may sometimes need to make custom tools.

LangChain offers a large selection of pre-built tools, but only so many problems can be solved using existing tools.

Create a file named request.py and include the following import statements.

import re
from datetime import date, datetime
import os
import base64
from typing import List, Union
import pandas as pd  # dataframe

from reportlab.platypus import SimpleDocTemplate, Paragraph
from reportlab.lib.pagesizes import letter
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.lib.units import inch

import requests
import urllib3
from dotenv import dotenv_values
import streamlit as st

from langchain import LLMChain, LLMMathChain
from langchain.agents import Tool, AgentExecutor, LLMSingleActionAgent, AgentOutputParser, create_pandas_dataframe_agent, initialize_agent, AgentType
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI  # , GPT4All
from langchain.prompts import BaseChatPromptTemplate
from langchain.prompts.few_shot import FewShotPromptTemplate
from langchain.prompts.prompt import PromptTemplate
from langchain.schema import AgentAction, AgentFinish, HumanMessage

from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import (
    Mail, Attachment, FileContent, FileName, FileType, Disposition)

import stapp  # file
urllib3.disable_warnings(
    urllib3.exceptions.InsecureRequestWarning
)
config = dotenv_values(".env")

Next, we set some variables such as the Strava URL to retrieve a user's Activity data, have the app access its API keys, and then chain LLMs together: the llm variable passes the desired gpt-4-32k model to use with ChatOpenAI and sets the temperature to 0.2–play around with this value. A higher temperature value means the LLM output will include more randomness. Lastly, pass LLM to a math chain to perform complex math problems.

activities_url = "https://www.strava.com/api/v3/athlete/activities"
os.environ["OPENAI_API_KEY"] = config.get('OPENAI_API_KEY')
os.environ["SENDGRID_API_KEY"] = config.get('SENDGRID_API_KEY')
llm = ChatOpenAI(model_name='gpt-4-32k', temperature=0.2)
llm_math_chain = LLMMathChain.from_llm(llm=llm, verbose=True)

Now make some functions for the custom tools to help the LLM convert meters to miles, calculate the number of days between two dates, and get the number of days in a Pandas dataframe object.

def convert_to_miles(num):
    return ((num/m_conv_factor)*1000)/1000.0

def calc_days_btwn(training_start_date, marathon_date): 
    training_start_date = training_start_date.split(',')[1].strip()
    training_start_date = datetime.strptime(training_start_date, '%Y-%m-%d') 
    training_start_date = datetime.date(training_start_date)

    delta = marathon_date - training_start_date
    return delta.days-1

#number of workouts in Strava data
def num_rows_in_dataframe(df):
    return len(df.index)

Beneath that, add a helper function to help validate user emails.

email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,7}\b'
def validate_email(email):
    # pass the regular expression and the string into the fullmatch() method
    if(re.fullmatch(email_regex, email)):
        return True
    else:
        return False

It's time to make a Custom Prompt Template in LangChain to make constructing prompts using dynamic inputs easier.

Make Custom Prompt Template

LangChain provides a set of default prompt templates to generate prompts for different tasks. However, for some projects like this one, developers might want to create a prompt template with specific dynamic instructions for the LLM–in instances like that, you can create a custom prompt template similar to the one below right inside request.py beneath validate_email:

#Set up a prompt template
class CustomPromptTemplate(BaseChatPromptTemplate):
    template: str
    # The list of tools available
    tools: List[Tool]

    def format_messages(self, **kwargs) -> str:
        # Get the intermediate steps (AgentAction, Observation tuples)
        # Format them in a particular way
        intermediate_steps = kwargs.pop("intermediate_steps")
        thoughts = ""
        for action, observation in intermediate_steps:
            thoughts += action.log
            thoughts += f"\nObservation: {observation}\nThought: "
        # Set the agent_scratchpad variable to that value
        kwargs["agent_scratchpad"] = thoughts
        # Create a tools variable from the list of tools provided
        kwargs["tools"] = "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
        # Create a list of tool names for the tools provided
        kwargs["tool_names"] = ", ".join([tool.name for tool in self.tools])
        formatted = self.template.format(**kwargs)
        return [HumanMessage(content=formatted)] 

class CustomOutputParser(AgentOutputParser):
    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:
        # Check if agent should finish
        if "Marathon Day" in llm_output:
            return AgentFinish(
                return_values={"output": llm_output.split("Marathon Day:")[-1].strip()},
                log=llm_output,
            )
        elif marathon_date in llm_output:
            return AgentFinish(
                # Return values is generally always a dictionary with a single `output` key
                # It is not recommended to try anything else at the moment :)
                return_values={"output": llm_output.split(marathon_date)[-1].strip()},
                log=llm_output,
            )
        # Parse out the action and action input
        regex = r"Action\s*\d*\s*:(.*?)\nAction\s*\d*\s*Input\s*\d*\s*:[\s]*(.*)"
        match = re.search(regex, llm_output, re.DOTALL)
        if not match:
            try:
                action = match.group(1).strip()
                action_input = match.group(2)
                # Return the action and action input
                return AgentAction(tool=action, tool_input=action_input.strip(" ").strip('"'), log=llm_output)
            except ValueError as e:
                response = str(e)
                if not response.startswith("Could not parse LLM output: `"):
                    raise e
                response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

That helps shape the output. Let's work on receiving the input.

Make a Streamlit web app in Python

To make a webpage to receive input about the user for their personal marathon training plan, we will use Streamlit.

In request.py, we set a title, subheader, the dates that are first displayed when the date input widget is rendered, and a Streamlit form containing two text_input fields for the user's Strava token and email, make the date input widget, and then write the input dates to the page and split up the input dates into two variables. This code goes directly beneath response = response.removeprefix("Could not parse LLM output: ").removesuffix("").

st.title('Personal Marathon Training plan generator')
st.subheader('enter details below')
today = datetime.now()
jul_30 = date(today.year, 7, 30)
dec_31 = date(today.year, 12, 31)

with st.form('my_form'):
    strava_token_input = st.text_input('Strava API token')
    email = st.text_input('Email to send plan to')
    dates = st.date_input(
        "Select your training dates for the year: it should be from the first training date to the marathon date",
        (jul_30, date(today.year, 12, 10)),
        jul_30,
        dec_31
    )
    st.write('Training date: ', dates[0], "Marathon date: ", dates[1])
    training_start_date = dates[0]
    marathon_date = dates[1]
    submitted = st.form_submit_button('Submit')

Run the command streamlit run request.py in your terminal. An application should open up on your localhost:8080:

Beneath the last line marathon_date = dates[1], add the following code to validate the user input:

if submitted:

        if not validate_email(email):
            st.error("invalid email", icon="🚨")
            st.cache_data()
            VALIDANSWERS = False
        elif training_start_date > marathon_date:
            st.error("Start date must be earlier than end date", icon="🚨")
            st.cache_data()
            VALIDANSWERS = False     
        try:
            r = requests.get('https://www.strava.com/api/v3/activities?access_token=' + strava_token_input)
            r.raise_for_status()
            VALIDANSWERS = True
        except requests.exceptions.HTTPError as e:
            print(e)
            st.error("Check Strava token")
            st.cache_data()
            VALIDANSWERS = False
        if VALIDANSWERS:

Get Strava data into Pandas and CSV form

The parameters and headers need to be set in order to call a Strava API request. my_dataset contains data from just the first page of user activities, so to retrieve more activity data, loop through four pages of Strava activities and add them to my_dataset before calling pd.json_normalize on it to turn the semi-structured JSON data into a flat table to work with it.

Directly beneath the if VALIDANSWERS: line, add the following code:

 header = {'Authorization': 'Bearer ' + strava_token_input} 
            params = {'per_page': 200, 'page': 1} #max 200 per page, can only do 1 page at a time
            my_dataset = requests.get(activities_url, headers=header, params=params).json() #activities 1st page
            page = 0
            for x in range(1,5): #loop through 4 pages of strava activities
                page +=1 
            params = {'per_page': 200, 'page': page}
            my_dataset += requests.get(activities_url, headers=header, params=params).json() 

            activities = pd.json_normalize(my_dataset)

Next, create a new Pandas Dataframe object with specific columns containing the categories you want to consider: this tutorial is only looking at the activity name, type (bike ride, run, swim, tennis, etc), distance, moving time, total elevation gain, and start date.

The following code goes beneath activities = pd.json_normalize(my_dataset) and creates a CSV file of the activities data after removing items from 2021 so it only includes workouts from 2022 and 2023. You can replace this with another year! It then makes a CSV file named data_files/runs.csv containing the runs and no other activity types.

cols = ['name', 'type', 'distance', 'moving_time', 'total_elevation_gain', 'start_date']
            activities = activities[cols]
            activities = activities[activities["start_date"].str.contains("YEAR-YOU-WANT-TO-IGNORE-WORKOUTS") == False] 
            activities.to_csv('data_files/activities.csv', index=False)

            runs = activities.loc[activities['type'] == 'Run']
            runs.to_csv('data_files/runs.csv', index=False)

Now, convert meters to miles and the moving time seconds to minutes and hours with the code below:

 data_run_df = pd.read_csv('data_files/runs.csv')
            data_activity_df = pd.read_csv('data_files/activities.csv')
            m_conv_factor = 1609

            data_run_df['distance'] = data_run_df['distance'].map(lambda x: convert_to_miles(x))
            data_activity_df['distance'] = data_activity_df['distance'].map(lambda x: convert_to_miles(x))
            #convert moving time secs to mins, hours
            data_run_df['moving_time'] = data_run_df['moving_time'].astype(str).map(lambda x: x[7:]) #slices off 0 days from moving_time
            data_run_df.to_csv('data_files/runs.csv')
            data_activity_df = pd.read_csv('data_files/activities.csv')

Next, this data will be passed to a LangChain agent.

Generate a Personal Marathon Plan with LangChain Agents

In the same request.py file, make an array called tools containing the tools the agent has access to. Each tool gets a name, function to run, and description. The description is optional, but recommended because it's used by an agent to determine tool use.

Beneath creating the data_activity_df variable, add the tools array:

tools = [
                Tool(
                    name = "rows in csv",
                    func = lambda df: num_rows_in_dataframe(df),
                    description="use to get the number of rows in csv file to calculate averages from running data"
                ),
                Tool(
                    name="Calculator",
                    func=llm_math_chain.run,
                    description="useful for when you need to answer questions about math"
                ),
                Tool(
                    name="days_between",
                    func=lambda day1, day2=marathon_date: calc_days_btwn(day1, day2),
                    description="useful for when you need to calculate the number of days between two dates of type date, like between the training start date and the marathon date"
                )
            ]

Now, create some Pandas dataframe agents to interact with the Strava data. These agents are similar to CSV agents which load data from CSV files instead of dataframes and perform queries. You could use both and see how the output differs.

Agents can be chained together to build more complex applications. Another agent used below is of type ZERO_SHOT_REACT_DESCRIPTION. Zero-shot means the agent functions only on the present action — it lacks memory. It uses the ReAct framework to decide which tool to use, solely based on the tool’s description.

The code runs the agents, telling them what to do and saving the output to variables for use later. Be specific with the prompts here and play around with different questions or commands.

Beneath the tools array, add:

pd_agent_run = create_pandas_dataframe_agent(OpenAI(temperature=0.3), data_run_df, verbose=True) 
pd_agent_activity = create_pandas_dataframe_agent(OpenAI(temperature=0.3), data_activity_df, verbose=True)
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
num_workouts = agent.run(f'Calculate the number of days between these two dates representing the number of workouts in the plan: {marathon_date} and {training_start_date}. You have access to tools')
pd_run_output = pd_agent_run.run("Calculate average distance and maximum distance in miles and average moving time and maximum moving time in minutes. Additionally, calculate any other running statistics from the data you think would be helpful to consider in marathon training.")

Next, make a prompt template, a "reproducible way to generate a prompt" containing a text string ("the template"). It accepts a set of parameters from the user and generates a prompt. This mirrors the items in stapp.py's examples array.

prefix contains directions and context to the LLM and suffix is the user input and output indicator. Finally, these, along with the input variables which are unique to the student the plan is being made for, are passed into FewShotPromptTemplate.

coach_template = """training_start_date:{training_start_date}, marathon_date:{marathon_date}, pd_run_output: {pd_run_output}, num_workouts: {num_workouts}, plan: {plan}
            """
example_prompt = PromptTemplate(input_variables=["training_start_date", "marathon_date", "pd_run_output", "num_workouts", "plan"], template=coach_template)

prefix = """
            You are a marathon trainer tasked with crafting one personalized marathon training plan containing  according to your student's previous runs and activities.
            For each week you should recommend 2 cross-training activities and no more than one run greater than 14 miles per week. The longest run you recommend should be 20 miles around 2 weeks before {marathon_date}.
            There should be {num_workouts} workouts for {num_workouts} days, and there should be easy workouts and rest days in the week leading up to {marathon_date}.
            Slightly modify the following example marathon training plans based on your student and the days beginning on {training_start_date} up until {marathon_date} to create one personalized training plan with each workout on a new line: 
            """
suffix = """
            training_start_date: {training_start_date}, marathon_date: {marathon_date}, pd_run_output: {pd_run_output}, num_workouts: {num_workouts}
            """
few_shot_prompt_template = FewShotPromptTemplate(
                examples=stapp.examples, 
                example_prompt=example_prompt, 
                prefix=prefix,
                suffix=suffix, 
                input_variables=["marathon_date", "training_start_date", "pd_run_output", "num_workouts"]
            )

Create a custom output parser object to help structure the response from the LLM and LLM chain, the most common type of chain. The LLM chain consists of the model and prompt. For this case it's an LLM but it could be a ChatModel. The LLM is passed to a LLMSingleActionAgent to complete a single action and structure the response from the LLM.

This agent is passed to an AgentExecutor. An executor differs from an agent in that it is a running instance of an agent used to execute tasks according to the agent’s decisions and configured tools. In short, agents decide which tools to use and the executors execute them. The output of the Executor (which is given the user input) is the personalized marathon training plan. Beneath the FewShotPromptTemplate() function, add the following code:

output_parser = CustomOutputParser()

#LLM chain consisting of the LLM and a prompt
llm_chain = LLMChain(llm=llm, prompt=few_shot_prompt_template)
tool_names = [tool.name for tool in tools]
agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\Marathon Day:"], 
    allowed_tools=tool_names
)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
plan = agent_executor({"marathon_date":marathon_date, "training_start_date": training_start_date, "pd_run_output": pd_run_output, "num_workouts": num_workouts})
print('plan being print ', plan)

Email the Plan as a PDF using Twilio SendGrid

We access the part of the plan containing the dates, replace new lines with breaks because the ReportLab Python library to generate and format PDFs ignores \n, and send an outbound email using Twilio SendGrid–the email body says "good luck at your marathon on {marathon_date}" with the plan attached.

plan = plan['output']
plan = str(plan).replace('\n','<br />') #each workout on new line
message = Mail(
                from_email='langchain_sendgrid_marathon_trainer@sf.com',
                to_emails=email,
                subject='Your AI-generated marathon training plan',
                html_content='<strong>Good luck at your marathon on %s</strong>!\n\nYour plan is attached.'%(marathon_date))

The email attachment is formatted and styled using ReportLab. The following code returns a stylesheet with a few basic heading and text styles to format the PDF attachment which is in the form of a SimpleDocTemplate document. The document is built using the Paragraph class that lets us format the text with inline font style (and optional color changes) using an XML style markup.

The encoded file is passed as an attachment to the SendGrid email. If the email is sent successfully, the Streamlit app displays a success message; otherwise, it displays an error message.

styleN = getSampleStyleSheet()['Normal']
            story = []

            pdf_name = 'plan.pdf'
            doc = SimpleDocTemplate(
                pdf_name,
                pagesize=letter,
                bottomMargin=.4 * inch,
                topMargin=.6 * inch,
                rightMargin=.8 * inch,
                leftMargin=.8 * inch)
            P = Paragraph(plan, styleN)
            story.append(P)

            doc.build(
                story,
            )
            with open(pdf_name, 'rb') as f:
                data = f.read()
                f.close()
            encoded_file = base64.b64encode(data).decode()

            attachedFile = Attachment(
                FileContent(encoded_file),
                FileName('personal_ai_generated_marathon_training_plan.pdf'),
                FileType('application/pdf'),
                Disposition('attachment')
            )
            message.attachment = attachedFile
            sg = SendGridAPIClient()
            response = sg.send(message)
            if response.status_code == 202:
                st.success("Email sent! Check your email for your personal training plan")
                print(f"Response Code: {response.status_code} \n Message sent!")
            else:
                st.warning("Email not sent--check email")

Run streamlit run request.py in your terminal. Fill in your information (like Strava Access Token, the email where you want to receive the generated plan, the training start date, and marathon race date!) on the Streamlit app, click Submit, and wait for your personal marathon training plan in your email.

If the Submit button is clicked, we check the input (dates and email). If the inputs are either:

invalid: the Streamlit app displays an error message and caches the function so the user can try valid input.
valid: the Streamlit app displays a success message and it's time to use the user's Strava API token to get their Strava workout data.

Play around with different agents, prompting, and different parameters like temperature (which affects the risks the agent takes and creativity of the output).

The complete code above can be found here on GitHub.

What's Next for Agents and SendGrid

So will I use this to train for my marathon? I haven't coded this much since college (or had this much fun doing so) and this was a great way to learn many of LangChain's features, but time will tell –work travel, tennis practices, and AI hackathons make consistent training difficult. I do want to continue trying different ways of changing the output--this is a lot of code which I feel like could be unnecessary, but it yielded the best results.

There's so much developers can do with communication APIs and AI–this is just the tip of the iceberg. You can use a different LLM, look at different Strava data, try different agents (or use chains instead!), embeddings, use LLaMA 2 or other similar models on Replicate, and more. You could use SMS or Twilio Conversations to collect user input instead of Streamlit, or make the LLM recommend other workouts for different goals or meals based on nutrition data!

Let me know online what you're building (and send running tips!)

Build an AI SMS Chatbot with Replicate, LLaMA 2, and LangChain

Lizzie Siegle — Tue, 12 Sep 2023 22:16:09 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

Recently, Meta and Microsoft introduced the second generation of the LLaMA LLM (Large Language Model) to help developers and organizations to build generative AI-powered tools and experiences. Read on to learn how to build an AI SMS chatbot that answers questions like Ahsoka (from Star Wars) using LangChain templating, LLaMa 2, Replicate, and Twilio Programmable Messaging!

Do you prefer learning via video more? Check out this TikTok summarizing this tutorial in 1 minute!

Prerequisites

A Twilio account - sign up for a free Twilio account here
A Twilio phone number with SMS capabilities - learn how to buy a Twilio Phone Number here
Replicate account to host the LlaMA 2 model – make a Replicate account here
Python installed - download Python here
ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access.

Replicate

Replicate offers a cloud API and tools so you can more easily run machine learning models, abstracting away some lower-level machine learning concepts and handling infrastructure so you can focus more on your own applications. You can run open-source models that others have published, or package and publish your own, either publicly or privately.

Configuration

Since you will be installing some Python packages for this project, you will need to make a new project directory and a virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir replicate-llama-ai-sms-chatbot  
cd replicate-llama-ai-sms-chatbot  
python3 -m venv venv 
source venv/bin/activate 
pip install langchain replicate flask twilio

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir replicate-llama-ai-sms-chatbot  
cd replicate-llama-ai-sms-chatbot   
python -m venv venv 
venv\Scripts\activate 
pip install langchain replicate flask twilio

Grab your default Replicate API Token or create a new one here.

On the command line run

export REPLICATE_API_TOKEN={replace with your api token}

Now it's time to write some code!

Code

Make a file called app.py and place the following import statements at the top.

from flask import Flask, request
from langchain import LLMChain, PromptTemplate
from langchain.llms import Replicate
from langchain.memory import ConversationBufferWindowMemory
from twilio.twiml.messaging_response import MessagingResponse

Though LLaMA 2 is tuned for chat, templates are still helpful so the LLM knows what behavior is expected of it. This starting prompt is similar to ChatGPT so it should behave similarly.

template = """Assistant is a large language model.

Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.

Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.

Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist. 

I want you to act as Ahsoka giving advice and answering questions. You will reply with what she would say.
SMS: {sms_input}
Assistant:"""

prompt = PromptTemplate(input_variables=["sms_input"], template=template)

Next, make a LLM Chain, one of the core components of LangChain. This allows us to chain together prompts and make a prompt history. The model is formatted as the model name followed by the version–in this case, the model is LlaMA 2, a 13-billion parameter language model from Meta fine-tuned for chat completions. max_length is 4096, the maximum number of tokens (called the context window) the LLM can accept as input when generating responses.

sms_chain = LLMChain(
    llm = Replicate(model="a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5"), 
    prompt=prompt,
    memory=ConversationBufferWindowMemory(k=2),
    llm_kwargs={"max_length": 4096}
)

Finally, make a Flask app to accept inbound text messages, pass that to the LLM Chain, and return the output as an outbound text message with Twilio Programmable Messaging.

app = Flask(__name__)


@app.route("/sms", methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body'].lower().strip()
    output = sms_chain.predict(sms_input=inb_msg)
    print(output)
    resp.message(output)
    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

On the command line, run python app.py to start the Flask app.

Configure a Twilio Number for the SMS Chatbot

You should see the screen above. Grab that ngrok Forwarding URL to configure your Twilio number: select your Twilio number under Active Numbers in your Twilio console, scroll to the Messaging section, and then modify the phone number’s routing by pasting the ngrok URL with the /sms path in the textbox corresponding to when A Message Comes In as shown below:

Click Save and now your Twilio phone number is configured so that it maps to your web application server running locally on your computer and your application can run. Text your Twilio number a question relating to the text file and get an answer from that file over SMS!

You can view the complete code on GitHub here.

What's Next for Twilio, LangChain, Replicate, and LLaMA 2?

There is so much fun for developers to have around building with LLMs! You can modify existing LangChain and LLM projects to use LLaMA 2 instead of GPT, build a web interface using Streamlit instead of SMS, fine-tune LLaMA 2 with your own data, and more! I can't wait to see what you build–let me know online what you're working on!

Build a SMS Bot that Answers questions over Docs with LangChain in Python

Lizzie Siegle — Thu, 15 Jun 2023 20:33:43 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

With Natural Language Processing (NLP), you can chat with your own documents, such as a text file, a PDF, or a website. Read on to learn how to build a generative question-answering SMS chatbot that reads a document containing Lou Gehrig's Farewell Speech using LangChain, Hugging Face, and Twilio in Python.

LangChain Q&A

LangChain is an open-source tool that wraps around many large language models (LLMs) and tools. It is the easiest way (if not one of the easiest ways) to interact with LLMs and build applications around LLMs.

LangChain makes it easy to perform question-answering of those documents. Picture feeding a PDF or maybe multiple PDF files to a machine and then asking it questions about those files. This could be useful, for example, if you have to prepare for a test and wish to ask the machine about things you didn’t understand.

Prerequisites

A Twilio account - sign up for a free Twilio account here
A Twilio phone number with SMS capabilities - learn how to buy a Twilio Phone Number here
Hugging Face Account – make a Hugging Face Account here
Python installed - download Python here
ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access. > ⚠️ ngrok is needed for the development version of the application because your computer is likely behind a router or firewall, so it isn’t directly reachable on the Internet. You can also choose to automate ngrok as shown in this article.: ## Configuration Since you will be installing some Python packages for this project, you will need to make a new project directory and a virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir lc-qa-sms 
cd lc-qa-sms 
python3 -m venv venv 
source venv/bin/activate 
!pip install langchain
!pip install requests
pip install flask
pip install faiss-cpu
pip install sentence-transformers
pip install twilio
pip install load_dotenv

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir lc-qa-sms  
cd lc-qa-sms 
python -m venv venv 
venv\Scripts\activate 
pip install langchain
pip install requests
pip install flask
pip install faiss
pip install sentence-transformers
pip install twilio
pip install load_dotenv

Set up Hugging Face Hub

The Hugging Face Hub offers over 120k models, 20k datasets, and 50k demos people can easily collaborate in their ML workflows. As mentioned earlier, this project needs a Hugging Face Hub Access Token to use the LangChain endpoints to a Hugging Face Hub LLM. After making a Hugging Face account, you can get a Hugging Face Access Token here by clicking on New token. Give the token a name and select the Read Role.

Alternatively, you could use models from, say, OpenAI.

On the command line in your root directory, run the following command on a Mac to set the token as an environment variable.

export HUGGINGFACEHUB_API_TOKEN=replace-with-your-huggingfacehub-token

For the Windows command, check out this blog post on environment variables.

Now let's build that LangChain question-answering chatbot application.

Answer Questions from a Doc with LangChain via SMS

Inside your lc-qa-sms directory, make a new file called app.py.

At the top of the file, add the following lines to import the required libraries.

import requests
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
from langchain.chains.question_answering import load_qa_chain
from langchain import HuggingFaceHub
from flask import Flask, request, redirect
from twilio.twiml.messaging_response import MessagingResponse

Beneath those import statements, make a helper function to write to a local file from a URL, and then load that file from the local file storage with LangChain's TextLoader library. Later the file will be passed over to the split method to create chunks.

def loadFileFromURL(text_file_url):
    output_file = "lougehrig.txt"
    resp = requests.get(text_file_url)
    with open(output_file, "w",  encoding='utf-8') as file:
      file.write(resp.text)

    # load text doc from URL w/ TextLoader
    loader = TextLoader('./'+output_file)
    txt_file_as_loaded_docs = loader.load()
    return txt_file_as_loaded_docs

In this tutorial, the file will be Lou Gehrig's famous speech, which is why the output_file is named "lougehrig.txt".

You could alternatively use a local file.

Next, add the following code for a helper function to split the document into chunks. This is important because LLMs can't process inputs that are too long. LangChain's CharacterTextSplitter function helps us do this–setting chunk_size to 1000 and chunk_overlap to 10 keeps the integrity of the file by avoiding splitting words in half.

def splitDoc(loaded_docs):
    # split docs into chunks
    splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=10)
    chunked_docs = splitter.split_documents(loaded_docs)
    return chunked_docs

Now convert the chunked document into embeddings (numerical representations of words) with Hugging Face and store them in a FAISS Vector Store. Faiss is a library for efficient similarity search and clustering of dense vectors.

def makeEmbeddings(chunked_docs):
    # Create embeddings and store them in a FAISS vector store
    embedder = HuggingFaceEmbeddings()
    vector_store = FAISS.from_documents(chunked_docs, embedder)
    return vector_store

That makeEmbeddings function will make it more efficient to retrieve and manipulate the stored embeddings to conduct a similarity search to get the most semantically-similar documents to a given input which the LLM needs to best answer questions in the following helper function.

def askQs(vector_store, chain, q):
    # Ask a question using the QA chain
    similar_docs = vector_store.similarity_search(q)
    resp = chain.run(input_documents=similar_docs, question=q)
    return resp

The final helper function defines and loads the Hugging Face Hub LLM to be used with your Access Token and starts the request on a similarity search embedded with some input question to the selected LLM, enabling a question-and-answer conversation.

def loadLLM():
    llm=HuggingFaceHub(repo_id="declare-lab/flan-alpaca-large", model_kwargs={"temperature":0, "max_length":512})
    chain = load_qa_chain(llm, chain_type="stuff")
    return chain

declare-lab/flan-alpaca-large is a LLM. You could use others from Hugging Face Hub such as google-flan-t5-xl. They are trained on different corpuses and this tutorial uses flan-alpaca-large because it's faster.

Lastly, make a Flask app using the Twilio REST API to call the helper functions and respond to inbound text messages with information pulled from the text file.

app = Flask(__name__)
@app.route('/sms', methods=['POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body'].lower().strip() #get inbound text body
    chain = loadLLM()
    LOCAL_ldocs = loadFileFromURL('https://raw.githubusercontent.com/elizabethsiegle/qanda-langchain-sms-lougehrig/main/lougehrig.txt')
    LOCAL_cdocs = splitDoc(LOCAL_ldocs) #chunked
    LOCAL_vector_store = makeEmbeddings(LOCAL_cdocs)
    LOCAL_resp = askQs(LOCAL_vector_store, chain, inb_msg)
    resp.message(LOCAL_resp)
    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

Your complete app.py file should look like this:

import requests
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
from langchain.chains.question_answering import load_qa_chain
from langchain import HuggingFaceHub
from flask import Flask, request, redirect
from twilio.twiml.messaging_response import MessagingResponse

def loadFileFromURL(text_file_url): #param: https://raw.githubusercontent.com/elizabethsiegle/qanda-langchain-sms-lougehrig/main/lougehrig.txt
    output_file = "lougehrig.txt"
    resp = requests.get(text_file_url)
    with open(output_file, "w",  encoding='utf-8') as file:
      file.write(resp.text)

    # load text doc from URL w/ TextLoader
    loader = TextLoader('./'+output_file)
    txt_file_as_loaded_docs = loader.load()
    return txt_file_as_loaded_docs

def splitDoc(loaded_docs):
    # split docs into chunks
    splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=10)
    chunked_docs = splitter.split_documents(loaded_docs)
    return chunked_docs

def makeEmbeddings(chunked_docs):
    # Create embeddings and store them in a FAISS vector store
    embedder = HuggingFaceEmbeddings()
    vector_store = FAISS.from_documents(chunked_docs, embedder)
    return vector_store

def askQs(vector_store, chain, q):
    # Ask a question using the QA chain
    similar_docs = vector_store.similarity_search(q)
    resp = chain.run(input_documents=similar_docs, question=q)
    return resp

def loadLLM():
    llm=HuggingFaceHub(repo_id="declare-lab/flan-alpaca-large", model_kwargs={"temperature":0, "max_length":512})
    chain = load_qa_chain(llm, chain_type="stuff")
    return chain


app = Flask(__name__)
@app.route('/sms', methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body'].lower().strip() #get inbound text body
    chain = loadLLM()
    LOCAL_ldocs = loadFileFromURL('https://raw.githubusercontent.com/elizabethsiegle/qanda-langchain-sms-lougehrig/main/lougehrig.txt')
    LOCAL_cdocs = splitDoc(LOCAL_ldocs) #chunked
    LOCAL_vector_store = makeEmbeddings(LOCAL_cdocs)
    LOCAL_resp = askQs(LOCAL_vector_store, chain, inb_msg)
    resp.message(LOCAL_resp)
    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

On the command line, run python app.py to start the Flask app.

Configure a Twilio Number for the SMS Chatbot

Now, your Flask app will need to be visible from the web so Twilio can send requests to it. ngrok lets you do this. With ngrok installed, run ngrok http 5000 in a new terminal tab in the directory your code is in.

You should see the screen above. Grab that ngrok Forwarding URL to configure your Twilio number: select your Twilio number under Active Numbers in your Twilio console, scroll to the Messaging section, and then modify the phone number’s routing by pasting the ngrok URL with the /sms path in the textbox corresponding to when A Message Comes In as shown below:

Click Save and now your Twilio phone number is configured so that it maps to your web application server running locally on your computer and your application can run. Text your Twilio number a question relating to the text file and get an answer from that file over SMS!

The complete code can be found here on GitHub.

What's Next for Twilio and LangChain

There's so much you can do with document-based question-answering. You can use a different LLM, use a longer document than a text file containing Lou Gehrig's famous speech, use other types of documents like a PDF or website (here's LangChain's docs on documents), store the embeddings elsewhere, and more. Other than a SMS chatbot, you could create an AI tutor, search engine, automated customer service agent, and more.

Let me know online what you're building!

Build a Basketball SMS Chatbot with LangChain Prompt Templates in Python

Lizzie Siegle — Wed, 31 May 2023 22:00:33 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

As I've played with the OpenAI API (and DALL·E and fine-tuning models) and gotten more into ML, developers I meet (as well as my wonderful coworker Craig Dennis who advised me on this tutorial) keep telling me to use LangChain, a powerful and flexible framework for developing applications powered by language models.

Read on to learn how to build an SMS chatbot using LangChain prompt templates, OpenAI, Twilio Programmable Messaging, and Python.

Large Language Models

Large Language Models are trained on large quantities of textual data (i.e. in ChatGPT's case, the entire internet up to 2021–so you can provide more context or data that the model is missing with prompt engineering and Prompt Templates–more on those later) to produce human-like responses to dialogue or other natural language inputs.

To yield these natural language responses, LLMs use deep learning (DL) models, which use multi-layered neural networks to process, analyze, and make predictions with complex data.

LangChain

LangChain came out late last year and already has over 43,000 stars on GitHub and a thriving community of contributors.

It is many things, but ultimately at its core is an open-source framework that simplifies the development of applications using large language models (LLMs), like OpenAI or Hugging Face. Developers can use it for chatbots, Generative Question-Answering (GQA), summarization, and more.

With LangChain, developers can “chain” together different LLM components to create more advanced use cases around LLMs. Chains can consist of multiple components from several modules:

Prompt Templates: Prompt templates are templates for different types of prompts. Like “chatbot” style templates, ELI5 question-answering, etc
LLMs: Large language models like GPT-3, Hugging Face, BLOOM, etc
Agents: Agents use LLMs to decide what actions should be taken. Tools like web search or calculators can be used, and all are packaged into a logical loop of operations.
Memory: Short-term memory, long-term memory. LangChain recognizes the power of prompts and has built an entire set of objects for them. It also provides a wrapper around different LLMs so you can easily change models, swapping them out with different templates. The chat model could be different, but running and calling it is the same–a very Java-like concept! ## LangChain Twilio Tool Recently, LangChain came out with a Twilio tool so your LangChain Agents are able to send text messages. For example, your LLM can understand the input in natural language, but Agents can let you complete different tasks like calling an API.

You'll need your own Twilio credentials and to install it with pip install twilio. The code for the tool would look something like this:

from langchain.utilities.twilio import TwilioAPIWrapper

twilio = TwilioAPIWrapper(
     account_sid="YOUR-ACCOUNT-SID",
     auth_token="YOUR-TWILIO-AUTH-TOKEN",
     from_number="YOUR-TWILIO-NUMBER,"
)
twilio.run("hello world", "NUMBER-TO-TEXT")

This tutorial will, however, show you how to use LangChain Prompt Templates with Twilio to make a SMS chatbot.

LangChain Prompt Templates

"Prompts” refer to the input to the model and are usually not hard-coded, but are more often constructed from multiple components. A Prompt Template helps construct this input. LangChain provides several classes and functions to make constructing and working with prompts easy.

Prompts being input to LLMs are often structured in different ways so that we can get different results. For Q&A, you could take a user’s question and reformat it for different Q&A styles, like conventional Q&A, a bullet list of answers, or even a summary of problems relevant to the given question. You can read more about prompts here in the LangChain documentation.

Prompt templates offer a reproducible way to generate a prompt. Like a reusable HTML template, you can share, test, reuse, and iterate on it, and it will update. When you update a prompt template, it updates for anyone else using it or the whole set of apps that could use it.

Prompt templates contain a text string (AKA “the template”) that can take in a set of parameters from the user and generate a prompt.

Importing and initializing a LangChain PromptTemplate class would look like so:

from langchain import PromptTemplate

template = """Question: {question}

Answer: """
prompt = PromptTemplate(
        template=template,
    input_variables=['question']
)

# user question
question = "Which NBA team won the finals in 1996?"

Since OpenAI LLMs lack data after September 2021, its models can't answer anything that occurred after without additional context. Prompt templates help provide additional context, but differ from fine-tuning - fine-tuning is like coaching the model with new data to get certain output, but prompt engineering with Prompt Templates provides the model specific data to help it get the output you want.

Let's use this template to build the chatbot.

Prerequisites

A Twilio account - sign up for a free one here
A Twilio phone number with SMS capabilities - learn how to buy a Twilio Phone Number here
OpenAI Account – make an OpenAI Account here
Python installed - download Python here
ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access.

⚠️ ngrok is needed for the development version of the application because your computer is likely behind a router or firewall, so it isn’t directly reachable on the Internet. You can also choose to automate ngrok as shown in this article.

Configuration

Since you will be installing some Python packages for this project, you will need to make a new project directory and a virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir lc-sms 
cd lc-sms 
python3 -m venv venv 
source venv/bin/activate 
!pip install langchain
!pip install openai
pip install Flask
pip install twilio
pip install load_dotenv

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir lc-sms 
cd lc-sms 
python -m venv venv 
venv\Scripts\activate 
pip install langchain
pip install openai
pip install Flask
pip install twilio
pip install load_dotenv

The last command uses pip, the Python package installer, to install the three packages that you are going to use in this project, which are:

The OpenAI Python client library, to send requests to OpenAI's GPT-3 engine. You could also use a different LLM, listed here in the LangChain docs.
The Twilio Python Helper library, to work with SMS messages.
The Flask framework, to create the web application in Python.
The load_dotenv library to load environment variables from a .env file
The LangChain library to building applications with LLMs through composability As mentioned above, this project needs an OpenAI API Key to use the LangChain endpoints to OpenAI. After making an OpenAI account, you can get an OpenAI API Key here by clicking on + Create new secret key. The Python application will need to have access to this key, so we are going to make a .env file where the API key can safely be stored. The application we create will be able to import this key as an environment variable soon.

Create a .env file in your project’s root directory and enter the following line of text, making sure to replace <OPENAI_API_KEY> with your actual key:

OPENAI_API_KEY= <YOUR-OPENAI-KEY>

Make sure that the OPENAI_API_KEY is safe and that you don't expose your .env file in a public location such as GitHub.

Now, your Flask app will need to be visible from the web, so Twilio can send requests to it. ngrok lets you do this: with ngrok installed, run ngrok http 5000 in a new terminal tab in the directory your code is in.

You should see the screen above. Grab that ngrok Forwarding URL to configure your Twilio number: select your Twilio number under Active Numbers in your Twilio console, scroll to the Messaging section, and then modify the phone number’s routing by pasting the ngrok URL in the textbox corresponding to when A Message Comes In as shown below:

Click Save and now your Twilio phone number is configured so that it maps to your web application server running locally on your computer. Let's build that application now.

Generate text through LangChain with OpenAI in Python via SMS

Inside your lc-sms directory, make a new file called app.py.

Copy and paste the following code into app.py to start off the ChatGPT-like SMS app to import the required libraries.

from flask import Flask, request, redirect
from dotenv import load_dotenv

from twilio.twiml.messaging_response import MessagingResponse

from langchain.llms import OpenAI
from langchain import PromptTemplate

import os

Then, make the OpenAI LLM object (which could be another LLM--this is where LangChain can make reusability easier!), passing it the model name and API key from the .env file, and create a Flask application.

load_dotenv()


llm = OpenAI(
    model_name="text-davinci-003",
    openai_api_key=os.environ.get('OPENAI_API_KEY')
)


app = Flask(__name__)

Make the start of the /sms webhook containing a TwiML response to respond to inbound SMS with.

@app.route("/sms", methods=['GET', 'POST'])
def sms_reply():
    # Start our TwiML response
    resp = MessagingResponse()

Add the following template code below to help shape the queries and answers from the LLM. In this tutorial, it's Warriors basketball-themed. (Yes, I know the 2023 NBA Finals are the Heat versus the Nuggets. A girl can dream.)

 template = """Answer the question based on the context below. If the question cannot be answered using the information provided, answer with "I don't know, but the Warriors are the best team in the NBA".
    Context: Steph Curry has won 4 NBA Finals series. His Golden State Warriors defeated the Cleveland Cavaliers three times and the Boston Celtics once. 

In 2015 Steph Curry and the Warriors defeated the Cleveland Cavaliers. The Cavs featured LeBron James, Kyrie Irving and not much else! 
In 2017 Steph Curry, Kevin Durant and the Warriors defeated the Cavs again. The Cavs still had Lebron and Kyrie.
In 2018 the Warriors, featuring Steph and KD again, defeated the Cavs for the third time in four years. The Cavs still had Kyrie and Lebron. 
In 2022 Steph and the Warriors defeated the Boston Celtics for his fourth title. The Celtics featured Jayson Tatum and Jaylen Brown. Steph Curry and the Golden State Warriors lost one NBA Finals series to the Cleveland Cavaliers and one to the Toronto Raptors. 

In 2016 Steph, Klay, Draymond and the rest of the Warriors lost to the Cleveland Cavaliers.  The Cavs starred LeBron James and Kyrie Irving. 
In 2019 the Steph and the Warriors, missing an injured KD, lost to the Toronto Raptors. The Raptors featured Kawhi Leonard in his only season in Canada alongside Kyle Lowry and Pascal Siakham. Steph Curry has only won one NBA Finals MVP up to this point in his career. . 

In 2022 Steph Curry averaged 31 points, 6 rebounds and 5 assists per game to win the MVP award in the Warriors 6-game defeat of the Boston Celtics.
In 2015 Andre Iguodala won the MVP in the Warriors defeat of the Cavs.
In both 2017 & 2018 Kevin Durant was Finals MVP in the Warriors victories over the Cavs. Steph Curry's 4-2 NBA Finals record puts him ahead of many NBA greats including Larry Bird (3-2) and LeBron James (4-6). Steph still comes up short of the greatest NBA Finals winners including Bill Russell (11-1) and Michael Jordan (6-0).  
    Question: {query}
    Answer: """
    basketball_query_template = PromptTemplate(
        input_variables=["query"],
        template=template
    )

Get the user's question via inbound text message, print out the answer by passing the question to the prompt template and formatting it, and then pass that to Twilio as the text message to send back to the user.

question = request.form['Body'].lower().strip()
    print(llm(
        basketball_query_template.format(
            query=question
        )
    ))
    resp.message(openai(
        basketball_query_template.format(
            query=question
        )
    ))

    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

In a new terminal tab (while the other terminal tab is still running ngrok http 5000), run python app.py. You can now text your Twilio number (questions about the Warriors because we provided the model Warriors context) configured from above.

The complete code can be found here on GitHub.

What's Next for Twilio and LangChain

LangChain can be used for chatbots (not just Warriors/basketball-themed), chaining different tasks, Generative Question-Answering (GQA), summarization, and so much more. Stay tuned to the Twilio blog for more LangChain and Twilio content, and let me know online what you're building with AI!

How to Fine-Tune an OpenAI ML Model with Node.js

Lizzie Siegle — Tue, 02 May 2023 17:40:54 +0000

This blog post was written for Twilio and originally published on the Twilio blog.
OpenAI, the creator of ChatGPT, offers a variety of Machine Learning models developers can use with different capabilities and price points. Their API lets developers interface with the models, adding features like text completion, code completion, text or image generation, speech-to-text, and more to their applications.

ChatGPT's model is powerful (this Twilio tutorial went over how to Build a Serverless ChatGPT SMS Chatbot with the OpenAI API), but still limited. This tutorial will go over how to fine-tune an existing model from OpenAI with your own data so you can get more out of it.

What is Fine-tuning?

OpenAI's models have been trained on a large amount of text from the web, but their training data is still finite and not as current as they could be: they still miss information on specific topics.

Transfer learning is a machine learning method where you use knowledge that was gained from solving one problem and apply it to a new but related problem. A model developed for a task is reused as the starting point for a model on a second task–for example, a model trained to specifically recognize pizza could be edited to recognize calzones.

Fine-tuning is similar to transfer learning in that it lets you modify and customize an existing ML model.

Why Fine-tune?

Fine-tuning helps you achieve better results on a wide number of tasks. Once a model is fine-tuned, you won't need to provide samples via prompt anymore, helping you save on costs and enabling lower-latency requests.

Assuming the original task the model was trained for is similar to the new task, fine-tuning a model that has already been designed and trained allows developers to take advantage of what the model has already learned without having to develop it from scratch.

Prerequisites

OpenAI Account - create an OpenAI Account here
Node.js installed - download Node.js here

After making an OpenAI account, you'll need an API Key. You can get an OpenAI API Key here by clicking on + Create new secret key.

Save that API key for later.

Make a New Node.js project

To get started, create a new Node.js project in an empty directory:

mkdir finetune-model
cd finetune-model
npm init --y

Get started with OpenAI

Make a .env file in your root directory and add the following line:

OPENAI_API_KEY=YOUR-OPENAI-API-KEY

Replace YOUR-OPENAI-API-KEY with the OpenAI API Key you took note of earlier. Now you will be able to access this API Key in your code with process.env.OPENAI_API_KEY.

Next, install the openai and fs npm packages:

npm install openai fs

Now it's time to prepare your data!

Prepare Custom Data

Training data is how GPT-3 learns what you want it to say.

Your data should be in a JSONL document, where each line is a prompt-completion pair corresponding to a training sample.

Make a jsonl-file called newdata.jsonl and include prompt-completion pairs that look something like this:

{"prompt": "<prompt text> ->", "completion": "<ideal generated text>\n"}
{"prompt": "<prompt text> ->", "completion": "<ideal generated text>\n"}
{"prompt": "<prompt text> -> ", "completion": "<ideal generated text>\n"}

The data I used includes fun facts about me and can be found here on GitHub.

OpenAI recommends having at least a couple hundred examples and found that each doubling of dataset size leads to a linear increase in model quality. Save at least thirty prompt-completion pairs (the more the better). This is not enough to make a fine-tuned model for a production-ready product, but is enough for the purposes of this tutorial.

Save your jsonl file, and on the command line, use OpenAI's CLI data preparation tool to help you easily convert your data into the jsonl file format.

openai tools fine_tunes.prepare_data -f newdata.jsonl

With your prepared data, it's time to write some code to fine-tune an existing OpenAI machine learning model.

Fine-tune an OpenAI Model

In this tutorial, you'll use davinci, but OpenAI currently lets you fine-tune its base models which also include curie, babbage, and ada. These are their original models and they lack any instruction after training (which text-davinci-003 has, for example).

In a file called finetune.js, add the following code to the top which will be used for each step:

const { Configuration, OpenAIApi } = require("openai");
const fs = require("fs");
const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

Beneath that, your first function uploads the jsonl file so you can use it next.

async function uploadFile() {
    try {
        const f = await openai.createFile(
            fs.createReadStream("newdata.jsonl"),
            "fine-tune"
        );
        console.log(`File ID ${f.data.id}`);
        return f.data.id;
    }
    catch (err) {
        console.log('err uploadfile: ', err);
    } 
}
uploadFile();

Once you run the file with node finetune.js, take note of the ID printed out for the next step and comment out the uploadFile() function call.

Beneath the uploadFile() function, add the following function:

async function makeFineTune() {
    try {
        const ft = await openai.createFineTune({
            training_file: 'YOUR-FILE-ID',
            model: 'davinci'
        });
        console.log(ft.data);
     }
    catch (err) {
        console.log('err makefinetune: ', err.response.data.error);
    }
}
makeFineTune();

Replace YOUR-FILE-ID with the file ID from the previous step, and rerun finetune.js to run the new function. After running it, comment out the makeFineTune() function call.

Wait a few minutes to let it process. You can check-in on the status of the fine tune, and additionally get the model ID, by calling the listFineTunes API method as shown below:

async function getFineTunedModelName() {
    try {
        const modelName = await openai.listFineTunes();
        console.table(modelName.data.data, ["id", "status", "fine_tuned_model"]);

    }
    catch (err) {
        console.log('err getmod: ', err)
    }
 }
getFineTunedModelName();

The fine-tuned model is not ready for use when status is running and fine_tuned_model is null, but once you see status is succeeded and fine_tuned_model is something like davinci:ft-personal-2023-05-01-22-05-28, then comment out getFineTunedModelName(), copy the model name (beginning with davinci), and you're good to go and use the model!

Test your Fine-Tuned OpenAI model

Add the following code to your finetune.js file.

async function run() {
    try {
        const comp = await openai.createCompletion({
            model: 'YOUR-FINETUNED-MODEL-NAME',
            prompt: 'What is Lizzie Siegle's favorite food', //replace this prompt according to your data
            max_tokens: 200
        });
        if (comp.data) {
            console.log('choices: ', comp.data.choices)
        }
    } catch (err) {
        console.log('err: ', err)
    }
}
run();

Replace YOUR-FINETUNED-MODEL-NAME with the model name from the previous step.

Then, rerun node finetune.js and you should see your new custom model deployed in action making a new completion according to the prompt you passed in!

Alternatively, you can test out your fine-tuned model in the OpenAI Playground.

Complete code can be found here on GitHub.

What's Next for Fine-tuning OpenAI models?

That's how to customize one of OpenAI's existing models to fit your own specific use case and data!

For next steps, you can

add more data to improve completions and make the model production-ready
try fine-tuning a different model other than davinci
use all the other Completions parameters like temperature, frequency_penalty, presence_penalty, etc., on your requests to fine-tuned models
fine-tune a model in another programming language
and more!

How to generate TwiML using Strings in C#

Niels Swimburger.NET 🍔 — Tue, 21 Feb 2023 00:00:00 +0000

This blog post was written for Twilio and originally published at the Twilio blog.

Over the decades, C# has added different ways to create a string, each with their own benefit. In this tutorial, you'll learn how to generate TwiML using the different C# string features with an ASP.NET Core Minimal API and compare it to the object oriented way of generating TwiML.

But first, let's get you up to speed on how Twilio uses webhooks and TwiML to respond to text messages and voice calls.

Prerequisites

Here's what you will need to follow along:

.NET 6 SDK, or .NET 7 SDK to use Raw String Literals (later versions will work too)
A code editor or IDE. I recommend JetBrains Rider, Visual Studio 2022 Preview (will also work in the non-preview Visual Studio in the future), or VS Code with the C# plugin
A free Twilio account (sign up with Twilio for free and get trial credit)
A Twilio Phone Number
The ngrok CLI, and optionally, a free ngrok account

Experience with ASP.NET Core and Minimal APIs is not required but recommended. Here's an article that brings you up to speed on how to integrate ASP.NET Core Minimal APIs with Twilio.

You can find the source code for this tutorial on GitHub. Use it if you run into any issues, or submit an issue, if you run into problems.

Before creating your application, let's get you up to speed on how Twilio uses webhooks and TwiML to respond to text messages and voice calls.

Webhooks and TwiML

Using Twilio, you can build programmatic text message and voice call applications. Twilio can do a lot of things like playing audio, gathering input, and recording a call. But Twilio doesn't know how you want to respond to voice calls and text messages. That's why when Twilio receives a call or text message, Twilio will send an HTTP request to your application asking for instructions.

A webhook is a user-defined HTTP callback. When an event happens in a service, your application is notified of that event using an HTTP request.

Twilio uses webhooks heavily throughout all its products. Here's a diagram of what it looks like when there's an incoming text message to your Twilio Phone Number and your application is handling the messaging webhook:

When your Twilio Phone Number receives a text message, Twilio will send an HTTP request with the message details to your application. Your application then has to respond with TwiML ( Twi lio M arkup L anguage) to instruct Twilio how to respond. TwiML is XML with special tags defined by Twilio to provide instructions on how to respond to messages and voice calls. In the diagram depicted above, Twilio will respond with "Hi!" because the app responded with the following TwiML:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Message>Hi!</Message>
</Response>

You can learn more about TwiML for Programmable Messaging here and TwiML for Programmable Voice here.

Create an ASP.NET Core Minimal API to handle voice calls

First, let's create an ASP.NET Core Minimal API to handle voice calls without using Raw String Literals.

Open a terminal and create a new ASP.NET Core Minimal API project using the .NET CLI:

dotnet new web -o TwimlStrings 
cd TwimlStrings

These commands will create the project in a folder named TwimlStrings and navigate into the folder.

Open the project in your editor, and update the Program.cs file with the following C# code:

using System.Security;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/what-does-the-fox-say", () => Results.Text("" +
"<?xml version=\"1.0\" encoding=\"utf-8\"?>" +
"<Response>" +
" <Gather action=\"/answer\" method=\"GET\" input=\"speech\"> " +
" <Say>What does the fox say?</Say>" +
" </Gather>" +
" <Say>Ring-ding-ding-ding-dingeringeding!</Say>" +
"</Response>",
 contentType: "application/xml"
));

app.MapGet("/answer", (string speechResult) => Results.Text("" +
"<?xml version=\"1.0\" encoding=\"utf-8\"?>" +
"<Response>" +
" <Say>You said: " + SecurityElement.Escape(speechResult) + "</Say>" +
"</Response>",
 contentType: "application/xml"
));

app.Run();

When your Twilio Phone Number receives a call, Twilio will send an HTTP GET request to the /what-does-the-fox-say endpoint, which will set the content-type response header to application/xml, and return the following TwiML in the response:

<?xml version="1.0" encoding="utf-8"?><Response><Gather action="/answer" method="GET" input="speech"><Say>What does the fox say?</Say></Gather><Say>Ring-ding-ding-ding-dingeringeding!</Say></Response>

That's not very readable, so here's the formatted version:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Gather action="/answer" method="GET" input="speech">
<Say>What does the fox say?</Say>
</Gather>
<Say>Ring-ding-ding-ding-dingeringeding!</Say>
</Response>

When Twilio receives the TwiML, it'll ask the caller "What does the fox say?" and listen for a speech response. When the caller responds, Twilio will send an HTTP request, this time to the /answer endpoint, with the transcription of what the caller said. If the caller doesn't respond, Twilio will say "Ring-ding-ding-ding-dingeringeding!" to the caller.

The /answer endpoint will retrieve the transcription from the query string parameter SpeechResult by binding it to the speechResult parameter. The speechResult is used to construct some more TwiML, but it is first escaped using SecurityElement.Escape to ensure no XML is in the variable. The /answer endpoint will generate the following TwiML when the caller says "ring ding ding":

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Say>You said: ring ding ding</Say>
</Response>

When Twilio receives this TwiML, Twilio will convert "You said: ring ding ding" to speech and stream the audio to the caller.

In case you're not familiar with the song I am referencing in this app, "What does the fox say?" originates from this famous song.

Start your .NET application using your editor or using the terminal using the .NET CLI:

dotnet run

Before integrating Twilio, open a browser and navigate to your web application URL with the suffix /what-does-the-fox-say, to verify the TwiML that is generated.

Make the application publicly accessible

For Twilio to be able to send HTTP requests to your local web server, the server needs to be publicly accessible. ngrok is a free and secure tunneling service that can make your local web server public.

To start ngrok, run the following ngrok command in a separate terminal:

ngrok http [YOUR_ASPNET_URL]

Replace [YOUR_ASPNET_URL] with the localhost URL from your .NET application. If you're using an HTTPS localhost URL, you'll need to authenticate ngrok. The ngrok command will display an HTTPS Forwarding URL that makes your local web server public.

Configure your Twilio Phone Number

Now it's time to update your Twilio Phone Number to send HTTP requests to your /what-does-the-fox-say endpoint via the ngrok Forwarding URL. The URL should look something like https://cd2f8809cbd0.ngrok.io/what-does-the-fox-say.

Go to the Active Phone Numbers section in the Twilio Console and click on your Twilio Phone Number.

This will take you to the configuration for the phone number. Find the Voice & Fax section and under the " A CALL COMES IN" label, set the dropdown to " Webhook". In the text field next to it, enter your ngrok forwarding URL with /what-does-the-fox-say appended to it. Select “ HTTP GET ” in the last dropdown. Finally, click the Save button at the bottom of the page.

Test that the application works

To test out your application, call your Twilio Phone Number. You should hear the question "What does the fox say?" which you can respond to, wait 5 seconds, and then your response will be read back to you.

After you've done that, stop your .NET application using your editor, or if you're using the terminal, press ctrl + c.

Use Verbatim Strings

The first way you can improve your code is by using Verbatim Strings. To turn your string into a Verbatim String, prefix your string with the "at" @ symbol.

@"The fox said: ""I like using verbatim strings,
because it supports multi-line text,
you don't need to escape forward slashes (/), and backslashes (\),
and you can easily escape double quotes by doubling the double quotes""."

As the Verbatim String above says, you don't need to escape forward slashes or backslashes, however you still need to escape double quotes. However, instead of using a backslash to escape the double quotes, you place two double quotes and the resulting character will be a single double quote.

Applying this knowledge to your TwiML Program.cs file, you will have the following endpoint code:

app.MapGet("/what-does-the-fox-say", () => Results.Text(
@"<?xml version=""1.0"" encoding=""utf-8""?>
 <Response>
 <Gather action=""/answer"" method=""GET"" input=""speech""> 
 <Say>What does the fox say?</Say>
 </Gather>
 <Say>Ring-ding-ding-ding-dingeringeding!</Say>
 </Response>",
 contentType: "application/xml"
));

app.MapGet("/answer", (string speechResult) => Results.Text(
@"<?xml version=""1.0"" encoding=""utf-8""?>
 <Response>
 <Say>You said: " + SecurityElement.Escape(speechResult) + @"</Say>
 </Response>",
 contentType: "application/xml"
));

The resulting TwiML will be formatted better, but still be indented oddly because the tabs used to nicely indent the code are also included in the string:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Gather action="/answer" method="GET" input="speech"> 
<Say>What does the fox say?</Say>
</Gather>
<Say>Ring-ding-ding-ding-dingeringeding!</Say>
</Response>

By using Verbatim Strings, you are able to use multi-line text so you don't have to concatenate the string for every line of TwiML. You may think that this will also improve performance as you aren't creating as many strings and then concatenating them. But the compiler will optimize your code so that there's no difference in the end result.

Use String Interpolation

By using Verbatim Strings, you were able to decrease the amount of opening and closing double quotes to construct your TwiML string which makes the code a lot easier to read. However, you still had to use string concatenation to embed the speechResult into your TwiML string for the /answer endpoint.

To embed variables in your string, you can use String.Format or, much better, String Interpolation. First, start your string with the dollar $ sign then use the curly brackets {} to embed C# expressions.

For example, the following code will write "What does the fox say?" to the console.

var animal = "fox";
Console.WriteLine($"What does the {animal} say?");

Applying this knowledge to the /answer endpoint, you can further improve the code like this:

app.MapGet("/answer", (string speechResult) => Results.Text(
$@"<?xml version=""1.0"" encoding=""utf-8""?>
 <Response>
 <Say>You said: {SecurityElement.Escape(speechResult)}</Say>
 </Response>",
 contentType: "application/xml"
));

C# 11 and .NET 7 only: Use Raw String Literals

C# 11 is still in preview, but you can try it out today using the .NET 7 previews. Make sure you have .NET 7 installed and are using .NET 7 in your project. Then, open your project file at TwimlStrings.csproj and add the <LangVersion>preview</LangVersion> to the first <PropertyGroup> node. This will enable the C# 11 preview features.

Your TwimlStrings.csproj in the project folder, should look something like this:

<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>preview</LangVersion>
</PropertyGroup>
</Project>

Now that you have enabled C# 11, let's talk about the new Raw String Literals. To use Raw String Literals, you start and end your string with at least 3 double quotes. For example:

"""
 I like using Raw String Literals,
 because it supports multi-line text,
 you don't need to escape forward slashes (/), backslashes (\), and double quotes\* ("),
 and you can indent your code better without having the indentation included in your string. 
"""

The syntax highlighting may be off in the next snippets and snippet above because the blog doesn't have C# 11 highlighting yet.

As with Verbatim Strings, you don't have to escape forward slashes and backslashes, but, as opposed to Verbatim Strings, in Raw String Literals you also don't have to escape your double quotes, more or less.

You can use double quotes as long as the number of subsequent double quotes is less than the amount of double quotes you started and ended your string with. This is clearer with an example:

// 3 start and end double quotes, so 1 and 2 subsequent double quotes is allowed
"""This " and this "" is allowed.""" 

// 3 start and end double quotes, so 3 subsequent double quotes is NOT allowed
"""This """ is not allowed."""

// 4 start and end double quotes, so 3 subsequent double quotes is NOT allowed
""""This """ is allowed.""""

Lines one and two are valid Raw String Literals because the number of subsequent double quotes is lower than the number of double quotes used to start and end the Raw String Literal.

Raw String Literals also makes it easier to format your strings. Take the following method that generates a Verbatim String as an example:

public string GenerateHtml()
{
return @"
 <p>
 This is a <b>multi-line Verbatim String</b> to generate some HTML
 </p>
";
}

The resulting string will be:


<p>
 This is a <b>multi-line Verbatim String</b> to generate some HTML
<p>

Notice the newlines and the tab indentation? I formatted the Verbatim String for the sake of readability, but as a result I also got the newlines and tabs in it which I do not want.

Let's take a look at the Raw String Literal version:

string GenerateString()
{
return """
 <p>
 This is a <b>multi-line Raw String Literal</b>
 </p>
""";
}

The resulting string now looks like this:

<p>
This is a <b>multi-line Raw String Literal</b>
</p>

The unnecessary newlines and tabs are gone. The way Raw String Literals indent your strings depends on where you place the starting and ending double quotes, and also how you indent the content of your string. It's a little funky and hard to explain all the variations, so I recommend experimenting with it and reading the Microsoft documentation on Raw String Literals to learn more.

Using this knowledge, you can apply Raw String Literals to your endpoints like this:

app.MapGet("/what-does-the-fox-say", () => Results.Text("""
 <?xml version="1.0" encoding="utf-8"?>
 <Response>
 <Gather action="/answer" method="GET" input="speech">
 <Say>
 What does the fox say?
 </Say>
 </Gather>
 <Say>Ring-ding-ding-ding-dingeringeding!</Say>
 </Response>
""",
 contentType: "application/xml"
));

app.MapGet("/answer", (string speechResult) => Results.Text($"""
 <?xml version="1.0" encoding="utf-8"?>
 <Response>
 <Say>You said: {SecurityElement.Escape(speechResult)}</Say>
 </Response>
""" ,
 contentType: "application/xml"
));

Now the resulting TwiML will be readable and not contain unnecessary newlines and tabs.

Other ways to generate TwiML

There are many other ways you could generate TwiML. Since TwiML is XML, you can use the APIs from the System.Xml and System.Xml.Linq namespace. Twilio also has a helper library for .NET that lets you generate TwiML in an object-oriented way, and the helper library for ASP.NET helps you return the TwiML in the HTTP response.

In the terminal where you ran your project, run the following commands to add the Twilio package and Twilio.AspNet.Core package:

dotnet add package Twilio
dotnet add package Twilio.AspNet.Core

Then, update the Program.cs file with the following code:

using Twilio.AspNet.Core.MinimalApi;
using Twilio.TwiML;
using Twilio.TwiML.Voice;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/what-does-the-fox-say", () =>
{
var response = new VoiceResponse();
var gather = new Gather(
 input: new List<Gather.InputEnum> {Gather.InputEnum.Speech},
 action: new Uri("/answer", UriKind.Relative),
 method: Twilio.Http.HttpMethod.Get
 );
 gather.Append(new Say("What does the fox say?"));
 response.Append(gather);
 response.Say("Ring-ding-ding-ding-dingeringeding!");
return Results.Extensions.TwiML(response);
});

app.MapGet("/answer", (string speechResult) =>
{
var response = new VoiceResponse()
 .Say($"You said: {speechResult}");
return Results.Extensions.TwiML(response);
});

app.Run();

The resulting TwiML will be the same as before. Notice, however, that you don't need to use the SecurityElement.Escape anymore to escape the user input, because the Twilio helper library will XML encode it for you. Under the hood, the helper library uses the APIs from System.Linq.Xml to generate the XML string which will also do the formatting for you.

Now, you may be wondering, why would you use strings when you can use the Twilio helper library, or maybe you're wondering why you would use the helper library when you can use strings.

The advantages of creating TwiML using strings, is that you don't need any dependencies and it's more memory efficient & performant. However, it is easier to make a mistake in your TwiML and there will be no compilation errors to prevent you from doing that. You also have to XML escape user input yourself to protect yourself from XML injection.

The advantages of using the Twilio helper library are that you are using fully typed objects and methods that provide IntelliSense in IDEs. If you make a typo, your project will not build and you will receive a compiler error telling you where your typo is. However, you do depend on the Twilio helper library and its dependencies, and you have to create a bunch of objects that are ultimately serialized to an XML string which is less memory efficient and slower.

They both have their own advantages and disadvantages, but you can mix and match based on your use case.

Next steps

You learned how Twilio uses webhooks and TwiML to give you control over how to respond to a call or text message. You then learned how to generate TwiML using different string features in C#, such as Verbatim Strings, String Interpolation, and C# 11's new Raw String Literal feature. You then looked at the difference between generating TwiML using strings and generating TwiML using the Twilio helper library.

Here are a couple more resources to further your learning on Minimal APIs and Twilio:

We can't wait to see what you build. Let us know!

Use Raw String Literals to generate TwiML in C# 11

Niels Swimburger.NET 🍔 — Wed, 15 Feb 2023 00:00:00 +0000

This blog post was written for Twilio and originally published at the Twilio blog.

Over the decades, C# added different ways to create a string, each with their own benefit, and C# 11 is adding another way: Raw String Literals. There are many use cases for Raw String Literals, and one of those use cases is to generate TwiML, but more on that later. In this tutorial, you'll learn how to generate TwiML with Raw String Literals with an ASP.NET Core Minimal API.

Prerequisites

Here's what you will need to follow along:

.NET 7 SDK (later versions will work too)
A code editor or IDE (I recommend JetBrains Rider, Visual Studio 2022 Preview (C# 11 is only supported in preview now, but will work in non-preview in the future.), or VS Code with the C# plugin)
A free Twilio account (sign up with Twilio for free and get trial credit)
A Twilio Phone Number
The ngrok CLI, and optionally, a free ngrok account

Experience with ASP.NET Core and Minimal APIs is not required but recommended. Here's an article that brings you up to speed on how to integrate ASP.NET Core Minimal APIs with Twilio.

You can find the source code for this tutorial on GitHub. Use it if you run into any issues, or submit an issue, if you run into problems.

Before creating your application, let's get you up to speed on how Twilio uses webhooks and TwiML to respond to text messages and voice calls.

Webhooks and TwiML

Using Twilio, you can build programmatic text message and voice call applications. Twilio can do a lot of things like playing audio, gathering input, and recording a call, but Twilio doesn't know what you want to do in response to voice calls and text messages. That's why when Twilio receives a call or text message, Twilio will send an HTTP request to your application asking for instructions.

A webhook is a user-defined HTTP callback. When an event happens in a service, your application is notified of that event using an HTTP request.

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Message>Hi!</Message>
</Response>

You can learn more about TwiML for Programmable Messaging here and TwiML for Programmable Voice here.

Now that you know what webhooks and TwiML are, let's create a voice call application with ASP.NET Core.

Create an ASP.NET Core Minimal API to handle voice calls

First, let's create an ASP.NET Core Minimal API to handle voice calls without using Raw String Literals.

Open a terminal and create a new ASP.NET Core Minimal API project using the .NET CLI:

dotnet new web -o TwimlRawStringLiterals 
cd TwimlRawStringLiterals

These commands will create the project in a folder named TwimlRawStringLiterals and navigate into the folder.

Open the project in your editor, and update the Program.cs file with the following C# code:

using System.Security;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/what-does-the-fox-say", () => Results.Text(
@"<?xml version=""1.0"" encoding=""utf-8""?>
 <Response>
 <Gather action=""/answer"" method=""GET"" input=""speech""> 
 <Say>What does the fox say?</Say>
 </Gather>
 <Say>Ring-ding-ding-ding-dingeringeding!</Say>
 </Response>",
 contentType: "application/xml"
));

app.MapGet("/answer", (string speechResult) => Results.Text(
$@"<?xml version=""1.0"" encoding=""utf-8""?>
 <Response>
 <Say>You said: {SecurityElement.Escape(speechResult)}</Say>
 </Response>",
 contentType: "application/xml"
));

app.Run();

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Gather action="/answer" method="GET" input="speech"> 
<Say>What does the fox say?</Say>
</Gather>
<Say>Ring-ding-ding-ding-dingeringeding!</Say>
</Response>

Take note of how the <Response> node and everything in it is indented with a tab too much.

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Say>You said: ring ding ding</Say>
</Response>

When Twilio receives this TwiML, Twilio will convert "You said: ring ding ding" to speech and stream the audio to the caller.

In case you're not familiar with the song I am referencing in this app, "What does the fox say?" originates from this famous song.

Start your .NET application using your editor or using the terminal using the .NET CLI:

dotnet run

Before integrating Twilio, open a browser and navigate to your web application URL with the suffix /what-does-the-fox-say to verify the TwiML that is generated.

Make the application publicly accessible

For Twilio to be able to send HTTP requests to your local web server, the server needs to become publicly accessible. ngrok is a free secure tunneling service that can make your local web servers public.

To start ngrok, run the following ngrok command in a separate terminal:

ngrok http [YOUR_ASPNET_URL]

Configure your Twilio Phone Number

Now it's time to update your Twilio Phone Number to send HTTP requests to your /what-does-the-fox-say endpoint via the ngrok Forwarding URL. The URL should look something like _ https://cd2f8809cbd0.ngrok.io/what-does-the-fox-say _.

Test that the application works

After you've done that, stop your application using your editor, or if you're using the terminal, press ctrl + c.

C# 11 and .NET 7 only: Use Raw String Literals

C# 11 is still in preview, but you can try it out today using the .NET 7 previews. Make sure you have .NET 7 installed and are using .NET 7 in your project. Then, open your project file at TwimlRawStringLiterals.csproj and add the <LangVersion>preview</LangVersion> to the first <PropertyGroup> node. This will enable the C# 11 preview features.

The TwimlRawStringLiterals.csproj file should look something like this:

<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net7.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>preview</LangVersion>
</PropertyGroup>
</Project>

Now that you have enabled C# 11, let's talk about the new Raw String Literals. To use Raw String Literals, you start and end your string with at least 3 double quotes. For example:

"""
 I like using Raw String Literals,
 because it supports multi-line text,
 you don't need to escape forward slashes (/), backslashes (\), and double quotes\* ("),
 and you can indent your code better without having the indentation included in your string. 
"""

The syntax highlighting may be off in the next snippets and snippet above because the blog doesn't have C# 11 highlighting yet.

You don't have to escape forward slashes, backslashes, and you also don't have to escape your double quotes, more or less. You can use double quotes as long as the number of subsequent double quotes is less than the amount of double quotes you started and ended your string with. This is clearer with an example:

// 3 start and end double quotes, so 1 and 2 subsequent double quotes is allowed
"""This " and this "" is allowed.""" 

// 3 start and end double quotes, so 3 subsequent double quotes is NOT allowed
"""This """ is not allowed."""

// 4 start and end double quotes, so 3 subsequent double quotes is NOT allowed
""""This """ is allowed.""""

Line one and two are valid Raw String Literals because the number of subsequent double quotes is lower than the number of double quotes used to start and end the Raw String Literal.

Raw String Literals also makes it easier to format your strings. Take the following method that generates a Verbatim String (@) as an example:

public string GenerateHtml()
{
return @"
 <p>
 This is a <b>multi-line Verbatim String</b> to generate some HTML
 </p>
";
}

The resulting string will be:


<p>
 This is a <b>multi-line Verbatim String</b> to generate some HTML
<p>

Notice the newlines and the tab indentation? I formatted the Verbatim String for the sake of readability, but as a result I also got the newlines and tabs in it which I do not want.

Let's take a look at the Raw String Literal version:

string GenerateString()
{
return """
 <p>
 This is a <b>multi-line Raw String Literal</b>
 </p>
""";
}

The resulting string now looks like this:

<p>
This is a <b>multi-line Raw String Literal</b>
</p>

Using this knowledge, you can apply Raw String Literals to your endpoints like this:

app.MapGet("/what-does-the-fox-say", () => Results.Text("""
 <?xml version="1.0" encoding="utf-8"?>
 <Response>
 <Gather action="/answer" method="GET" input="speech">
 <Say>
 What does the fox say?
 </Say>
 </Gather>
 <Say>Ring-ding-ding-ding-dingeringeding!</Say>
 </Response>
""",
 contentType: "application/xml"
));

app.MapGet("/answer", (string speechResult) => Results.Text($"""
 <?xml version="1.0" encoding="utf-8"?>
 <Response>
 <Say>You said: {SecurityElement.Escape(speechResult)}</Say>
 </Response>
""" ,
 contentType: "application/xml"
));

Now the resulting TwiML will be readable and not contain unnecessary newlines and tabs.

Other ways to generate TwiML

In the terminal where you ran your project, run the following commands to add the Twilio package and Twilio.AspNet.Core package:

dotnet add package Twilio
dotnet add package Twilio.AspNet.Core

Then, update the Program.cs file with the following code:

using Twilio.AspNet.Core.MinimalApi;
using Twilio.TwiML;
using Twilio.TwiML.Voice;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/what-does-the-fox-say", () =>
{
var response = new VoiceResponse();
var gather = new Gather(
 input: new List<Gather.InputEnum> {Gather.InputEnum.Speech},
 action: new Uri("/answer", UriKind.Relative),
 method: Twilio.Http.HttpMethod.Get
 );
 gather.Append(new Say("What does the fox say?"));
 response.Append(gather);
 response.Say("Ring-ding-ding-ding-dingeringeding!");
return Results.Extensions.TwiML(response);
});

app.MapGet("/answer", (string speechResult) =>
{
var response = new VoiceResponse()
 .Say($"You said: {speechResult}");
return Results.Extensions.TwiML(response);
});

app.Run();

The resulting TwiML will be the same as before. Notice, however, that you don't need to use the SecurityElement.Escape anymore to escape the user input, because the Twilio helper library will XML encode it for you.

Under the hood, the helper library uses the APIs from System.Linq.Xml to generate the XML string which will also do the formatting for you.

Now, you may be wondering, why would you use strings when you can use the Twilio helper library, or maybe you're wondering why you would use the helper library when you can use strings.

They both have their own advantages and disadvantages, but you can mix and match based on your use case.

Next steps

You learned how Twilio uses webhooks and TwiML to give you control over how to respond to a call or text message. You then learned how to generate TwiML instructions using C# 11's new Raw String Literal feature, and compared it to generating TwiML with the Twilio helper library.

Here are a couple more resources to further your learning on Minimal APIs and Twilio:

We can't wait to see what you build. Let us know!

Use Visual Studio dev tunnels to handle Twilio Webhooks

Niels Swimburger.NET 🍔 — Wed, 08 Feb 2023 00:00:00 +0000

This blog post was written for Twilio and originally published at the Twilio blog.

Visual Studio recently introduced a new feature called dev tunnels. By using dev tunnels, Visual Studio will create a new public URL (tunnel URL) for you, and HTTP requests sent to the tunnel URL will be forwarded to your ASP.NET Core project running on localhost.

Dev tunnels has a lot of use cases. You could use this to easily test your web application on other devices like mobiles phone and tablets. You could also use this to make your application temporarily, publicly available for doing interactive demos and inviting your audience to participate.

How to use dev tunnels with Twilio

The most exciting use case for Twilio is that you can use dev tunnels to test webhooks. What are webhooks again?

" Webhooks are user-defined HTTP callbacks. They are triggered by some event in a web application and can facilitate integrating different applications or third-party APIs, like Twilio."

At Twilio, we heavily use webhooks throughout all of our products. You can use webhooks to respond to text messages or to manage voice calls. Here's a diagram of what it looks like when there's an incoming text message to your Twilio Phone Number and your application is handling the messaging webhook:

When your Twilio Phone Number receives a text message, Twilio will send an HTTP request with the message details to your application. Your application then has to respond with TwiML (Twilio Markup Language) to instruct Twilio how to respond. TwiML is XML with special tags defined by Twilio to provide instructions on how to respond to messages and voice calls. In the diagram depicted above, Twilio will respond with "Hi!" because the app responded with TwiML that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<Response>
<Message>Hi!</Message>
</Response>

You can learn more about TwiML for Programmable Messaging here.

Some other types of webhooks will inform you of an event and not expect instructions as a response. However, webhook requests can only be made to publicly available URLs , and when you're developing and testing applications, you are usually doing so locally. But with Visual Studio dev tunnels, you can quickly make your web application public, so you can develop your webhooks!

So when a text message comes in, instead of Twilio sending an HTTP request directly to your application, Twilio can send it to your Visual Studio tunnel which will forward it to your application running on localhost.

In this tutorial, you'll learn how to use Visual Studio to create an ASP.NET Core project that will handle the Twilio SMS webhook, use dev tunnels to make the locally running application public, and then configure a Twilio Phone Number to send webhook requests to the public tunnel URL whenever a text message comes in so you can respond to it.

Prerequisites

You'll need the following things in this tutorial:

A Windows machine (there is no information about support for Visual Studio for Mac yet.)
Visual Studio 2022 Preview, with the ASP.NET and web development workload installed
In Visual Studio, you need to enable the dev tunnels preview feature

You can find details on how to do these steps in the dev tunnels announcement post by Microsoft.

Lastly, you'll also need a Twilio account (trial or upgraded). Try Twilio for free with promotional trial credit.

You can find the source code for this tutorial on GitHub. Use it as a reference if you run into any issues, or submit an issue if you need assistance

Create an empty ASP.NET Core web application with dev tunnels

Open Visual Studio 2022 preview and click " Create a new project", then search for the " ASP.NET Core Empty" project template and click Next.

On the next screen, give your project a meaningful name and click Next. Then on the last screen, leave everything as is and click Create.

Visual Studio will now generate an empty ASP.NET Core application that returns "Hello World!" using Minimal APIs. Go ahead and run your project to see it in action by pressing the green play arrow with your project name.

Visual Studio will run your application, open your browser, and then browse to the web application URL, which will look similar to https://localhost:5000. However, port number 5000 will be a random port number instead. The "Hello World!" application works as expected, but is only reachable on your machine. No other devices can reach it.

Back in Visual Studio, click on the debug dropdown menu, and then select the Dev Tunnels menu item. Now, the Dev Tunnels fly out menu shows you different options, to either select a dev tunnel (but there are none yet), to create a new tunnel, or to open the dev tunnels window.

Click on Create A Tunnel... which opens a dialog to create your tunnel.

Just like the form above, select an Account , enter a Name for the tunnel, set the Tunnel Type to Persistent , and the Access to Public. Then click OK.

When you set the Tunnel Type to Persistent, the URL of your dev tunnel is preserved until you delete the tunnel. When set to Temporary, the URL of the tunnel changes whenever you restart Visual Studio. The Tunnel Access has three options:

Private : Only you can access the web application by logging in with your Visual Studio account. This is great for personally testing your application on other devices.
Organizational : Anyone within your organization can access the web application. You can use this to demo the application to your teammates.
Public : Anyone can access the web application, and you don't need to log in. Make sure it is safe for you to share your application publicly! This is what you need to receive HTTP requests from Twilio webhooks.

Just like before, Visual Studio will open your web browser and browse to your web application. However, this time it will browse to the public tunnel URL, which looks something like https://0pbvlk3m-7032.use.devtunnels.ms/.

Now you can open the same URL on any other device and send it to your friends and colleagues. As long as your project is running, they will be able to use the web application that is running locally on your machine.

Now that your application is publicly available, let's add some Twilio magic to it ☎️

Respond to SMS with ASP.NET Core

Buy a Twilio Phone Number

If you haven't done so already, you'll need to buy a Twilio phone number. To do that, log in to the Twilio Console, select Phone Numbers, and then click on the “ Buy a number ” button. Note that if you have a free account, you will be using your trial credit for this purchase.

On the “ Buy a Number ” page, select your country and check SMS in the “ Capabilities ” field. If you’d like to request a number that is local to your region, you can enter your area code in the “ Number ” field.

Click the “ Search ” button to see what numbers are available, and then click “ Buy ” for the number you like from the results. After you confirm your purchase, click the “ Close ” button.

Handle Twilio SMS webhook in ASP.NET Core

Since TwiML is XML, you can use any technique to produce the XML as a response to the Twilio webhook request. However, you can use the Twilio .NET SDK and the Twilio helper library for ASP.NET Core to make it easier.

First, make sure your application isn't running anymore. You can stop your app by pressing the red square icon in the Visual Studio top toolbar, or close the browser window.

In Visual Studio, right-click on the Dependencies node of your project, and click " Manage NuGet Packages". This will open the NuGet Package Manager window in Visual Studio. Click on the Browse tab and search for " Twilio.AspNet.Core". Click on the Twilio.AspNet.Core package by Twilio Labs, and click the Install button to install the latest version.

By installing the Twilio.AspNet.Core package, you also implicitly install the Twilio NuGet package which has the Twilio SDK. You can also explicitly install the Twilio NuGet package to choose the package version yourself.

Now, replace the code from Program.cs with the following code:

using Twilio.AspNet.Core.MinimalApi;
using Twilio.TwiML;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapPost("/message", () =>
{
var messagingResponse = new MessagingResponse();
 messagingResponse.Message("Ahoy!");
return Results.Extensions.TwiML(messagingResponse);
});

app.Run();

The "Hello World!" GET / endpoint has been replaced with the /message endpoint. The /message endpoint creates a MessagingResponse. messagingResponse.Message("Ahoy!") will create a Message TwiML verb to respond with "Ahoy!".

Lastly, the Results.Extensions.TwiML method will create a TwiMLResult object which takes care of serializing the object to XML and setting the correct content-type headers.

Start your project again. Visual Studio will open the browser again, but you'll see a HTTP 404 error which is fine, this is expected. Copy the URL somewhere as you'll need it in the next section.

Configure your SMS webhook

Now that your web application is running and publicly available through the tunnel, you can now configure the messaging webhook on your Twilio Phone Number.

Back in the Twilio Console, navigate to Phone Numbers > Manage > Active Numbers , or click this link to the Active Numbers page, then click on your Twilio Phone Number to navigate to the configuration page. Scroll down to the Messaging section and where it says " A MESSAGE COMES IN", pick the " Webhook" option in the first dropdown, paste your Visual Studio tunnel URL and add the /message path, and then pick the " HTTP POST" option from the next dropdown.

Click the Save button.

Test your SMS application

Now that everything has been set up, it's time to test out your SMS application. Pull out your personal phone, and send an SMS to your Twilio Phone Number. You should receive a response saying "Ahoy!".

One thing I always do to gain a better understanding is to put a breakpoint in the application and see when the breakpoint is hit. Feel free to put a breakpoint inside the /message endpoint, and play around with the application to see what you can do with Twilio Messaging.

Alternatively, you can test your messaging and voice applications using the Twilio Dev Phone.

Next steps

Congratulations on building this SMS application! Since your application is now public, that also means anyone on the internet can reach it which comes with its own risks. To mitigate these risks, you can validate that the incoming requests originated from Twilio and not some bad actor. In this tutorial, you manually updated the webhook URL in the Twilio console, but you can also do this programmatically. Check out this follow-up tutorial on how to automatically configure your Twilio webhooks using Visual Studio dev tunnels.

Want to learn what else you can do with Twilio? Here are some more tutorials for inspiration:

Shout-out to Sayed I. Hashimi for answering my questions and taking my feedback.

Build a ChatGPT-like SMS Chatbot with OpenAI and Python

Lizzie Siegle — Thu, 12 Jan 2023 21:48:48 +0000

This blog post was written for Twilio and originally published on the Twilio blog.
ChatGPT went viral recently. This conversational machine learning (ML) chatbot developed by OpenAI can answer questions, admit its mistakes, challenge incorrect premises, generate stories and poetry, and more. Read on to learn how to build a ChatGPT-like SMS chatbot using the OpenAI API and Twilio Programmable Messaging with Python.

Read this blog post if you'd like to learn how to build the same application but using Node.js.

ChatGPT and Python

GPT-3 (short for “Generative Pre-training Transformer 3”) is a natural language processing (NLP) model trained on human-generated text. Given some text input, it can generate its own human-like text based in a variety of languages and styles. Here, I ask it to "give me some bars about SendGrid."

You can test it out in the browser here.

The GPT-3 model uses a transformer architecture. This multi-layer neural network is good for processing sequential data, like text. Language-related tasks it can perform include translation, summarization, and question answering, as well as text generation comparable to human text generation.

To make a ChatGPT-like application via SMS with Python, you must use the OpenAI API.

Prerequisites

A Twilio account - sign up for a free one here
A Twilio phone number with SMS capabilities - learn how to buy a Twilio Phone Number here
OpenAI Account – make an OpenAI Account here
Python installed - download Python here
ngrok, a handy utility to connect the development version of our Python application running on your machine to a public URL that Twilio can access. This is needed for the development version of the application because your computer is likely behind a router or firewall, so it isn’t directly reachable on the Internet. You can also choose to automate ngrok as shown in this article. ### Configuration Since you will be installing some Python packages for this project, you will need to make a new project directory and a virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir chatgpt-sms-python
cd chatgpt-sms-python
python3 -m venv venv
source venv/bin/activate
pip install openai twilio flask python-dotenv

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir chatgpt-sms-python
cd chatgpt-sms-python
python -m venv venv
venv\Scripts\activate
pip install openai twilio flask python-dotenv

The last command uses pip, the Python package installer, to install the four packages that you are going to use in this project, which are:

The OpenAI Python client library, to send requests to OpenAI's GPT-3 engine.
The Twilio Python Helper library, to work with SMS messages.
The Flask framework, to create the web application in Python.
The python-dotenv package, to read a configuration file. As mentioned above, this project needs an OpenAI API Key. After making an OpenAI account, you can get an OpenAI API Key here by clicking on + Create new secret key. The Python application will need to have access to this key, so we are going to make a .env file where the API key can safely be stored. The application we create will be able to import this key as an environment variable soon.

Create a .env file in your project’s root directory and enter the following line of text, making sure to replace with your actual key:

OPENAI_API_KEY= <YOUR-OPENAI-KEY>

Make sure that the OPENAI_API_KEY is safe and that you don't expose your .env file in a public location such as GitHub.

Now, your Flask app will need to be visible from the web so Twilio can send requests to it. ngrok lets you do this. With ngrok installed, run ngrok http 5000 in a new terminal tab in the directory your code is in.

You should see the screen above. Grab that ngrok Forwarding URL to configure your Twilio number: select your Twilio number under Active Numbers in your Twilio console, scroll to the Messaging section, and then modify the phone number’s routing by pasting the ngrok URL in the textbox corresponding to when A Message Comes In as shown below:

Click Save and now your Twilio Phone Number is configured so that it maps to your web application server running locally on your computer. Let's build that application now.

Build your ChatGPT-like SMS Python App

Inside your chatgpt-sms-python directory, make a new file called app.py.

Copy and paste the following code into app.py to start off the ChatGPT-like SMS app. It imports the required modules, sets the OpenAI API key from the .env file, and creates a Flask app.

from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse
import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
app = Flask(__name__)

Directly beneath, add the following code:

@app.route("/sms", methods=['POST'])
def chatgpt():
    """get incoming message"""
    inb_msg = request.form['Body'].lower()
    print(inb_msg)
    response = openai.Completion.create(
        model="text-davinci-003",
        prompt=inb_msg,
        max_tokens=3000,
        temperature=0.7
    )
    """Respond to incoming calls with a simple text message."""
    # Start our TwiML response
    resp = MessagingResponse()
    # Add a message
    resp.message(response["choices"][0]["text"])
    print(response["choices"][0]["text"])

    return str(resp)

if __name__ == "__main__":
    app.run(debug=True)

Inside of the /sms webhook, this code creates a variable inbMsg from the inbound text message users will text in and prints it out. It then calls the openai.Completion.create method to use one of their language models to generate text based on inbMsg.

In order to specify our completion, you pass in some properties: model identifies the OpenAI language model used to generate an answer for the text which we assign to the prompt property. In this tutorial, you use the text-davinci-003 language model. It's the same language model used in the background by ChatGPT. The OpenAI docs list the other language models they offer for use.

Optional properties max_tokens and frequency_penalty specify the maximum completion length and the effort the model will make to not repeat itself. You can see more optional properties for completion here in the OpenAI documentation.

You can view the complete app on GitHub here.

Run the Python Flask app by running the following command

python app.py

Now take out your phone and text your Twilio Phone Number a question or prompt so that OpenAI can answer it or generate text!

What's Next for Twilio and ChatGPT?

The development possibilities offered by ChatGPT and Twilio are endless! You can build an SMS chatbot with Python, call an AI friend, chat with an AI chef over WhatsApp, use the OpenAI API with Node.js and Twilio Serverless, and more. Let me know what you're working on with OpenAI or Python–I can't wait to see what you build.

Build a Serverless ChatGPT SMS Chatbot with the OpenAI API

Lizzie Siegle — Mon, 09 Jan 2023 20:48:02 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

ChatGPT went viral recently! This conversational machine learning (ML) chatbot developed by OpenAI can answer questions, admit its mistakes, challenge incorrect premises, generate stories and poetry, and more. Read on to learn how to build a serverless SMS chatbot using ChatGPT and the OpenAI API, Twilio Programmable Messaging, the Twilio Serverless Toolkit, and Node.js.

You can test this chatbot out yourself by texting a question or prompt (like in the GIF above) to +17622490430.

Do you prefer learning via video more? Check out this TikTok summarizing this tutorial in under three minutes!

ChatGPT and Node.js

You can test out ChatGPT in the browser here.

To use ChatGPT in a Node.js application, you must use the OpenAI API.

Prerequisites

A Twilio account - sign up for a free Twilio Account here
A Twilio Phone Number with SMS capabilities - learn how to buy a Twilio Phone Number here
OpenAI Account – make an OpenAI Account here
Node.js installed - download Node.js here ### Get Started with OpenAI After making an OpenAI account, you'll need an API Key. You can get an OpenAI API Key here by clicking on + Create new secret key. Save that API key for later to use the OpenAI client library in your Twilio Function. ### Get Started with the Twilio Serverless Toolkit The Serverless Toolkit is CLI tooling that helps you develop Twilio Functions locally and deploy them to Twilio Functions & Assets. The best way to work with the Serverless Toolkit is through the Twilio CLI. If you don't have the Twilio CLI installed yet, run the following commands on the command line to install it and the Serverless Toolkit:

npm install twilio-cli -g
twilio login
twilio plugins:install @twilio-labs/plugin-serverless

Afterwards, create your new project and install our lone requirement openai:

twilio serverless:init chatgpt-sms --template=blank
cd chatgpt-sms
npm install -s openai

Set an Environment Variable with Twilio Functions and Assets

Open up your .env file for your Functions project in your root directory and add the following line:

OPENAI_API_KEY=YOUR-OPENAI-API-KEY

Replace YOUR-OPENAI-API-KEY with the OpenAI API Key you took note off earlier. Now you can access this API Key if you'd like to do so in your code with context.OPENAI_API_KEY.

Make a Twilio Function with JavaScript

Make a new file in the /functions directory called chatgpt.js containing the following code:

const { Configuration, OpenAIApi } = require("openai");
exports.handler = async function(context, event, callback) {
  const twiml = new Twilio.twiml.MessagingResponse();
  const inbMsg = event.Body.toLowerCase().trim();
  const configuration = new Configuration({
    apiKey: process.env.OPENAI_API_KEY
  });
  const openai = new OpenAIApi(configuration);
  const response = await openai.createCompletion({
      model: "text-davinci-003",
      prompt: inbMsg,
      temperature: 0.7, //A number between 0 and 1 that determines how many creative risks the engine takes when generating text.
      max_tokens: 3000, // Maximum completion length. max: 4000-prompt
      frequency_penalty: 0.7 // # between 0 and 1. The higher this value, the bigger the effort the model will make in not repeating itself.
    });
  twiml.message(response.data.choices[0].text);
  callback(null, twiml);
};

This code imports openAI and makes an async function containing a Twilio Messaging Response object and a variable inbMsg from the inbound text message users will text in. It then initializes a Configuration object and passes it an object to the Configuration constructor containing the property apiKey. The function then calls the openai.createCompletion function to use one of their language models to generate text based on inbMsg.

In order to specify our completion, you need to pass in a configuration object containing two properties: model and prompt. model identifies the OpenAI language model used to generate an answer for the text which we assign to the prompt property. In this tutorial, you'll use the text-davinci-003 language model. It's the same language model used in the background by ChatGPT. The OpenAI docs list the other language models they offer for use.

You can view the complete source code on GitHub here.

Configure the Function with a Twilio Phone Number

To deploy your app to Twilio, run twilio serverless:deploy from the chatgpt-sms root directory. You should see the URL of your Function at the bottom of your terminal:

Grab the Function URL corresponding to your app (the one that ends with /chatgpt) and configure a Twilio Phone Number with it as shown below: select the Twilio number you just purchased in your Twilio Phone Numbers console and scroll down to the Messaging section. Paste the link in the text field for A MESSAGE COMES IN webhook making sure that it's set to HTTP POST. When you click Save, it should look like this!

The Service is the Serverless project name, environment provides no other options, and Function Path is the file name. Now take out your phone and text a question or prompt to your Twilio number.

What's Next for Twilio Serverless and ChatGPT?

The development possibilities offered by ChatGPT and Twilio are endless! You can build an SMS chatbot with Python, call an AI friend, chat with an AI chef over WhatsApp, and more. Let me know what you're working on with OpenAI–I can't wait to see what you build.

Pull Congressional Data via SMS with the Congress API and JavaScript

Lizzie Siegle — Mon, 31 Oct 2022 23:27:27 +0000

This blog post was written for Twilio and originally published on the Twilio blog.

Congress recently came out with an official API so the public can "view, retrieve, and re-use machine-readable data from collections available on Congress.gov!" Read on to learn how to read congressional data using the Congress API, Twilio Functions, and the Twilio Serverless Toolkit.

Text "bill", "amendment", or "summaries" to +12029337044 to receive data about a random bill or amendment!

Prerequisites

A Twilio account - sign up for a free one here
A Twilio phone number with SMS capabilities - configure one here
A Congress.gov API Key – get one here
Postman (you could alternatively make cURL requests from the command line)
Node.js installed - download it here

Make a congress.gov Request

You can look over the different Congress API endpoints offered here. To get someone's profile information, the URL would look like [https://api.congress.gov/v3/bill/117/s/4693/summaries](https://api.congress.gov/v3/bill/117/s/4693/summaries) where 117 is the [Congress numberhttps://en.wikipedia.org/wiki/117th_United_States_Congress, "s" is the type of bill, and "4693" is the bill number. Open Postman and paste that URL into the URL bar.

Add your API Key under Params: the key is api_key and the value is your Congress API Key.

Click Send to hit it with a GET request to see the following data returned:

Again, you can see more Congress API endpoints here you can play around with!

Get Started with the Twilio Serverless Toolkit

The Serverless Toolkit is CLI tooling that helps you develop locally and deploy to Twilio Functions & Assets. The best way to work with the Serverless Toolkit is through the Twilio CLI. If you don't have the Twilio CLI installed yet, run the following commands on the command line to install it and the Serverless Toolkit:

npm install twilio-cli -g
twilio login
twilio plugins:install @twilio-labs/plugin-serverless

Afterwards, create your new project and install our lone requirement [undici](https://github.com/nodejs/undici) to make HTTP requests in Node.js by running:

twilio serverless:init congress-api-sms –template=blank
cd congress-api-sms
npm install -s undici

The Functions and Assets team recommends using undici with your Functions.

Set an Environment Variable with Twilio Functions and Assets

Open up your .env file for your Functions project in your root directory and add the following line:

CONGRESS_API_KEY=YOUR-CONGRESS-API-KEY

Now you can access this API Key if you'd like to do so in your code with context.CONGRESS_API_KEY.

Make a Twilio Function with JavaScript

Make a new file in the /functions directory called congress.js containing the following code:

exports.handler = async (context, event, callback) => {
  const { request } = await import('undici');
  const inbMsg = event.Body.toLowerCase().trim();
  const twiml = new Twilio.twiml.MessagingResponse();
  if(inbMsg.includes("bill")) {
    const {
      body
    } = await request(`https://api.congress.gov/v3/bill?api_key=${context.CONGRESS_API_KEY}`);
    const data = await body.json();
    const billsLength = data.bills.length;
    const randNum = Math.floor(Math.random() * (billsLength - 1)) + 1;
    const bill = data.bills[randNum];
    const { latestAction, title, type, number, originChamber } = bill;

    const billMap = {
        HR: "house bill",
        S: "senate bill",
        HJRES: "house joint resolution",
        SJRES: "senate joint resolution",
        HCONRES: "house concurrent resolution",
        SCONRES: "senate concurrent resolution",
        HRES: "house simple resolution",
        SRES: "senate simple resolution"
    };
    twiml.message(`Bill: ${title}.\nType: ${billMap[type]}\nAssigned bill or resolution number:   ${number}\nIt originated in the ${originChamber} but the latest action was ${latestAction.text} on ${latestAction.actionDate}`);
  }
  else if(inbMsg.includes("amendment")) {
    const {
      body
    } = await request(`https://api.congress.gov/v3/amendment?api_key=${context.CONGRESS_API_KEY}`);
    const data = await body.json();
    const amendmentLength = data.amendments.length;
    const randNum = Math.floor(Math.random() * (amendmentLength - 1)) + 1;
    const amendment = data.amendments[randNum];
    const { latestAction, purpose, congress, number } = amendment;
    twiml.message(`Amendment purpose: ${purpose}\nAssigned amendment or resolution number:${number}\n Latest action was on ${latestAction.actionDate} to ${latestAction.text}.\n Congress: ${congress}.`);
  }
  else if(inbMsg.includes("summaries")) {
    const CURRENT_CONGRESS = 117;
    const {
      body
    } = await request(`https://api.congress.gov/v3/summaries/${CURRENT_CONGRESS}?api_key=${context.CONGRESS_API_KEY}`);
    const data = await body.json();
    const sumLength = data.summaries.length;
    const randNum = Math.floor(Math.random() * (sumLength - 1)) + 1;
    const summaries = data.summaries[randNum];
    const { bill, currentChamber, text, actionDate, actionDesc } = summaries;
    const regexToStripHtmlTags = /<(.|\n)*?>/g;
    const textWithoutHtmlTags = text.replace(regexToStripHtmlTags, '');
    const msg = `Summary title: ${bill.title}.\nStarted in: ${bill.originChamber} on ${actionDate}, currently in the ${currentChamber} chamber in Congress ${bill.congress} and it was ${actionDesc}\n${textWithoutHtmlTags}`; 
    twiml.message(msg);
  }
  else {
    twiml.message(`Send "bill", "amendment", or "summaries"`);
  }
  callback(null, twiml);
};

This code imports undici, makes a Twilio Messaging Response object, creates a variable inbMsg from the inbound text message users will text in and checks if that message input contains bill, amendment, or summaries (three endpoints of the Congress API). The code then generates a random number based on the total amount of items returned from the Congress API to select a random Bill or Amendment, parses the responses from that given endpoint (as we saw using Postman above) into variables such as the type of Amendment or Bill, the most recent date some action was taken, and more!

If someone does not send a message containing "bill", "summaries", or "amendment" (the three topics they can learn about), we return a message saying so. Else, we parse the Congress object returned for information such as the latest action taken on a bill, type of bill, purpose of the given random amendment, origin chamber of a bill, and more. We return a text message containing that data!

You can view the complete app on GitHub here.

Configure the Function with a Twilio Phone Number

To open up our app to the web with a public-facing URL, run twilio serverless:deploy from the congress-api-sms root directory. You should see this at the bottom of your terminal:

Grab the Function URL corresponding to your app (the one that ends with /congress) and configure a Twilio phone number with it as shown below: select the Twilio number you just purchased in your Twilio phone numbers console and scroll down to the Messaging section. Paste the link in the text field for A MESSAGE COMES IN webhook making sure that it's set to HTTP POST. When you click Save it should look like this!

The Service is the Serverless project name, environment provides no other options, and Function Path is the file name. Now take out your phone and text "bill" or "amendment" or "summaries" to your Twilio number.

What's Next for Twilio Serverless and the Congress API?

Twilio's Serverless Toolkit makes it possible to deploy web apps quickly, and Functions & Assets seamlessly handles servers for you. You can do a lot with this data, and you can see what actions Congress are taking--it's fun data to play around with! Let me know online what you're building with Twilio Functions&Assets and don't forget to vote this November!