DEV Community: Tony Hung

Predicting the price of Heating Oil using PyCaret

Tony Hung — Wed, 07 Jul 2021 17:55:52 +0000

Predicting the price of Heating Oil using PyCaret

In this notebook, we'll go over how to perform a time series forecasting on the price of heating oil.

Background

In update New York, we are unable to get natural gas to service our home for heating. This is because the rock is mostly made of shale, which makes it tough to get pull natural gas. So we have to rely on oil for heat.

As the price of gas goes up and down, the price of heating oil is the same. There are many companies that distribute oil and all of them have different prices. Also, these prices change between seasons. So, I wanted to build an application that predicts the price of oil, so that I know when is the best time to buy.

Data

Before building the models, i needed to get data. I have been unable to find a dataset with the current price of oil, therefore I had to build my own. The website cheapestoil.com shows the price of oil for many companies in the northeast United States. The site shows the latest prices for these companies, but they do not show the previous prices.

So in order to get the prices, I build a AWS lambda function that scrapes the price of oil daily. I used AWS CloudWatch events to run a lambda function every 12 hours, in order to fetch the prices for that time. This lambda extracted the last updated date, price and supplier, and save these results as JSON and save to an S3 bucket. After the json data is saved, I have another lambda function, attached as a trigger, to read each json file, and save into DynamoDB. Please see this GitHub repo for more detail on how I build these lambda functions.

I started this project back in Dec 2020 in order to build up my dataset. The lambda function has been running for about 6 months, and I have a decent amount of data to work with. In order to expand my dataset, I was able to pull more data using The Wayback Machine on web.archive.org. The Wayback Machine stores a snapshot of many pages on the internet. It doesn't have every site, but it did have some snapshots from cheaptestoil.com. To get that data, I used https://github.com/hartator/wayback-machine-downloader to download the archive data. The archive only had 7 snapshots, between the dates of Aug 2020 and Oct 2021.

In all, I have about 5k records of all the oil prices from the website

Fetching Saved Data

I used DynamoDB to store the oil price data, and used Boto3 to fetch the data, which I then save to a CSV.

import boto3
from boto3.dynamodb.conditions import Key
import pandas as pd
dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('heating_oil_prices')
response = table.scan()
df = pd.DataFrame(response["Items"])
df.to_csv("data.csv")

def get_data():
    df = pd.read_csv("data.csv", usecols=["last_updated", "price150","price500", "price300", "supplier", "state"])
    df["last_updated"] = pd.to_datetime(df["last_updated"])
    df = df.set_index("last_updated")
    df["state"] = df["state"].apply(lambda x: "NewYork" if str(x) == "nan" else x)
    df = df.sort_index()
    return df
df = get_data()

On Cheapestoil.com, the have the price of oil in gallons, but the price is slightly different for how many gallons you buy. If we get suppliers in the state of New York, we'll see the following

    state = "NewYork"
    suppliers_by_state = df[ (df["state"] == state)].dropna()
    suppliers_by_state.iloc[1]

price500          1.449
price300          1.469
price150          1.549
supplier    Suffolk Oil
state           NewYork
Name: 2020-08-03 15:11:16, dtype: object

The row above shows that the price for 500 gallons(price500) is $1.449 per gallon, 300 gallons(price300) is $1.69 and 150 gallons(price150) is $1.549

Lets see how many suppliers we have for New York

suppliers_by_state["supplier"].value_counts().sum()

Since we have so many suppliers, a forecasr for the average price of oil for all the suppliers might be a better way to go, since the prices are simlar between every company

df = df.reset_index()
data = df[ df["state"] == state].resample('d', on='last_updated').mean().dropna()
data = data.reset_index()

Here is the mean price of oil over all the suppliers in New York.

data.head()

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	last_updated	price500	price300	price150
0	2020-08-03	1.672000	1.719400	1.76140
1	2020-08-04	1.632429	1.625750	1.68075
2	2020-10-29	1.852500	1.819455	1.85300
3	2020-11-11	1.786250	1.759000	1.76900
4	2020-11-12	1.420000	1.460000	1.54000

For a quick check on our data, let's plot the price for 500 gallons

data["price500"].plot()

<AxesSubplot:>

Model

Now, we can start building our model. We'll be using PyCaret to build our time series forecast.
Before modeling, we need to update the dataset to remove the date and replace with numeric values. To do this, I've included fastai's add_datepart function to convert the data is series of features, split by year, month, day, day of week, and much more

from fastai.tabular.all import *
add_datepart(data, field_name="last_updated")

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	price500	price300	price150	last_updatedYear	last_updatedMonth	last_updatedWeek	last_updatedDay	last_updatedDayofweek	last_updatedDayofyear	last_updatedIs_month_end	last_updatedIs_month_start	last_updatedIs_quarter_end	last_updatedIs_quarter_start	last_updatedIs_year_end	last_updatedIs_year_start	last_updatedElapsed
0	1.672000	1.719400	1.761400	2020	8	32	3	0	216	False	False	False	False	False	False	1.596413e+09
1	1.632429	1.625750	1.680750	2020	8	32	4	1	217	False	False	False	False	False	False	1.596499e+09
2	1.852500	1.819455	1.853000	2020	10	44	29	3	303	False	False	False	False	False	False	1.603930e+09
3	1.786250	1.759000	1.769000	2020	11	46	11	2	316	False	False	False	False	False	False	1.605053e+09
4	1.420000	1.460000	1.540000	2020	11	46	12	3	317	False	False	False	False	False	False	1.605139e+09
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
56	2.220000	2.260000	2.340000	2021	6	26	28	0	179	False	False	False	False	False	False	1.624838e+09
57	2.632621	2.626935	2.646290	2021	7	26	2	4	183	False	False	False	False	False	False	1.625184e+09
58	2.606429	2.582633	2.611437	2021	7	27	5	0	186	False	False	False	False	False	False	1.625443e+09
59	2.619422	2.609042	2.632478	2021	7	27	6	1	187	False	False	False	False	False	False	1.625530e+09
60	2.571294	2.545455	2.575238	2021	7	27	7	2	188	False	False	False	False	False	False	1.625616e+09

61 rows × 16 columns

The add_datepart function generates lots of feature for the date, but we don't need to use all of them. For this model, we'll use last_updatedYear, last_updatedMonth and last_updatedDay. In future models, we can try to use other features.

data = data[['price500', 'last_updatedYear', 'last_updatedMonth', 'last_updatedDay']]

mask = round(len(data) * 0.7)
train = data[:mask]
test = data[mask:]
train.shape, test.shape

((43, 4), (18, 4))

After we split the data into a train, test set, we then must initalize the regression setup in PyCaret. This includes suppling the dataset, the feature we are predicting on (price500) and the other features to use(last_updatedYear,last_updatedMonth and last_updatedDay)

from pycaret.regression import *
s = setup(data = train, test_data = test, target = 'price500', fold_strategy = 'timeseries', numeric_features = ['last_updatedYear','last_updatedMonth', 'last_updatedDay'], fold = 3, transform_target = True, session_id = 42)

Next we call the compare_models function to find the best model using the Mean Absolute Error(MAE), which is the mean of the absolute difference between the models prediction and expected values.

best = compare_models(sort = 'MAE')

From the results above, it looks like the Passive Aggressive Regressor has the lowest MAE error(0.0605), so we'll use that model on the test set

preds = predict_model(best)

Next, we'll use this model and generate a forecast for the next 7 days worth of prices

forecast_df = pd.DataFrame()
forecast_df["last_updated"] = pd.date_range(start='2021-07-08', periods=8)
add_datepart(forecast_df, 'last_updated')
forecast_df = forecast_df[['last_updatedYear', 'last_updatedMonth', 'last_updatedDay']]
forecast_df

predictions = predict_model(best, data=forecast_df)
predictions

Once we make our predictions, we'll then merge the predictions to the orginal data and plot the last 15 days in the dataframe

def pad_value(day):
    value = str(int(day))
    print()
    if len(value) ==1:
        return f'0{value}'
    return value

results_df = pd.concat([data,predictions], axis=0)
dates = []
for idx, x in results_df.iterrows():
    date_str =  f'{pad_value(x["last_updatedYear"])}-{pad_value(x["last_updatedMonth"])}-{pad_value(x["last_updatedDay"])}'
    dates.append(pd.to_datetime(date_str, format='%Y-%m-%d'))
results_df["date"] = dates

results_df.drop(["last_updatedYear", "last_updatedMonth", "last_updatedDay"], axis=1,inplace=True)
results_df = results_df.set_index('date')

results_df[-15:].plot()

The blue line above shows the actual prices and the orange are the predicitons, which do not look the best. If we zoom in to the predictions. we'll see a increasing in price per day

results_df[-8:].plot()

Saving The Model

After we trained our model, we can now save and use for forecasting on other data

save_model(best, "model")

Transformation Pipeline and Model Succesfully Saved





(Pipeline(memory=None,
          steps=[('dtypes',
                  DataTypes_Auto_infer(categorical_features=[],
                                       display_types=True, features_todrop=[],
                                       id_columns=[], ml_usecase='regression',
                                       numerical_features=['last_updatedYear',
                                                           'last_updatedMonth',
                                                           'last_updatedDay'],
                                       target='price500', time_features=[])),
                 ('imputer',
                  Simple_Imputer(categorical_strategy='not_available',
                                 fill_value_ca...
                                                  regressor=PassiveAggressiveRegressor(C=1.0,
                                                                                       average=False,
                                                                                       early_stopping=False,
                                                                                       epsilon=0.1,
                                                                                       fit_intercept=True,
                                                                                       loss='epsilon_insensitive',
                                                                                       max_iter=1000,
                                                                                       n_iter_no_change=5,
                                                                                       random_state=42,
                                                                                       shuffle=True,
                                                                                       tol=0.001,
                                                                                       validation_fraction=0.1,
                                                                                       verbose=0,
                                                                                       warm_start=False),
                                                  shuffle=True, tol=0.001,
                                                  validation_fraction=0.1,
                                                  verbose=0,
                                                  warm_start=False)]],
          verbose=False),
 'model.pkl')

Training Model for all Prices

After we have a intial model, we can now train 3 seperate models for each price(price150, price300 and price500)
We'll refactor the model training code into a function that trains each price, gets the model with the best score, and saves the model to a seperate file

def train(data, price, state="NewYork"):
    data = data.reset_index()
    data = data[ data["state"] == state].resample('d', on='last_updated').mean().dropna()
    data = data.reset_index()

    add_datepart(data, field_name="last_updated")
    data = data[[price, 'last_updatedYear', 'last_updatedMonth', 'last_updatedDay']]   
    mask = round(len(data) * 0.7)
    train = data[:mask]
    test = data[mask:]
    print("train",train.shape)
    print("test",test.shape)

    s = setup(data = train, test_data = test, target = price, fold_strategy = 'timeseries', numeric_features = ['last_updatedYear','last_updatedMonth', 'last_updatedDay'], fold=2, transform_target = True, session_id = 42)
    best = compare_models(sort = 'MAE')
    save_model(best, f'{price}_model')

for feature in ["price150", "price300", "price500"]:
    df = get_data()
    train(df, feature)

Transformation Pipeline and Model Succesfully Saved

Conclusion and What's Next

In this post, we've seen how to build a time series model for forecasting the price of heating oil. In the next post, we'll go over how to deploy these models into a StreamLit application.
We'll also go over how the process on how the data was collected.
We'll also go over how the process on how the data was collected.

Adding two-factor authentication to your iOS app using TypingDNA

Tony Hung — Sun, 28 Feb 2021 00:07:56 +0000

By now, most people use a password as a way to login into an account. But we’ve all seen the problems with passwords. For example, they need to be memorable but not easy to guess.

Many people, including myself, have used passwords that are easy to remember. Unfortunately, this makes it very easy for other people to guess your password and hack your account.

Luckily, in recent years, there have been great strides made in security that allows users to generate very hard-to-guess passwords and store them securely. But passwords themselves have been around for a long time, and maybe there’s another way to have access to an account without having to remember a string of text.

What if you could use something unique to you that can not be hacked or stolen?

The team over at TypingDNA has thought about this problem in depth. Luckily for us, they came up with a great way to authenticate you as a user, using the way you type.

The following post will cover how the TypingDNA Authentication API works and how to integrate into an IOS application in four easy steps.

Create a TypingDNA Account
Integrate the TypingDNARecorderMobile framework
Integrate the TypingDNA Authentication API
Put it all Together

The source code for the IOS is available on GitHub. I’ve also created a demo of the application, which is available on YouTube.

TypingDNA overview

Here’s how TypingDNA’s technology works: Every person has a unique way of typing. This includes many factors, which are a key to verifying your identity.

The TypingDNA Authentication API has a way of recording your typing pattern. After a few attempts of training to the way you type, you can use that typing pattern to login into an application. Behind the scenes, TypingDNA uses a machine learning model to categorize how you type and saves this pattern as your biometric.

After three or so attempts of learning, the TypingDNA system can record that pattern for future use when you try to log in.

Create a TypingDNA account

Before we build out the iOS application, we’ll need to know how to interact and use the TypingDNA Authentication APIs.

To do that, you first need to create an account here. Once authenticated, navigate to the API dashboard here and copy the apiKey and apiSecret.

When we build out our application, we will need to add these keys into the app. For now, just save them for later.

iOS app

Learning about how TypingDNA works is great. But as a developer, I usually like to see things in action. So I’ve worked on an iOS demo application that shows how to use the TypingDNA API in a Signup/Login scenario.

If you want to go straight to the code, please have a look at this GitHub repo. You will need to have the latest version of Xcode installed and be on a Mac. (Sorry, Windows people.)

App overview

This demo app shows how to integrate the TypingDNA Authentication API into your application. For this app, I’ve created a SignupViewController that allows a user to enter an email address and a password.

The user will need to verify their pattern by entering in their username and a password at least three times. This allows the TypingDNA system to learn your typing pattern.

After the third time the username and password are entered, the user will be able to login with the same username and password and be directed to the Authenticated ViewController, which shows the status of the authentication as well as how many times the user tried to verify themselves.

Next, we’ll build out the interface in the main storyboard.

Our design looks like this:

We first have a NavigationController, which loads our second ViewController, which I called SignupViewController.swift. This view contains two text fields that allow the user to enter a username and a password. Once the user is authenticated, they will be brought to the rightmost screen, which is the AuthenticateViewContoller.swift. This simulates when a user is logged in to an application.

Integrating TypingDNARecorderMobile framework

Before we start calling the TypingDNA Authentication API, we need a way to record the typing pattern of the user. First, you will need to install the Typing DNA recorder framework. Download this repo, and inside the TypingDNARecorderMobile folder, copy theTypingDNARecorderMobile.swift file into your Xcode project.

This framework will allow you to enter a set of text and will return a typing pattern. This pattern, as a string, will be used in the TypingDNA Authentication API to learn your typing pattern.

When using the typing pattern recorder on a mobile device, the way you physically hold the device is taken into account when generating a pattern. For example, if you try to signup with a username and password and hold the device in portrait, you would not be able to login when using the same username and password while holding the device in landscape mode.

For more information about this, please see the Mobile Positions FAQ and the Mobile integration process tutorial.

Before we can use this framework, we first need to add two text fields to our storyboard and create an outlet. In the SignupViewController, inside ViewDidLoad, we need to tell the recorder what to listen to. We do that by adding the following:

@IBOutlet weak var typing_pattern: UITextField!
 override func viewDidLoad() {
        super.viewDidLoad()
    TypingDNARecorderMobile.addTarget(user_txt_field)
    TypingDNARecorderMobile.addTarget(password_txt_field
    }

The addTarget function is a listener that is attached to both the username and password text fields. When we add a target to both fields, this tells the TypingDNARecorderMobile framework to capture both text patterns, which will increase the number of characters used for generating a typing pattern. The recommended number of characters is around 30. Using both text fields will help us get closer to that number rather than using the password textfield alone.

As the user begins to type, the TypingDNARecorderMobile framework will record the typing pattern for both inputs. Once the text is entered and the Next button is tapped, we grab the pattern using the following function:

TypingDNARecorderMobile.getTypingPattern()

For our application, we will set type to 1 and set all the other parameters to their defaults.

When we set type to 1, we will use the getTypingPattern function to expect short, identical texts, which are the username and password.

We then call this function inside our IBAction:

@IBAction func signupButtonTapped(_ sender: Any) {
    guard let userId = user_id.text, userId.count > 6 else {
        return
    }
    let typingPattern = TypingDNARecorderMobile.getTypingPattern(1, 0, "", 0, nil);

}

Once the user enters their username and password, the getTypingPattern will return the user's typing pattern. Now, we can start integrating the TypingDNA Authentication API.

Integration of TypingDNA Authentication API

Auto API

Now, in Xcode, we’ll create a new class called NetworkManager.swift. This class will make the requests to the API. Once we have our pattern, we need to start the authentication process. We will be using the /auto endpoint to send our pattern.

For this demo, we’ll be making the API calls within our iOS application. However, making these requests server-side would be the better approach if you are using this in a production setting.

In NetworkManager.swift, we first need to add the TypingDNA API key and secret, which should have been created previously. On lines 13 and 14 in NetworkManager.swift, update the api_key and api_secret to your keys.

Next, we’ll need a way to send the pattern to the API. Inside NetworkManager.swift,
we’ll have a function called save_pattern, which accepts the username and typing pattern.

We need to send the username and the typing pattern to the endpoint. To send it, the TypingDNA Authentication API requires us to use a unique identifier for the user. In our case, we will use MD5 to hash the username. Finally, we can make a request to the /auto endpoint.

class NetworkManager {  
    private let base_url = "https://api.typingdna.com"
    var api_key = "xxx"
    var api_secret = "xxx"

    func save_pattern(user_id:String, typing_pattern:String, completionHandler:@escaping (_ result:[String: Any]) -> Void) {
            let hashed_user_id = MD5(string: user_id)
            let serviceUrl = URL(string: base_url+"/auto/"+hashed_user_id)
            let parameterDictionary = ["tp" : typing_pattern]
             var request = URLRequest(url: serviceUrl!)
             request.httpMethod = "POST"
             request.setValue("Application/json", forHTTPHeaderField: "Content-Type")
                let authString = encode_auth()
                request.setValue(authString, forHTTPHeaderField: "Authorization")
                guard let httpBody = try? JSONSerialization.data(withJSONObject: parameterDictionary, options: []) else {
                    return
                }
             request.httpBody = httpBody
             let session = URLSession.shared
             session.dataTask(with: request) { (data, response, error) in
                 if let data = data {
                     do {
                        let json = try JSONSerialization.jsonObject(with: data, options: [])
                        completionHandler(json as! [String : Any])
                     } catch {
                         print(error)
                     }
                 }
             }.resume()
        }

}

In the code above, we make a POST request to /auto endpoint, passing in the MD5 version of the username and send the typing_pattern(tp) in the body of the request. For authentication, we base64 our API key and secret and add that to the Authorization header of the request.

After we send the first request, we should receive a JSON payload:

{"message": Pattern(s) enrolled. Not enough patterns for verification., "action": enroll, "message_code": 10, "status": 200, "enrollment": 1}

In the action parameter, we receive the word enroll. This means that we have begun the authentication process.

Check API

The next step is to make a request to the /check endpoint. This will allow us to verify that the pattern was accepted and allow us to log in, as long as we have sent at least three typing patterns.

func check_user(user_id:String, completionHandler:@escaping (_ result:[String: Any]) -> Void) {

        let hashed_user_id = MD5(string: user_id)
        let serviceUrl = URL(string: base_url+"/user/"+hashed_user_id + "?type=1")

         var request = URLRequest(url: serviceUrl!)
         request.httpMethod = "GET"
         request.setValue("Application/json", forHTTPHeaderField: "Content-Type")

            let authString = encode_auth()
            request.setValue(authString, forHTTPHeaderField: "Authorization")

         let session = URLSession.shared
         session.dataTask(with: request) { (data, response, error) in

             if let data = data {
                 do {
                     let json = try JSONSerialization.jsonObject(with: data, options: []) as? [String: Any]
                    completionHandler(json!)
                 } catch {
                     print(error)
                 }
             }
         }.resume()
    }

This is very similar to the /auto endpoint call. We pass the MD5 version of the username and a type parameter, which is set to 1. This is to tell the /check endpoint that our typing pattern came from a set of repeated text, rather than being a random sentence. If we set the type to 0, it would mean that we allow the user to enter any text they want.

Putting it all together

After we finished writing our two API calls, we can now put it all together.

In the SignupViewController, after we enter a username and record a typing pattern, we can now request the /auto endpoint to send the typing pattern. Once we send the typing pattern, we need to imminently call the /check API to verify that the enrollment process has begun.

When using a free account, the API calls are limited to one call per second. Therefore, we need to add a one-second delay when making a request to the /check endpoint.

We will use the response from the /check endpoint to know what the status of the authentication process is and when we can have the user login.

First, let’s make a request to /auto endpoint and send the username as well as the typing pattern.

let typingPattern = TypingDNARecorderMobile.getTypingPattern(1, 0, "", 0, typing_pattern);
networkManager.save_pattern(user_id: userId, typing_pattern: typingPattern) { (json) in
user.pattern_response = json
}

This will return the following JSON response:

{"status":200,"message_code":10,"action":"enroll","message":"Pattern(s) enrolled. Not enough patterns for verification.”,"enrollment":1}

The full documentation for the response is here.

For our application, we are considered the action parameter. From the documentation:

“Returns one of the following values, corresponding to the action performed on the request: enroll, verify, verify;enroll. Enroll is returned if the typing profile has less than 3 previous enrollments. Verify is returned if initial enrollments are met, only a verification was performed, and the pattern was not additionally enrolled. If both a verification and enrollment are performed, the value is verify;enroll.”

Next, we need to call the /check endpoint. This is used to get the number of times the user has tried to authenticate. The JSON payload from the /check app looks like this:

{"mobilecount":1,"type":1,"count":0,"message":"Done","status":200,"success":1,"message_code":1}

For our example, we need to know the mobilecount, which is the number of times the user tried to authenticate. Please note that the mobilecount parameter will only increment when on a mobile device. If you are using the APIs on the web, the “count” parameter will be incremented.

After the user makes the first attempt of authenticating, the “action” parameter will return “enroll”, which means that the user is in the process of authentication. After the user makes two more attempts of authenticating, the “action” parameter will return “verify;enroll”, which means that the user has been authenticated and can now log in.

Once they log in using the same username and password, they are directed to the AuthenticatedView Controller, and can now use the application as a logged-in user. The following is a video of the demo application:

Conclusion

The TypingDNA Authentication API allows you as a developer to easily add authentication to your iOS application—without having to worry about storing encrypted passwords, which makes it easier for you to focus on building your application.

At the same time, users won’t need to worry about security breaches or hacks that can access your data. Also they no longer need another service to save their password.

To access their accounts, all they need to do is type like they do all day long. It’s the easiest way to ensure only authorized individuals can access their own accounts.

Introduction to GPT-3

Tony Hung — Mon, 05 Oct 2020 13:21:52 +0000

If you haven’t noticed, AI is everywhere, and we’ve finally come to a point where it is in almost everything we interact with. From Amazon product recommendations to Netflix suggestions to autonomous driving, and writing excellent blog posts… Don’t worry, this post was written by a human, for now.

Many people see how AI is used, but have you ever wondered how it comes to be?

This post will look at a very popular model used for many tasks, including generating news articles, image generation, and even building HTML sites.

Introduction to AI

First, let’s go through a basic summary of AI: It’s a series of algorithms that can learn a specific task and make predictions on similar tasks. One of these tasks could predict if an image contains a picture of a cat or a dog.

In this example, we gather lots of these images and feed them into an algorithm. We then label each image, as in “This is a photo of a dog” or “This is a cat photo”. The algorithm “learns” which images contain a dog or cat. The model makes assumptions of what constitutes a dog (big ears, fluffy tail) and a cat (whiskers, eye shape) and can learn these differences.

We give our model hundreds of images known as “training,” with these, the model forms a good idea of what a dog and cat look like. Finally, we will give a model a new image, and it should be able to tell us if this is an image of a dog or cat.

If you are interested in learning how to build a model to identify dogs, please look at this post.

The idea of training a model with examples (images of cats and dogs) and being able to make predictions on new images is Deep Learning, which is a subset of AI.

We won’t be going over how to train a model for this post, but we will go over a very popular model that can do more interesting things.

In June 2020, a company called OpenAI (founded by Elon Musk), released a new model called GPT-3, which is capable of generating new content, given a small number of examples of input data.

Examples of how you could use this include:

Question and Answering
Summarizing sentences
Translation
Text Generation
Image Generation
Performing three-digit arithmetic
Unscrambling words

About GPT-3

GPT-3 is a deep learning language model, meaning that this model is trained on thousands of articles from Wikipedia, web sites, and books.

When a model is trained, its output is a series of parameters, commonly a multidimensional array of numbers. These numbers represent what the model has learned.

GPT-3 contains 175 billion parameters. For perspective, Microsoft also came out with a language model that uses only 10 billion parameters.

For a model to learn from the given data, it needs to be trained. This training is done by feeding the model each word of a given text and then predicting the next word.

This training is computationally expensive and requires many GPUs to train. According to one estimate, training of the GPT-3 model costs $4.6 million.

How Does GPT-3 Work

GPT-3 is known as a (G)enerative (P)re-trained (T)ransformer. By being generative, it means that it can generate new text, given an input of text.

For example, if we give the model the following text:

“The sky is”

The model should be able to predict that the next word is “blue”.

If I give it another sentence:

“The quick brown fox”

The model would first make a prediction “jumped,” then using the previous sentence (“The quick brown fox jumped”), it should predict the word “over,” and so on.

Another part of the GPT-3 is the transformer. The transformer is an architecture, developed by Google, that allows a model to remember or give higher weight to a phrase or set of phrases in a given sentence that has the most importance.

Language models are built using a Recurrent Neural Network. This neural network architecture takes a sentence, word by word, and feeds into the network. What makes it recurrent, is that the output from the previous word is an input to the next word in the sentence.

These models can only deal with numbers. Therefore, the text needs to be converted into a number. One way of converting text into a number is though word embeddings.

A word embedding turns words into a 3d vector space that can capture the meaning of a word using its relation to other words. An excellent example of a word embedding is a way to understand the similarities between the words “brother” and “sister” as compared to “man” and “women”.

In the above image, we can see that the word “brother” resides in the same physical space as the word “man”. This embedding was also learned by a model to read lots of text and come up with these similarities.

This word embedding is fed into the next word embedding, allowing the model to “remember” the previous words using its embedding.

One of the big problems is that RNNs are generally not good at remembering long sentences.

Let us take this sentence:

“Tony Hung is a software Engineer at Vonage. He likes to write about Artificial Intelligence over on the Vonage blog. He lives in upstate New York, with his wife, 4-year-old daughter, and dog.”

AN RNN would take each word (“Tony”, “Hung”, “is”), and feed into the network as a word embedding. Over time the model may forget the word “Tony” since it was the first word in the sentence. If I ask the model the following question, “Who works for Vonage”, it would need to go back in the sentence, find the word “Vonage”, and try to find the noun associated with the question. Since the word “Tony” is so far in the past, the RNN may not be able to find it.

The Transformer architecture helps solves this problem, which was proposed in the paper Attention is All You Need and uses a concept called Attention.

Attention is part of a neural network layer that can focus on specific parts of the sentence. As we stated before, an RNN model can capture every word in the sentence, but if the embedding is too large, the model may not remember everything.

With Attention, each embedding also now contains a score on how important the specific word is. So now, the RNN does not have to remember every word embedding, but rather, just the word embeddings with a higher score than the other word embeddings.

For a more detailed description of the transformer and Attention, check out Jay Alammar’s visual transformer blog post.

GPT-3 Samples

Still with me? Great! Let’s get into some examples of how GPT-3 is used.

With access to the OpenAI API, you can supply training data, which contains a sample input and what the output should be. You might be saying, “This is great, how can I get started and use it?”, OpenAI has not made the model publicly available, but only as an API which is in closed beta access, which means you would have to request access to use the API. At the time of this writing, I have not been accepted yet to the beta.

The good news is that many people have and can give a detailed explanation of how the API works.

Let’s go through some of the examples of other developers using GTP-3.

Text To HTML Using GTP-3

One of many usages of GPT-3 is to generate HTML from a given string.

The input into the OpenAI API would consist of a string, as well as its HTML equivalent.

Input: bold the following text. “GPT-3”

We would supply the output of:

<b>GPT-3</b>

The OpenAI API allows a developer to supply these input and output texts to GPT-3. Then, on the OpenAI servers, the API will send this input and output to GPT-3 to “learn” the supplied input and what the output should be.

Then, if we supply a new series of text to the OpenAI API:

“Center and bold the word GPT-3”.

Its output will be

<center><b>GPT-3</b></center>

We did not tell GPT-3 anything about the <center> tag since GPT-3 most likely contained HTML strings during its training process.

Here is an example of what this looks like:

This is mind blowing.

With GPT-3, I built a layout generator where you just describe any layout you want, and it generates the JSX code for you.

W H A T pic.twitter.com/w8JkrZO4lk

— Sharif Shameem (@sharifshameem) July 13, 2020

Text Adventure

Other examples of developers using GPT-3 is a text adventure game from aidungeon.io

AI dungeon generates a story in which you can navigate using text — also known as multi-user dungeon. In this example, entering the words “Look Around” will generate a new set of text about the scenery. This feature is only using GPT-3 to generate text after each input.

Text to Regex

This example is the one that got me. By supplying a set of text input and its regex equivalent, you can generate valid regular expressions using readable English.

I once had a problem and used regex. Then I had two problems

Never again. With the help of our GPT-3 overlords, I made something to turn English into regex. It's worked decently for most descriptions I've thrown at it. Sign up at https://t.co/HtTpJ16V4F to play with a prototype pic.twitter.com/trJA7VRrsf

— Parthi Loganathan (@parthi_logan) July 25, 2020

More examples of developers using GPT-3 to build exciting things can be found at buildgpt3.com.

Conclusion

Through this post, we’ve been able to go through a basic understanding of what GPT-3 is and how it is built. However, we didn’t dive in too much on this and other AI models’ technical aspects. To get a deep dive into GPT-3, look at Jay Alammar’s Video on How GPT-3 Works. It is a great starting point on how AI models can be trained.

If you are new to the technical aspects of AI, which includes Deep Learning, please look at Fast.ai, a free course that goes over what deep learning is and how to get started.

I hope this post has helped you understand what GPT-3 is, from both a technical and non-technical standpoint. If you are interested in learning more about GPT-3 and what other projects OpenAI is doing, please check them out at OpenAI.com.

The post Introduction to GPT-3 appeared first on Vonage Developer Blog.

Building a Machine Learning Model for Answering Machine Detection

Tony Hung — Fri, 22 Mar 2019 10:54:12 +0000

Did you ever need a way to detect when an answering machine was on a voice call? No? Thats ok. I did!

Prerequisites

This post assumes you have basic Python experience, as well as having a very basic understanding of machine learning. We'll go over a few basic concepts on machine learning, and we have linked to more resources throughout this post.

A few weeks ago, I received a request from one of our sales engineers about an answering machine detection service for a client. They wanted a way to send a message to a answering machine when the call went to voicemail.

I've done some research on this, and it does seem possible, but I couldn't find anything on HOW this was done. So I decided to figure it out...

The first thought was to build a machine learning model that detects when the beep sound in an answering machine is heard. In this post, we'll go over how the model was trained and deployed into a application.

Training Data

Before we can start building a machine learning model, we need to have some data. For this problem, we need to have a bunch of audio files with the answering machine beep sounds, like this:
Beep sound 1
or this:
Beep sound 2

We also need to include samples that don't include the beep sound:
Non-Beep sound 1
or this:
Non-Beep sound 2

Since this kind of data doesn't seem to exist on the internet, we needed to gather as many samples as possible of beeps and other sounds from calls, in order to train our model. To do this, I built a webpage that allows anyone to record their voicemail greeting message.

When you call the Nexmo number, the application will create an outbound call to the same number. When the call is received, you just need to send the call directly to voicemail. From there, we record the call using the record action and save the file into a Google Cloud Storage bucket. After gathering a lot of examples, we can start looking at the data.

In any machine learning project, one of the first things to do is to look at the data and make sure it's something we can work with.

Since it's audio, we can't look at it directly, but we can visualize the audio files using a mel-spectrogram, which looks like this:

A mel-spectrogram shows a range of frequencies (lowest at the bottom of the display, highest at the top) and shows how loud events are at different frequencies. In general, loud events will appear bright and quiet events will appear dark.

We'll need to load a few files of both types of sounds, plot them, and see how they look. To show the mel-spectrogram, we'll use a Python package called Librosa to load the audio recording, then plot the mel-spectrogram using matplotlib, another Python package to plot charts and graphs.

import glob
import librosa
import matplotlib.pyplot as plt
%matplotlib inline

def plot_specgram(file_path):
  y, sr = librosa.load(file_path)
  S = librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128,fmax=8000)
  plt.figure(figsize=(10, 4))
  librosa.display.specshow(librosa.power_to_db(S,ref=np.max),y_axis='mel', fmax=8000,x_axis='time')
  plt.colorbar(format='%+2.0f dB')
  plt.title(file_path.split("/")[-2])
  plt.tight_layout()

sound_file_paths = [
                    "answering-machine/07a3d677-0fdd-4155-a804-37679c039a8e.wav",
                    "answering-machine/26b25bb7-6825-43e7-b8bd-03a3884ed694.wav",
                    "answering-machine/2a685eda-8dd9-4a4d-b00e-4f43715f81a4.wav",
                    "answering-machine/55b654e5-7d9f-4132-bc98-93e576b2d665.wav",
                    "speech-recordings/110ac98e-34fa-42e7-bbc5-450c72851db5.wav",
                    "speech-recordings/3840b850-02e6-11e9-aa3d-ad1a095d8d72.wav",
                    "speech-recordings/55b654e5-7d9f-4132-bc98-93e576b2d665.wav",
                    "speech-recordings/81270a2a-088b-4e3c-9f47-fd927a90b0ab.wav"
                    ]

for file in sound_file_paths:
  plot_specgram(file)

Let's see what each audio file looks like.

You can clearly tell which audio file is a beep and which is just speech.

Before we train our model, we will take all the recordings that we have for both beeps and non-beeps, which are labeled as speech, and convert each recording into a vector of numbers, since our model will only accept numbers, not images.

To compute the data, we'll use the mel-frequency cepstral coefficients (MFCCs) of each sample. Then, we'll save this value into a csv so that we do not have to re-compute the MFCC's over again.

For each audio sample, the csv will contain the path to the audio sample, the label of audio sample(beep, or speech), the MFCC, and the duration of the audio sample (using the get_duration function in librosa). We also tried a few other audio characteristics including chroma, contrast and tonnetz). However, these features were not used in the latest version of the model.

Let's now take a look at the first 5 rows of the csv, just to see what the data looks like.

Each row contains a 1 dimension vector of each of the audio features. This is what we'll use to train our model.

Training

Now we'll take this data and train a model with it. We'll be using the Scikit-learn package to do our training. Scikit-learn is a great package that allows you to build simple machine learning models without having to be a machine learning expert.

For each model, we took our dataframe, which contained the label of each audio file, (beep, speech), with the MFCC for each sample, split it into a train and test dataset, and ran each model through the data.

def train(features, model):
  X, y = generateFeaturesLabels(features)
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

  model.fit(X_train, y_train)
  print("Score:",model.score(X_test, y_test))

  cross_val_scores = cross_val_score(model, X, y, cv=5, scoring='f1_macro')
  print("cross_val_scores:", cross_val_scores)
  print("Accuracy: %0.2f (+/- %0.2f)" % (cross_val_scores.mean(), cross_val_scores.std() * 2))

  predictions = model.predict(X_test)

  cm = metrics.confusion_matrix(y_test, predictions)
  plot_confusion_matrix(cm, class_names)

  return model

The function train takes a list of features that we want to use, which is just MFCC of the audio sample, as well as the model we want to train on. Then we print our score, which is how well the model performed. We also print the cross validation score. This makes sure that our model was trained correctly. The plot_confusion_matrix function plots a confusion matrix that shows exactly what the model got correct and incorrect.

We then tried the following models and included their accuracy (0-100% score on how well the model did).

All these models performed very well, except Support Vector Machines. The best was Gaussian Naive Bayes, so we will use that model. In our Confusion Matrix from above, out of the 67 examples, 40 samples that were predicted as a beep were actually beeps, and 22 samples that were predicted to be speech were, in fact, speech examples. However, 1 example that was predicted to be a beep was actually speech.

After we have our model, we need to save it to a file, then import this model into our VAPI application.

import pickle
filename = "model.pkl"
pickle.dump(model, open(filename, 'wb'))

Building the Application

The last part is to now integrate our model into a VAPI application.
Source

When building an application, you have to first create a Nexmo application and purchase a Nexmo number.

We'll build an application that lets a user dial a Nexmo number. We'll then ask the user to enter a phone number to call. Once that number is entered, we'll connect that call into the current conversation and connect to our websocket. Using Nexmo websockets, we are able to stream the audio call into our application.

First, we need to load our model into our application.

loaded_model = pickle.load(open("models/model.pkl", "rb"))

When the user first dials the Nexmo number, we return a NCCO with the following:

class EnterPhoneNumberHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        ncco = [
              {
                "action": "talk",
                "text": "Please enter a phone number to dial"
              },
              {
                "action": "input",
                "eventUrl": ["https://3c66cdfa.ngrok.io/ivr"],
                "timeOut":10,
                "maxDigits":12,
                "submitOnHash":True
              }

            ]
        self.write(json.dumps(ncco))
        self.set_header("Content-Type", 'application/json; charset="utf-8"')
        self.finish()

We first send a Text-To-Speech action into the call asking the user to enter a phone number. When the phone number is entered, we get those digits from the https://3c66cdfa.ngrok.io/ivr url.

class AcceptNumberHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def post(self):
        data = json.loads(self.request.body)
        ncco = [
             {
             "action": "connect",
              "eventUrl": ["https://3c66cdfa.ngrok.io"/event"],
               "from": NEXMO_NUMBER,
               "endpoint": [
                 {
                   "type": "phone",
                   "number": data["dtmf"]
                 }
               ]
             },
              {
                 "action": "connect",
                 "eventUrl": ["https://3c66cdfa.ngrok.io/event"],
                 "from": NEXMO_NUMBER,
                 "endpoint": [
                     {
                        "type": "websocket",
                        "uri" : "ws://3c66cdfa.ngrok.io/socket",
                        "content-type": "audio/l16;rate=16000"

                     }
                 ]
               }
            ]
        self.write(json.dumps(ncco))
        self.set_header("Content-Type", 'application/json; charset="utf-8"')
        self.finish()

After the phone number is entered, we will receive a callback from the https://3c66cdfa.ngrok.io/ivr url. Here we take the phone number the user entered from data["dtmf"] and perform a connect action to that phone number, then perform another connect action into our websocket. Now our websocket is able to listen in on the call.

As the call is streamed into the websocket, we need to capture chunks of speech using Voice Activity Detection, save into a wave file, and make our predictions on that wav file using our trained model.

class AudioProcessor(object):
    def __init__(self, path, rate, clip_min, uuid):
        self.rate = rate
        self.bytes_per_frame = rate/25
        self._path = path
        self.clip_min_frames = clip_min // MS_PER_FRAME
        self.uuid = uuid
    def process(self, count, payload, id):
        if count > self.clip_min_frames:  # If the buffer is less than CLIP_MIN_MS, ignore it
            fn = "{}rec-{}-{}.wav".format('', id, datetime.datetime.now().strftime("%Y%m%dT%H%M%S"))
            output = wave.open(fn, 'wb')
            output.setparams((1, 2, self.rate, 0, 'NONE', 'not compressed'))
            output.writeframes(payload)
            output.close()
            self.process_file(fn)
            self.removeFile(fn)
        else:
            info('Discarding {} frames'.format(str(count)))
    def process_file(self, wav_file):
        if loaded_model != None:
            X, sample_rate = librosa.load(wav_file, res_type='kaiser_fast')
            mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=40).T,axis=0)
            X = [mfccs]
            prediction = loaded_model.predict(X)
            if prediction[0] == 0:
                beep_captured = True
                print("beep detected")
            else:
                beep_captured = False

            for client in clients:
                client.write_message({"uuids":uuids, "beep_detected":beep_captured})

        else:
            print("model not loaded")
    def removeFile(self, wav_file):
         os.remove(wav_file)

Once we have a wav file, we use librosa.load to load in the file, and then use the librosa.feature.mfcc function to generate the MFCC of the sample. We then call loaded_model.predict([mfccs]) to make our prediction. If the output of this function is 0, a beep was detected. If it outputs 1, then it's speech. We then generate a JSON payload of whether a beep was detected, and the uuids of the conversation. This way, our client application can send a TTS into the call, using the uuids.

Websocket Client

The final step is to build a client that connects to the websocket, observes when a beep is detected, and sends a TTS into the call, when the voicemail is detected.

Source

First, we need to connect to the websocket.

ws = websocket.WebSocketApp("ws://3c66cdfa.ngrok.io/socket",
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open

ws.run_forever()

Next, we just listen for any incoming message from our websocket.

def on_message(ws, message):
    data = json.loads(message)
    if data["beep_detected"] == True:
        for id in data["uuids"]:
            response = client.send_speech(id, text='Answering Machine Detected')

        time.sleep(4)
        for id in data["uuids"]:
            try:
                client.update_call(id, action='hangup')
            except:
                pass<a href="https://www.nexmo.com/wp-content/uploads/2019/02/amd-confusion-matrix.png"><img src="https://www.nexmo.com/wp-content/uploads/2019/02/amd-confusion-matrix-600x300.png" alt="" width="300" height="150" class="alignnone size-medium wp-image-28012" /></a>

<a href="https://www.nexmo.com/wp-content/uploads/2019/02/amd-df.png"><img src="https://www.nexmo.com/wp-content/uploads/2019/02/amd-df-600x300.png" alt="" width="300" height="150" class="alignnone size-medium wp-image-28015" /></a>

<a href="https://www.nexmo.com/wp-content/uploads/2019/02/amd-eda.jpg"><img src="https://www.nexmo.com/wp-content/uploads/2019/02/amd-eda-600x300.jpg" alt="" width="300" height="150" class="alignnone size-medium wp-image-28018" /></a>

We'll parse the incoming message as JSON, then check the beep_detected property is True. If it is, then a beep was detected. We will then send a TTS into the call saying 'Answering Machine Detected', then perform a hangup action into the call.

Conclusion

We've shown how we built a answering machine detection model with 96% accuracy, using a few audio samples of beeps and speech in order to train our model. Hopefully, we've shown how you can use machine learning in your projects. Enjoy!