DEV Community: Joshua Oladokun

Building a Recommender System with Flask and AWS Personalize (Part 2): Setting Up Amazon Personalize

Joshua Oladokun — Wed, 11 May 2022 15:11:23 +0000

In the previous part of this series, we learnt what a recommender system is. We also learnt about some use cases of a recommender system, and how Amazon Personalize works. Here, we would be going through the steps required to set up Amazon Personalize.

Setting Up Amazon Personalize

To get started with Amazon Personalize, you need to have an Amazon Web Service (AWS) account. In case, you don't have an account, follow the instructions here to create an account.

Now that you've created your AWS account, you need to create an AWS Identity and Access Management (IAM) admin user. An IAM user is an entity created to represent a person or application interacting with an AWS. An IAM user with administrative permission has unrestricted access to AWS services in your account. To create an IAM user, follow the set of instructions here.

Before we can use Amazon Personalize, you have to set up permissions that allow your IAM user access to Amazon Personalize Console and API operations. To set up the IAM user policy, follow the instructions here.

You now have to create an IAM role for Amazon Personalize. An IAM role is an IAM identity that you create in your account that has specific permissions. An IAM role is similar to an IAM user, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, an IAM role is not uniquely associated with one person, a role is intended to be assumable by anyone who needs it. To create the IAM Role, follow the instructions here.
Use these configurations while creating the IAM Role:

For Choose the service that will use this role, choose Personalize.
For Attach permissions policies, either choose the IAM user policy you created previously, or choose AmazonPersonalizeFullAccess from the list of Amazon-managed policies.

After creating the IAM role for Amazon Personalize, you have to prepare your training data and upload it to Amazon S3. We would be using a Custom dataset group. Download the user-interactions file here.
You would have to format the user_interaction.csv so that Amazon Personalize can understand the data. We have to change the header rows from user_id, movie_idand timestamp to USER_ID, ITEM_ID, and TIMESTAMP. You can manually enter it with an Excel file or use the Python Pandas package.
Follow these steps to change the header row using pandas

import pandas as pd
# Read user_interactions.csv into a Pandas Dataframe
df = pd.read_csv("user_interactions.csv", encoding='latin-1')
#Rename column name
df = df.rename(columns={"user_Id":"USER_ID","movie_Id": "ITEM_ID","timestamp": "TIMESTAMP"})
#Save dataframe into a new CSV file named interactions.csv
df.to_csv("interactions.csv", index=False)

We would have to upload our interaction.csv file into an AWS S3 bucket. To create an S3 bucket and upload files to it follow the set of instructions here.
For Region, choose US West (Oregon) us-west-2.

After uploading your file to Amazon S3, you have to set up permission that allows Amazon Personalize to access the Amazon S3 bucket. We would be doing two things to set up the permission:

Attaching an Amazon S3 policy to your Amazon Personalize service role.
Attaching an Amazon Personalize access policy to your Amazon S3 bucket.

To attach an Amazon S3 policy to your Amazon Personalize service role, do the following:

Sign in to the IAM console.
In the navigation pane, choose Policies, and choose Create Policy.
Choose the JSON tab, and update the policy as follows. Replace the bucket name with the name of your S3 bucket.

{
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

Choose Next: Tags. Optionally add any tags and choose Review.
Give the policy a name.
For Description, enter a short sentence describing this policy, for example, Allow Amazon Personalize to access its Amazon S3 bucket.
Choose the Create Policy.
In the navigation pane, choose Roles, and choose the role you created for Amazon Personalize.
For Permissions, choose Attach Policies.
To display the policy in the list, type part of the policy name in the Filter policies filter box.
Choose the check box next to the policy you created earlier in this procedure.
Choose Attach policy.

Now you have to attach a bucket policy to the Amazon S3 bucket containing your data. This permission allows Amazon Personalize access to the S3 bucket.

To create a bucket policy:

In the Buckets list, choose the name of the bucket that you want to create a bucket policy.
Choose Permissions.
Under Bucket Policy, choose Edit. This opens the Edit bucket policy page.
On the Edit bucket policy page, replace the JSON in the Policy section, with the following:

{
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "personalize.amazonaws.com"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

Replace bucket-name with the name of the S3 bucket.

Choose the Save Changes to save the policy.

Before starting with Amazon Personalize, you have to determine your use case. Determining your use case will help identify what recipe would be used to train our model. Recipes are Amazon Personalize algorithms that are prepared for specific use cases.
The available use cases for Amazon Personalize include:

Recommending items for users using the USER_PERSONALIZATION recipe.
Ranking items for a user using the PERSONALIZED_RANKING recipe.
Recommending similar items using the RELATED_ITEMS recipe.
Getting user segments using the USER_SEGMENTATION recipes.

Our use cases are for recommending items to users and recommending similar items. To start with Amazon Personalize, we have to create a Custom Dataset group, to do this follow the set of instructions here.
After creating a new Dataset group, you would be directed to the Create interactions dataset page. Follow the following steps to create an interaction dataset and schema:

Specify a name for the dataset group.
Choose Create new schema.
Enter a name to distinguish our schema.
Paste the schema JSON in the Schema definition.

{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    {
      "name": "USER_ID",
      "type": "string"
    },
    {
      "name": "ITEM_ID",
      "type": "string"
    },
    {
      "name": "TIMESTAMP",
      "type": "long"
    }
  ],
  "version": "1.0"
}

Choose Create dataset and continue

Next, we have to import our interaction data into Amazon Personalize from Amazon S3. To do these, we would have to create an import job. Follow the steps here to create a data import job.
After creating the dataset import job, you will be redirected to the Dashboard Overview page.

On the create dataset row, you would see the Interaction data would have Pending status. Amazon Personalize would take about 5-15 mins to create the Interaction dataset.
While we wait for Amazon Personalize to create the Interaction dataset, we would import the Item dataset. Before we import our Item dataset into Personalize, we need to have our Item metadata. You can download the Item metadata file we would be using in this tutorial here. Click here, to learn more about creating an Item dataset.
After downloading our Item metadata we have to upload the file to our previously Amazon S3 bucket. To upload the Item metadata to S3:

Open the Amazon S3 console.
In the Buckets list, choose the bucket's name to which you want to upload your files to.
Choose Upload.
In the Upload window, drag and drop the item metadata file to the Upload window.
Choose Upload at the bottom of the page to upload the file to our Amazon S3 bucket.

To import our Item dataset into Amazon Personalize;

Select the Import Item data ** on the **Overview Dashboard.
Enter a dataset name of your choice that will be easily distinguishable.
Select Create new Schema.
Enter the following JSON schema, in the Schema Definition.

{
    "type": "record",
    "name": "Items",
    "namespace": "com.amazonaws.personalize.schema",
    "fitelds": [
        {
            "name": "ITEM_ID",
            "type": "string"
        },
        {
            "name": "YEAR_RELEASED",
            "type": "int"
        },
        {
            "name": "GENRE",
            "type": "string",
            "categorical": true
        },
        {
            "name": "ACTORS",
            "type": "string",
            "categorical": true
        },
        {
            "name": "DIRECTOR",
            "type": "string"
        }
    ],
    "version": "1.0"
}

Choose Next, you would be directed to the Configure dataset import job page.
Enter the dataset import job name the Configure dataset import job page. The dataset import job name must be between 1-63 characters and shouldn't contain space.

Enter the S3 location of your data. If you don't know the location of the file in the S3 bucket, go to the file location in the S3 console. Check the box of the file and choose the Copy S3 URI.
Paste the S3 URI into the S3 location box on the Configure Dataset import job page.

In the IAM Role section, choose Enter a custom IAM Role ARN. You have to enter the ARN of the IAM Role previously created. To get the ARN, go to the IAM console. Choose Role and choose the IAM role created earlier. On the IAM Role page, you would see ARN. Copy the ARN and paste it into the IAM Role ARN on the Configure dataset import job. Choose Start Import.

After importing data, we have to create a Solution. A Solution is a combination of Amazon Personalize recipe, customized parameters, and one or more solution versions (trained models) that Amazon Personalize uses to generate recommendations. Amazon Personalize provides recipes based on common use cases, for training models. Recipes are Amazon Personalize algorithms that are prepared for specific use cases. You can read more about Amazon Personalize recipes, here

To create the Solution we would be using, head over to the Amazon Personalize Overview page, and choose Create Solution.

On the Create Solution page. Enter a name for the solution. The solution name must have 1-63 characters with no spaces. (Valid characters: a-z, A-Z, 0-9, and _).

For Solution type, choose Item recommendation.

For Recipe, select aws-similar-items. The aws-similar-items or Similar-Items recipe generates recommendations for items that are similar to an item you specify.

Select Create and train solutions. You would then be redirected to the Overview page.

After preparing and importing our data and creating our solutions, we are ready to deploy our solution version to generate recommendations in real-time. A solution version is a trained model that is created as part of a solution in Amazon Personalize. We can deploy our solution version by creating an Amazon Personalize campaign. A campaign is a deployed solution version (trained model) with provisioned dedicated transaction capacity for creating real-time recommendations for your application users. After you create a campaign, you use the GetRecommendations or GetPersonalizedRanking API operations to get recommendations.

Follow the instructions, here to create a campaign. This campaign will be used to get recommendations of items similar to a selected item.
In the next part of this series, we would creating a movie API and integrating Amazon Personalize into our API.

Building a Recommender System with Flask and Amazon Personalize. (Part 1): Introduction

Joshua Oladokun — Fri, 29 Apr 2022 14:48:17 +0000

Have you ever used a website, and gotten a recommendation for similar videos or items when viewing an item? Have you ever visited a site homepage, and gotten item recommendations based on your search history. What is responsible for this is a recommendation system.

A recommender system is a system designed to recommend items to users based on different factors. These factors vary from user's interactions (behavior and preferences) to user's metadata (age, location, gender) to item's metadata. Companies like YouTube, Netflix, Spotify, and Amazon use recommender systems to improve users' experience on their websites.

Common use cases of recommender systems include:

recommendation of similar products on the product detail page.
personalized re-ranking of products for users.
personalized push notifications and marketing email.
recommend products based on user's interaction history on user's home page.
recommend relevant new products to users.

In this tutorial series, you would be using Amazon Personalize and Flask to create a recommender system. Amazon Personalize is a machine learning service managed by Amazon, that makes it easy for developers to deliver personalized experiences to users.
Flask is a Python microframework used for creating web applications.

How does Amazon Personalize Work

In Amazon Personalize, you start by creating a dataset group. A dataset group is a container for storing Amazon Personalize components such as datasets, event trackers, solutions, filters, and campaigns. A dataset group helps organize resources into an independent collection, to prevent resources from one dataset group from interfering with another dataset group. There are two types of dataset groups:

Domain dataset group
Custom dataset group

A Domain dataset group is a dataset group consisting of pre-configured resources for different business use cases. There are 2 pre-configured business domain groups in Amazon Personalize:

ECOMMERCE domain: for getting recommendations for best-selling products on an e-commerce application.
VIDEO_ON_DEMAND domain: for getting similar video recommendations.

With Domain dataset groups, Amazon Personalize manages the life cycle of training models and deployment. When you create a Domain dataset group, you choose your business domain, import your data, and create recommenders for each of your use cases. You use your recommender in your application to get recommendations with the GetRecommendations operation. If you start with a Domain dataset group, you can add custom resources later during the lifecycle of the model. Use a Domain dataset group if you have a video on demand or e-commerce application and want Amazon Personalize to find the best configurations for your use cases.

A Custom dataset group is a dataset group containing custom resources such as solutions, solution versions, filters, and campaigns. When creating a Custom dataset group, you determine your use case, import data, train a model and deploy a campaign. You use the campaign in your application to get recommendations with the GetRecommendation operation. A Custom dataset group is used if you want to configure and manage only custom resources or want to get recommendations in a batch workflow. The developer is responsible for managing the lifecycle of training models and deployment. If you start with a Custom dataset group, you cannot add a domain resource later.

After choosing the dataset group, we would import data into a dataset. A dataset is a container for storing data imported into Amazon Personalize. There is 3 type of datasets in Amazon Personalize: Users, Interactions, and Items. The Items dataset is the only required dataset to create the recommender, with the other two datasets being optional.

Amazon Personalize creates a recommender or a solution depending on the dataset group you are using. It creates and deploys a customized API for getting recommendations in real-time and a batch workflow for getting recommendations in batch.
In the next series, we will be going through steps to set up Amazon Personalize.

Creating A Flask REST API (Part 1):Setting Up a Basic Flask App

Joshua Oladokun — Tue, 22 Feb 2022 16:08:23 +0000

In this tutorial, we would be learning how to create a REST API using Flask. For those who might not know what REST is, REST is an acronym for REpresentational State Transfer. REST is an architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. Like other architectural styles, REST has its guiding principles and constraints.
API is an acronym for Application Programming Interface, it is a set of defined rules that explain how computers or application communicate with each other. APIs provides means for different entity of code (which may always not be written in the same language), to communicate with each other without any need to understand how they both go about carrying out their operations. APIs that comply with REST architectural style guidelines and constraints are called RESTful APIs.
Creating our RESTful APIs in this tutorial, we would be using Flask framework. Flask is a Python web application microframework. The "micro" in framework does not mean that your entire web application has to fit into a single Python file (although it certainly can), nor does it mean that Flask is lacking in functionality. The “micro” in microframework means Flask aims to keep the core simple but extensible. Flask won’t make many decisions for you, such as what database to use. Those decisions that it does make, such as what templating engine to use, are easy to change. Everything else is up to you, so that Flask can be everything you need and nothing you don’t.

Some of the extension we would use in this tutorial includes:

flask-restful - extension for building RESTful APIs.
flask-sqlalchemy - extension that adds support for SQLAlchemy (Python's ORM)
flask-migrate - extension that adds handles SQLAlchemy database migration.
flask-jwt-extended - extension that handles authentication and authorization of users.

In this tutorial we would be building a catalogue API where the users can search for books, borrow available books and return borrowed book, search for books based on genre, authors and publishers. Users would be able to carry out CRUD operations (i.e create, read, update and delete) on their profiles. Our application would have an admin user who can carry out CRUD operation on the library database (i.e the admin user would be able to create, read, update and delete books from our library database).

To begin our project we would create a directory called catalogue_api, then change our working directory to the catalogue_api directory. This is done by entering the command in our command line;

mkdir catalogue_api
cd catalogue_api

The next thing to do is create a virtual environment in our library_api directory for our project. A virtual environment is an isolated Python environment where a project's dependencies are installed in a different directory from those installed in the system's default Python path and other virtual environments. In order to create our virtual environment we would run the venv module as a script with the directory path.

python3 -m venv venv

This would create a venv directory if it doesn't exist before, and also create directories inside it containing a copy of the Python interpreter and various supporting files. Next we have to activate the virtual environment.
On Windows, run:

venv\Scripts\activate.bat

On Mac, run:

source venv/bin/activate

Now we install flask into our virtual environment using pip. We use the pip command:

pip install flask

We create an app.py file in our current working directory were we would create our first Flask Hello World app. Enter the following code into the app.py.

from flask import Flask

app = Flask(__name__)

@app.route("/")
def index():
    return "Hello World"

if __name__ == "__main__":
   app.run()

In the code above, first we import the Flask class from the flask package. Next we create an instance of the Flask class that take in the __name__ as its only argument, Flask uses the __name__ argument to find templates and static files. Then we make use of a built-in route decorator to register the root URL "/" to the index function. The @app.route decorator is used to register a URL to a view function. A view function is simply a function that handles client's request, and values returned are known as response. Whenever a request is sent to the root URL "/" endpoint, the index function is automatically called to handle the request.
The app.run() function is used to run the Flask application on a local development server.

To run the our Flask application, we enter the following code in the terminal.

python app.py

The output looks like this:

We can see our app is running on http://127.0.0.1:5000/. If we enter the URL into any browser of our choice. We would get a response looking like this :

This shows that our app is running well.
But we can see that there is a warning when running our server that says this is a development server and shouldn't be run in a production deployment. The reason for this warning is because Flask thinks we are running our app in a production environment on a development server. To remove the warning we have to let Flask know we are running our app in a development environment, to do this we have to create a new file called .flaskenv in our catalogue_api directory. Enter the following code into the .flaskenv

FLASK_APP = app.py
FLASK_ENV = development

The .flaskenv file is used to store Flask specific configurations. In the first line, we tell Flask to check the app.py file to run our application, and in the second line we set our environment to a development environment. In order for Python to read our .flaskenv file we would need to install a Python library python-dotenv. To install the library into our virtual environment, we enter the following code into our terminal.

pip install python-dotenv

The library reads information from .env files or .flaskenv as key-value pairs. To run our flask app we write the following code into the terminal.

flask run

The output will look like this

We haves successfully set up our Flask app in a the development environment. In the next tutorial we would be looking at creating models and persisting data into our database using Flask-SQLAlchemy, Flask_Migrate and Postgres.