DEV Community: ICCHA Technologies

Sentiment Analysis: A Practical Guide to Data Collection, Preparation, and Model Training

ICCHA Technologies — Tue, 23 May 2023 02:05:36 +0000

Sentiment Analysis has become an essential tool in the field of data science and machine learning. It enables us to understand the underlying sentiments in text data, which can be valuable for various tasks like product reviews, customer feedback, or social media analysis. In this guide, we will walk you through the steps of creating a sentiment analysis model, starting from data collection to model evaluation. For this example, we will focus on product reviews as our topic.

Data Collection and Preparation

The first step is to collect a dataset containing product reviews. Online platforms like Amazon or Yelp are good sources for such data, as are publicly available datasets on platforms such as Kaggle or UCI Machine Learning Repository. For this example, we will use a small synthetic dataset of product reviews. We'll make sure to include some 'dirty' elements, such as missing values, irrelevant information, and unclean text.

import pandas as pd
import numpy as np

# Creating a synthetic dataset
data = {
    'user_id': ['user1', 'user2', 'user3', 'user4', 'user5', 'user6'],
    'timestamp': ['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04', '2023-01-05', '2023-01-06'],
    'review': ['This product is amazing!', 'I hated it, would not recommend.', 'Best purchase I\'ve ever made!!!', 'Absolutely terrible. Stay away!', 'Pretty good.', np.nan],
    'rating': [5, 1, 5, 1, 4, 3],
}

df = pd.DataFrame(data)

After collecting the dataset, the next step is data preprocessing. This step is critical as the quality of data affects the performance of the machine learning model. Here are some common preprocessing steps:

Remove Irrelevant Information

First, we will drop the columns that are not relevant to the sentiment of the review. In this case, the 'user_id' and 'timestamp' columns:

df = df.drop(['user_id', 'timestamp'], axis=1)

Handle Missing Values

Next, we need to handle the missing review text. Since we can't infer the sentiment of a missing review, we'll drop any rows with missing 'review' values:

df = df.dropna(subset=['review'])

Text Cleaning

We'll now clean the text data. This involves removing punctuation, converting all text to lowercase, and removing any unnecessary whitespace. Let's use Python's built-in string methods:

df['review'] = df['review'].str.lower()  # Convert to lowercase
df['review'] = df['review'].str.replace('[^\w\s]', '')  # Remove punctuation
df['review'] = df['review'].str.strip()  # Remove unnecessary whitespace

Now, our data is clean and ready for labelling.

Labelling the Data

In this step, we assign labels to the reviews indicating whether they are positive or negative. For our product reviews, we could use a simple criterion like the star rating: reviews with 4 or 5 stars are labelled as 'positive', and reviews with 1 or 2 stars as 'negative'. For more nuanced criteria, you could manually review and label a subset of the data or use a pre-trained sentiment analysis model to assign initial labels. We'll create a new column, 'label', to store these:

df['label'] = df['rating'].apply(lambda x: 'positive' if x > 3 else 'negative')

Our data is now preprocessed and labelled, and ready for feature extraction.

Feature Engineering

Great! Now that we have preprocessed and labelled our data, the next step is to extract features from our text data that will be used to train the machine learning model.

Imagine you're a detective investigating a crime. You have a bunch of evidence, including fingerprints, footprints, DNA samples, and security camera footage. Your goal is to extract useful features from these pieces of evidence to help you identify the culprit.

In sentiment analysis, feature engineering is similar to the detective's task. Instead of physical evidence, we have a collection of text documents (such as reviews, tweets, or customer feedback) that express opinions or sentiments. The aim is to extract meaningful and relevant features from this text data that can be used to determine the sentiment associated with each document.

Feature extraction involves creating informative features from the text data that can help classify the sentiment of the reviews. Here are a few common text feature extraction methods:

Bag of Words: This is the simplest form of text representation, where each document is represented as a vector in a high-dimensional space.
TF-IDF: This is a more sophisticated method that weighs each word by how unique it is to a document in comparison to the entire corpus.
Word Embeddings: This is an advanced method where each word is represented as a high-dimensional vector capturing semantic information.

We'll use the TF-IDF (Term Frequency-Inverse Document Frequency) method for this task. TF-IDF is a numerical statistic that reflects how important a word is to a document in a corpus. We will use the TfidfVectorizer from the Scikit-Learn library. This will convert our text data into a matrix of TF-IDF features.

from sklearn.feature_extraction.text import TfidfVectorizer

# Create a vectorizer object
vectorizer = TfidfVectorizer()

# Fit and transform the 'review' column
tfidf_matrix = vectorizer.fit_transform(df['review'])

# Convert the matrix into a DataFrame for better readability
tfidf_df = pd.DataFrame(tfidf_matrix.toarray(), columns=vectorizer.get_feature_names_out())

Splitting the Data

Now that we've transformed our text data into numerical features using TF-IDF, we're ready to split our data into training and testing sets. This is an essential step in building any machine learning model. The model is trained on the training set and then evaluated on the unseen testing set. This process helps us understand how well our model can generalize to new, unseen data.

We'll use the train_test_split function from Scikit-Learn to do this. We'll follow the common practice of using 80% of the data for training and 20% for testing.

from sklearn.model_selection import train_test_split

# Split the data
X_train, X_test, y_train, y_test = train_test_split(tfidf_df, df['label'], test_size=0.2, random_state=42)

Let's delve deeper into the details of what's happening here:

train_test_split function: This function from Scikit-learn is used to divide our dataset into training and testing sets. It takes in several parameters:

tfidf_df: This is the first parameter, representing the features our model will learn from.
df['label']: The second parameter, which is the target variable or labels that our model will aim to predict.
test_size=0.2: An optional parameter that specifies the proportion of the dataset to be included in the test split. Here, we have set it to 0.2, implying that 20% of the data will be used for testing.
random_state=42: Another optional parameter that manages the shuffling applied to the data prior to executing the split. The number 42 is arbitrary and can be set to any integer. It ensures that the splits you generate are reproducible.

Outputs of the train_test_split function:

X_train and X_test: These are the portions of tfidf_df assigned for training and testing, respectively.
y_train and y_test: Similarly, these are the portions of df['label'] assigned for training and testing, respectively.

Model Selection and Training

At this stage, we need to pick a suitable machine learning algorithm for our sentiment classification task. Here are a few potential options:

Logistic Regression: This is a simple, yet potent algorithm for binary classification problems. It's efficient to train and usually offers good baseline performance.
Naive Bayes: This is a probabilistic classifier that makes strong independence assumptions. It tends to be particularly effective when dealing with high-dimensional text data.
Decision Trees: These models are straightforward to understand and interpret, and they can handle both numerical and categorical data.
Deep Learning Models: For more intricate tasks or larger datasets, deep learning models like Recurrent Neural Networks (RNNs) or Transformers can deliver superior results. However, they demand more computational resources and additional time for training.

After deciding on a model, we'll train it using our labeled training data and the features we extracted earlier.

For our specific task, we're going to use a Logistic Regression model. Logistic Regression is often a good starting point for text classification tasks due to its simplicity and efficiency. Let's proceed to train a Logistic Regression model with Scikit-Learn:

from sklearn.linear_model import LogisticRegression

# Create a Logistic Regression model
model = LogisticRegression()

# Train the model on our training data
model.fit(X_train, y_train)

In the above code, we first import the LogisticRegression class from the Scikit-Learn library. We then create an instance of this class, which represents our model. The fit method is used to train the model on our training data.
At this point, our Logistic Regression model has been trained on our training data. The model has learned to associate the TF-IDF features with the 'positive' and 'negative' labels.
The next step is to use this trained model to make predictions on our testing data, and then evaluate how well it performed. Let's proceed with the model evaluation.

Model Evaluation

With our trained model ready, we can now evaluate its performance. This is done by making predictions on our test data and comparing those predictions to the actual labels.

We will use common metrics for binary classification problems to assess the model's performance: accuracy, precision, recall, and F1-score. These metrics offer different viewpoints on how well our model performs:

Accuracy: This is the ratio of total predictions that are correct.
Precision: This is the ratio of positive predictions that are indeed correct.
Recall: This is the ratio of actual positive instances that our model correctly identified.
F1-score: This is the harmonic mean of precision and recall, providing a balance between these two metrics.

Let's see how to generate these metrics using Scikit-Learn:

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Generate predictions on the test data
y_pred = model.predict(X_test)

# Calculate metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, pos_label='positive')
recall = recall_score(y_test, y_pred, pos_label='positive')
f1 = f1_score(y_test, y_pred, pos_label='positive')

# Print the metrics
print('Accuracy:', accuracy)
print('Precision:', precision)
print('Recall:', recall)
print('F1-score:', f1)

With this, we've evaluated our model's performance. These metrics provide a starting point to understand how well our model is performing. Remember, in practice, the choice of metrics depends on your specific problem and objectives.
This completes our guide to building a sentiment analysis model. Remember that this is a simplified example. Real-world data will often require more extensive preprocessing and feature engineering, and you may need to try multiple models or tune their parameters to achieve optimal performance.

OAuth Bearer Token Using Python

ICCHA Technologies — Mon, 11 Jul 2022 19:50:00 +0000

As you build your own APIs, examining your use cases will help you decide which security methods to implement for them. For some use cases, API keys are sufficient; in others, you’ll want the additional protection and flexibility of tokens.

Solution

A bearer Token would include an embedded digital signature and be encrypted. The digital signature confirms that the token came from a trusted source. The use of encryption means only systems meant to be able to read the token can read it.

Recipe

We are going to use the API from https://developer.here.com/, so please follow these steps to get your OAuth token.

You will get a 'credentials.properties' file created from above, it contains two values of interest, the Access Key ID and the Access Key Secret. We will be needing them going forward.

It looks like this:
```
here.user.id = HERE-USER_ID
here.client.id = YOUR_CLIENT_ID
here.access.key.id = YOUR_ACCESS_KEY
here.access.key.secret = YOUR_ACCESS_KEY_SECRET
here.token.endpoint.url = https://account.api.here.com/oauth2/token
```
Install Python and the environment required to run the file.
```
  pip install virtualenv
```
Then create your main folder, and create the environment.
```
    cd your_folder
    virtualenv venv -p python3
```
before starting the environment, create at the main folder your "app.py"
to put your code.

You will now have a venv folder, go to quotes/venv/Scripts and run "activate" to active the virtual environment.

Add the imports required to run the request.

    import requests #Needed for making HTTP requests
    import time #Needed to generate the OAuth timestamp
    import urllib.parse #Needed to URLencode the parameter string
    from base64 import b64encode  #Needed for create_signature function
    import hmac  #Needed for create_signature function
    import hashlib #Needed for create_signature functionx
    import binascii#Needed for create_signature function

Be sure to install the request modules in your environment

    pip install requests

Creating the Oauth Signature

For requesting a token, we need to pass the OAuth signature in the Authorization Header of a request. Signature plays an important role as it is used for authenticating a user or application. It requires us to create a base string containing various parameters and then pass it into an HMAC-SHA256 hashing algorithm.

Create the Parameter String

We need 6 key-value pair for generating a signature

grant_type - Value always remains same, "client_credentials"
oauth_consumer_key - The Access Key ID value we acquired from credentials.properties file
oauth_nonce - A unique string which never repeats
oauth_signature_method - Always use "HMAC-SHA256"
oauth_timestamp - The number of seconds since the Unix epoch, in simple words, the current time
oauth_version - Always use "1.0"

The values of the parameter looks like this.

    grant_type = 'client_credentials'
    oauth_consumer_key = 'HERE.ACCESS.KEY.ID'  # replace  your credentials.properties file
    access_key_secret = 'HERE.ACCESS.KEY.SECRET'  # replace credentials.properties file
    oauth_nonce = str(int(time.time()*1000))
    oauth_timestamp = str(int(time.time()))
    oauth_signature_method = 'HMAC-SHA256'
    oauth_version = '1.0'

The important thing here is that the type of all 6 parameters has to be string. For calculating nonce and timestamp we have used the time module of Python.

Next, we alphabetically combine all the parameters as a single string, separating each key value pair with an ampersand character ("&") and then URL-encoding it.


    def create_parameter_string(grant_type, oauth_consumer_key, oauth_nonce, oauth_signature_method, oauth_timestamp, oauth_version):
        parameter_string = ''
        parameter_string = parameter_string + 'grant_type=' + grant_type
        parameter_string = parameter_string + '&oauth_consumer_key=' + oauth_consumer_key
        parameter_string = parameter_string + '&oauth_nonce=' + oauth_nonce
        parameter_string = parameter_string + \
            '&oauth_signature_method=' + oauth_signature_method
        parameter_string = parameter_string + '&oauth_timestamp=' + oauth_timestamp
        parameter_string = parameter_string + '&oauth_version=' + oauth_version
        return parameter_string

    parameter_string = create_parameter_string(grant_type, oauth_consumer_key,oauth_nonce,oauth_signature_method,oauth_timestamp,oauth_version)

    encoded_parameter_string = urllib.parse.quote(parameter_string, safe='')

The parameter_string is a simple concatenated output containing key-value pair separated by an ampersand character. With the help of the urllib python library, we got our URL-encoded output in encoded_parameter_string. It looks like this:

parameter_string

    grant_type=client_credentials&oauth_consumer_key=XXXXXXXXXX&oauth_nonce=1585745318447&oauth_signature_method=HMAC-SHA256&oauth_timestamp=1585745318&oauth_version=1.0

encoded_parameter_string

    grant_type%3Dclient_credentials%26oauth_consumer_key%3DXXXXXXXXXX%26oauth_nonce%3D1585745318447%26oauth_signature_method%3DHMAC-SHA256%26oauth_timestamp%3D1585745318%26oauth_version%3D1.0

Create Signature Base String

Next we need to add the HTTP method (POST), base URL and encoded parameter string into a single string called base string.

    url = 'https://account.api.here.com/oauth2/token'

    encoded_base_string = 'POST' + '&' + urllib.parse.quote(url, safe='')
    encoded_base_string = encoded_base_string + '&' + encoded_parameter_string

Base string looks like this.

    POST&https%3A%2F%2Faccount.api.here.com%2Foauth2%2Ftoken&grant_type%3Dclient_credentials%26oauth_consumer_key%3DXXXXXXXXXX%26oauth_nonce%3D1585747084344%26oauth_signature_method%3DHMAC-SHA256%26oauth_timestamp%3D1585747084%26oauth_version%3D1.0

POST, URL and encoded parameter string separated by ampersand ("&")

Create Signgin Key

The signing key is the URL-encoded access key secret, followed by an ampersand ("&"). The access key secret is the value of "here.access.key.secret" property in credentials.properties file.

    access_key_secret = 'HERE.ACCESS.KEY.SECRET'#From credentials.properties file
    signing_key = access_key_secret + '&'

Combine all to create OAuth Signature

The signature base string and the signing key created above, are passed to the HMAC-SHA256 Hashing Algorithm and the output is converted to a base64 string. Finally, we have our OAuth Signature.

    def create_signature(secret_key, signature_base_string):
    encoded_string = signature_base_string.encode()
    encoded_key = secret_key.encode()
    temp = hmac.new(encoded_key, encoded_string, hashlib.sha256).hexdigest()
    byte_array = b64encode(binascii.unhexlify(temp))
    return byte_array.decode()

    oauth_signature = create_signature(signing_key, encoded_base_string)

    encoded_oauth_signature = urllib.parse.quote(oauth_signature, safe='')

The create_signature method takes secret key and base string as an input, performs hashing, converts the output into a base64 string. Then we URL-encode the output for further usage.

Request Token

Once we have our signature, the rest of the process is straight forward. All we require now is to create an Authorization header for the request and then make a request.

For this, first we will combine -

oauth_consumer_key - The value of "here.access.key.id" from credentials.properties file
oauth_nonce - Already have
oauth_signature - The value of encoded_oauth_signature from above
oauth_signature_method - "HMAC-SHA256"
oauth_timestamp - Already have
oauth_version - "1.0"

and append them to a string beginning with “OAuth”.

    body = {'grant_type' : '{}'.format(grant_type)}

    headers = {
                'Content-Type' : 'application/x-www-form-urlencoded',
                'Authorization' : 'OAuth oauth_consumer_key="{0}",oauth_nonce="{1}",oauth_signature="{2}",oauth_signature_method="HMAC-SHA256",oauth_timestamp="{3}",oauth_version="1.0"'.format(oauth_consumer_key,oauth_nonce,encoded_oauth_signature,oauth_timestamp)
              }

    response = requests.post(url, data=body, headers=headers)

    print(response.text)

The request body must contain grant_type value as 'client_credentials', always.

The output of the code looks like this-

    {
        "access_token":"eyJhbGci...",
        "token_type":"bearer",
        "expires_in":86399
    }

Add requirements file
```
    (venv) pip freeze > requirements.txt
```
be sure to install requests module

The Complete Code

import requests #Needed for making HTTP requests
import time #Needed to generate the OAuth timestamp
import urllib.parse #Needed to URLencode the parameter string
from base64 import b64encode  #Needed for create_signature function
import hmac  #Needed for create_signature function
import hashlib #Needed for create_signature functionx
import binascii#Needed for create_signature function

grant_type = 'client_credentials'
oauth_consumer_key = 'HERE.ACCESS.KEY.ID' #From credentials.properties file
access_key_secret = 'HERE.ACCESS.KEY.SECRET'#From credentials.properties file
oauth_nonce = str(int(time.time()*1000))
oauth_timestamp = str(int(time.time()))
oauth_signature_method = 'HMAC-SHA256'
oauth_version = '1.0'
url = 'https://account.api.here.com/oauth2/token'

# HMAC-SHA256 hashing algorithm to generate the OAuth signature
def create_signature(secret_key, signature_base_string):
    encoded_string = signature_base_string.encode()
    encoded_key = secret_key.encode()
    temp = hmac.new(encoded_key, encoded_string, hashlib.sha256).hexdigest()
    byte_array = b64encode(binascii.unhexlify(temp))
    return byte_array.decode()

# concatenate the six Oauth parameters, plus the request parameters from above, sorted alphabetically by the key and separated by "&"
def create_parameter_string(grant_type, oauth_consumer_key,oauth_nonce,oauth_signature_method,oauth_timestamp,oauth_version):
    parameter_string = ''
    parameter_string = parameter_string + 'grant_type=' + grant_type
    parameter_string = parameter_string + '&oauth_consumer_key=' + oauth_consumer_key
    parameter_string = parameter_string + '&oauth_nonce=' + oauth_nonce
    parameter_string = parameter_string + '&oauth_signature_method=' + oauth_signature_method
    parameter_string = parameter_string + '&oauth_timestamp=' + oauth_timestamp
    parameter_string = parameter_string + '&oauth_version=' + oauth_version
    return parameter_string

parameter_string = create_parameter_string(grant_type, oauth_consumer_key,oauth_nonce,oauth_signature_method,oauth_timestamp,oauth_version)
encoded_parameter_string = urllib.parse.quote(parameter_string, safe='')
encoded_base_string = 'POST' + '&' + urllib.parse.quote(url, safe='')
encoded_base_string = encoded_base_string + '&' + encoded_parameter_string

# create the signing key
signing_key = access_key_secret + '&'

oauth_signature = create_signature(signing_key, encoded_base_string)
encoded_oauth_signature = urllib.parse.quote(oauth_signature, safe='')

#---------------------Requesting Token---------------------
body = {'grant_type' : '{}'.format(grant_type)}

headers = {
            'Content-Type' : 'application/x-www-form-urlencoded',
            'Authorization' : 'OAuth oauth_consumer_key="{0}",oauth_nonce="{1}",oauth_signature="{2}",oauth_signature_method="HMAC-SHA256",oauth_timestamp="{3}",oauth_version="1.0"'.format(oauth_consumer_key,oauth_nonce,encoded_oauth_signature,oauth_timestamp)
        }

response = requests.post(url, data=body, headers=headers)

print(response.text)

Feel free to see this git project as reference.

Flask + Serverless — API in AWS Lambda the Easy Way

ICCHA Technologies — Wed, 06 Jul 2022 06:33:12 +0000

If you either need to quickly deploy a small API or just decided to migrate your codebase to leverage the advantages of AWS Lambda. There are of course other guides on how to accomplish that, like packing and uploading your app by yourself. The process gets a bit more tricky when you realize some dependencies are not compatible (because of Linux on lambda) and you need to handle that too.

Solution

Use a powerful combo of Flask and Serverless framework. Any WSGI application such as Django can work too. We will use a quotes service.

Recipe

Download and install Python in your device, then install pip and virtualenv package.
```
  pip install virtualenv
```
Install Serverless framework, see full instructions
```
  $ npm install -g serverless
```
Create your main folder, can call it "quotes". Then create the python venv.
```
    mkdir quotes
    cd quotes
    virtualenv venv -p python3
    (venv) pip install Flask
    (venv) pip freeze > requirements.txt
```
You will now have a venv folder, go to quotes/venv/Scripts and run "activate" to active the virtual environment.

At the main folder /quotes create an app.py file to contain our quotes service,then add the information below.

    from collections import namedtuple
    from random import choice

    from flask import Flask, jsonify

    Quote = namedtuple("Quote", ("text", "author"))

    quotes = [
        Quote("Talk is cheap. Show me the code.", "Linus Torvalds"),
        Quote("Programs must be written for people to read, and only incidentally for machines to execute.", "Harold Abelson"),
        Quote("Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live",
            "John Woods"),
        Quote("Give a man a program, frustrate him for a day. Teach a man to program, frustrate him for a lifetime.", "Muhammad Waseem"),
        Quote("Progress is possible only if we train ourselves to think about programs without thinking of them as pieces of executable code. ",
            "Edsger W. Dijkstra")
    ]

    app = Flask(__name__)

    @app.route("/")
    def hello_from_root():
        return jsonify(message='Hello from root!')

    @app.route("/quote", methods=["GET"])
    def get_random_quote():
        return jsonify(choice(quotes)._asdict())

Wire it with the Serverless framework. To do so we need to create a serverless.yml file (in our root) manually. The file will look like this:

    service: quotes #name this whatever you want

    provider:
    name: aws
    runtime: python3.6
    region: us-east-1
    memorySize: 128

Install two Serverless plugins. The first thing we need is to make Lambda understand WSGI (protocol Flask/Django is using), second is to make Serverless pack our python requirements into our deployment package.
```
  $ sls plugin install -n serverless-wsgi
  $ sls plugin install -n serverless-python-requirements
```
Update the serverless.yml file with the plugins and further information needed.
```
service: quotes-serverless #name this whatever you want

provider:
name: aws
runtime: python3.6
region: us-east-1
memorySize: 128

plugins:
- serverless-wsgi
- serverless-python-requirements

custom:
wsgi:
app: app.app
packRequirements: false

functions:
app:
handler: wsgi_handler.handler
events:
    - httpApi: '*'

deprecationNotificationMode: error #add this for debugging
configValidationMode: error #add this for debugging
    `);
```
You can see a new section (custom) which holds configuration for those plugins. The wsgi part is saying where your app is and turning off packing of requirements. The last part (functions) declares what our service contains. We can have more functions within one service and also require specific permissions. In this case, we are just saying that all requests will be served through a WSGI handler, which is provided by our installed plugin.
Update requirements.txt
```
Flask==1.1.4
Werkzeug==1.0.1
markupsafe==2.0.1
```
Check 'Werkzeug' version, can cause error while deploying.
Do a Local deployment to test the service, run:
```
    $ serverless wsgi serve
```
You can open the server to the specific path, for example http://localhost:5000/

use /quotes to see the random quotes.
Before deployment our project to AWS you need to create an account.
1. Login to your AWS account and go to the Identity & Access Management (IAM) page.
2. Click on Users and then Add user. Enter a name in the first field to remind you this user is related to the Serverless Framework, like serverless-admin. Enable Programmatic access by clicking the checkbox. Click Next to go through to the Permissions page. Click on Attach existing policies directly. Search for and select AdministratorAccess then click Next: Review. Check to make sure everything looks good and click Create user.
3. View and copy the API Key & Secret to a temporary place. These are your AWS access keys.
see more instruction in here
Add AWs credentials to the project, the easiest way is create an .env file in our root folder and add the information below.

Edit the create a new file nodemon.json in the root of your project folder, that file will be used to store our environment variables. In case that you are not using nodemon , create an .env file.

    set AWS_ACCESS_KEY_ID=<your-key-here>
    set AWS_SECRET_ACCESS_KEY=<your-secret-key-here>
    # AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are now available for serverless to use
    serverless deploy

    # In Linux use 'export' instead of 'set'

Deploy our project, run the below command.
```
    $ serverless deploy
```
Now we can see the server in our endpoint, at AWS Cloudformation we now have our Stack of the deployed project.

If you get an internal error or want to see recent logs, open CloudWatch in your AWS console or just type: serverless logs -f app

Feel free to visit this repository with all what this blog contains.

How to Build a Verification Code Based Sign Up

ICCHA Technologies — Fri, 01 Jul 2022 23:17:38 +0000

While creating an account at first a system may allow a user to add any email address without verifying if it even exists or if the user owns this mail account.

Solution

Create verification code containing 4 random digits sent to the email of the user, the app will request now to type the code in the verification page, once it is approved the account is created.

Recipe

Server side with node

First create a constant to store 6 random digits, need to be a string.

const randomCode = Math.floor(100000 + Math.random() * 900000).toString()

Encrypt the 6 digits and then store into the database with all the information needed.

const hash = await bcrypt.hash(randomCode, Number(10))

Check bcrypt library at https://www.npmjs.com/package/bcrypt used for the encryption.

  await new Token({
    emailId: email,
    token: hash,
    createdAt: Date.now(),
  }).save()

DB example

  const schema = new Schema({
    emailId: {
      type: String,
    },
    token: {
      type: String,
      required: true,
    },
    createdAt: {
      type: Date,
      expires: 3600,
      default: Date.now,
    },
  })

Send the email.

const emailOptions = {
    subject: 'CoNectar: Verify Your Account',
    data: {
      verification_code: randomCode,
      name: name,
    },
    toAddresses: [email],
    fromAddress: process.env.AWS_SES_SENDER || '',
    templateUrl: path.join(__dirname, '..', 'templates', 'verification-code.html'),
  }

  const sendEmailResponse = await emailService.send(emailOptions)

At the email service

For the email sent procces , AWS is our option handled a email html template is needed, see basic template here
Configure your AWS access and SES functionality.

let AWS = require('aws-sdk')

AWS.config.update({
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  region: process.env.AWS_REGION,
})

const SES = new AWS.SES({ region: process.env.AWS_REGION })

At the corresponding service let's start loading the template.

async function getTemplate(templateUrl) {
  return fs.readFileSync(templateUrl, 'utf8')
}

Add a function that builds the body with the template.

function buildList(listName, list, template) {
  let newTemplate = template

  const startTag = `{{${listName}}}`
  const valueTag = `{{${listName}-value}}`
  const endTag = `{{${listName}-end}}`

  const startTagPos = newTemplate.indexOf(startTag)
  if (startTagPos === -1) return template

  const contentEndPos = newTemplate.indexOf(endTag)
  if (contentEndPos === -1) return template

  const contentStartPos = startTagPos + startTag.length
  const endTagPos = contentEndPos + endTag.length

  const content = newTemplate.slice(contentStartPos, contentEndPos)

  let expandedContent = ''
  list.map((value) => (expandedContent += content.replace(valueTag, value)))

  newTemplate = newTemplate.slice(0, startTagPos) + expandedContent + newTemplate.slice(endTagPos)
  return newTemplate
}

Add a function that runs the build of the template.

function transformContent(content, data) {
  if (!content) return ''

  for (let key in data) {
    if (data.hasOwnProperty(key)) {
      if (Array.isArray(data[key])) {
        content = buildList(key, data[key], content)
        continue
      }
      const replacer = `[[${key}]]`
      const value = `${data[key]}`
      content = content ? content.replace(replacer, value) : ''
    }
  }

  return content
}

Mix all the functions and create the send function needed to the sign up process. > NOTE: Amazon SES does not like undefined as value so, do not send the field at all in case the value is undefined or, at least send empty string.

async function send(options) {
  let template, htmlBody

  if (!options.textOnly) {
    template = options.template || (await getTemplate(options.templateUrl))
    htmlBody = options.data ? transformContent(template, options.data) : template
  }

  const plaintext = options.data
    ? transformContent(options.plaintext, options.data)
    : options.plaintext || ''
  let params = {
    Destination: {
      ToAddresses: options.toAddresses,
    },
    Message: {
      Body: {
        ...(options.textOnly
          ? {
              Text: {
                Charset: 'UTF-8',
                Data: plaintext,
              },
            }
          : {
              Html: {
                Charset: 'UTF-8',
                Data: htmlBody,
              },
            }),
      },
      Subject: {
        Charset: 'UTF-8',
        Data: options.subject,
      },
    },
    Source: options.fromAddress || process.env.CDP_SENDER_EMAIL,
  }

  return SES.sendEmail(params).promise()
}

Check the email respone to handle it.

  if (!sendEmailResponse || !sendEmailResponse.MessageId) {
    throw Boom.conflict('Could not send email')
  }

Email Preview

Client side with React

Create a sign up page containing a form with the information needed to create an account, send the information using location and history features.

  let userPayload = {
  name: userLogin.name.value,
  username: userLogin.username.value,
  email: userLogin.email.value,
  password: userLogin.password.value,
  photo: profileImage && profileImage instanceof File ? profileImage : null,
}
history.push({ pathname: '/verify-code', state: { ...userPayload } })

NOTE: Read reactrouter documentation https://v5.reactrouter.com/web/api/history

Create the verifyCode react component and get the information from location.

const history = useHistory()
const location = useLocation()

const [verificationCode, setVerificationCode] = useState('') // Needed to store the code
const [email, setEmail] = useState('')
const [name, setName] = useState('')
const [payload, setIsPayload] = useState({})

Below useEffect will load the information from location if exists, in case that there is no information , the page will be redirected.

useEffect(() => {
  if (
    !location.state ||
    !location.state.email ||
    !location.state.name ||
    !location.state.username ||
    !location.state.password
  ) {
    history.push('/')
  } else {
    setEmail(location.state.email)
    setName(location.state.name)
    setIsPayload(location.state)
  }
}, [location, history])

Create the form needed to fill the verification code.

Note: we use react-hook-form to handle the verification form, see https://react-hook-form.com/ for reference.

  const {
  handleSubmit,
  reset,
  formState: { isSubmitting },
  } = useForm()

Note: We are using some features from ChakraUI, see the documentation below:
https://chakra-ui.com/guides/first-steps
Imported: FormControl, Center, useToast, PinInput, PinInputField.

Note: We are using some features from TailwindCSS, see the documentation below:
https://tailwindcss.com/docs/installation

JSX component for the form usages, we use PinInput to retrieve the code value.

return (
  <div className="flex flex-1 justify-center items-center h-full w-full">
    <div className="flex flex-col w-full max-w-md px-4 py-8 bg-white rounded-lg shadow-2xl dark:bg-gray-800 sm:px-6 md:px-8 lg:px-10">
      <div className="self-center mb-2 text-4xl font-medium text-gray-600 sm:text-3xl dark:text-white">
        Verification code
      </div>
      <div className="self-center mb-4 text-sm font-medium text-gray-400 dark:text-white">
        Please check your email for the verification code.
      </div>
      <div className="my-4">
        <form onSubmit={handleSubmit(onSubmit)} action="#" autoComplete="off">
          <FormControl>
            <div className="flex flex-col mb-6">
              <div className="flex-auto mb-2">
                <Center>
                  <PinInput
                    value={verificationCode}
                    onChange={handleChange}
                    className="flex-auto"
                  >
                    <PinInputField className="text-gray-600" />
                    <PinInputField className="text-gray-600" />
                    <PinInputField className="text-gray-600" />
                    <PinInputField className="text-gray-600" />
                    <PinInputField className="text-gray-600" />
                    <PinInputField className="text-gray-600" />
                  </PinInput>
                </Center>
              </div>
            </div>
          </FormControl>
          <div className={'flex w-full'}>
            <Button
              disabled={verificationCode.length < 6}
              text="Verify"
              isLoading={isSubmitting}
              type="submit"
            />
          </div>
        </form>
      </div>
      <div className="my-4">
        <div className="flex w-full">
          <p className="text-sm font-medium text-gray-600">
            Didn&#x27;t receive the code?&nbsp;
            <button
              onClick={() => onRequestCode(true)}
              className="text-purple-600 hover:text-purple-800 focus:text-gray-600"
            >
              click here to request a new code
            </button>
          </p>
        </div>
      </div>
    </div>
  </div>
)

Verification code example Preview

Create the reference for UseToast , this chakraUI feature let us handle the errors easily.
```
  const toast = useToast()
```

Create the remaining functions to retrieve the infomration from the server, onRequestCode (it will request the code and it will be send to the user's email ) and onSubmit (if the codes matches the new acocunt will be created)

OnRequestCode

const onRequestCode = useCallback(
  async (forceSend = false) => {
    try {
      if (email && name) {
        const response = await requestVerificationCode({
          email: email,
          name: name,
          forceSend: forceSend,
        })
        if (response.statusText === 'Created') {
          toast({
            duration: 5000,
            status: 'success',
            position: 'top-right',
            variant: 'left-accent',
            title: 'SUCCESS: Check your email for the verification code',
            description: 'Please check your inbox messages.',
          })
        } else if (response.status === 400) {
          toast({
            duration: 5000,
            status: 'error',
            position: 'top-right',
            variant: 'left-accent',
            title: 'WARNING: Verification code already sent',
            description: 'Please check your email or try again later.',
          })
        }
      }
    } catch (error) {
      toast({
        duration: 5000,
        status: 'error',
        position: 'top-right',
        variant: 'left-accent',
        title: 'ERROR: Oops! Something Went Wrong',
        description: 'Please contact support at support@conectar.ch',
      })
    } finally {
      reset()
    }
  },
[email, name, toast, reset]
)

This function refers to a service called "requestVerificationCode", means to request the code to the server and it to be sent to the referenced email address.

It has a value call "forceSend", this allow the page to request a code thorugh an action once set to "true" because the server only allows to send a code every 5 minutes by default.

Becareful with the error handling, it need to match the server's response.

This function is call by an useEffect once per load, that's why its recommended to set the function as a callback using useCallback.

  useEffect(() => {
  onRequestCode(false)
  }, [onRequestCode])

onSubmit and OnSignup

   const onSubmit = async (data) => {
    try {
      const response = await tokenVerificationCode({
        email,
        verificationCode,
      })
      if (response.data?.checkCode) {
        toast({
          duration: 5000,
          status: 'success',
          position: 'top-right',
          variant: 'left-accent',
          title: 'SUCCESS: your verification code has been verified',
        })
        onSignUp()
      }
    } catch (error) {
      reset()
      if (error.response.data.statusCode === 400) {
        toast({
          duration: 5000,
          status: 'error',
          position: 'top-right',
          variant: 'left-accent',
          title: 'ERROR: Invalid or expired verification code',
        })
      }
    }
  }

This "onSubmit" function will use a service that check if the code matches with the one from the server, if it is maching it will now be forwarded to the below function "onSignUp"

  const onSignUp = async () => {
    try {
      const response = await signup(payload)
      if (response.ok) {
        history.push({ pathname: '/login' })
        toast({
          duration: 5000,
          status: 'success',
          position: 'top-right',
          variant: 'left-accent',
          title: 'SUCCESS: Your account has been created',
          description: 'Please login.',
        })
      } else {
        toast({
          duration: 5000,
          status: 'error',
          position: 'top-right',
          variant: 'left-accent',
          title: 'ERROR: Email is already in use!',
          description: 'Please contact support at support@conectar.ch',
        })
        history.push({ pathname: '/login' })
      }
    } catch (error) {
      toast({
        duration: 5000,
        status: 'error',
        position: 'top-right',
        variant: 'left-accent',
        title: 'ERROR: Oops! Something Went Wrong',
        description: error.message + ', Please contact support at support@conectar.ch',
      })
    } finally {
      reset()
    }
  }

This "onSignUp" function will create the new account if does'nt exist.

Finally, be sure to clean the location value once the component did unmount.

    useEffect(() => {
      return () => {
        reset()
        location.state = null
      }
    }, [location, reset])

Using GraphQL for Your APIs

ICCHA Technologies — Fri, 01 Jul 2022 23:12:53 +0000

Most of the APIs request an endpoint to access a predefined data structure. If you want to access other resources, it is necessary to request another endpoint, it makes the process kind of tricky.

We only define a single endpoint (for example http://example/graphql).
Since this is a query language, all actions are done through a POST.

Solution

GraphQL allow us to retrieve only the data we need by using a query language for web APIs.

Recipe

CRUD GraphQL API with Nodejs, Express & MongoDB

Create a new directory to store the project, run npm init to configure the new project
Run npm install to create our package.json file
Create a server.js file (entry point for our server)
Create the src folder and the below required folders and files:
1. Create src/schema directory and an index.js file (will contain the business logic)
2. Create a src/resolvers directory and an index.js file.
3. Create an src/models directory and a post.js which holds what a post should look like.
```
├── src
│   ├── schema
│   │   └── index.js
│   ├── resolvers
│   │   └── index.js
│   └── models
│       └── post.js
├── package-lock.json
├── package.json
└── server.js
```

Install the dependencies

Using npm

  # npm
  $ npm install --save express express-graphql graphql body-parser

Using yarn

  # yarn
  $ yarn add --save express express-graphql graphql body-parser

You can also install "nodemon" locally to avoid having to restart your server with each change

  $npm install --save-dev nodemon

Also you need to update your "package.json" for using "nodemon".

  "scripts": { "start": "nodemon server.js"}

Edit the schema, it will allow us to define our GraphQL objects and list the different actions available through our API.
```
// ./src/schema/index.js
const { buildSchema } = require('graphql');

module.exports = buildSchema(`
  """
  A Post refers to available attributes for a Post
  """
  type Post {
    _id: ID!
    body: String!
    createdAt: String!
  }
  input PostType {
    body: String!
  }
  type RootQuery {
    posts: [Post!]
    post(_id: String!): Post!
  }
  type Mutation {
    createPost(post:PostType): Post,
    deletePost(_id: String): Post,
    updatePost(_id: String, body: String): String
  }
  schema {
    query: RootQuery
    mutation: Mutation
  }
`);
```
We set up two queries, one to fetch all posts, and the other one to retrieve a post by id.

To be able to retrieve posts, we must first have them in our database. That implies we’ll have to create them. We achieve this with the help of a GraphQl mutation. It’s a query that (creates/updates/deletes).
1. The "createPost" mutation is used to make a new post in this case. It takes an item of type PostInput and creates a post from it.
2. The "deletePost" mutation to delete a post by id.
3. The "updatePost" mutation is made to update a post, It takes two params _id and the new body.

Create Moongoose Model, install the mongoose package:

npm install mongoose

Then edit the file "src/model/post.js" in order to add the model.

//./src/models/post.js
const mongoose = require("mongoose")

const Schema = mongoose.Schema

const postSchema = new Schema(
  {
    body: {
      type: String,
      required: true,
    },
  },
  { timestamps: true }
)

module.exports = mongoose.model("Post", postSchema)

Edit the resolver file, it is a set of functions that generates a GraphQL query response. It’s a GraphQL query handler, the name of every query or mutation must exactly match the name of the resolver function.

Add this code below in the "src/resolvers/index.js"

//./src/resolvers/index.js
const Post = require("../models/post")

module.exports = {
  posts: async () => {
    try {
      const postsFetched = await Post.find()
      return postsFetched.map(post => {
        return {
          ...post._doc,
          _id: post.id,
          createdAt: new Date(post._doc.createdAt).toISOString(),
        }
      })
    } catch (error) {
      throw error
    }
  },

  post: async (_id) => {
    try {
      const postFetched = await Post.findById(_id);
      return {
        ...postFetched._doc,
        _id: postFetched.id,
        createdAt: new Date(postFetched._doc.createdAt).toISOString(),
      }
    } catch (error) {
      throw error
    }
  },

  createPost: async args => {
    try {
      const { body } = args.post
      const post = new Post({
        body,
      })
      const newPost= await post.save()
      return { ...newPost._doc, _id: newPost.id }
    } catch (error) {
      throw error
    }
  },

  deletePost: async (id) => {
    try {
      const deletedPost = await Post.findByIdAndDelete(id);
      return {
        ...deletedPost._doc,
        _id: deletedPost.id,
        createdAt: new Date(deletedPost._doc.createdAt).toISOString(),
      }
    } catch (error) {
      throw error
    }
  },

  updatePost: async args => {
    try {
      const { _id, body } = args
      const updatedPost = await Post.findByIdAndUpdate(_id, { body: body });
      return `Post ${updatedPost.id} updated Successfully!!!`
    } catch (error) {
      throw error
    }
  },
}

Based on the "src/shcema/index.js", we created a query named posts that returns an array of posts. The posts method sends the request to MongoDB using the model created with mongoose.

The mutation described before in our Schema will be handled by the second resolver function createPost. It takes the post object as an input and uses it to create a new post based on the Post model.
And to save it in MongoDB, we just need to utilize another mongoose helper, the save() function, and return the newly generated post as anticipated in the Schema.

We now have a schema and resolvers for our API, which is almost everything we need to go on to the next phase and construct a server and endpoint.

Create MongoDB

Go to https://cloud.mongodb.com/ and create you own cluster, as recommendation download https://www.mongodb.com/es/products/compass to connect to the DB.

Save the "Connection String" from your MongoDB.
Create a new file nodemon.json in the root of your project folder, that file will be used to store our environment variables. In case that you are not using nodemon , create an .env file
```
{ 
  "env": { 
            "MONGO_DB": "your_Connection String"
        }
}
```

Create the Server & connect MongoDB

Add this to "server.js" file

const express = require("express");
const { graphqlHTTP } = require("express-graphql");
const graphQlSchema = require("./src/schema");
const graphQlResolvers = require("./src/resolvers");
const mongoose = require("mongoose");

const app = express();

app.use(
  "/graphql",
  graphqlHTTP({
    schema: graphQlSchema,
    rootValue: graphQlResolvers,
    graphiql: true,
  })
);

const uri = `${process.env.MONGO_DB}?retryWrites=true&w=majority`;
const options = { useNewUrlParser: true, useUnifiedTopology: true };
mongoose
  .connect(uri, options)
  .then(() => app.listen(4000, console.log("Server is listening on 4000")))
  .catch((error) => {
    throw error;
  });

Imported the schema and resolvers that were previously created. graphqlHTTPis needed to use them. It’s a express-graphql method that expects some options, in this case, are the schema and the resolvers. I also enabled graphiql, a useful tool for query testing.

The last part of the code is to establish the connection to MongoDB

Run the project, use "npm start" at the console, Then open at your browser: "http://localhost:4000/graphql".

We can now try our queries directly from the interface.

You can run this code to create the first post:
```
mutation {
  createPost(post: {body:"My first post"}){
    body,
    createdAt
  }
}
```

Feel free to visit this repository containing all what this blog contains: https://github.com/ICCHA-Technologies/GraphQL-api

Internationalization with ReactJS and i18n

ICCHA Technologies — Fri, 01 Jul 2022 23:02:49 +0000

There are a lot of languages around the world, any person should be able to understand the content of our page, the aim of internationalization is to remove the barriers to localizing or deploying an application internationally.

Solution

Use React i18next https://www.i18next.com/ and https://github.com/isaachinman/next-i18next to add internationalization to your React applications.

Recipe

For NextJS

Initialize your NextJS project see https://nextjs.org/docs/getting-started documentation
Install the i18n package, this time as recommendation use "next-i18next"

Using npm
```
  # npm
  $ npm install next-i18next
```
Using yarn
```
  # yarn
  $ yarn add next-i18next
```
Create the next-i18next configuration file as "next-i18next.config.js" in the main folder, it might contain the locale configuration matching the one added in the next config file.
```
module.exports = {
  i18n: {
    locales: ["en", "es"],
    defaultLocale: "en", // default locale will fallback to 'en'
  },
};
```

You need to specify the routes that next will allow, this time only English and Spanish are going to be used.

Configure the next config file "next.config.js", next has it own route specified in the locale value, as we already configured it in the next-i18next config file we
only need to import it.
```
const { i18n } = require("./next-i18next.config.js");
const nextConfig = {
  reactStrictMode: true,
  i18n,
};

module.exports = nextConfig;
```
for changing the language open the link referring to /en or /es

Add the wrapper component "appWithTranslation" into the app file, it allows us to use the t (translate) function in our components via hooks.

import { appWithTranslation } from "next-i18next";

function MyApp({ Component, pageProps }) {
  return <Component {...pageProps} />;
}

export default appWithTranslation(MyApp);

Create the translation files, by default, next-i18next expects your translations to be organized as such:

.
└── public    
    └── locales
        ├── en
        |   └── common.json
        └── es
            └── common.json

see the public > locales > en > common.js file

{
  "title": "i18n NextJS project",
  "spanish": "Spanish",
  "english": "English"
}

see the public > locales > es > common.js file

{
  "title": "Proyecto i18n NextJS",
  "spanish": "Español",
  "english": "Ingles"
}

Then we add "serverSideTranslation" to getStaticProps or getServerSideProps (depending on your case) in our page-level components. In this case we will use the "serverSideTranslation" one.

import { serverSideTranslations } from "next-i18next/serverSideTranslations";

export async function getStaticProps({ locale }) {
  return {
    props: {
      ...(await serverSideTranslations(locale, ["common"])), //['common', 'other'] if more files are needed
      // Will be passed to the page component as props
    },
  };
}

This code need to be added in every file using the translations.

Start using the translations by first importing the next-i18next package in the desired component

import { useTranslation } from "next-i18next";

Then create the const inside the respective component where the text will be translated

const { t } = useTranslation("common");

And this is how the translation is being used

{t("title")}

see a sample Home file:

import Head from "next/head";
import styles from "../styles/Home.module.css";

import { serverSideTranslations } from "next-i18next/serverSideTranslations";
import { useTranslation } from "next-i18next";

const Home = () => {
  const { t } = useTranslation("common");
  return (
    <div className={styles.container}>
      <Head>
        <title>i18n NextJS project</title>
        <meta name="description" content="Generated by create next app" />
        <link rel="icon" href="/favicon.ico" />
      </Head>

      <main className={styles.main}>
        <h1 className={styles.title}>{t("title")}</h1>

        <div className={styles.grid}>
          <a href="/en" className={styles.card}>
            <p>{t("english")}</p>
          </a>

          <a href="/es" className={styles.card}>
            <p>{t("spanish")}</p>
          </a>
        </div>
      </main>
    </div>
  );
};

export async function getStaticProps({ locale }) {
  return {
    props: {
      ...(await serverSideTranslations(locale, ["common"])),
      // Will be passed to the page component as props
    },
  };
}

export default Home;

Feel free to check this codesanbox.io project

For React.js

Create your react project

Install the i18n required packages

Using npm

  # npm
  $ npm install i18next --save
  $ npm install react-i18next i18next --save
  npm install i18next-browser-languagedetector

Using yarn

  # yarn
  $ yarn add i18next
  $ yarn add react-i18next
  $ yarn add i18next-browser-languagedetector

Create the translation file, it contains the languages and configuration, recommend to create a folder called "i18n" and inside a JS file called "i18n.js"

import i18n from "i18next";
import { initReactI18next } from "react-i18next";
import LanguageDetector from "i18next-browser-languagedetector";

i18n
  // detect user language
  // learn more: https://github.com/i18next/i18next-browser-languageDetector
  .use(LanguageDetector)
  // pass the i18n instance to react-i18next.
  .use(initReactI18next)
  // init i18next
  // for all options read: https://www.i18next.com/overview/configuration-options
  .init({
    debug: true,
    fallbackLng: "en",
    interpolation: {
      escapeValue: false, // not needed for react as it escapes by default
    },
    resources: {
      en: {
        translation: {
          title: "i18n React project",
          english: "English",
          spanish: "Spanish",
        },
      },
      es: {
        translation: {
          title: "Proyecto i18n en React",
          english: "Ingles",
          spanish: "Español",
        },
      },
    },
  });

export default i18n;

Import the i18n.js file to the index.js file
```
import "./i18n/i18n";
```
Start using the translations by first importing the react-i18next package in the desired component
```
import { useTranslation } from 'react-i18next'
```
Then create the const inside the respective component where the text will be translated
```
 const { t } = useTranslation()
```
And this is how the translation is being used
```
  {t(title)}
```

Finally Configure the changelanguage function at app.js component

First get thi18n element from useTranslation

const { t, i18n } = useTranslation(); // we used the t before

now we can change the language bt using the changelanguage function:

<button onClick={() => i18n.changeLanguage("en")} className="App-link">
  {t("english")}
</button>
<button onClick={() => i18n.changeLanguage("es")} className="App-link">
  {t("spanish")}
</button>

Feel free to check this codesanbox.io project