DEV Community: Piyush Goyani

Build CLI App in Python with Click and Publish to PyPI

Piyush Goyani — Fri, 24 Feb 2023 10:45:39 +0000

Background

Of course, you all are aware of CLI(Command Line Interface) tools, because you might use any OS like Linux-based, Mac or Windows, you have used these tools in your life. These are Git CLI, language interpreters and runtimes(Python, Node, Ruby, Java, Go etc), system commands and network utilities(reboot, restart, ping etc), editors like nano, vim and there many other tools and command line software that we use nowadays which are minimalist to fulfil our needs and save time. In this tutorial, I'll explain how to implement a CLI app that gives answers to your input about me based on CLI inputs. Let's dive in.

Setup

If you don't want to set up and go with development/install/publish flow and directly want to try CLI, then you can clone or direct install the package.

GitHub Source

First, we'll be using Python 3 for this project. let's install basic dependencies.

Install PIP, python package manager, used to install Python packages.
Install a virtual environment for this project, which isolates python packages.
Make the project directory( click-piyush ), set up virtualenv and activate. Then install click with pip, the library to make CLI apps.

Let's Code

Let's create a python file named clickpiyush.py where we'll write the below code.

import click

@click.command()
def age():
    """Get Age"""
    click.echo("I'm 2X years older, shouldn't be matter.")

if __name__ == ' __main__':
    age()

Here we are importing the package click that is used to make the CLI app.

Then we are adding one command to get the age and method for that command, which prints the age. Then call the method when the script is executed.

$ python test.py 
I'm 2X years older, shouldn't be matter.

Here, as you can see, you are not seeing any feeling like the CLI app but a simple script is running and printing some stuff, just like a normal python program does.

Now you can register for options and arguments(next section) or make command groups for adding multiple commands or nesting of commands. Let's do that.

import click

@click.group()
def cli():
    pass

@click.command()
def age():
    """Get Age"""
    click.echo("I'm 2X years older, shouldn't be matter.")

cli.add_command(age)

if __name__ == ' __main__':
    cli()

In this, we have made slight changes. We are adding multi-command using group functionality of click. inside that, we added one method which passes invocation as it's empty. Using that cli() method, we are registering the command using add_command and passing an argument of the commanding method(age). Now you can see the output below. As you can see click auto-generate beautiful Usage guide, options and help pages.

$ python test.py 
Usage: test.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help Show this message and exit.

Commands:
  age Get Age

Now we'll add different types of commands and functionality to our CLI app.

@click.command()
@click.option('--name', prompt='Identify youself by name')
def greet(name):
    """Describe this tool with colors to You."""
    click.secho(f"Hello {name}", bold=True, bg='green', fg='black')
    click.secho(
        "This is Command Line Interface which gives information of maker named Piyush.", bg='blue', fg='white')

cli.add_command(greet)

In this greet command, we are asking for --name option using the runtime prompt, you can configure it as you want. which is stored in the name variable for further access to the method. Inside that, we are printing two lines with different background colours and font colours in CLI using click.secho(). It combines echo and style functionality of click in one method and styles of the font as bold.

Let's add another command.

@click.command()
@click.option('--desc', default=False, show_default=True, help='Show detailed Info.')
def bio(desc):
    """Basic Bio"""
    if desc:
        click.echo("Hey, this is piyush here, thanks for your interest.\nTo know more about me please run this CLI and different commands.\nThank You!")
    else:
        click.echo("This is me, and describing self.")

cli.add_command(bio)

In the bio command, the user can get information about me, but basic if not input is passed. To get detailed info --desc true flag must be passed. This argument can be configured with the @click.option method. In which you can describe the default value, enabling the default value in the help doc and help text.

Full Code and Output

Here is the full code of my package script clickpiyush.py and the output is below for your reference.

import click
import time

@click.group()
def cli():
    pass

@click.command()
@click.option('--name', prompt='Identify youself by name')
def greet(name):
    """Describe this tool with colors to You."""
    click.secho(f"Hello {name}", bold=True, bg='green', fg='black')
    click.secho(
        "This is Command Line Interface which gives information of maker named Piyush.", bg='blue', fg='white')

@click.command()
@click.option('--desc', default=False, show_default=True, help='Show detailed Info.')
def bio(desc):
    """Basic Bio"""
    if desc:
        click.echo("Hey, this is piyush here, thanks for your interest.\nTo know more about me please run this CLI and different commands.\nThank You!")
    else:
        click.echo("This is me, and describing self.")

@click.command()
def age():
    """Get Age"""
    click.echo("I'm 2X years older, shouldn't be matter.")

@click.command()
def skills():
    """Get my dummy skills set with progress bar"""
    skill_set = ["JavaScript", "Python", "GraphQL", "C", "TypeScript", "VueJS",
                 "NuxtJS", "Socket Programming", "Machine Learning", "NodeJS", "Linux"]

    with click.progressbar(skill_set, label='Getting skills', length=len(skill_set)-1, show_eta=False, color='blue') as skill_list:
        for skill in skill_list:
            click.echo(f" {skill}")
            time.sleep(0.3)

@click.command()
def blog():
    """Get My blog URL"""
    blog_url = "https://blog.thesourcepedia.org/"
    click.echo(f"{blog_url} Wanna visit [y/n]: ", nl=False)
    c = click.getchar()
    click.echo(c)
    if c == 'y':
        click.echo("Launching Piyush's Blog")
        click.launch(blog_url)

@click.command()
def portfolio():
    """Launch my Portfolio!"""
    click.launch("https://piyushgoyani.thesourcepedia.org/")

# @click.command()
# @click.Choice()
cli.add_command(greet)
cli.add_command(bio)
cli.add_command(age)
cli.add_command(skills)
cli.add_command(blog)
cli.add_command(portfolio)

if __name__ == ' __main__':
    cli()

Running CLI Locally

To run the and see output execute clickpiyush.py file with no argument or option so it'll show help and all command as below.

python clickpiyush.py
Usage: clickpiyush.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help Show this message and exit.

Commands:
  age Get Age
  bio Basic Bio
  blog Get My blog URL
  greet Describe this tool with colors to You.
  portfolio Launch my Portfolio!
  skills Get my dummy skills set with progress bar

In this, we are launching a website in a browser based on the user input, showing a progress bar in skill command and more.

Source Repository

Prepare for Package

Let's create a setup.py file and paste the below configuration. This file contains the name of the package, version, inside modules, requirements, entry points and more configurable things.

from setuptools import setup

setup(
    name='clickpiyush',
    version='1.0.1,
    py_modules=['clickpiyush'],
    install_requires=['Click',],
    entry_points={
        'console_scripts': [
            'clickpiyush = clickpiyush:cli'
        ]
    }
)

Install Local Package

We have created a setup file, now we can install our CLI app and run it just like a normal program, so don't need to invoke a python interpreter.

pip install --editable .


Obtaining file:///home/piyush/Documents/prs/click-piyush
Requirement already satisfied: Click in ./venv/lib/python3.8/site-packages (from clickpiyush==1.0.1) (8.1.3)
Installing collected packages: clickpiyush
  Running setup.py develop for clickpiyush
Successfully installed clickpiyush

As you can see clickpiyush package is installed successfully, now you can use it with the name of the CLI app.

clickpiyush

Publish to PyPI

Till now whatever we make, run and install was on our system. Now we want to publish our package to PyPI(Python Package Index) so other users can download our CLI app and use it in their system.

python setup.py sdist

Once we run the above command, we have a new directory dist/, which contains our distribution or simply archived package.

running sdist
running egg_info
writing clickpiyush.egg-info/PKG-INFO
writing dependency_links to clickpiyush.egg-info/dependency_links.txt
writing entry points to clickpiyush.egg-info/entry_points.txt
writing requirements to clickpiyush.egg-info/requires.txt
writing top-level names to clickpiyush.egg-info/top_level.txt
reading manifest file 'clickpiyush.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'clickpiyush.egg-info/SOURCES.txt'
running check
creating clickpiyush-1.0.1
creating clickpiyush-1.0.1/clickpiyush.egg-info
copying files to clickpiyush-1.0.1...
copying LICENSE -> clickpiyush-1.0.1
copying README.md -> clickpiyush-1.0.1
copying clickpiyush.py -> clickpiyush-1.0.1
copying setup.cfg -> clickpiyush-1.0.1
copying setup.py -> clickpiyush-1.0.1
copying clickpiyush.egg-info/PKG-INFO -> clickpiyush-1.0.1/clickpiyush.egg-info
copying clickpiyush.egg-info/SOURCES.txt -> clickpiyush-1.0.1/clickpiyush.egg-info
copying clickpiyush.egg-info/dependency_links.txt -> clickpiyush-1.0.1/clickpiyush.egg-info
copying clickpiyush.egg-info/entry_points.txt -> clickpiyush-1.0.1/clickpiyush.egg-info
copying clickpiyush.egg-info/requires.txt -> clickpiyush-1.0.1/clickpiyush.egg-info
copying clickpiyush.egg-info/top_level.txt -> clickpiyush-1.0.1/clickpiyush.egg-info
Writing clickpiyush-1.0.1/setup.cfg
Creating tar archive
removing 'clickpiyush-1.0.1' (and everything under it)

Now let's install twine. It's a utility to publish python packages to PyPI and then upload generated distribution.

pip install twine
twine upload dist/*

After you run the command, you'll show that distribution is uploading to PyPI and asking for your credentials. Once you enter those, the package will be published successfully and listed in PyPI as this.

https://pypi.org/project/clickpiyush/

Now anyone can install this package by using the below command.

pip install clickpiyush

$ clickpiyush
Usage: clickpiyush [OPTIONS] COMMAND [ARGS]...

Options:
  --help Show this message and exit.

Commands:
  age Get Age
  bio Basic Bio
  blog Get My blog URL
  greet Describe this tool with colors to You.
  portfolio Launch my Portfolio!
  skills Get my dummy skills set with progress bar

I hope you enjoyed this article. Share, subscribe and stay tuned for more!

Displaying Hashnode Posts on Personal Site with RSS

Piyush Goyani — Fri, 21 Oct 2022 07:30:02 +0000

Introduction

Hello,

I hope you are writing articles on Hashnode, either in Personal, community or company blogs. You may want to link and display your written articles in your personal websites or portfolio where others can see only your article with other content. Here we'll see two ways with which you can achieve above task easily.

Solutions

Hashnode Official API

Hashnode provides official GraphQL API from which you can retrieve various content and display. You can test their playground here. With this API you can get various type content like featured posts, users posts and publications, manipulate articles and related data. You may require PAT(Personal Access Token) for some of access.

For more information, refer this article by Catalin Pit.

RSS Parsing

Another way is RSS Parsing. In this, RSS URL of publication is required to retrieve and process data. Here I'll take my blogs rss and process it and display articles which are written by me(aka Piyush Goyani).

First, I have created Backend NodeJS API in which I have installed rss-parser package to parse RSS feed of blog.

import Parser = require('rss-parser');

async getRss() {
  const parser: Parser = new Parser();
  const url = 'https://blog.thesourcepedia.org/rss.xml';
  const feed = await parser.parseURL(url);
  return feed.items.map((item) => {
    return {
      title: item['title'],
      coverImage: item['cover_image'],
      creator: item['creator'],
      link: item['link'],
      pubDate: item['pubDate'],
    };
  });
}

Here, as you can see, RSS feed URL is parsed and parser return whole array, which further mapped with specific key/value object and return as API response.

In the frontend side, which is Vue component where All articles are being rendered by calling API.

<template>
  <div>
    {{ items }} // render posts
  </div>
</template>
export default {
  data() {
    return {
      items: [],
    };
  },
  mounted() {
    fetch("https://api.thesourcepedia.org/blog/getRss")
      .then((res) => res.json())
      .then((data) => {
        this.items = data.filter((post) => post.creator === "Piyush Goyani");
      });
  },
};

Here I'm calling /getRss API which returns article array, and here I can filter which specific author(e.g myself) and display only those posts which I have written in my personal site.

You can filter and process RSS in different way as per you need.

Conclusion

I hope you enjoyed this article and found useful. Give a thumb and subscribe for more interesting stuff.

Preliminary Go Language for Beginners

Piyush Goyani — Wed, 19 Oct 2022 07:32:36 +0000

Go is the programming language that was Invented and backed by Google and is now used by many companies and programmers worldwide.

There aren't any use-case limitations with Go to build something as you can build anything with it, But it's mostly used for CLI Tooling, Network Services and APIs, Web applications and DevOps.

The Go Language is known for the following features:

Performance
Built-in Concurrency Support
Safety
Garbage Collection
Scalability
Portability / Cross Platform
Binary Generation
Standard Library and Package Management
Built-in Testing Support

The language is not intended for replacement of any other languages like C, C++, Python, Rust or any other as these languages have their own beauty and places in the market and people who adopted them for a long time.

Let's dive into installation and Basic Go Code.

Installation

Download Go

Download the Go archive from official site as per your choice. Here, I'm using Linux so the rest steps are accordingly.

Go to the downloaded directory and run this command from the terminal.

sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.19.2.linux-amd64.tar.gz

This will remove the old installation of Go and extract the downloaded version to the /usr/local directory.

Add PATH

Edit .bashrc file and sudo nano .bashrcadd PATH and save.export PATH=$PATH:/usr/local/go/bin

Check the installed version of GO with the following command.

go version
// go version go1.19.2 linux/amd64

Demo

Create a demo project directory and go inside the directory.

mkdir demo && cd demo

Run the below command to enable Dependency tracking in your project.

go mod init example/demo

This will create a file named go.mod with the below content.

module example/demo

go 1.19

This file is generally used to manage internal and external dependencies, which stay within the project.

Create init.go file with your favourite text editor,

nano init.go

and paste the below code and save.

package main

import "fmt"

func main() {
    fmt.Println("First program of GoLang!")
}

Explaination

package main : It declares the main package, which may contain all sub-functionality within the same directory linked to this file.

import "fmt" : Importing fmt, one of the standard library packages of Go, which contains functionality for console output and text formatting.

func main(): Defining main() function, which by default called first when program get executed, similar like main in C/C++.

Running Go Code

To run the Go program execute the below command in the terminal and see Output.

go run .
// First program of GoLang!

Conclusion

I hope you enjoyed this article. Stay tuned and subscribe for more Go Language posts and other useful tutorials.

AWS Amplify: All in One Framework that you need

Piyush Goyani — Mon, 19 Sep 2022 15:43:47 +0000

Background

Amazon Amplify is a framework for building full-stack applications within no time. It can be integrated with various frameworks and libraries for Mobile, Web and Desktop Application Development including JavaScript, iOS, Ionic, Flutter, React, Vue and many more.

In this article, We are going to learn How to use AWS Amplify for Developing a Full Stack Text Summarization Application in which we'll include Amplify CLI, Auth, UI Library, API, Serverless Functions, Deployment, and Debugging of Application.

Prerequisites

Amazon AWS Account
System from where you can access your AWS account

Installation

Install vue-cli and create vue project.

npm install -g @vue/cli
vue create ml-with-amplify
// select vue3 version
cd ml-with-amplify

This was the front end of our project. Now we'll set up backend API and services for our project.

Install the Amplify CLI.

The Amplify CLI is a command line toolchain that runs locally in order to communicate with your app backend.

npm install -g @aws-amplify/cli@latest
// OR with npm
curl -sL https://aws-amplify.github.io/amplify-cli/install | bash && $SHELL

Install & Configure Amplify UI

To install Amplify UI with Vue integration run the below command,

npm i @aws-amplify/ui-vue aws-amplify

Edit src/main.js file and add below content.

import { createApp } from "vue";
import App from "./App.vue";
import './assets/style.css'
import AmplifyVue from "@aws-amplify/ui-vue";
import router from "./router";

const app = createApp(App);
app.use(router);
app.use(AmplifyVue);
app.mount("#app");

To set up amplify backend in our project, run the following command from your project's root folder (ml-with-amplify): you can customize the selections. See the output below.

amplify init

Add Auth

To set up Amplify Auth in our project run the below command and review selections.

amplify add auth

Amplify Auth uses Amazon Cognito for Auth mechanisms like Sign Up, Sign In and other Access Control. You can customize it as per your need.

Integrate Auth with Amplify UI

Edit src/App.vue

<template>
  <authenticator :form-fields="formFields">
    <template v-slot="{ user, signOut }">
      <template v-if="user">
        <AppHeader :user="user" :sign-out="signOut"></AppHeader>
        <router-view class="scrollbar" :user="user"></router-view>
      </template>
    </template>
  </authenticator>
</template>

<script setup>
import "@aws-amplify/ui-vue/styles.css";
import { Amplify, Auth } from "aws-amplify";
import awsconfig from "./aws-exports";
import AppHeader from "./components/app-header.vue";

Amplify.configure(awsconfig);
Auth.configure(awsconfig);

const formFields = {
  signIn: {
    username: {
      labelHidden: true,
      placeholder: "Enter your username",
      isRequired: true,
      label: "",
    },
  },
  signUp: {
    username: {
      labelHidden: true,
      placeholder: "Create username",
      isRequired: true,
      label: "",
    },
    email: {
      labelHidden: true,
      placeholder: "Enter Your Email Here",
      isRequired: true,
      label: "",
    },
  },
  resetPassword: {
    username: {
      labelHidden: true,
      placeholder: "Enter your username",
      isRequired: true,
      label: "",
    },
  },
};
</script>

Here we are Configuring Amplify Auth with awsConfig(aws-exports.js file) which is auto-generated by Amplify. We use component which is Amplify Auth component for Vue, which is used for Sign Up, Sign In, Reset Password, Social Login and Customization Auth Options. It can be customizable. Here in formFields object we are customizing Authenticator SignIn, Sign Up, and Reset Password Form's fields. In app-header.vue we have passed signOut method of the authenticator component to split functionality.

Push Deployment

Run the below command to deploy our frontend and backend app in the cloud in a single workflow.

amplify push

List all deployed apps in Amplify Console. You can create a new app from here.

Review detailed view of our application's backend environment. Here you can see deployment details, Added categories, actions and commands to install this backend in a new project.

Connect Branch

In the Hosting environment section, you can connect any Git Version Control provider and connect the repository branch to link the code. Here we are selecting GitHub as a provider. Click Connect Branch.

Add Repository

After successful authorization, we need to select the repository where our code is hosted and the name of the branch. Click Next.

Create role

In this section, we need to create an AWS role to Allow Access to resources on our behalf so it can auto-create instances, and pools, and deploy them. After review click Create Role.

Build Settings

In this section you can configure the build section, it will auto-detect frameworks, here Vue is detected as Frontend Framework and Amplify as Backend. After the necessary configuration click next.

Review

Review final details and settings and all ok then click Save and deploy.

After deploying go back to the Hosting section app detail page, where you can see the pipeline is running with four steps which include Provision, Build, Deploy, and Verify. From here you can see deployment process steps, its logs, and perform various actions like adding a custom domain, password-protect site etc.

Once deployment is successful you'll see the URL where an app is hosted, click on URL. If it is still not reflected you'll see the below screen otherwise your app will be served.

SignUp

Enter the confirmation code that you received on the email to verify the email and complete registration, and then log in.

If you try to attempt login with the wrong credential you will see amplify will throw a respective error.

User Management

Visit Amplify Studio Dashboard and Goto User Management Tab to view and manage Users.

You can also see and manage Users in Cognito Dashboard.

Amplify API

Add Backend API

To add backend service run the below command.

amplify add api

It will ask for the type of service REST/GraphQL, endpoint, name of a resource, path, lambda source, lambda function name, runtime, template and other advanced settings.

Provision resources

Once added locally push to the cloud for provisioning resources.

amplify push

It will create necessary resources like lambda instance, permissions, environment variables and configs, endpoints etc.

You can create a function URL from Lambda Dashboard as below from the Function URL tab which is optional though.

Remove API and De-provision Resources

To remove amplify API and lambda function run the below commands in the following order. First, you need to remove API and then the Lambda function because a function is dependent on API.

amplify api remove apiexpress
amplify function remove mlwithamplifyexpress

Text Summarization Logic

We have added a lambda function to get a summary of the text, which takes input from the front end and processes text in lambda and another custom external API and returns a response to the front end. Here is the function code.

/**
 * @type {import('@types/aws-lambda').APIGatewayProxyHandler}
 */

const stopwords = ["i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very", "s", "t", "can", "will", "just", "don", "should", "now"];

function remove_stopwords(str) {
  let res = [];
  let words = str.split(" ");
  for (let i = 0; i < words.length; i++) {
    let word_clean = words[i].split(".").join("");
    if (!stopwords.includes(word_clean)) {
      res.push(word_clean);
    }
  }
  return res.join(" ");
}
exports.handler = async (event) => {
  console.log("LAMBDA FUNTION INVOKED");
  const natural = require("natural");
  const fetch = require("node-fetch");
  let TfIdf = natural.TfIdf; // term-frequency inverse document frequency
  let tfidf = new TfIdf();
  const sent_tokenizer = new natural.SentenceTokenizer(); // sentence tokenizer
  const word_tokenizer = new natural.WordTokenizer(); // word tokenizer
  const body = JSON.parse(event.body);
  const doc = body.doc;
  console.log(`EVENT: ${JSON.stringify(event)}`);
  const sents = sent_tokenizer.tokenize(doc); // tokenize sentences
  let word_freq = {};
  // sentences without stopwords
  sents.forEach((sent) => {
    tfidf.addDocument(remove_stopwords(sent));
  });
  // remove stopwords from document
  let stopWRDoc = remove_stopwords(doc);
  let wordArr = [];

  wordArr = word_tokenizer.tokenize(stopWRDoc);

  // find frequency of words
  wordArr.forEach((word) => {
    if (!word_freq[word]) word_freq[word] = 0;
    if (wordArr.indexOf(word) === -1) word_freq[word] = 1;
    else word_freq[word] += 1;
  });

  // get maximum frequency
  const MAX_FREQ = Math.max(...Object.values(word_freq));

  // calculate weighted frequency
  Object.keys(word_freq).forEach((key) => {
    word_freq[key] = word_freq[key] / MAX_FREQ;
  });

  // calculate sentence scores
  let sent_scores = {};

  const word_freq_keys = Object.keys(word_freq);
  sents.forEach((sent) => {
    word_tokenizer.tokenize(sent.toLowerCase()).forEach((word) => {
      if (word_freq_keys.indexOf(word) !== -1) {
        // shorter sentence for summary X length
        if (sent.split(" ").length < body.sentLength) {
          if (Object.keys(sent_scores).indexOf(sent) === -1) {
            sent_scores[sent] = word_freq[word];
          } else {
            sent_scores[sent] += word_freq[word];
          }
        }
      }
    });
  });
  // get summary
  const summary = await fetch("https://heapq-api.vercel.app/nlargest", {
    method: "POST",
    headers: { Accept: "application/json", "Content-Type": "application/json" },
    body: JSON.stringify({
      size: 5,
      sent_scores: sent_scores,
    }),
  }).then((res) => res.json());

  return {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin": "*",
      "Access-Control-Allow-Headers": "*",
    },
    body: JSON.stringify({ flag: true, data: summary }),
  };
};

In this for text summarization and NLP(Natural Language Processing), we have used natural library which does basic Sentence and Word tokenization, finding Term Frequency Inverse Document Frequency(TfIdf).

In this, I have deployed the serverless function of Python library heapq API wrapper to find nlargest. I need to separate this in Python due to the erroneous and lack of compatibility in the NodeJS ecosystem that I have tried.

Visit the Demo Site

In this, you can enter a large paragraph/document in the left input box and get a summary in the right pane.

Register yourself better for auth flow understanding or use Demo Credentials

Username: amplifydemo

Password: amplifydemo

In this, you can customise sentence length, which is passed to the lambda function. Further can customize heapq-nlargest scoring which is currently 5 or change logic or something else.

Debug Amplify App with CloudWatch

To monitor your application, API and Lambda function you can use Amazon CloudWatch. In each deployment, a new log stream is created for easier access and monitoring.

You can see a list of lambda functions and a log of each deployment with a date and time based on the last event.

In case of an exception or error, you can get details info from CloudWatch logs that from where the error is occurring what it is and the cause of it. You can also see console logs here written in lambda functions.

Source Code and Local Setup

https://github.com/piyush-multiplexer/ml-with-amplify

https://github.com/piyush-multiplexer/heapq-api

Clone the repository and go to the project directory.

git clone https://github.com/piyush-multiplexer/ml-with-amplify.git
cd ml-with-amplify

You will amplify CLI to be installed for further operation.

npm install -g @aws-amplify/cli

Configure Amplify Project

amplify configure

This will ask several questions and try to create an IAM role with the chosen region. Once Amplify is successfully configured run amplify push to provision cloud resources and deploy your app. Visit your dashboard and see the preview URL to check if everything is working fine or not.

Conclusion

So in the end we got to know how to use AWS Amplify CLI, Hosting, and UI for building full-stack applications with UI and authentication and set up Amplify REST API and lambda function for Text Summarization. I hope you find this article useful. Please leave a comment if any doubt. Like, comment, share and subscribe.

Introduction to WebAssembly(Wasm) with Rust for Beginners

Piyush Goyani — Tue, 30 Aug 2022 07:31:00 +0000

Introduction

WebAssembly also known as WASM, is a technology that enables developers to run a set of code on a web browser with native-like performance. It is low-level assembly-like code that runs in the browser alongside JavaScript to achieve performance.

As we all know in terms of performance lower level languages are the fastest like Binary, Assembly language, Higher level languages like C, C++, JavaScript, Python, Java, Ruby, Rust, Swift, and all that most of us use for daily purpose programming. Because these are more human-friendly than machines. There are a set of tools like interpreters, compilers, assemblers and others which convert these languages into machine code to make this understandable to real hardware.

There are programming languages in which we can write our code and ship WebAssembly or Binary(WASM) files. Majorly used ones are Rust, TinyGo(Go), Emscripten(C/C++), and AssemblyScript. All of these languages are known for better performance and memory optimization.

We will use Rust language as a compilation target for our demo of programming.

Demo

Installation

cargo install wasm-pack

In this cargo is a Rust package manager just like npm in node. In Rust, libraries are known as crates like packages in Node. Here above command will install ( wasm-pack )[https://github.com/rustwasm/wasm-pack] crate/lib in our system.

Initialization

Create a new directory and go to that directory.

mkdir wasm-rust-demo && cd wasm-rust-demo

Create Cargo Package.

cargo init

Above command will create Cargo.toml manifest file(similar like package.json in node) and src directory with main.rs basic rust file.

Edit Cargo.toml

Paste the below code in the Cargo.toml file. It defines crate-type as cdylib, which is used when compiling a dynamic library which needs to be loaded from another language like JavaScript in our case. It includes dependency as wasm-bindgen. This crate/package provides communication between Rust and JavaScript with other features.

[package]
name = "wasm-rust-demo"
version = "0.1.0"
authors = ["Piyush Goyani <thesourcepedia@gmail.com>"]
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
wasm-bindgen = "0.2"

Go to the src/ directory, rename main.rs to lib.rs , and add the following content.

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn calc(a: i32, b: i32, op: char) -> i32 {
    if op == '+' {
        return a + b;
    } else if op == '-' {
        return a - b;
    } else if op == '*' {
        return a * b;
    } else if op == '/' {
        return a / b;
    } else {
        return 0;
    }
}

Here we are defining a Rust function named calc which takes three parameters, two numbers and 3rd for an operator to perform math action and return the result.

Build

 wasm-pack build --target web

This command will compile Rust code to wasm and generate a pkg directory which can be published to npm as a package and provides files which can be used in JavaScript communication.

Error during build

[INFO]: Installing wasm-bindgen...

Error: failed to download from https://github.com/WebAssembly/binaryen/releases/download/version\_90/binaryen-version\_90-x86-linux.tar.gz

To disable wasm-opt, add wasm-opt = false to your package metadata in your Cargo.toml.

Add this line in Cargo.toml

[package.metadata.wasm-pack.profile.release]
wasm-opt = false

Running WebAssembly/WASM Code.

1. By Loading Generated JS package

<script type="module">
        import init, { calc } from "./pkg/wasm_with_rust.js"
        init().then(() => {
            console.log(calc(10, 2, "+")) // 12
            console.log(calc(10, 2, "-")) // 8
            console.log(calc(10, 2, "*")) // 20
            console.log(calc(10, 2, "/")) // 5
        })
</script>

In this, we are loading a javascript file which is generated from wasm-pack which does all the job of loading WebAssembly code and all complex memory-related stuff and export functions that we wrote in Rust(e.g calc()) and init() to initialize WebAssembly. In this, we are passing two integer parameters and 3rd is an operator which decides which math operation to perform.

2. By Directly Loading .wasm File

<script>
  WebAssembly.instantiateStreaming(fetch('./pkg/wasm_with_rust_bg.wasm'))
  .then((results) => {
            const calc = results.instance.exports.calc
            console.log(calc(10, 2, 43)) // 12
            console.log(calc(10, 2, 45)) // 8
            console.log(calc(10, 2, 42)) // 20
            console.log(calc(10, 2, 47)) // 5
        })
</script>

In this, we are loading the wasm binary file directly which is compiled from wasm-pack. We are loading using fetch and instantiate using instantiateStreaming() method of WebAssembly class available globally. Here we are passing the same parameter as above but as you can see 3rd parameter is numeric. That is ASCII value of operator respectively. Here we can not pass string value because are not passing it to JavaScript so it can further convert to Machine Code, but Passing it to directly WASM file.

You can see the WASM Machine code that looks like below.

Source

Resources on WebAssembly

Advantages

The Memory of WebAssembly is Linear. It is an expandable continuous buffer of unsigned bytes that can be read and stored from JavaSCript and WASM synchronously.
Can import JavaScript functions and export custom function logic specified in WASM.
Compiled to Binary file, Assembly-like language which gives native speed and performance in the browser.
and many more based on requirements.

Conclusion

In this tutorial, you get the basic idea of what is WebAssebly or wasm is, its purpose, make wasm binary from Rust Language and how to use it with JavaScript. I hope you enjoyed it give it a thumb, share, and subscribe. Have any doubts or thoughts share them in the comment section below. I am available here.

Natural Language Processing with JavaScript using Compromise

Piyush Goyani — Fri, 26 Aug 2022 07:31:01 +0000

Introduction

Natural Language Processing(NLP) is a mostly used and discussed concept worldwide. But in the field of programming languages like Python, R and are Java widely used for this concept because of their large library support and community. Today we will show how to use the concept of NLP using JavaScript.

Installation

If you are setting up Node Project use npm.

npm install compromise

Add the following file in package.json if you face a module/import error.

"type": "module"

On the client side with CDN.

<script src="https://unpkg.com/compromise"></script>

Setup

Now let's import the library into the project.

import nlp from "compromise";

Initialise the Instance of nlp object with the demo string.

const text = "I was there, suddenly raining starts. I am still sick!";
const doc = nlp(text);

This returns the document object of an input string.

Examples

Tenses

Convert document sentences to different tenses e.g past, present, and future tense.

console.log(`Past Tense: ${doc.sentences().toPastTense().text()}`);
// Past Tense: I was there, suddenly raining starts. I was still sick!

console.log(`Present Tense: ${doc.sentences().toPresentTense().text()}`);
// Present Tense: I am there, suddenly raining starts. I am still sick!

console.log(`Future Tense: ${doc.sentences().toFutureTense().text()}`);
// Future Tense: I will be there, suddenly raining starts. I will be still sick!

Negative Statement

Convert regular or positive statements to negative statements.

console.log(doc.sentences().toNegative().text())
// I was not there, suddenly raining starts. I am not still sick!

Sentence Metadata

Let's look into the detail or meta data of the sentence. It returns array of objects for every sentence, which includes, text string, sentence detail(subject, verb, predicate, noun etc), and terms array.

console.log(doc.sentences().json())
// [
// ({
// text: "I was there, suddenly raining starts.",
// terms: [[Object], [Object], [Object], [Object], [Object], [Object]],
// sentence: { subject: "i", verb: "was", predicate: "there starts" },
// },
// {
// text: "I am still sick!",
// terms: [[Object], [Object], [Object], [Object]],
// sentence: { subject: "i", verb: "am still", predicate: "sick" },
// })
// ]

Metadata Details

As we see above, there is an array of objects named terms for every sentence, lets look inside that. It returns detail of every word and its attributes like text, pre/post symbols, tags, index, id, chunks, and dirty flag.

console.log(doc.sentences().json()[1].terms);
// [
// {
// text: 'I',
// pre: '',
// post: ' ',
// tags: ['Noun', 'Pronoun'],
// normal: 'i',
// index: [1, 0],
// id: 'i|00700100W',
// chunk: 'Noun',
// dirty: true
// },
// {
// text: 'am',
// pre: '',
// post: ' ',
// tags: ['Verb', 'Copula', 'PresentTense'],
// normal: 'am',
// index: [1, 1],
// id: 'am|00800101O',
// chunk: 'Verb',
// dirty: true
// },
// {
// text: 'still',
// pre: '',
// post: ' ',
// tags: ['Adverb'],
// normal: 'still',
// index: [1, 2],
// id: 'still|00900102C',
// dirty: true,
// chunk: 'Verb'
// },
// {
// text: 'sick',
// pre: '',
// post: '!',
// tags: ['Adjective'],
// normal: 'sick',
// index: [1, 3],
// id: 'sick|00A00103B',
// dirty: true,
// chunk: 'Adjective'
// }
// ]

Adjectives

Finding adjectives from the text.

console.log(doc.adjectives().text());
// sick

Adverbs

Looking for Adverbs that are describing the Adjectives.

console.log(doc.adjectives().adverbs().text());
// still

Adjectives Metadata

Same as a sentence, adjectives also have metadata that can retrieve as below, both have different formats of results.

console.log(doc.adjectives().json());
// [
// {
// text: 'sick!',
// terms: [[Object] ],
// normal: 'sick!',
// adjective: {
// adverb: 'sickly',
// noun: 'sickness',
// superlative: 'sickest',
// comparative: 'sicker'
// }
// }
// ]

Pre Post Sentences

Here we will add a specific symbol in starting(/) and ending() of each sentence.

console.log(doc.pre("/").text())
console.log(doc.post("\\ ").text())

// /I was there, suddenly raining starts. /I am still sick!
// /I was there, suddenly raining starts\ /I am still sick\

Whitespace

Add hyphens to white spaces in sentences with the inbuilt hyphenate method.

console.log(doc.hyphenate().text())
// I-was-there-suddenly-raining-starts. I-am-still-sick!

Number Game

const str = "Price of an Apple is $1.5, per KG may around 6 to 7.5 USD";
const numDoc = nlp(str);

Parsing numbers in a given string and getting details with prefixes and suffixes.

console.log(numDoc.numbers().parse());
// [
// { prefix: "$", num: 1.5, suffix: "", hasComma: false, unit: "" },
// { prefix: "", num: 6, suffix: "", hasComma: false, unit: "" },
// { prefix: "", num: 7.5, suffix: "", hasComma: false, unit: "usd" },
// ];

Increment/Decrement numbers in a sentence.

console.log(numDoc.numbers().increment().text());
// $2.5, 7 8.5

Convert numbers or digits to text format.

console.log(numDoc.numbers().toText().text());
// one point five dollars, six seven point five

Conclusion

Thanks for reading. In this article, we get a basic idea of how to process textual data for NLP with JavsSCript using compromise library. These were very basic uses of a library you can review its documentation further. If you enjoyed the article give it a thumb, subscribe and stay tuned for more.

Dedicated and Shared Web Worker with Performance Testing

Piyush Goyani — Thu, 18 Aug 2022 11:39:52 +0000

Overview

Web Workers are threads attached in Browser and run on the client side. It runs in a separate space from the main application thread. Simply It gives the ability to write multi-threaded Javascript applications. It is generally used to do heavy lifting tasks(in terms of computational time), e.g process huge data from API, calculations, batching of real-time chart data before drawing in the background.

Web Workers perform all processes in the Worker thread aka separate environment which is separate from the main JavaScript thread without blocking UI or affecting performance thus does not have access to DOM, window and document objects, and for the same reason, it can not simply manipulate DOM.

The life span of a Worker is limited to a browser tab, when a browser tab is closed Worker also dies. The data are passed between Worker and the main thread via messages. To send data postMessage() method is used and to listen to messages onmessage event is used on both sides.

A single web page/app can have more than one Web Worker regardless of the number of tabs. It can be Shared or Dedicated workers. We'll go through demo examples for both workers and use cases.

Demo

In this demo, there are two examples, in first you can test that the 50k array is being sorted with bubble sort algorithm with and without Web Worker(Dedicated) and see the difference between both. Second, you can test Shared Worker which is used by two client sources for similar functionality. Both workers use Network APIs for the processing which is made in Node/ExpressJS.

Preview Demo

Source of Demo

Dedicated Workers

Dedicated Workers can only establish a single connection. It can be initialized using the following syntax.

let worker = new Worker("worker.js");

// receive data from web worker sent via postMessage()
worker.onmessage = function (e) {
  console.log(e.data);
}

// send data to web worker
.postMessage("start")

To receive data onmessage event is used and to pass data postMessage method is used. In this demo, we have used a Dedicated worker to Bubble Sort 50k length of array data. In UI there are two options for sorting with or without Web Workers.

Without Web Worker

When the user clicks the without web worker option, the script starts and gets data from the API of the 50k array and starts sorting in a main, during this you may see a frozen progress bar, and the rest things are stuffed like not able to select the text, the mouse pointer changes to the cursor and other UI blocking effects because it is performing a heavy task to sort a large array. When sorting is done you'll see processing time and an animated section regarding the process below.

With Web Worker

When the user clicks the web worker option, the worker initiate and gets data from the API of the 50k array and starts sorting in a separate thread, during this you can see a progress bar and no UI blocking like text-selection and cursor etc. Everything is smooth. When sorting Done you'll see processing time and an animated section regarding the process below.

So Web Workers overcome this problem. Here is the index.html file.

<body>
    <div id="wrap">
        <div class="container">
            <div class="row pt-3">
                <div class="col">
                    <h2>Dedicated Worker</h2>
                    <div class="text-secondary h4">
                        Sorting Array with Bubble Sort(50k)
                    </div>
                    <div>
                        <button class="btn btn-large btn-primary" onclick="nonWebWorker();">Without Web Worker</button>
                        <button class="btn btn-large btn-success" onclick="withWebWorker();">With Web Worker</button>
                    </div>

                    <div id="progressbar" class="progress hide">
                        <div class="progress-bar progress-bar-striped progress-bar-animated" role="progressbar"
                            aria-valuenow="100" aria-valuemin="0" aria-valuemax="100" style="width: 100%"></div>
                    </div>
                    <div id="resultBox" class="hide bg-info rounded px-2">
                        <p class="muted">
                            Array sorted in:
                        </p>
                        <h1 id="timespent"></h1>
                        <p id="withoutWW" class="output hide">
                            As you can see, without Web Worker, your browser may be able to sort the 50K Array but you
                            can't work with your browser while sorting and also your browser won't render anything until
                            sorting ends, that's why you can't see the animated progress bar on the page.
                        </p>
                        <p id="withWW" class="output hide">
                            Your browser sorted 50K Array without any crash or lagging because your browser supports
                            Web Worker. When you do a job with Web Worker, it's just like when you run a program in
                            another thread. Also, you can see the animated progress bar while sorting.
                        </p>
                    </div>
                </div>
                <div class="col-1">
                    <div class="vr h-100">
                    </div>
                </div>
            </div>
        </div>
        <script src="./utils.js"></script>
</body>

Dedicated Worker file worker.js.

onmessage = async function (e) {
  if (e.data[0] === "start") {
    let a = [];

    async function getData() {
      return fetch(`${e.data[1]}/getData`)
        .then((res) => res.json())
        .then((data) => {
          a = data;
        });
    }

    function bubbleSort(a) {
      let swapped;
      do {
        swapped = false;
        for (let i = 0; i < a.length - 1; i++) {
          if (a[i] > a[i + 1]) {
            let temp = a[i];
            a[i] = a[i + 1];
            a[i + 1] = temp;
            swapped = true;
          }
        }
      } while (swapped);
    }
    let start = new Date().getTime();
    getData()
      .then(() => {
        bubbleSort(a);
      })
      .then(() => {
        let end = new Date().getTime();
        let time = end - start;
        postMessage(time);
      });
  }
};

Shared Workers

The Shared Worker can establish multiple connections as they are accessible by multiple scripts even in separate windows, iframes(demo) or workers. It can be spawned using the below syntax.

let sharedWorker = new SharedWorker("shared-worker.js");
sharedWorker.port.postMessage("begin");
sharedWorker.port.onmessage = function (e) {
  console.log(e.data)
}

In Shared Worker similar concept apply as Worker for data passing but via port object(explicit object) which is done implicitly in dedicated workers for communication. In this demo, there is a Shared Worker which does multiply/square of numbers. It is used in two places. The first is on the main page and the second is on another HTML page which is included in the main page via IFRAME. On the main page user input two number for multiplication and passes them to Worker and get multiplied output as below.

In the second case, the user inputs single number input which is given to the same Shared Worker and gets a squared input number as a result. Now within the worker, it takes input as the number and calls API to do Math Operation and returns the result as below.

Main HTML file to load IFRAME index.html.

<div class="col">
                    <h2>Shared Worker</h2>
                    <div class="text-secondary h4">
                        Multiply/Square Numbers with Shared Resource
                    </div>
                    <div>
                        <input type="text" id="number1" class="form-control" placeholder="Enter number 1" />
                        <input type="text" id="number2" class="form-control mt-1" placeholder="Enter number 2" />
                        <input type="button" class="btn btn-dark mt-2" value="Calculate" onclick="multiply();" />
                        <p class="result1 text-success pt-2"></p>
                        <iframe id="iframe" src="shared.html"></iframe>
                    </div>
</div>

Second HTML file shared.html which load in IFRAME in parent.

<body>
    <script type="text/javascript">
        const endpoint = window.origin;
        if (typeof (Worker) === "undefined") {
            alert("Oops, your browser doesn't support Web Worker!");
        }

        function getSquare() {
            let third = document.querySelector("#number3");
            let squared = document.querySelector(".result2");

            if (!!window.SharedWorker) {
                let myWorker = new SharedWorker("shared-worker.js");
                myWorker.port.postMessage([third.value, third.value, window.origin]);
                myWorker.port.onmessage = function (e) {
                    squared.textContent = e.data;
                };
            }
        }
    </script>
    <div class="container">
        <h1>
            Shared Web Worker(iframe)
        </h1>
        <div class="row">
            <div class="col">
                <input type="text" id="number3" class="form-control" placeholder="Enter a number" />
                <input type="button" id="btn" class="btn btn-primary" value="Submit" onclick="getSquare()" />
                <p class="result2"></p>
            </div>
        </div>
    </div>
</body>

Here is shared-worker.js file.

onconnect = function (e) {
  let port = e.ports[0];

  port.onmessage = function (e) {
    fetch(`${e.data[2]}/multiply?number1=${e.data[0]}&number2=${e.data[1]}`)
      .then((res) => res.json())
      .then((data) => {
        port.postMessage([data.result]);
      });
  };
};

Utility functions file utils.js that handles worker-related stuff.

const endpoint = window.origin;
if (typeof Worker === "undefined") {
  alert("Oops, your browser doesn't support Web Worker!");
}

function nonWebWorker() {
  cleanWindowAndStart();
  let a = [];
  async function getData() {
    return fetch(`${endpoint}/getData`)
      .then((res) => res.json())
      .then((data) => {
        a = data;
      });
  }

  function bubbleSort(a) {
    let swapped;
    do {
      swapped = false;
      for (let i = 0; i < a.length - 1; i++) {
        if (a[i] > a[i + 1]) {
          let temp = a[i];
          a[i] = a[i + 1];
          a[i + 1] = temp;
          swapped = true;
        }
      }
    } while (swapped);
  }

  let start = new Date().getTime();
  getData()
    .then(() => {
      bubbleSort(a);
    })
    .then(() => {
      let end = new Date().getTime();
      let time = end - start;
      afterStop(time, false);
    });
}

function withWebWorker() {
  cleanWindowAndStart();
  let worker = new Worker("worker.js");
  worker.onmessage = function (e) {
    afterStop(e.data, true);
  };
  worker.postMessage(["start", endpoint]);
}

function cleanWindowAndStart() {
  $("#resultBox").hide(500);
  $("#withWW").hide();
  $("#withoutWW").hide();
  $("#progressbar").addClass("d-flex").show(500);
}

function afterStop(spentTime, mode) {
  $("#timespent").html(spentTime + "ms");
  $("#progressbar")
    .hide(500, function () {
      mode ? $("#withWW").show() : $("#withoutWW").show();
      $("#resultBox").show(500);
    })
    .removeClass("d-flex");
}

function multiply() {
  let first = document.querySelector("#number1");
  let second = document.querySelector("#number2");

  let multiplied = document.querySelector(".result1");

  if (!!window.SharedWorker) {
    let myWorker = new SharedWorker("shared-worker.js");

    myWorker.port.postMessage([first.value, second.value, endpoint]);

    myWorker.port.onmessage = function (e) {
      multiplied.textContent = e.data;
    };
  }
}

Node/Express API server.js

app.get("/getData", (req, res) => {
  res.send(
    Array(50000)
      .fill(0)
      .map(() => Math.floor(Math.random() * 100))
  );
});

app.get("/multiply", (req, res) => {
  const multiply = req.query.number1 * req.query.number2;
  const result = isNaN(multiply) ? "Invalid input" : multiply;
  res.send({ result });
});

Performance Testing

Now time for numbers. Let's test our Sorting demo with Chrome Performance Test in Browser DevTools. We will record and profile both with and without web workers so we can differentiate performance and resource utilization.

Without Web Worker

Open Dev Tools and navigate to the Performance tab and start recording. Once recording starts in UI click on the Without Worker button in Dedicated Worker Section and waits till sorting is done. Once you see the result stop recording and the preview will be there as below.

This is an overview of the Performance tab which looks a little complicated. We'll use some of these for over understanding of the use case. In the first section, you can see the Frame rate, Network, CPU and Memory utilization chart. Below is the Network section where requests are recorded with the timeline. Below is Main Section which shows all tasks, macro tasks, functions and thread-related information that was executed in the tab during that time. You can click for a detailed view of each. Below is the CPU activity breakdown in a pie chart, which show the type of tasks with the time taken.

This is a detailed activity view of the bubbleSort script, and here you can see that this function consumes most of the resources and time in a thread(96.5% - 5663 ms) and other processes like rendering dom and manipulation, network calls consumed rest of all. You can save your profile if you want or delete it.

With Web Worker

Now once you test with Without Worker Profiling, start with With Worker button. The process is the same, start recording -> click on With Web Worker button -> stop recording once sorting is done. Before that make sure the Performance tab is cleared and you will see a similar result as below.

When you look in the profile With Workers in Performance tab you'll see a significant difference from the previous summary of the consumed time by task type. Here the script is consume very less time.

Again, with a worker as you can see in activity details, you won't see bubbleSort or any other script taking a long time this breakdown is of the main thread but sorting is done in a worker thread so it doesn't affect the main thread resources.

Advantages & Limitations with Usecases

Can be used to perform CPU-intensive tasks without blocking UI.
Has access to fetch, so can communicate over Netowork to make API requests.
Doesn't have access to DOM, so can't manipulate therefore, tasks like canvas, images, SVG, video or any element-related drawing/direct manipulation is not possible.
Can be used in Real-time data processing in the Stock market and related fields e.g Crypto, and NFT.
To process large data-sets like DNA and Genetics
Media manipulation in the background e.g image compress/decompress, video etc.
Can be used to process textual data in the background e.g NLP
Caching: Prefetching data for later use

Conclusion

In this article, we have got the basic idea of Web Workers including Dedicated and Shared Workers and How they affect user experience if used properly in a project. We also debug it with the Chrome DevTools Performance tab as proof of the consumed resources that our app consumed.

If you enjoyed the article and found it useful give it a thumb. Let us know your thoughts on this in the comment section below. You can find me on Twitter

Build ChatBot with ChatterBot and Python - Part 1

Piyush Goyani — Mon, 01 Aug 2022 09:20:46 +0000

Introduction

ChatterBot is a machine-learning based conversational dialogue engine build in Python which makes it possible to generate responses based on collections of known conversations. The language-independent design of ChatterBot allows it to be trained to speak any language. Initially, it has no knowledge, then based on training and communication it tries to respond to the user's query accurately. It matches the closest matching response by searching a statement that matches the input and how frequently the response is used. These things are highly customizable with logic adapters, training processes, preprocessors, filters, and other custom behaviour.

Installation

We'll install a chatterbot package with pip. So first, open your terminal and type the following command with the appropriate Python version. Here I'm using v3.7.10 specifically.

pip install chatterbot

Check if the installation is successful, run the below command:

pip show chatterbot

Once done with installation let's start with a basic example.

Create a Bot

First, create a file named greet_bot.py. import ChatBot class from chatterbot library.

from chatterbot import ChatBot

Create a new chatbot named "GreetBot", you can give any name you want.

chatbot = ChatBot("GreetBot")

Training your Bot

Import ListTrainer from sets of available trainers to train bot with greet data so it can respond to our inputs.

from chatterbot.trainers import ListTrainer

Create trainer instance of ListTrainer with chatbot parameter, and train with list of sample greeting data.

trainer = ListTrainer(chatbot)
train_data = [
    "Hello",
    "Hi",
    "Hi, how are you?",
    "I'm doing great.",
    "Greetings!",
    "How do you do?",
    "Great, thanks for asking!",
    "What's up"
] 
trainer.train(train_data)

Retrieve response from Bot

To retrieve a response from trained bot use get_response method of bot object with input parameter.

response = chatbot.get_response('hi')
print(response)

Final Code

from chatterbot import ChatBot
from chatterbot.trainers import ListTrainer

chatbot = ChatBot('GreetBot')

trainer = ListTrainer(chatbot)

train_data = [
    "Hello",
    "Hi",
    "Hi, how are you?",
    "I'm doing great.",
    "Greetings!",
    "How do you do?",
    "Great, thanks for asking!",
    "What's up"
]

trainer.train(train_data)

response = chatbot.get_response('Hi')
print(response)

To run a bot, run the script greet_bot.py

python greet_bot.py

Conclusion/Notes

Thank You for reading this article. I hope this helps in some ways and get a basic idea of how to get started with bot building with Python and the Chatterbot library. This was a very basic explanation and demo. Share, Like, Subscribe and stay tuned for the same and other bot-building tutorials.

Setup Ghost Webhook for Custom Email Newsletter

Piyush Goyani — Fri, 29 Jul 2022 05:24:59 +0000

Hello people,

This guide is to set up and send the newsletter to subscribers in Ghost cms with Custom Webhook support. As you know Ghost provides only integration of Mailgun and they don't have a free tier or simply say costly pricing structure for me.

So I needed an alternate approach from many of proposed solutions. I have tried with Zappier Gmail integration but it was causing the same issue that will occur in this setup too. I opened a discussion regarding that in Ghost Forum but no response till now.

Anyway, We will try a custom solution with Webhooks. There any many webhooks available in Ghost that you can build your custom solution or integration as needed. This guide follows a self-hosted Ghost instance on a cloud server, not a managed one. Let's begin.

Step 1: Ghost Webhooks

Refer to this Offical Webhook Doc - Ghost to understand and see a list of available webhooks in Ghost, their triggers and which to use.

Step 2: Create New Integration

From Ghost Admin, Goto Settings -> Integrations. Click on the Add Custom integration Button and you will see the below dialogue. In this dialogue give the name of your integration(e.g Custom Email hooks) or whatever you want like Newsletter something and click Create.

Step 3: Configure your Integration

When you click on Create you will be redirected to the configuration page, where you have two sections. Configuration and Webhooks.

3.1 Basic Configuration

In the Configuration section, you can add/update things like Name, Description, and Icon of Integration. Other important things are credentials like Content API key, Admin API key and API URL as below. You can regenerate API keys if you want in case of loss/compromise.

3.2 Add New webhook

Click on the Add webhook button to create a new webhook where you'll see the following popup. Add Name of the hook, Select Event e.g I have selected Page Published event from many available because we want to trigger an event when a page is published and send a mail with our custom logic and finally set Target URL where you want to pass webhook payload which will further process it and send mail, it can be any running server/less API. Click on Create and your webhook should be created.

3.3 Review/Update Webhooks

When you create a webhook you will see it is listed here, all your webhooks are listed here in this specific integration. From here you can edit/edit webhook, and see when was last it was triggered and other basic information.

Notes:

As Ghost is tightly coupled with Mailgun, and in most releases, their features are also connected to it e.g during publishing a post you can't send Newsletter if Mailgun is not configured.
With this setup, you won't be able to see stats of emails/newsletters-related stuff, If I say simply that you can only send emails when specified webhook triggers.
As I said earlier, when you send a mail with this you'll lose CSS/UI in the mail, but you can build an email template for this by referring to internal logic in Ghost.
In this, my newsletter emails are sent using NodeMailer with Gmail(SMTP), which should not be used in production, but implement queue management or an alternate approach on the server side.

Thank you for reading this article, I hope this guide was informative and useful to you, please like/share and leave your feedback.