DEV Community: Lalit Vavdara

How I Decreased API Response Time by 89.30% in Python

Lalit Vavdara — Thu, 13 May 2021 14:08:04 +0000

Make your APIs faster

API response time is an important factor to look for if you want to build fast and scalable applications.

While working on a client’s project I had a task where I needed to integrate a third-party API for the project. This means that the Client’s project API response time will now also depend on this third-party API.

Everything works fine if we just need to make a single query to the API and return the results for that query, things start getting worse when we need to make multiple queries and return results for each query we made.

For demonstrating how to make multiple requests efficiently we will use this simple Rhyming word API.

For example, suppose we want to find rhyming words for all the words present in [“hello”, “mellow”, “cat", "rat", “dog”, “frog”, “mouse”, “sparrow”, “man”, “women”] one might think of implementing a code like this.

import requests, time

words = ["hello", "mellow", "cat", "rat", "dog", "frog", "mouse", "sparrow", "man", "women"]

def make_req_syncronously(words_arr):
    final_res = []
    for word in words_arr:
        url = f"https://api.datamuse.com/words?rel_rhy={word}&max=100"
        response = requests.get(url)
        json_response = response.json()
        for item in json_response:
            rhyming_word = item.get("word", "")
            final_res.append({"word": word, "rhyming_word": rhyming_word})
    return final_res

without_async_start_time = time.time()
response = make_req_syncronously(words)
time_without_async = time.time() - without_async_start_time

print("total time for with synchronous execution >> ", time_without_async, " seconds")

you will see that this approach takes approximately 9.479 seconds to search rhyming words for each word present in the Words list. Can you imagine you open a site and it takes 10 seconds to load? just because someone wrote this horrible code in the backend.

To better understand this look at this graph of how we make requests in this synchronous approach.

Synchronous Requests vs. Time Graph

As you see we make 1st request for (“hello”) word then wait for it to return the result and then append it to the list we want to return, and the same for each word in the list. This is an extremely time-consuming way of implementing these.

So, how can we improve this?

Well, that’s where Async/wait comes to our help. Let’s understand how we can make multiple requests simultaneously with the following Python code.

import asyncio
import aiohttp  # external library
import time

words = ["hello", "mellow", "cat", "rat", "dog", "frog", "mouse", "sparrow", "man", "women"]

async def main():
    headers = {'content-type': 'application/json'}
    async with aiohttp.ClientSession(headers=headers) as session:
        tasks = []  # for storing all the tasks we will create in the next step
        for word in words:
            task = asyncio.ensure_future(get_rhyming_words(session, word))  # means get this process started and move on
            tasks.append(task)

        # .gather() will collect the result from every single task from tasks list
        # here we use await to wait till all the requests have been satisfied
        all_results = await asyncio.gather(*tasks)
        combined_list = merge_lists(all_results)
        return combined_list


async def get_rhyming_words(session, word):
    url = f"https://api.datamuse.com/words?rel_rhy={word}&max=1000"
    async with session.get(url) as response:
        result_data = await response.json()
        return result_data


async_func_start_time = time.time()
response2 = asyncio.get_event_loop().run_until_complete(main(words))
time_with_async = time.time() - async_func_start_time

print("\nTotal time with async/await execution >> ", time_with_async, " seconds")

print("\nTotal time with async/await execution >> ", time_with_async, " seconds")

here we create a Task for each request also called an Couritine in Python. In this implementation, we start a task but don’t wait for it to complete we simply move on and start a new task.

Finally, after we are done with creating all the tasks, we wait till we have result from each task, then we simply merge all the results from tasks using merge_lists function and return a combined list containing all the results.

The time it takes to search rhyming words for all the words in words list is approximate 1 second which is way better than what we had in the previous approach.

The requests vs time graph will now look like

Asynchronous Requests vs. Time Graph

As you can see in the graph by the time we initiate the final request we will have some of the previous requests already completed.

Here is the Python code for you to compare both implementations.

_import_ requests, time

words = ["hello", "mellow", "cat", "rat", "dog", "frog", "mouse", "sparrow", "man", "women"]

_def_ make_req_syncronously(words_arr):
    final_res = []
    _for_ word _in_ words_arr:
        url = f"https://api.datamuse.com/words?rel_rhy={word}&max=100"
        response = requests.get(url)
        json_response = response.json()
        _for_ item _in_ json_response:
            rhyming_word = item.get("word", "")
            final_res.append({"word": word, "rhyming_word": rhyming_word})
    _return_ final_res

without_async_start_time = time.time()
response = make_req_syncronously(words)
time_without_async = time.time() - without_async_start_time
_#  
print_ ("total time for with synchronous execution >> ", time_without_async, " seconds")

_import_ asyncio
_import_ aiohttp _# external library_


_def_ merge_lists(results_from_fc):
    _"""  
 Function for merging multiple lists  
 """_ combined_list = []
    _for_ li _in_ results_from_fc:
        combined_list.extend(li)

    _return_ combined_list


_async def_ main():
    headers = {'content-type': 'application/json'}
    _async with_ aiohttp.ClientSession(headers=headers) _as_ session:
        tasks = [] _# for storing all the tasks we will create in the next step  
 for_ word _in_ words:
            task = asyncio.ensure_future(get_rhyming_words(session, word)) _# means get this process started and move on_  
tasks.append(task)

        _# .gather() will collect the result from every single task from tasks list  
 # here we use await to wait till all the requests have been satisfied_ all_results = _await_ asyncio.gather(*tasks)
        combined_list = merge_lists(all_results)
        _return_ combined_list


_async def_ get_rhyming_words(session, word):
    url = f"https://api.datamuse.com/words?rel_rhy={word}&max=1000"
    _async with_ session.get(url) _as_ response:
        result_data = _await_ response.json()
        _return_ result_data


async_func_start_time = time.time()
response2 = asyncio.get_event_loop().run_until_complete(main())
time_with_async = time.time() - async_func_start_time

_print_("\nTotal time with async/await execution >> ", time_with_async, " seconds")

total_improvement = (time_without_async - time_with_async) / time_without_async * 100
_print_(f"\n{'*' * 100}\n{' ' * 32}Improved by {total_improvement} %\n{'*' * 100}")

Thank you for taking the time to read my blog.

How to Integrate twillio SendGrid web API with Nextjs?

Lalit Vavdara — Sun, 18 Apr 2021 11:56:42 +0000

Whenever developing any web app whether it be some static site or some dynamic site, in most of the cases we require some type of contact form.

What is Next.js?

It is used to build server-side rendering and static web applications using React.

What is SendGrid?

SendGrid is a cloud-based SMTP provider that allows you to send email without having to maintain email servers. SendGrid manages all of the technical details, from scaling the infrastructure to ISP outreach and reputation monitoring to whitelist services and real-time analytics. In short, it is used to send emails.

Assuming that you already have a next.js app setup let's start with integrating SendGrid API to send emails, if you don't have one check out this guide, on how to create next.js app.

There are two ways in which you can achieve this, one is by using external library from SendGrid and the second is by making a POST request to https://api.sendgrid.com/v3/mail/send with all the required data, this is more suitable if you don't want to include any new library in your project for just sending emails.

Let's see how can you send emails with SendGrid web API in Nextjs, create a file sendMail.js in the utils folder of your project's root directory. Now, your project's directory structure should look like this,

add the following code to your sendMail.js file

import fetch from "node-fetch";

const SENDGRID_API_URL = "https://api.sendgrid.com/v3/mail/send";
const SENDGRID_API_KEY = process.env.NEW_SENDGRID_API_KEY;

const sendMailToMe = async (
   recepient_email, // email_address to send mail
  name_, // from name on email
  subject = "sample subject",
  client_message, // value we receive from our contact form
  client_email // value we receive from our contact form
) => {
  const sgResponse = await fetch(SENDGRID_API_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${SENDGRID_API_KEY}`,
    },
    body: JSON.stringify({
      personalizations: [
        {
          to: [
            {
              email: recepient_email,
            },
          ],
          subject: subject,
        },
      ],
      from: {
        email: "YOUR VERIFIED SENDGRID MAIL HERE",
        name: "YOUR NAME",
      },
      content: [
        {
          type: "text/html",
          value: `<strong>Client Name: ${name_} </strong> \n <p> 
                  sent you a query regarding <strong>${subject} </strong></p>
                  \n <p>Client's Message: <strong>${client_message}</strong><\p> 
                  <p>Client's Email : <strong> ${client_email} </strong></p>`,
        },
      ],
    }),
  });
  return sgResponse;
};

export { sendMailToMe };

here we are using node-fetch for making a POST request so you need to install it through npm i node-fetch it is a lightweight module that brings window.fetch to Node.js, and also this function expects some values that we will include from our contact form. You will need SendGrid API key and store it in next.config.js as env variable to send emails and also verify a Sender Identity

Then, we need to create an API endpoint in Next.js that we will use to send data from our contact form, this is done by creating a new file in pages/api folder. This api folder is a special folder in Nextjs which is used to create all the api endpoints of your Nextjs app, and these endpoints are only invoked when they are required.

so add senMail.js to the pages/api folder of your app like this.

add following code into this file,

import { sendMailQueryToMe } from "../../utils/sendMailToMe";

export default async function handler(req, res) {
  if (req.method === "POST") {
    // req.body carries all the data

    try {
      const { email, name_, subject, client_message } = req.body;

      if (
        typeof (email || name_ || subject || client_message) === "undefined"
      ) {
        console.log(" ************* Invalid Data received ************ ");

        return res
          .status(400)
          .send({ error: "bad request, missing required data!" });
      } else {
        //  Data received as expected
        try {
          const sendGridResponse = await sendMailQueryToMe(
            "your_mail@gmail.com",
            name_,
            subject,
            client_message,
            email
          );

          return res.status(200).send({
            sg_response: sendGridResponse,
          });
        } catch (err) {
          console.log(
            "ERROR WHILE SENDING MAIL TO *YOU* THROUGH WEB API >> ",
            err
          );

          return res.status(400).send({
            err_message: "bad request",
          });
        }
      }
    } catch (err) {
      console.log("Err while sending Mail through send grid >> ", err);
      return res
        .status(400)
        .send({ error: "Error in sendgrid Service.", errMsg: err });
    }
  }

  res.status(400).send({ error: "bad request" });
}

now finally we need to create some UI form from which users can send mail. For this, Create a contact.js file in the pages folder of your app and add the following code to it.

import React, { useState } from "react";
import MailOutlineIcon from "@material-ui/icons/MailOutline";
import { MenuItem, Input } from "@material-ui/core";
import TextField from "@material-ui/core/TextField";
import https from "https";

function contact() {
  const [formError, setFormError] = useState({ error: "" });
  const [querySubject, setQuerySetsubject] = useState("");
  const [name_, setName_] = useState("");
  const [clientEmail, setClientEmail] = useState("");
  const [clientMsg, setClientMsg] = useState("");

  const serviceOptions = [
    {
      value: "option1",
      label: "option1",
    },
    {
      value: "option2",
      label: "option2",
    },
    {
      value: "option3",
      label: "option3",
    },
    {
      value: "option4",
      label: "option4",
    },
  ];

  const sendMail = async (
    client_name,
    client_email,
    client_message,
    client_subject
  ) => {
    const data = JSON.stringify({
      name_: client_name,
      email: client_email,
      client_message: client_message,
      subject: client_subject,
    });

    const options = {
      path: "/api/sendMail",
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Content-Length": data.length,
      },
    };

    const req = https.request(options, (res) => {
      // console.log(`statusCode: ${res.statusCode}`);

      res.on("data", (d) => {
        // process.stdout.write(d);
        // console.log("data  from API >> ", JSON.parse(d));
      });
    });

    req.on("error", (error) => {
      setFormError({
        error: "Unable to send your message please try after some time.",
      });
    });

    req.write(data);
    req.end();
  };

  return (
    <div>
      <form
        style={{ display: "flex", flexDirection: "column", padding: "50px" }}
      >
        <Input
          style={{ width: "100%", color: "black" }}
          type="text"
          value={name_}
          placeholder="What is your name ?"
          onChange={(e) => setName_(e.target.value)}
          required
        />
        <Input
          style={{ width: "100%", color: "black" }}
          value={clientEmail}
          type="email"
          placeholder="What is your email ?"
          onChange={(e) => setClientEmail(e.target.value)}
          required
        />
        <TextField
          style={{ width: "100%", color: "black" }}
          id="standard-select-Interest"
          select
          label="What are you interested in ?"
          value={querySubject}
          onChange={(e) => setQuerySetsubject(e.target.value)}
          required
        >
          {serviceOptions.map((option) => (
            <MenuItem key={option.value} value={option.value}>
              {option.label}
            </MenuItem>
          ))}
        </TextField>

        <TextField
          style={{ width: "100%", color: "black" }}
          id="client-message-textarea"
          label="Message"
          multiline
          rows={4}
          value={clientMsg}
          onChange={(e) => setClientMsg(e.target.value)}
        />
        {formError.error ? (
          <div className="form__error">
            <span style={{ color: "black" }}>{formError.error}</span>
          </div>
        ) : (
          ""
        )}
        <div>
          <button
            disabled={!name_ || !clientEmail || !clientMsg || !querySubject}
            type="submit"
            style={
              !name_ || !clientEmail || !clientMsg || !querySubject
                ? {
                    backgroundColor: "#878a8f",
                    color: "white",
                    transform: "scale(1)",
                    cursor: "default",
                    margin: "50px 0",
                  }
                : { margin: "50px 0" }
            }
            onClick={(e) => {
              e.preventDefault();
              sendMail(name_, clientEmail, clientMsg, querySubject);
            }}
          >
            <MailOutlineIcon /> Send
          </button>
        </div>
      </form>
    </div>
  );
}

export default contact;

And that's all folks you have a full-featured contact form from which you can send or receive emails.

I have implemented this in my contact form which you can try at my Site

How to create Real Deepfakes ?

Lalit Vavdara — Mon, 05 Apr 2021 09:57:16 +0000

First things first !

What is a deepfake ?

Deepfakes are basically fake videos of real people or objects generated using complex machine learning algorithms, that require some good computational power as well.

So, I was just wondering on youtube and suddenly youtube algorithm came up with some unexpected recommendation, it was Jordan peele and Barack Obama's deep fake video. which you can watch it here
and I decided to make one like that.

Now, I don't have any experience with machine learning or deep learning stuff, neither I have that much computational power, but still I found one really cool Github repo out there which can help with all those problems.

AliaksandrSiarohin / first-order-model

This repository contains the source code for the paper First Order Motion Model for Image Animation

Now, as this Repository states that it contains the source code for the paper First Order Motion Model for Image Animation by Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci and Nicu Sebe. So all with all credits to them they have done some really great work.

Now we can easily clone this repo to our pc and start working on it, but that will require you to install all the heavy requirements too, which I think will take much more time if you have never worked with all that machine learning stuff. And also not to forget the GPU power that's required.

So, I think in that case we should use the Colab Demo that they have provided which is much easier to work with. Now all you need is google drive account to add source image and driving video to the modal.

Also you will need to add first-order-motion-modal folder from the link provided in colab demo file to your G-drive which you can download and upload it to your drive or you can simply create a copy to your drive.

This folder includes some default data in it to try but you can easily add your data to the modal by changing the path in source_image and reader.

Change the path in both the variable to your g-drive location like for example if your source_image and driving video are located in 'My drive' than your path will be

source_image = imageio.imread('/content/gdrive/My Drive/source_image.png')

reader = imageio.get_reader('/content/gdrive/My Drive/driving_video.mp4')

Now there are some limitations to this modal which you have to keep in mind in order to make it work as expected. For the source_image it should be a proper square image i.e the aspect ratio of the source image should be exactly 1:1 . similarly the dirving video should also be a proper square video you can choose any of online services to crop video to 1:1 aspect ratio.

Once you have done that just run the remaining code blocks from colab-demo and you will have your deepfake video.