DEV Community: Kritebh Lagan Bibhakar

Gaining Performance 🚀

Kritebh Lagan Bibhakar — Fri, 09 Feb 2024 05:30:38 +0000

I am currently working at Rubick.ai OMS (order management system) and we encountered few performance bottlenecks when loading images from AWS S3.

Initially, we directly fetched these images from S3, a process that generally worked well but occasionally resulted in longer loading times, with certain images taking up to 1 second to load.

Upon some research, a noteworthy discovery emerged—AWS S3 utilizes the HTTP 1.1 protocol. This led me to a deeper understanding of the challenges posed by HTTP 1.1, particularly the inherent limitation of opening only six connections per domain.

Consequently, when the browser-initiated requests for numerous resources from S3, it often became stalled, necessitating the opening of new connections after the initial six. This behavior was identified as a key contributor to the observed delays in image loading times.

To solve this issue, I implemented a Content Delivery Network (CDN), a widely recognized solution for optimizing the delivery of web content.

AWS has a CDN solution as CloudFront, so it was configured to serve all images and files, mitigating the limitations imposed by S3's use of HTTP 1.1.

The impact of this transition was vividly demonstrated when a profile picture, initially taking approximately 1.5 seconds to load from S3, was seamlessly reduced to a mere ~155 milliseconds after the migration to CloudFront.

Different Http Protocol:

HTTP/1.0:

Simple request-response model.
Each request opens a new connection, leading to performance inefficiencies.

HTTP/1.1:

Persistent connections to reuse a single connection for multiple requests.
Limited to six simultaneous connections per domain, potentially causing delays.

HTTP/2:

Multiplexing allows multiple requests and responses over a single connection simultaneously.
Header compression reduces overhead, improving performance.
Prioritization of requests for optimized resource loading.

HTTP/3:

Based on Google's QUIC protocol.
Focuses on improving performance over unreliable networks.
Multiplexing and encryption are integral components.

Compressing the Response

Furthermore, we addressed another performance bottleneck related to API responses, where the size ranged from 25 KB to 100 KB.

In this context, we implemented gzip compression, a technique aimed at reducing the size of transmitted data. Gzip compression resulted in an impressive 4x to 9x reduction in response size, contributing to improved overall system efficiency.

To dive deeper into compression, it is a process of encoding data to reduce its size for efficient transmission over networks. Gzip is one of the widely used compression algorithms, applying the DEFLATE algorithm to compress data.

Other compression algorithms, such as Brotli, are also gaining popularity due to their superior compression ratios. The benefits of compression extend beyond reduced data transmission size; they include faster loading times, lower bandwidth consumption, and ultimately, an enhanced user experience.

These optimizations not only resolved the challenges associated with S3's limitations but also significantly enhanced the overall speed, efficiency, and user experience of our system.

Unleash the Data Goldmine with Web Scraping

Kritebh Lagan Bibhakar — Wed, 06 Sep 2023 09:42:12 +0000

Data is the new oil and to extract oil from digital desert we can use scrapping, which means to extract data from web pages and there is various way to achieve this. There are many popular packages such as selenium, scrapy, beautiful soup etc.

But in this episode, we will be scrapping data with puppeteer a very famous package in node eco system which provide us a user-friendly Api to interact with web pages.

One other package is famous for scraping is cheerio, but I am not going to use that because it has some limitations and kind of not so user friendly for all developers because it uses jQuery syntax to filter the element and also it will not work on CSR (Client-Side Rendering) website.

Crafting Our Extractor's Den

Let's start by preparing our coding playground. Create a folder with a fun name like "scrape-book". Next, let's initiate our project: simply type npm init -y. Now, to summon Puppeteer, enter npm i puppeteer. Easy-peasy!

Now, let's get ready to code by creating a file named "scrape.js". Voilà!

mkdir scrape-book
cd scrape-book
npm init -y
npm i puppeteer
touch scrape.js

for this episode I am going to scrape the books collection of https://books.toscrape.com

now let's import the puppetter

now I will make a function scrapeBooks which will take a parameter link.

Inside this function let's initialize the browser and open a new page with the given link.

const puppeteer = require("puppeteer");
const scrapeBooks = async (link) => {
    const browser = await puppeteer.launch({headless: false}); //pass "new" for headless
    const page = await browser.newPage();
    await page.setViewport({ width: 1366, height: 768, deviceScaleFactor: 1 });
    await page.goto(link);

    await browser.close();
};

A quick tip: There is a "headless" option that can be passed in puppeteer.launch, which means the GUI version of the browser will not be opened. This is fine when we are developing locally so that we can see what's happening. You can set it to false or use the "new" option.

And the timeout challenge! Puppeteer gives us 30 seconds to open the page. If it's taking too long, we can speed things up by skipping images and unnecessary stuff. And if the page is fancy with scripts, let's give them a break.

//abort the request for image, css and javascript file
page.on('request', (request) => {
        if (request.resourceType() === 'image' || request.resourceType() === 'stylesheet' || request.resourceType() === 'script') {
            request.abort();
        } else {
            request.continue();
        }
    });

okay let's continue.

now we can execute some query in the page, for that I will make another function called script this function will be executed on that page which just opened so think of it as frontend environment and don't try to include any nodejs module in this function.

It won't work because it's environment is different that's why I have created a separate function to keep it separate.

Now grab all the products element on the page and run a loop to extract all the information we need.

I am going to store all of them in an array and then return it.

const script = () => {
    let allProductsNode = document.querySelectorAll(".product_pod");

    let allProducts = [];

    allProductsNode.forEach((p) => {
        let title = p.querySelector("h3 a").innerHTML;
        let price = p.querySelector(".price_color").innerHTML;
        let availability = p
            .querySelector(".instock.availability")
            .textContent.trim();
        let imageUrl = p.querySelector("img").src;
        let bookLink =
            `https://books.toscrape.com/catalogue/` +
            p.querySelector(".image_container a").getAttribute("href");
        allProducts.push({
            title,
            price,
            availability,
            imageUrl,
            bookLink,
        });
    });
    return allProducts;
};

we have to update our scrapeBooks function also, we will wait till the products class is available on the page and then run that script using page.evaluate.

    //wait until products class is available
    await page.waitForSelector(".product_pod")

    //excecute script to grab all the data
    let allBooks = await page.evaluate(script);

On the website there are 1000 books, 20 books per page so we can loop through all the pages and we will get all the details of 1000 books.

let's write another scrape function which will extract data from each pages.

I will simply run a loop from 1 to 50 and call the scrapeBooks function, and also I will need a global array to store all of them.

after grabbing all the data we can store it in a file so we can do this easily with fs module.

so just import fs module and create a new file books.json.

 await fs.writeFile("./books.json", JSON.stringify(data), "utf-8");

Our Book object will look like this -

Instead of writing it into a file you can also store these data in database directly.

Note:- Sometimes it is not a good idea to store all that data in a single array which will occupy lots of memory so you can also write into file in each iteration and after that you can modify that file to a single array.

now the whole code looks like this :-

const puppeteer = require("puppeteer");
const fs = require("fs/promises");
let data = [];
const script = () => {
    let allProductsNode = document.querySelectorAll(".product_pod");

    let allProducts = [];

    allProductsNode.forEach((p) => {
        let title = p.querySelector("h3 a").innerHTML;
        let price = p.querySelector(".price_color").innerHTML;
        let availability = p
            .querySelector(".instock.availability")
            .textContent.trim();
        let imageUrl = p.querySelector("img").src;
        let bookLink =
            `https://books.toscrape.com/catalogue/` +
            p.querySelector(".image_container a").getAttribute("href");
        allProducts.push({
            title,
            price,
            availability,
            imageUrl,
            bookLink,
        });
    });
    return allProducts;
};

const scrapeBooks = async (link) => {
    const browser = await puppeteer.launch({ headless: false }); //pass "new" for headless
    const page = await browser.newPage();
    await page.setViewport({ width: 1600, height: 1000, deviceScaleFactor: 1 });
    await page.goto(link);

    //wait until products class is available
    await page.waitForSelector(".product_pod");

    //excecute script to grab all the data
    let allBooks = await page.evaluate(script);

    data = [...data, ...allBooks];

    await browser.close();
};

async function scrap() {
    try {
        for (let i = 1; i <= 50; i++) {
            await scrapeBooks(`https://books.toscrape.com/catalogue/page-${i}.html`);
        }

        await fs.writeFile("./books.json", JSON.stringify(data), "utf-8");
    } catch (error) {
        console.log(error)
    }
}

scrap();

If you have any suggestions or questions, please let me know in the comment section.

I made a Boilerplate Script for Django Project Setup

Kritebh Lagan Bibhakar — Sun, 20 Jun 2021 07:27:50 +0000

Recently I started to learn to React and the best part about React is when I want to set up a new project I have 'create-react-app' which make a smoother experience in creating a project.

It installs all the required dependencies, initialized git and added some extra file also.

But there is no such tool for Django or I don't know

Whenever I have to set up a new Project in Django, I have to do some repetitive task like :

python -m venv env
pip install django
git init
touch .gitignore
code .

Yack...

So I have created a PowerShell script that will create a Virtual Environment and install Django in a folder provided by you in the current directory.

After this It will also create a .gitignore file with "/env" and finally your project will open in VS Code

Prerequisite

Python Installed
Git Installed
Visual Studio Code Installed

Python Download

Git Download

Visual studio Code Download

How to use this script

First of all get the script.

kritebh / django-boilerplate-shell-script

A simple Shell Script to create Django Project in Virtual Environment

Note : Currently, I have created only three variant of this and download the most suitable script for you.

You can modify it according to your preferences.

Like if you don't want to open Visual Studio Code after creating the project then just remove the "code ."

Similarly, if you want to run the server after creating the project then you can use third script

Let's say you want to install more module like django-rest-framework or Pillow then just add the code like this after pip install django.

...
...
pip install django
pip install djangorestframework
pip install pillow
...
...

Put your script in the directory where you want to create a new folder and set up your Django Project.

Finally, Run the script by right-clicking on it and click on "Run with PowerShell"

Here is the screenshot of script in action :

If you not creating a lot of project like me then it will fine for you to set up Project on your own.

But I think for those who are learning Django and creating a lot of projects then this script will help them 😊

If you have any doubt regarding this or have any suggestions you can comment below.

Thanks for reading!

How to use Open Graph

Kritebh Lagan Bibhakar — Sun, 13 Jun 2021 05:51:56 +0000

Now tell me which one more likely to be clicked by a normal user.

First Link, but WHY ??

Because at first glance I got what's in that link and it is more appealing than the second one.

Open graph was originally developed by Facebook and now it is used by Twitter, Pinterest and LinkedIn.

The benefit of Open Graph is simple, it converts any webpage into a social rich object which becomes more attractive on Social Media upon sharing.

Open Graph Meta Tag

Implementing Open Graph is very simple, You just need to write some meta tag in the head section of an HTML document.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta property="og:title" content="Open Graph Uses">
    <meta property="og:type" content="article">
    <meta property="og:url" content="http://example.com">
    <meta property="og:image" content="https://ogp.me/logo.png">
    <meta property="og:image:type" content="image/png">
    <meta property="og:image:width" content="300">
    <meta property="og:image:height" content="300">
    <meta property="og:image:alt" content="The Open Graph logo">
    <meta property="og:description" content="The Open Graph protocol enables any web page to become a rich object in a social graph.">
    <title>Open Graph</title>
</head>

You can see the meta property with "og" is the Open Graph Tag.

There are different types of Open Graph Meta Tag which you can use as per your requirement.

Some of the most commonly used meta tags are -

og:title
og:image
og:url
og:type
og:description
og:locale

Whole List of Tags - Open Graph Tags

Validate the Tags

So, if you have any suggestions or feedback let me know in the comment.

Thanks for reading.

Django Signals

Kritebh Lagan Bibhakar — Sat, 05 Jun 2021 02:13:46 +0000

Signal is one of the useful feature provided by Django, so let's talk about it.

What is Signal
How to use Built-in Signal
How to create a custom one

What is Signal

Yes, it is the same as you are thinking, as its name suggests "Signal" gives notification to a sender at a specific event.
Let's say you want to get notified every time a guest visited your house.

You can set up a doorbell which will give you a signal that someone has arrived.

Django Signals works in the same manner whenever some event happens it sends a signal to do some defined task.
So you can use Signal when many codes are interested in a particular event.

Built-in Signals

There are awesome Built-in signals available which you can easily use.
Let's say I want to do something before an instance saved in my Post model.
First of all, I will create a Post model and then import a signal i.e. "pre_save"

My models.py files will look like this -

from django.db import models
from django.db.models.signals import pre_save,post_save  #Inbuilt Signals
# Create your models here.

class Post(models.Model):
    title = models.CharField(max_length=30)

    def __str__(self):
        return self.title

def save_post(sender,instance,**kwargs):
    print("Pre save")

pre_save.connect(save_post,sender=Post) # This will trigger after the saving data into Post model

So when I save something in Post model I get the message "Pre Save"

Similarly you can do something after saving an instance using post_save signals.

Simple...

Here is the list for all Built-in Django Signals

There is another famous one request_finished which you can use after a request is finished.

Custom Signals

So far so good but how we can create a custom one.

wherever you want to use signals you have to import signals and receiver.

Then create an object of Signals class and then define a sender and receiver.

views.py file

from django.shortcuts import render,HttpResponse
from django.core.signals import request_finished
from django.dispatch import receiver,Signal
from .models import Post

# Create your views here.
time_stamp = Signal(providing_args=['timestamp'])
def home(request):
    time_stamp.send(sender=None,timestamp="30/05/2021")
    return HttpResponse("Home Page")

## Custom Signal
@receiver(time_stamp)
def own_track(sender,**kwargs):
    print(kwargs)

So, if you have any suggestions or feedback let me know in the comment.

Thanks for reading.

Cover Image - Photo by Chinh Le Duc on Unsplash

How to extend User Model in Django

Kritebh Lagan Bibhakar — Sat, 29 May 2021 06:46:18 +0000

In the previous post, I discussed how you can create a custom User model in Django but what if you don't want to create a custom one.

In that case, You can just extend the User model with a OneToOne model relationship.

We can easily add another field such as Location, Date of Birth etc.

Create a Django Project

Setup your Virtual environment and create a Django Project.

Note - Set up code is available at previous post.

Extending User Model

Import the User model from "django.contrib.auth.models" and we will create another model which will relate to this User model

from django.db import models
from django.contrib.auth.models import User
from django.db.models.base import Model
from django.db.models.deletion import CASCADE
# Create your models here.
class ExtendUser(models.Model):
    r = models.OneToOneField(User,on_delete=models.CASCADE)
    date_of_birth = models.DateField(null=True)
    city = models.CharField(max_length=30)
    def __str__(self):
        return self.r.username

Add this model to the admin.py

Showing Data

Now the question is how we can access this data and show it at the frontend.

It's easy..

In views.py file I have created a single function which will send the data as context.

def home(request):
    data = request.user
    return render(request,'core/index.html',{'data':data})

Now in index.html you can access this data

<p>{{data.username}}</p>
<p>{{data.extenduser.city}}</p>
<p>{{data.extenduser.date_of_birth}}</p>

Cons

By this method, you can't change the primary login method which is username.

Here is the GitHub repo for the code

kritebh / extend-user-model-django

How to extend User model in Django

How to create custom User model in Django

Kritebh Lagan Bibhakar — Sun, 23 May 2021 05:05:54 +0000

We can't imagine a web app without a custom User model, so today I will show you how you can create a custom User model in Django which will override the default User model.
Here are the main features of this custom model -

Login by email instead of username
Add own fields like Date of birth, Address, Id etc.

Let's quickly set up a new Django-Project

python -m venv env
cd env/Scripts
./activate
cd ../..
pip install django
django-admin startproject custom_user_model
cd custom_user_model
python manage.py startapp custom

Editing models.py

Import the necessary modules like AbstractBaseUser, BaseUserManager, PermissionsMixin etc.

from django.contrib.auth.models import AbstractBaseUser,BaseUserManager,PermissionsMixin
from django.utils.translation import gettext_lazy as _
from django.db import models

I am going to create a model first and then its manager.

class Account(AbstractBaseUser,PermissionsMixin):
    email         = models.EmailField(_('email address'),max_length=60,unique=True)
    username      = models.CharField(max_length=30,unique=True)
    first_name    = models.CharField(max_length=30,blank=True)
    last_name     = models.CharField(max_length=30,blank=True)
    date_of_birth = models.DateField(verbose_name='date of birth',null=True,blank=True)
    city          = models.CharField(max_length=50,blank=True)
    date_joined   = models.DateTimeField(verbose_name='date_joined',auto_now_add=True)
    last_login    = models.DateTimeField(verbose_name='last login',auto_now=True)
    is_admin      = models.BooleanField(default=False)
    is_active     = models.BooleanField(default=False)
    is_staff      = models.BooleanField(default=False)
    is_superuser  = models.BooleanField(default=False)

    objects = AccountManager()
    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = ['username','first_name']

    def __str__(self):
        return self.email

Now create a Manager, here's the code for it

class AccountManager(BaseUserManager):
    def create_user(self,email,username,first_name,password,**other_fields):
        if not email:
            raise ValueError(_("Users must have an email address"))
        if not username:
            raise ValueError(_("Users must have an unique username"))
        email=self.normalize_email(email)
        user=self.model(email=email,username=username,first_name=first_name,**other_fields)
        user.set_password(password)
        user.save()

    def create_superuser(self,email,username,first_name,password,**other_fields):
            other_fields.setdefault('is_staff',True)
            other_fields.setdefault('is_superuser',True)
            other_fields.setdefault('is_active',True)
            if other_fields.get('is_staff') is not True:
                raise ValueError('is_staff is set to False')
            if other_fields.get('is_superuser') is not True:
                raise ValueError('is_superuser is set to False')
            return self.create_user(email,username,first_name,password,**other_fields)

Our Models is ready but for see it we have to customize the admin.py file

Editing admin.py

Import the Account model and all other necessary modules and register the model at admin.site

from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from .models import Account 
# Register your models here.

class UserAdminConfig(UserAdmin):
    list_display=['email','username','first_name','date_of_birth','city']
    search_fields=['email','username','city']
    readonly_fields=['date_joined','last_login']
    fieldsets = (
        (None, {'fields': ('email', 'password')}),
        ('Personal info', {'fields': ('username','first_name','last_name','date_of_birth','city')}),
        ('Activity', {'fields': ('date_joined','last_login')}),
        ('Permissions', {'fields': ('is_admin','is_active','is_staff','is_superuser')}),
    )
    add_fieldsets = (
        (None, {
            'classes': ('wide',),
            'fields': ('email','first_name', 'date_of_birth','city', 'password1', 'password2'),
        }),
    )

admin.site.register(Account,UserAdminConfig)

So our Custom User Model is ready to use or test.

If you just want to add some extra field to User model then you can just extend the User model by the OneToOne relationship which I will talk in the next post.

Here is the full code at Github, You will find models.py and admin.py in a custom folder which we have discussed above.

kritebh / custom-user-model-django

How to Create Custom User Model in Django

Love Calculator in Django

Kritebh Lagan Bibhakar — Wed, 05 May 2021 15:01:04 +0000

We all have made some basic project like To Do App or Weather app so let's build something fun and interesting today!

I was just scrolling through the directory of RapidAPI and found an awesome API by Ajith.

Open VS Code or your favourite text editor then terminal and set up a virtual environment for creating a Django Project by following commands

Setup Venv and Django Project

python -m venv env
cd env/Scripts
./activate
cd ../..
pip install django
pip install requests
django-admin startproject lovecalculator
cd lovecalculator
python manage.py startapp core
python manage.py runserver

We have set up our Project and created an app and now make sure you have added "core" app in the INSTALLED_APPS of settings.py file.

Setup the templates

Create a "templates" folder and then "core" folder within the templates folder.

Now create "home.html" and paste the Bootstrap starter template

Create a function for the homepage in "views.py"

def home(request):
    return render(request,'core/home.html')

Later we will add all the logic of our app here.

Before we run the server just create an URL for the homepage in "lovecalculator/urls.py"

from core import views
path('',views.home,name='home'),

And finally, run the server

python manage.py runserver

we can see the "Hello, world!" text on our homepage.

Get the API key

Visit Love Calculator and get your API key or you can just copy the code snippets.

url = "https://love-calculator.p.rapidapi.com/getPercentage"
querystring = {"fname":"John","sname":"Alice"}
headers = {
     'x-rapidapi-key': "Your RAPID API KEY",
     'x-rapidapi-host': "love-calculator.p.rapidapi.com"}
response = requests.request("GET", url, headers=headers, 
params=querystring).json()
print(response)

Paste this code in "home" function.
Hit the homepage again and you will get the response in the terminal.

Now we are getting our desired data, so just arrange it and send it to templates.

In the "views.py" I have created a separate function for fetching the data from API but you can simply add in the "home" function also.
home function

try:
     if request.method=='POST':
        f = request.POST.get('fname',False)
        s = request.POST.get('sname',False)
        res = calculator(f,s)
        context ={
        'fname':res['fname'],
        'sname':res['sname'],
        'percent':int(res['percentage']),
        'result':res['result'],
            }
        return render(request,'core/result.html',context {'data':context})
        return render(request,'core/home.html')
    except:
        return render(request,'core/error.html')

def calculator(f,s):
    url = "https://love-calculator.p.rapidapi.com/getPercentage"

    querystring = {"fname":f"{f}","sname":f"{s}"}

    headers = {
        'x-rapidapi-key': "Your RAPID API Key",
        'x-rapidapi-host': "love-calculator.p.rapidapi.com"}

    response = requests.request("GET", url, headers=headers, params=querystring).json()
    return response