DEV Community: Mageshwaran

How did I build a web app in a day and deployed

Mageshwaran — Fri, 08 Jan 2021 15:21:25 +0000

Tellasecret

You came here because of the title, yes this the article about how I build a Django app in a weekend.
Any suggestion or feedback is highly welcome.

https://tellasecret.live/
This is the website build in a day and gone to live with multiple iterations. At first, I thought it can be used to collect the anonymous message from friends but it can do more than that, this can be used to collect feedback from clients to improve the service, feedback for teachers and many more. Please command your suggestion.

The goal is to build a secret messaging app with Django, then I set a deadline to complete it within a day. I start to build and map the design of the APIs and tables, I did split the work into multiple parts like APIs, tables, Ui development at the first stage. And within 3 hours, I did complete the Apis level logics and the table schema.
Next thing is to build the frontend, in Django the frontend is built using Django templates. I'm not a frontend expert this is the part where I took some extra time, changing the UI multiple time and then I did build that too.

Then its time to deploy the code in EC2, django app is connected to the Nginx server. Check this article for reference django-nginx-gunicorn-virtualenv-supervisor, this the first time I ever try to deploy the django with Nginx. The project failed multiple times and did some late night work for more than two days and then finally deployed.

I learn a lot in this, its just a simple website but the learning I got in this is huge, listed some extra things below.
Breaking the parts

Apis
Database
Frontend
Content for page
SEO tags, OGP
Django sitemap
Deploying the app in EC2
Connecting Django app with the Nginx

Thanks for reading, please do check this https://tellasecret.live/ and share it with your friends.

My secret link to message me, the sender will stay unknown.
https://tellasecret.live/link/VYaFTTz

Please leave your suggestion in command.

Bubble sort in python

Mageshwaran — Wed, 18 Nov 2020 13:45:48 +0000

Bubble sort is the simplest sorting algorithm, compares the adjacent value and swap the element based on the condition.

In ascending order sorting basicly what it do is, comapres the two values and change the position, iterate it through the values and place the highest value at the end.

For this example we are taking this value as input
Values list -- > [6, 5, 3, 1, 8, 7]

Tracing

STAGE 1

Input
[6, 5, 3, 1, 8, 7]

compare position 0 and 1 values
---> 6>5 position 0 is greate than 1, swap it
[5, 6, 3, 1, 8, 7]

compare position 1 and 2 values
---> 6>3 position 1 is greate than 2, swap it
[5, 3, 6, 1, 8, 7]

compare position 2 and 3 values
---> 6>1 position 2 is greate than 3, swap it
[5, 3, 1, 6, 8, 7]

compare position 3 and 4 values
---> 6>8 position 3 is less than 4, no swap
[5, 3, 1, 6, 8, 7]

compare position 4 and 5 values
---> 8>7 position 4 is greate than 5, swap it
[5, 3, 1, 6, 7, 8]

LAST POSITION VALUE 8 IS LOCKED

STAGE 2

Input
[5, 3, 1, 6, 7]

compare position 0 and 1 values
---> 5>3 position 0 is greate than 1, swap it
[3, 5, 1, 6, 7]

compare position 1 and 2 values
---> 5>1 position 1 is greate than 2, swap it
[3, 1, 5, 6, 7]

compare position 2 and 3 values
---> 5>6 position 2 is less than 3, no swap
[3, 1, 5, 6, 7]

compare position 3 and 4 values
---> 6>7 position 3 is less than 4, no swap
[3, 1, 5, 6, 7]

LAST POSITION VALUE 7 IS LOCKED

STAGE 3

Input
[3, 1, 5, 6]

compare position 0 and 1 values
---> 3>5 position 0 is less than 1, no swap
[3, 5, 1, 6]

compare position 1 and 2 values
---> 5>1 position 1 is greate than 2, swap it
[3, 1, 5, 6]

compare position 2 and 3 values
---> 5>6 position 2 is less than 3, no swap
[3, 1, 5, 6]

LAST POSITION VALUE 6 IS LOCKED

STAGE 4

Input
[3, 1, 5]

compare position 0 and 1 values
---> 3>1 position 0 is greate than 1, swap it
[1, 3, 5]

compare position 1 and 2 values
---> 3>1 position 1 is less than 2, no swap
[1, 3, 5]

LAST POSITION VALUE 5 IS LOCKED

STAGE 5

Input
[1, 3]

compare position 0 and 1 values
---> 3>1 position 0 is less than 1, no swap
[1, 3]

LAST POSITION VALUE 3 IS LOCKED

Sorted result 1, 3, 5, 6, 7, 8

In this example we are using the python for loop to iterate over the list values, create a python function called bubbleSort where the logic for bubble sort will be written.

def bubbleSort(k):
    last = 0
    for i in range(0,len(k)):
        # reduce the last value to avoid over lapping
        for j in range(0,len(k)-1-last):
            # compare the adjustent value
            if k[j] > k[j+1]:
                k[j], k[j+1] = k[j+1], k[j]

        last+=1
        print(k)
    return k

k = [3,40,2,0,10]
bubbleSort(k)

"""
result
[3, 2, 0, 10, 40]
[2, 0, 3, 10, 40]
[0, 2, 3, 10, 40]
[0, 2, 3, 10, 40]
[0, 2, 3, 10, 40]

[0, 2, 3, 10, 40]
"""

Another method

If the values are already sorted or in the mid of the process if it get sorted then you don't have to do the process till the end. We can stop it when values got sorted.

def bubbleSort(k):
    last = 0
    for i in range(0,len(k)):
        # when the list is already sorted, its better to stop this will increase the process time
        done = True
        # reduce the last value to avoid over lapping
        for j in range(0,len(k)-1-last):
            # compare the adjustent value
            if k[j] > k[j+1]:
                k[j], k[j+1] = k[j+1], k[j]
                done = False
        if done:
            break
        last+=1
        print(k)
    return k

k = [3,40,2,0,10]
bubbleSort(k)

"""
result
[3, 2, 0, 10, 40]
[2, 0, 3, 10, 40]
[0, 2, 3, 10, 40]

[0, 2, 3, 10, 40]
"""

You can see the difference in the result first method run till the end, in second method once the list sorted, it stop.

Time Complexities

Bubble sort has a worst-case and average complexity of О(n2), Best case of O(n) if the values already sorted.

Learn python list slicing in programdoc

Django For Beginners - Life Cycle P2

Mageshwaran — Sat, 07 Nov 2020 11:30:36 +0000

When client sends a request(http synchronous) and waits for a response from the service, a lot happens in the server side, server will get the request and process it and return the response to the client.

Note 1

When client made a request, it will be sent to the server like Nginx(in development, django will use the default development server), then it will connect to the wsgi(standard interface for communication between an application and the web server) module in django to talks to the django project

Note 2

Then the request will be send through the middleware, django has its own premade middleware for things like authendication, session and many more. Middleware is like a gatekeeper, request has to cross each one in order and vice versa. you can even write your own custom middleware.

Note 3

urls is the router in django, django will look for the requested url and calls the view

Note 4

view in django where the logic will be written, view will get the requesting data and process it, queries the model and return back the HttpResponse as template or any other format like json, xml. view is the place where you work, most of the time to build the logic for the application.

Note 5

Templates in django are normal html files with extract features included, it will be send to client as html file pre rendered in the server. Template can be reused in another templates by including it (like reactjs component).

Note 6

Response has to cross the Middleware in the reverse order, check Note 2. you can validate the response or do any kind of validation in the response middleware.

Thanks you, have a great day ahead.😎

Django For Beginners - Project Setup P1

Mageshwaran — Mon, 26 Oct 2020 14:56:55 +0000

Django is a python web framework used to build web app and api services. Its comes under Model Template Views architectural pattern. Django is battery included tool, shipped with many built in feature like authentication, admin panel, session handling, ORM, etc.

Virtualenv

Advantage of using virtualenv is machine can have multiple django projects which are running in different version of django.

create Virtual environment using virtualenv

virtualenv --python=python3 venv

to know more about Virtual environment

Activate Virtual environment

This command will work on bith Mac and Linux

source venv/bin/activate

Use this to activate on Windows

venv\Scripts\activate

To deactivate Virtual environment, this command common for all the platforms

deactivate

Install Django

After activating the virtual environment install the latest django version in venv

pip install django

To check django is installed, list the pip packages, this command will list all the installed packages in venv

pip freeze

or you can check by entering this command in terminal. if django installed, you should see the version of your installation.

python -m django --version

Creating a project

Commad will create a project called mysite, its just a directory. mysite will act as the root directory for the project.

django-admin startproject mysite

mysite/ ---> root directory
    manage.py
    mysite/ ---> django package directory
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Alternative command, both do the same process with a small difference. Dot . will create the project at the at location from where the command is being executed. Then the directory will act as root directory

django-admin startproject mysite .

Now lets check the project, go to the root directory and run this command to start the django project

python manage.py runserver

Above one will run the django in http://127.0.0.1:8000/ Ip and port, we can change the running ip and port.

python manage.py runserver 192.168.43.135:8080

This will run the project in http://192.168.43.135:8080

runserver 192.168.43.135:8080
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
October 26, 2020 - 19:25:29
Django version 3.1.2, using settings 'mysite.settings'
Starting development server at http://192.168.43.135:8080/
Quit the server with CTRL-BREAK.

Open the link in browser with http://192.168.43.135:8080/, you will get an error this is because you need to add the ip in allowed host in settings.py.

For now use the default 127.0.0.1, run the project

python manage.py runserver

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
October 26, 2020 - 19:38:32
Django version 3.1.2, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.

???

You have 18 unapplied migration(s).

Django shipped with build in feature like admin page, session, auth for that it has to apply the migration table to the DB.

By default django shipped with SQlite DB.

Lets apply the migrate to the database

python manage.py migrate

You can see default table structure are applied to SQlite database

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK

Thanks you, have a great day ahead.😎

Open Graph Protocol Analyzer in Python

Mageshwaran — Mon, 19 Oct 2020 14:58:36 +0000

What is Open Graph protocol used for?

The Open Graph protocol enables any web page to become a rich object in a social graph. For instance, this is used on Facebook to allow any web page to have the same functionality as any other object on Facebook.

Open Graph Protocol was first introduced by Facebook that allows integration between Facebook and its user data and a website. By integrating Open Graph meta tags into your website will help social network to crawl the data given in the web page, you can identify which elements of your page you want to show when someone share's your page.

You may seen this while sharing a web link in social network, like twitter card, Facebook link share, Whatsapp link card.

To learn more about Open Graph protocol
Online testing tool Open Graph Tester

We are going to build a simple Open Graph protocol analyzer, which will fetch the OGP data from websites. By using the python libraries BeautifulSoup and requests.

Install beautifulsoup4 and requests

pip install requests
pip install beautifulsoup4

The website like user here is only for learning purposes.

import requests
from bs4 import BeautifulSoup
url = "https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/"
r = requests.get(url=url)
# Create a BeautifulSoup object
soup = BeautifulSoup(r.text, 'html.parser')

Above code will fetch the website content and load it as a BeautifulSoup object, from that we can extract the data.

# webpage can have more meta tags

# return the first meta tag
soup.find("meta")

# return all meta tag
soup.find_all("meta")

# return all meta tag with property as og:title
soup.find("meta",  property="og:title")

soup.find_all("meta") find all the meta tags, return the value in python list format, using this then filter out the individual tags. Iterate over the meta tag value, filter out the tag values using python if statement based on tag properties.

# data holder
data = {
    "tag": {},
    "ogp": {}
}
# find all the meta tags in the web page
for i in soup.find_all("meta"):
    # extract individual tag with the property value
    if i.get("property", None) == "og:title":
        data["tag"]["title"] = i
        data["ogp"]["title"] = i.get("content", None)
    if i.get("property", None) == "og:url":
        data["tag"]["url"] = i
        data["ogp"]["url"] = i.get("content", None)
    if i.get("property", None) == "og:description":
        data["tag"]["description"] = i
        data["ogp"]["description"] = i.get("content", None)
    if i.get("property", None) == "og:image":
        data["tag"]["image"] = i
        data["ogp"]["image"] = i.get("content", None)
    if i.get("property", None) == "og:type":
        data["tag"]["type"] = i
        data["ogp"]["type"] = i.get("content", None)
    if i.get("property", None) == "og:site_name":
        data["tag"]["site_name"] = i
        data["ogp"]["site_name"] = i.get("content", None)
    if i.get("property", None) == "og:locale":
        data["tag"]["locale"] = i
        data["ogp"]["locale"] = i.get("content", None)
print(data)

{'tag': {'title': <meta content="Flutter &amp; Dart - The Complete Guide [2020 Edition]" property="og:title"/>, 'url': <meta content="https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/" property="og:url"/>, 'description': <meta content="A Complete Guide to the Flutter SDK &amp; Flutter Framework for building native iOS and Android apps" property="og:description"/>, 'image': <meta content="https://img-a.udemycdn.com/course/480x270/1708340_7108_4.jpg?mTkNpG_o5Wh0tcZgEWDnLLfndz7BG87EWBPuhbZij4iaIzFjeWC9AwmBEt4sTy0ioCD3r8w-Wtzfac00nfnb-TGMYVhafN8EXUpihTvhffAbcaEuTbQgRQvPORm5i1bX" property="og:image"/>, 'type': <meta content="udemy_com:course" property="og:type"/>, 'site_name': <meta content="Udemy" property="og:site_name"/>, 'locale': <meta content="en_US" property="og:locale"/>}, 'ogp': {'title': 'Flutter & Dart - The Complete Guide [2020 Edition]', 'url': 'https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/', 'description': 'A Complete Guide to the Flutter SDK & Flutter Framework for building native iOS and Android apps', 'image': 'https://img-a.udemycdn.com/course/480x270/1708340_7108_4.jpg?mTkNpG_o5Wh0tcZgEWDnLLfndz7BG87EWBPuhbZij4iaIzFjeWC9AwmBEt4sTy0ioCD3r8w-Wtzfac00nfnb-TGMYVhafN8EXUpihTvhffAbcaEuTbQgRQvPORm5i1bX', 'type': 'udemy_com:course', 'site_name': 'Udemy', 'locale': 'en_US'}}

this is good, but ogp has more properties like og:image, og:audio, og:determiner, og.local, etc. For more detail https://ogp.me/#optional .

without explicit specifying individuals ogp property value, check the property attribute has a og value or else exclude it. Store the values in python dictionary variable called data

data = {
    "tag": {},
    "ogp": {}
}
for i in soup.find_all("meta"):
    if i.get("property", None) is not None:     
        if i.get("property", None).split(":")[0] == "og":
            data["tag"][i.get("property", None)] = i
            data["ogp"][i.get("property", None)] = i.get("content", None)
print(data)

{'tag': {'og:title': <meta content="Flutter &amp; Dart - The Complete Guide [2020 Edition]" property="og:title"/>, 'og:url': <meta content="https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/" property="og:url"/>, 'og:description': <meta content="A Complete Guide to the Flutter SDK &amp; Flutter Framework for building native iOS and Android apps" property="og:description"/>, 'og:image': <meta content="https://img-a.udemycdn.com/course/480x270/1708340_7108_4.jpg?mTkNpG_o5Wh0tcZgEWDnLLfndz7BG87EWBPuhbZij4iaIzFjeWC9AwmBEt4sTy0ioCD3r8w-Wtzfac00nfnb-TGMYVhafN8EXUpihTvhffAbcaEuTbQgRQvPORm5i1bX" property="og:image"/>, 'og:type': <meta content="udemy_com:course" property="og:type"/>, 'og:site_name': <meta content="Udemy" property="og:site_name"/>, 'og:locale': <meta content="en_US" property="og:locale"/>}, 'ogp': {'og:title': 'Flutter & Dart - The Complete Guide [2020 Edition]', 'og:url': 'https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/', 'og:description': 'A Complete Guide to the Flutter SDK & Flutter Framework for building native iOS and Android apps', 'og:image': 'https://img-a.udemycdn.com/course/480x270/1708340_7108_4.jpg?mTkNpG_o5Wh0tcZgEWDnLLfndz7BG87EWBPuhbZij4iaIzFjeWC9AwmBEt4sTy0ioCD3r8w-Wtzfac00nfnb-TGMYVhafN8EXUpihTvhffAbcaEuTbQgRQvPORm5i1bX', 'og:type': 'udemy_com:course', 'og:site_name': 'Udemy', 'og:locale': 'en_US'}}

After this blog post in submitted Dev will generate Open Graph protocol for this page, you can check this by View page source.

https://dev.to/magesh236/open-graph-protocol-analyzer-4dk0

** Conclusion:** This comes under web scraping technique, so use it with caution. Not all the website allows you to scrape their content in that case use tool like selenium to render the website, after that get the web page content and pass it to the web scraping tool.

Compress image in python

Mageshwaran — Wed, 14 Oct 2020 05:11:09 +0000

Yesterday i wrote a article in Dev about

Google Chrome Extension

I upload sample images of the Google Chrome Extension - New Tab a New Quote but it fails, i tried it multiple time, because of the file size and the network problem which i faced.
So i decided to compress the image using python

   from PIL import Image
   path = "img/Screenshot 6.png"
   image = Image.open(path)
   image = image.convert('RGB')
   image.save("img/Screenshot 6e.jpg",quality=20,optimize=True)

you can see the difference

8MB

275KB

Thanks you, have a great day ahead.😎

Build a Google Chrome Extension

Mageshwaran — Tue, 13 Oct 2020 17:58:02 +0000

In this article I will show you how to create a simple google chrome extension using HTML and Javascript which will show the current time and a random quotes with nature background image.

Background image will be fetched from Pexels.

Manifest

First let's create a folder called ChromeExte and then inside create a manifest.json file with the meta details

{
  "manifest_version": 2,

  "name": "Background",
  "description": "Background Image Changer",
  "version": "1.0",

  "chrome_url_overrides" : {
    "newtab": "Background.html"
  }
}

chrome_url_overrides is used to change the default page in chrome, it can be used to change newtab, bookmarks, history pages. By using a html file.

When a newtab is clicked it will look for the Background.html

Base html

we need to create it inside the ChromeExte directory

<!doctype html>
<html>
  <head>
    <title>Background BG</title>
  </head>
  <style type="text/css">
    body {
      background: url('pexels-photo-775201.jpeg') no-repeat center center fixed;
      -moz-background-size: cover;
      -webkit-background-size: cover;
      -o-background-size: cover;
      background-size: cover;
    }
    .content{
      width: 50%;
      position: absolute;
      top: 50%;
      left: 50%;
      transform: translate(-50%, -50%);
    }
    #show_time, #quote {
      color: white;
      text-align: center;
      text-shadow: 2px 2px 5px black;
    }
    #show_time {
      font-size: 55px;
    }
  </style>
  <body>

    <div class="content">
      <h1 id="show_time"></h1>
      <h1 id="quote"></h1>
    </div>

    <script src="script.js"></script>
  </body>
</html>

body tag will hold the background image from Pexels.
show_time used to show the current system time by using javascript.
quote used to show quotes.

Now let's add the functionality to this, for this we need javascript.

Random quotes logic

quotes will hold the list of quotes, to get the random quote from quotes we used Math in this.

Math.random() - will return a random decimal value, multiple it with the length of quotes this will result in some value and then floor the value, it will reduce to lower nearest. Use this value as index to get the quote.

const quotes = ["Always believe that something wonderful is about to happen.",
"Act as if what you do makes a difference. It does.", 
"Success is not final, failure is not fatal: it is the courage to continue that counts.", 
"Never bend your head. Always hold it high. Look the world straight in the eye.", 
"What you get by achieving your goals is not as important as what you become by achieving your goals.", 
"When you have a dream, you've got to grab it and never let go.", 
"I can't change the direction of the wind, but I can adjust my sails to always reach my destination.", 
"No matter what you're going through, there's a light at the end of the tunnel.",
"It is our attitude at the beginning of a difficult task which, more than anything else, will affect its successful outcome.",
"Life is like riding a bicycle. To keep your balance, you must keep moving."];

// random quote index
const quotes_i = Math.floor(Math.random() * quotes.length);
// set a quote
document.getElementById("quote").innerHTML = quotes[quotes_i];

Pexels Image 🏜⛰🏔🏕

https://www.pexels.com/api/?locale=en-US
Get the access token to use the api, token has to be passed in the header. We will use plain javascript for Ajax call.

We only get one image per page in the Api. To get the random image Math.floor(Math.random() * 200) + 1 this will return a value from 0 to 200 use this as page number, you can change the value as you want.

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX replace this with your access token from pexels

function loadPexels() {
  var xhttp = new XMLHttpRequest();
  // random page number
  page = Math.floor(Math.random() * 200) + 1
  // get one image per page
  xhttp.open("GET", "https://api.pexels.com/v1/search?query=hd nature wallpaper&per_page=1&page=" + page, true);
  // send Authorization in header
  xhttp.setRequestHeader("Authorization", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
  // response is in json
  xhttp.responseType = 'json';
  xhttp.send();

  xhttp.onreadystatechange = function() {
    // when the request is success get the image and update it as background
    if (this.readyState == 4 && this.status == 200) {
      document.body.style.backgroundImage = "url('"+ this.response.photos[0].src.large2x +"')";
    }
  };
}

body backgroundImage will get updated.
this.response.photos[0].src has the image source with different resolution.

Time shower ⏱

using Date method get the current time in AM PM format and set the value to show_time element, we need to update it so call it with setTimeout which is asynchronous function in javascript. It will call the setTime with the interval of 0.5 Sec.

// this function is asynchronous
function setTime(){
    t = new Date().toLocaleString('en-US', { hour: 'numeric', minute: 'numeric', second:'numeric', hour12: true });
    document.getElementById("show_time").innerHTML = t;
    // call itself for each 0.5Sec
    var j = setTimeout(setTime, 500);
}

Full JavaScript code


const quotes = ["Always believe that something wonderful is about to happen.",
"Act as if what you do makes a difference. It does.", 
"Success is not final, failure is not fatal: it is the courage to continue that counts.", 
"Never bend your head. Always hold it high. Look the world straight in the eye.", 
"What you get by achieving your goals is not as important as what you become by achieving your goals.", 
"When you have a dream, you've got to grab it and never let go.", 
"I can't change the direction of the wind, but I can adjust my sails to always reach my destination.", 
"No matter what you're going through, there's a light at the end of the tunnel.",
"It is our attitude at the beginning of a difficult task which, more than anything else, will affect its successful outcome.",
"Life is like riding a bicycle. To keep your balance, you must keep moving."];

// random quote index
const quotes_i = Math.floor(Math.random() * quotes.length);
// set a quote
document.getElementById("quote").innerHTML = quotes[quotes_i];


function loadPexels() {
  var xhttp = new XMLHttpRequest();
  // random page number
  page = Math.floor(Math.random() * 200) + 1
  // get one image per page
  xhttp.open("GET", "https://api.pexels.com/v1/search?query=hd nature wallpaper&per_page=1&page=" + page, true);
  // send Authorization in header
  xhttp.setRequestHeader("Authorization", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
  // response is in json
  xhttp.responseType = 'json';
  xhttp.send();

  xhttp.onreadystatechange = function() {
    // when the request is success get the image and update it as background
    if (this.readyState == 4 && this.status == 200) {
      document.body.style.backgroundImage = "url('"+ this.response.photos[0].src.large2x +"')";
    }
  };
}
// load a random image
loadPexels();

// this function is asynchronous
function setTime(){
    t = new Date().toLocaleString('en-US', { hour: 'numeric', minute: 'numeric', second:'numeric', hour12: true });
    document.getElementById("show_time").innerHTML = t;
    // call itself for each 0.5Sec
    var k = setTimeout(setTime, 500);
}
// set current time
setTime();

script.js--Linked To-->>Background.html--Linked To-->>manifest.json

File structure

To test this working you can just open the html file in browser as normal.

Add extension to chrome

Go to
chrome://extensions/
then enable the Developer Mode, at the top right

Load the extension, click on Load unpacked

select the ChromeExte directory
Extension got loaded

Samples Images

Thanks you, have a great day ahead.🤪😎

Captcha image generator

Mageshwaran — Wed, 07 Oct 2020 17:29:14 +0000

We will build a logic to create some random captcha image using python with the help of pillow package.

For more detailed reference about pillow check this.

install this package using this command

   pip install pillow

To learn more about virtual environment

Create new image

this piece of code will create a white image with 90X60 and saved a blank_image.png

   from PIL import Image
   img = Image.new('RGB', (90, 60), color = 'white')
   img.save('blank_image.png')

blank_image.png

Draw text on image

now we created a image and saved it, lets add some text to it

   from PIL import Image, ImageDraw
   draw = ImageDraw.Draw(img)
   draw.text((20,20), "Hello", fill=(38, 38, 38))
   img.save('blank_image.png')

Hello is written in img object at poin 10X10

Draw lines

Let draw some line on the image

   draw.line((40,30, 150,300), fill=120, width=1)
   draw.line((20,30, 30,20), fill=88, width=2)

fill is the color and you set the width size,
(40,30) is the starting point for line 1 and ends at (150,300).
You can notice that the end point is larger than the image size, it will be omitted.

Same as line 1, line 2 starts at (20,30) and ends in (30,20)

Draw points

Let add some more stuff
pass points as python tuples of tuples to draw points in the image

   draw.point(((30, 40), (10, 20), (14, 25), (35, 10), (14, 28), (17, 25)), fill="black")

this one will throws the dot points in the image at the given XY coordinates.

Generate the captcha string

This same logic is used in Url Shortener, you can check it here

Build Url Shortener API In Django - Redis

random_string function generate a random string of length 5, you can adjust it.

import string, random 

# generate thr random captcha string
def random_string():
    # hash length
    N = 5
    s = string.ascii_uppercase + string.ascii_lowercase + string.digits
    # generate a random string of length 5
    random_string = ''.join(random.choices(s, k=N))
    return random_string

Python lambda function

what is lambda?
Is a special function with one line, and with one expression. It can get multiple arguments.

# Normal funtion
def square(a):
    return a*a

square(5) ---> 25

# lambda funtion
square = lambda a: a*a
square(5) ---> 25

lambda function to get the random location in the image

   # lambda function - used to pick a random loaction in image
getit = lambda : (random.randrange(5, 85),random.randrange(5, 55))

this will spite out the random value between 5 and 84,
Note: 85 not taken

   random.randrange(5, 85)

values for colors which will be randomly picked

   # pick a random colors for points
   colors = ["black","red","blue","green",(64, 107, 76),(0, 87, 128),(0, 3, 82)]
   fill_color = [120,145,130,89,58,50,75,86,98,176,]
   # pick a random colors for lines
   fill_color = [(64, 107, 76),(0, 87, 128),(0, 3, 82),(191, 0, 255),(72, 189, 0),(189, 107, 0),(189, 41, 0)]

To get a random value from list, this will get a random color value one

   random.choice(colors)

Full logic to generate the captcha image

from PIL import Image, ImageDraw, ImageFont
import string, random 

# generate thr random captcha string
def random_string():
    # hash length
    N = 5
    s = string.ascii_uppercase + string.ascii_lowercase + string.digits
    # generate a random string of length 5
    random_string = ''.join(random.choices(s, k=N))
    return random_string

# lambda function - used to pick a random loaction in image
getit = lambda : (random.randrange(5, 85),random.randrange(5, 55))

# pick a random colors for points
colors = ["black","red","blue","green",(64, 107, 76),(0, 87, 128),(0, 3, 82)]

# fill_color = [120,145,130,89,58,50,75,86,98,176,]
# pick a random colors for lines
fill_color = [(64, 107, 76),(0, 87, 128),(0, 3, 82),(191, 0, 255),(72, 189, 0),(189, 107, 0),(189, 41, 0)]

# generate thr random captcha string
def random_string():
    # hash length
    N = 5
    s = string.ascii_uppercase + string.ascii_lowercase + string.digits
    # generate a random string of length 5
    random_string = ''.join(random.choices(s, k=N))
    return random_string

# generate the captcha image
def gen_captcha_img():
    # create a img object
    img = Image.new('RGB', (90, 60), color="white")
    draw = ImageDraw.Draw(img)

    # get the random string
    captcha_str = random_string()
    # get the text color
    text_colors = random.choice(colors)
    font_name = "fonts/SansitaSwashed-VariableFont_wght.ttf"
    font = ImageFont.truetype(font_name, 18)
    draw.text((20,20), captcha_str, fill=text_colors, font=font)

    # draw some random lines
    for i in range(5,random.randrange(6, 10)):
        draw.line((getit(), getit()), fill=random.choice(fill_color), width=random.randrange(1,3))

    # draw some random points
    for i in range(10,random.randrange(11, 20)):
        draw.point((getit(), getit(), getit(), getit(), getit(), getit(), getit(), getit(), getit(), getit()), fill=random.choice(colors))

    # save image in captcha_img directory
    img.save("captcha_img/"+ captcha_str +".png")

    return True

                          **WE MADE IT** 🥳🥳🥳

You will notice the code for i in range(5,random.randrange(6, 10)): which is used to randomise the line and points in each new image.
font used in this is from google fonts
https://fonts.google.com/specimen/Sansita+Swashed

Note: To validate the value(Backend) in the string you can store the string in the session or you can use your own way.
To generate more detailed image used some small images with different shapes in it and throws it randomly.

Build Url Shortener API In Django - Redis

Scrape IMDB movie rating and details using python

Build a News app using django - App like google news

Django News App Deployment On Heroku

Thanks you, have a great day ahead.🤪😎

Django News App Deployment On Heroku

Mageshwaran — Wed, 30 Sep 2020 16:46:32 +0000

Yesterday I write a post about Building a News App in django, you can find it here

https://dev.to/magesh236/news-app-in-django-2dcd

Now we are going to deploy this django app on heroku

This is the first time I'm trying to deploy django app on heroku

Go to the heroku and signup as a new user, after activating the account you will be redirected to dashboard, it will ask you to create app

after this will you get page like this, with the details how to deploy the project

click the open app, then it will prompt to the url where you will access the news app, for now you will see the default content.

Instructions to deploy apps in heroku

We will use heroku cli to deploy the project, install the heroku cli using this command, i'm using mac it may vary for you, depends which operating system you are using

   brew tap heroku/brew && brew install heroku

To check heroku is installed, use the heroku --version command

   heroku --version
   heroku/7.43.2 darwin-x64 node-v12.16.2

   heroku login

go to the root directory of the project in this case newsapp directory

   cd newsapp

and then initialize a git repository

   git init
   heroku git:remote -a APPNAME

same as git, commit the code

   git add .
   git commit -am "first commit"

Deploy your application

   git push heroku master

you may get error in this, You can now change your main deploy branch from "master" to "main" for both manual and automatic deploys.

create a new branch called main

   git checkout -b main

if you want you can delete the old branch

   git branch -D master

in this project deployment I'm not using the static files so I'm disabling it, for further reference check here Collectstatic error while deploying Django app to Heroku

   heroku config:set DISABLE_COLLECTSTATIC=1

Deploy News App Application

   git push heroku main

you can see the status in terminal or in Activity Feed

We made it.... Our app is live.

Don't forget to check this post Building a News App in django.

Thanks you, have a great day ahead.🤪😎

Build a News app using django

Mageshwaran — Tue, 29 Sep 2020 18:25:40 +0000

In this article we will build a simple news app using django, requests, bootstrap and with the help of the news feed from NewsApi

This app is an inspiration from Google News

Installing the dependency for the project

   pip install django
   pip install requests

Setting up the django project

for detailed reference about django visit the page
Now we can create our project

   django-admin startproject newsapp
   cd newsapp

To check the project is working, run this command

   python manage.py runserver

You will see a rocket page that means project creation is success, even though you will see warnings is the terminal like unapplied migrations, to remove that run this command.
This command will move the default migration to database like session, user table

  python manage.py migration

In this directory you can find manage.py file and newsapp directory, after this create a app call reader

   python manage.py startapp reader

now we need to add the app reader in setting.py
this will helps django to identity the record app

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'reader',
]

Don't forget to get the Api key from https://newsapi.org/ .
Then add the api key in setting.py

APIKEY = "XXXXXXXXXXXXXXXXXXX"

View and Urls
Where we will write the logic

create a urls.py inside reader directory, after that connect the reader/urls.py file to newsapp/urls.py,
this one will act as master all the urls

from django.contrib import admin
from django.urls import path
from django.conf.urls import include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include("reader.urls")),
]

NewsApi

This api will get the top news from all the sources.

url = https://newsapi.org/v2/top-headlines?country=us&apiKey=XXXXXXXXXXXXXXXXXXX

to get the data we are going to use python requests package to read data in python dictionaries

   r = requests.get(url=url) 
   data = r.json()

data will look like this

{
"status": "ok",
"totalResults": 38,
-"articles": [
-{
-"source": {
"id": null,
"name": "CNET"
},
"author": "Amanda Kooser",
"title": "Ancient underground lakes discovered on Mars - CNET",
"description": "Mars could be home to more liquid water than we originally thought.",
"url": "https://www.cnet.com/news/ancient-underground-lakes-found-on-mars/",
"urlToImage": "https://cnet2.cbsistatic.com/img/OJOEhYPBRrJmmwv03JD_1RF2wPI=/1200x630/2015/02/09/3a560b3c-d9eb-4f0c-af5c-9e9671844c93/mars-ice-cap.jpg",
"publishedAt": "2020-09-29T16:14:00Z",
"content": "This beautiful..."
}]}

let create a view to show the content


def home(request):
    page = request.GET.get('page', 1)
    search = request.GET.get('search', None)

    if search is None or search=="top":
        # get the top news
        url = "https://newsapi.org/v2/top-headlines?country={}&page={}&apiKey={}".format(
            "us",1,settings.APIKEY
        )
    else:
        # get the search query request
        url = "https://newsapi.org/v2/everything?q={}&sortBy={}&page={}&apiKey={}".format(
            search,"popularity",page,settings.APIKEY
        )
    r = requests.get(url=url)

    data = r.json()
    if data["status"] != "ok":
        return HttpResponse("<h1>Request Failed</h1>")
    data = data["articles"]
    context = {
        "success": True,
        "data": [],
        "search": search
    }
    # seprating the necessary data
    for i in data:
        context["data"].append({
            "title": i["title"],
            "description":  "" if i["description"] is None else i["description"],
            "url": i["url"],
            "image": temp_img if i["urlToImage"] is None else i["urlToImage"],
            "publishedat": i["publishedAt"]
        })
    # send the news feed to template in context
    return render(request, 'index.html', context=context)

the above home view will return the trending news or the requested query and then, pass it to the template, to render it in the frontend for that we need to created a template file inside /reader/templates/index.html

<!-- to get the next page content -->
<!-- in each ajax success request page value gets incremented -->
var page = 2;
<!-- we are using infinity scrolling in this page to get data -->
<!-- to prevent the unnecessary call -->
var window_scroll = true;
<!-- to get the news related to the page on -->
var search = "{{ search }}";

<!--  this will listen when scrolling -->
window.addEventListener('scroll', function(e) {
   console.log("Hi Dude...");
}

<!DOCTYPE html>
<html lang="en">
<head>
  <title>News App</title>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css">
  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.16.0/umd/popper.min.js"></script>
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js"></script>
</head>
<body>

    <div>
        {% include 'sidebar.html' %}
    </div>
    <div class="container" id="articles-container">
        {% for value in data %}
        <div class="card mb-3 box" style="max-width: 640px; margin:auto;">
            <div class="row">
                <div class="col-md-8">
                    <div class="card-body">
                        <h5 class="card-title"><a href="{{value.url}}" target="_blanck">{{value.title}}</a></h5>
                        <p class="card-text">{{value.description}}</p>
                        <p class="card-text"><small class="text-muted">{{value.publishedat}}</small></p>
                    </div>
                </div>

                <div class="col-md-4 img-box">
                    <img src="{{value.image}}" class="card-img" alt="..." height="100%">
                </div>
            </div>
        </div>
        {% endfor %}
    </div>
    <h1 id="loading">Loading....</h1>

</body>
<style>
#articles-container{
    margin-left: 250px;
}
.box{
    transition-property: box-shadow, transform;
    transition-duration: 1s;
}
.box:hover{
    /* box-shadow: 2px 2px 5px #8080807a; */
    box-shadow: 1px 2px 8px 0px #00bcffb0;
    transform: translateY(-9px);
}
.img-box{
    padding: 20px;
}
.img-box img{
    border-radius: 15px;
    object-fit: cover;
}
#loading{
    text-align: center;
    display: none;
}
</style>
<script>

$(document).ready(function() {
    console.log( "ready!" );
    el = document.getElementsByClassName("text-muted")
    for (i = 0; i < el.length; i++) {
        t = document.getElementsByClassName("text-muted")[i].innerText
        var d = new Date(t)
        document.getElementsByClassName("text-muted")[i].innerText = d.toDateString()
    }
});
var page = 2;
var window_scroll = true;
var search = "{{ search }}";

window.addEventListener('scroll', function(e) {
    // scroll check
    console.log("scroll check...")
    if(window_scroll){
        if((window.innerHeight + window.scrollY) >= (document.body.scrollHeight-200)){
            window_scroll = false;
            document.getElementById("loading").style.display = "block";
            $.ajax({
                url: '/next?page=' + page + '&search=' + search,
                dataType: 'json',
                success: function (data) {
                    if (data["success"]) {
                        articles = data["data"]
                        articles_html = ''
                        for (i = 0; i < articles.length; i++) {
                            articles_html += ' \
                            <div class="card mb-3 box" style="max-width: 640px; margin:auto;">\
                                <div class="row">\
                                    <div class="col-md-8">\
                                        <div class="card-body">\
                                            <h5 class="card-title"><a href="'+ articles[i]["url"] +'" target="_blanck">'+ articles[i]["title"] +'</a></h5>\
                                            <p class="card-text">'+ articles[i]["description"] +'</p>\
                                            <p class="card-text"><small class="text-muted">'+ articles[i]["publishedat"] +'</small></p>\
                                        </div>\
                                    </div>\
                                            \
                                    <div class="col-md-4 img-box">\
                                        <img src="'+ articles[i]["image"] +'" class="card-img" alt="..." height="100%">\
                                    </div>\
                                </div>\
                            </div>\
                            '
                        }
                        $("#articles-container").append(articles_html);
                        page += 1
                        window_scroll = true;
                        document.getElementById("loading").style.display = "none";
                    }
                    else {
                        console.log("Failed")
                        window_scroll = true;
                        document.getElementById("loading").style.display = "none";
                    }
                }
            });
        }
    }
})
</script>
</html>

after this create a another template file inside /reader/template/sidebar.html

django way of send the query params in Get method

<a href="{% url 'Home' %}?search=Business">Business</a>

<style>
    .sidebar{
        position: fixed;
        width: 200px;
        height: 100%;
        top: 0px;
        left: 0px;
    }
    .sidebar a{
        width: 250px;
        display: block;
        margin: 24px 0px;
        padding: 10px 5px;
        font-weight: bolder;
        font-size: large;
        text-decoration: none;
        background-color: #d9e4ffc7;
        border-top-right-radius: 30px;
        border-bottom-right-radius: 30px;
        box-shadow: 0px 0.5px 1px grey;
        text-align: center;
    }
</style>

<div class="sidebar">
    <a href="{% url 'Home' %}">Home</a>
    <a href="{% url 'Home' %}?search=top">Top News</a>

    <a href="{% url 'Home' %}?search=Business">Business</a>
    <a href="{% url 'Home' %}?search=Technology">Technology</a>
    <a href="{% url 'Home' %}?search=Entertainment">Entertainment</a>
    <a href="{% url 'Home' %}?search=Sports">Sports</a>
    <a href="{% url 'Home' %}?search=Science">Science</a>
    <a href="{% url 'Home' %}?search=Health">Health</a>
</div>

In the home view function it render the html page, it also require an additional api to update the news while the user scrolls the page, this will make the web more intractable without loading the entire page.
Use ajax request to get the json value from this loadcontent api

def loadcontent(request):
    try:
        page = request.GET.get('page', 1)
        search = request.GET.get('search', None)
        # url = "https://newsapi.org/v2/everything?q={}&sortBy={}&page={}&apiKey={}".format(
        #     "Technology","popularity",page,settings.APIKEY
        # )
        if search is None or search=="top":
            url = "https://newsapi.org/v2/top-headlines?country={}&page={}&apiKey={}".format(
                "us",page,settings.APIKEY
            )
        else:
            url = "https://newsapi.org/v2/everything?q={}&sortBy={}&page={}&apiKey={}".format(
                search,"popularity",page,settings.APIKEY
            )
        print("url:",url)
        r = requests.get(url=url)

        data = r.json()
        if data["status"] != "ok":
            return JsonResponse({"success":False})
        data = data["articles"]
        context = {
            "success": True,
            "data": [],
            "search": search
        }
        for i in data:
            context["data"].append({
                "title": i["title"],
                "description":  "" if i["description"] is None else i["description"],
                "url": i["url"],
                "image": temp_img if i["urlToImage"] is None else i["urlToImage"],
                "publishedat": i["publishedAt"]
            })

        return JsonResponse(context)
    except Exception as e:
        return JsonResponse({"success":False})

we made the view and templates, to access it we need to provide the urls

from django.urls import path
from reader import views

urlpatterns = [
    path('', views.home, name="Home"),
    path('next', views.loadcontent, name="Loadcontent"),
]

You can use this api only to get the Json data, send the query params accordingly

   path('next', views.loadcontent, name="Loadcontent"),

Now run the app
Finally we made it

This is the project structure

Thanks you, have a great day ahead.🤪😎

Build Url Shortener API In Django - Redis

Mageshwaran — Mon, 21 Sep 2020 18:43:39 +0000

We will build a simple Url Shortener using django and redis, data will be stored in SQLite and redis will be used as cache storage.

Setup a virtual environment for this project

Setup a virtual environment for this project, so it will not affect the projects in your machine.

Installing the dependency for the project

pip install django
pip install redis

above command will install the latest version of the packages

Setting up the django project

for detailed reference about django visit the page
Now we can create our project

django-admin startproject urlshortener
cd urlshortener

To check the app running successfully, run this it will open a rocket page(in terminal you can see unapplied migration, we will see this in migrate)

python manage.py runserver

The below command will create a app call short(where logic will be written) inside the urlshortener

python manage.py startapp short

now we have to add the app short to the settings.py, this will help django to identify the changes in models

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'short',
]

What is django model?

You can think model as a table in django, in SQL we use tables to store the data, right, In the same way model is being used. Table schema can be created in model then it can be migrated to the database.

by default django shipped with SQLite DB, you can change it to whatever data source you want to connect

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

models

Now add this code in short/models.py

from django.db import models

class URL(models.Model):
    full_url = models.TextField()
    hash = models.TextField(unique=True)

    class Meta:
        db_table = 'url_shortener'

in full_url we will hold the given url
in hash will store the unique key
db_table holds the name of the table

Run this

python manage.py makemigrations short

By running makemigrations, you’re telling Django that you’ve made some changes to your models, the above command only looks for the changes in short app, to load changes from all the app models just run this

 python manage.py makemigrations

this will creates migrations files inside the story/migrations directory, after that to move the change to SQLite db run this

python manage.py migrate

when you run this command this will apply the admin, session related migration to the database, all the warnings in the terminal will be gone

View and Urls

Where we will write the logic

create a urls.py inside short directory, after that include the file in urlshortener/urls.py, just like this

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include("short.urls")),
]

now lets write a simple HttpResponse view called as home, in short/views.py

from django.http import HttpResponse

def home(request):
    current_site = get_current_site(request)
    return HttpResponse("<h1>Url Shortener</h1>")

then link the view with a url

from django.urls import path
from short import views

urlpatterns = [
    path('', views.home, name="Home"),
]

open the home url (http://127.0.0.1:8000/) you will get the HttpResponse

project is working fine, now lets write some logic for the shortener
for detailed reference check this Designing a URL Shortening service like TinyURL

this will generate a random key value of length 7 with the combination of uppercase, lowercase and numbers. N can be adjusted as we want

import string, random 
# hash length
N = 7
s = string.ascii_uppercase + string.ascii_lowercase + string.digits
# generate a random string of length 7
url_id = ''.join(random.choices(s, k=N))

convert the logic to a function and it will create a new record in table for each new url short request

import string, random 
from short.models import URL

def shortit(long_url):
    # hash length
    N = 7
    s = string.ascii_uppercase + string.ascii_lowercase + string.digits
    # generate a random string of length 7
    url_id = ''.join(random.choices(s, k=N))
    # check the hash id is in
    if not URL.objects.filter(hash=url_id).exists():
        # create a new entry for new one
        create = URL.objects.create(full_url=long_url, hash=url_id)
        return url_id
    else:
        # if hash id already exists create a new hash
        shortit(url)

this view function will get the request and use the above function to convert it

from django.http import JsonResponse
from django.contrib.sites.shortcuts import get_current_site
from django.views.decorators.csrf import csrf_exempt

@csrf_exempt
def short_url(request):
    long_url = request.POST.get("url")
    # get the unique id for long url
    hash = shortit(long_url)
    # get the host url
    current_site = get_current_site(request)
    data = {
        "success": True,
        "id": hash,
        "link": "http://{}/{}".format(current_site, hash),
        "long_url": long_url
    }
    return JsonResponse(data)

link this view function to short/urls.py urlpatterns list

   path('shorten', views.short_url, name="ShortUrl"),

Now lets do the retrieving logic
redis - in-memory data structure store
we will use redis as a cache storage, I'm using redis in my local system

import redis
# host - running host
# port - the port in which its running
# db - by default 0 is the default db, you can change it
rds = redis.Redis(host='localhost', port=6379, db=0)

just run it using redis-server, once the redis server is stoped data will be cleared.

from django.shortcuts import redirect

def redirector(request,hash_id=None):
    # get the value from redis key, if value not in return None
    hash_code = rds.get(hash_id)
    if hash_code is not None:
        return redirect(hash_code.decode('ascii'))

    if URL.objects.filter(hash=hash_id).exists():
        url = URL.objects.get(hash=hash_id)
        # set the value in redis for faster access
        rds.set(hash_id,url.full_url)
        # redirect the page to the respective url
        return redirect(url.full_url)
    else:
        # if the give key not in redis and db
        return JsonResponse({"success":False})

Note: redis in python 3 always return binary string for that we used decode('ascii')

link the redirector view function in short/urls.py urlpatterns list

    path('<str:hash_id>/', views.redirector, name="Redirector"),

hash_id is a URL parameters used to get the value from url

DONE RIGHT, this is the project structure

Thanks you, have a great day ahead.🤪😎

Scrape IMDB movie rating and details using python

Mageshwaran — Wed, 16 Sep 2020 17:23:40 +0000

This article is only for educational purposes

IMDB providing its own api to get the movie details you can use that, this article is a web scraping example.

we will get the movie rating, number of rating, name and many more, for this we use BeautifulSoup and Requests packages

import requests
from bs4 import BeautifulSoup

boys_url = "https://www.imdb.com/title/tt1190634/"

r = requests.get(url=boys_url)
# create a BeautifulSoup object
soup = BeautifulSoup(r.text, 'html.parser')

to get the title of the html page

#page title
title = soup.find('title')
print(title.string)
data["title"] = title.string

soup.find this will look for the title tag and to get the value from it use title.string, this will return the string

Suppose you want to get all the div tags use find_all

soup.find_all('div')

to get a tag with class value

soup.find("div",{'class':'titleBar'})

full code to get the details in python dict, this function will get the html page in get method using requests then it is parsed as BeautifulSoup object and you can extract the data as you want

def getMovieDetails(url):
    data = {}
    r = requests.get(url=url)
    # Create a BeautifulSoup object
    soup = BeautifulSoup(r.text, 'html.parser')

    #page title
    title = soup.find('title')
    data["title"] = title.string

    # rating
    ratingValue = soup.find("span", {"itemprop" : "ratingValue"})
    data["ratingValue"] = ratingValue.string

    # no of rating given
    ratingCount = soup.find("span", {"itemprop" : "ratingCount"})
    data["ratingCount"] = ratingCount.string

    # name
    titleName = soup.find("div",{'class':'titleBar'}).find("h1")
    data["name"] = titleName.contents[0].replace(u'\xa0', u'')

    # additional details
    subtext = soup.find("div",{'class':'subtext'})
    data["subtext"] = ""
    for i in subtext.contents:
        data["subtext"] += i.string.strip()

    # summary
    summary_text = soup.find("div",{'class':'summary_text'})
    data["summary_text"] = summary_text.string.strip()

    credit_summary_item = soup.find_all("div",{'class':'credit_summary_item'})
    data["credits"] = {}
    for i in credit_summary_item:
        item = i.find("h4")
        names = i.find_all("a")
        data["credits"][item.string] = []
        for i in names:
            data["credits"][item.string].append({
                "link": i["href"],
                "name": i.string
            })
    return data

movies
tenet_url = "https://www.imdb.com/title/tt6723592/"
joker_url = "https://www.imdb.com/title/tt7286456/"
series
boys_url = "https://www.imdb.com/title/tt1190634/"
to get the movie details call this function

getMovieDetails(boys_url)

{'title': 'The Boys (TV Series 2019– ) - IMDb',
 'ratingValue': '8.7',
 'ratingCount': '173,133',
 'name': 'The Boys            ',
 'subtext': '18+|1h|Action,Comedy,Crime|TV Series (2019– )',
 'summary_text': 'A group of vigilantes sets out to take down corrupt superheroes who abuse their superpowers.',
 'credits': {'Creator:': [{'link': '/name/nm0471392/', 'name': 'Eric Kripke'}],
  'Stars:': [{'link': '/name/nm0881631/', 'name': 'Karl Urban'},
   {'link': '/name/nm4425051/', 'name': 'Jack Quaid'},
   {'link': '/name/nm1102278/', 'name': 'Antony Starr'},
   {'link': 'fullcredits/', 'name': 'See full cast & crew'}]}}

Additional content
to get all the crew and cast member with their role

import re
def getCrewData(url):
    crew_data = {
        "crew": []
    }
    r = requests.get(url=url)

    # Create a BeautifulSoup object
    soup = BeautifulSoup(r.text, 'html.parser')

    #page title
    title = soup.find('title')
    crew_data["title"] = title.string
    cast_list = soup.find("table", {"class" : "cast_list"})

    trows = cast_list.find_all('tr')

    for tr in trows:
        td = tr.find_all('td')
        if len(td)==4:
            row = [i.text for i in td]
            crew_data["crew"].append({
                "name":re.sub("[^a-zA-Z' ]+", '', row[1]).strip(),
                "character":re.sub("[^a-zA-Z' ]+", '', row[3]).strip()
            })
    return crew_data

series
boys_url = "https://www.imdb.com/title/tt1190634/fullcredits/"
Movie
tenet_url = "https://www.imdb.com/title/tt6723592/fullcredits/"

getCrewData(tenet_url)

Conclusion: Web scraping is always not stable when changes made to the web pages it will affects the code logic which is previously build. Not all website allow you to scrape their contents, some use javascript to render the page(build one vue, react js) in that case use selenium to get the page rendered. Use this only for educational use.

DEV Community: Mageshwaran

How did I build a web app in a day and deployed

Bubble sort in python

Tracing

Another method

Time Complexities

Django For Beginners - Life Cycle P2

Note 1

Note 2

Note 3

Note 4

Note 5

Note 6

Thanks you, have a great day ahead.😎

Django For Beginners - Project Setup P1

Virtualenv

Activate Virtual environment

Install Django

Creating a project

???

Open Graph Protocol Analyzer in Python

What is Open Graph protocol used for?

Compress image in python

8MB

275KB

Thanks you, have a great day ahead.😎

Build a Google Chrome Extension

Manifest

Base html

Random quotes logic

Pexels Image 🏜⛰🏔🏕

Time shower ⏱

Full JavaScript code

File structure

Add extension to chrome

Samples Images

Thanks you, have a great day ahead.🤪😎

Captcha image generator

Create new image

Draw text on image

Draw lines

Draw points

Generate the captcha string

Full logic to generate the captcha image

More articles

Thanks you, have a great day ahead.🤪😎

Django News App Deployment On Heroku

Deploy News App Application

Build a News app using django

Setting up the django project

NewsApi

Build Url Shortener API In Django - Redis

Setup a virtual environment for this project

Setting up the django project

What is django model?

models

View and Urls

Scrape IMDB movie rating and details using python

This article is only for educational purposes