DEV Community: Shreyas Mohite

Medium Blog Scraper 🤖

Shreyas Mohite — Sat, 10 Jun 2023 03:55:00 +0000

Have you ever wonder what it's like to save you favorite blogs in you local machine to read it later on in offline mode or share directly to your own personal blogging site .

Well I came up with some scripts which will help us to save our blogs in markdown format which will help us to publish these blogs on your personal blog posting website.

Let's get started 🚀

First of all we required some dependencies for our scripts
lets create one file and name it as requirements.txt

requirements.txt

beautifulsoup4==4.11.1
bs4==0.0.1
certifi==2022.9.24
charset-normalizer==2.1.1
deta==1.1.0
idna==3.4
markdownify==0.11.6
python-dotenv==0.21.0
requests==2.28.1
six==1.16.0
soupsieve==2.3.2.post1
urllib3==1.26.12

save all the dependencies in file. And now just install it all

pip install -r requirements.txt

Now create a new python file named it as scrap.py

scrap.py

import requests
from bs4 import BeautifulSoup
import re


def ScrapBlogs(urls):

    response = requests.get(str(urls)).text.encode('utf8').decode('ascii', 'ignore')
    soup = BeautifulSoup(response, 'html.parser')
    find_all_a = soup.find_all("a")
    urls=set()

    for x in find_all_a:
        if 'https' not in  x['href'] and \
            "/tag/" not in x['href'] and \
            "/?source" not in x['href'] and \
            "/plans?" not in x['href'] and \
            not re.match("/@\w*[a-zA-Z0-9]\?", x['href']):
            urls.add(x['href'])

    url_list = list(urls)
    return url_list[1:7]

Once it's done we need another file for our main script
named it as main.py

main.py

from scrap import ScrapBlogs
import re
import requests
from bs4 import BeautifulSoup
import os
import markdownify



urls = "https://medium.com/tag/node-js-api-development"
dir=urls.split("/")[-1]
urls = ScrapBlogs(urls)
if os.path.isdir(dir):pass
else:os.mkdir(dir)


def blogs_scrap():

    try:

        for url in urls:

            scrap_url = f"https://medium.com{url}"
            find_name=re.findall('\w+', url)
            name_list=[]
            for x in find_name:
                if x.isalpha():
                    name_list.append(x)

            file_name="_".join(name_list)

            response = requests.get(scrap_url).text.encode('utf8').decode('ascii', 'ignore')
            soup = BeautifulSoup(response, 'html.parser')
            find_section = soup.find("section")
            markdown_section=markdownify.markdownify("""{0}""".format(find_section), heading_style="ATX")
            print(scrap_url)
            if find_section is None:
                continue
            else:
                with open(f"{dir}/{file_name}.md","w") as f:
                    f.write(f"{scrap_url}")
                    f.write("\n")
                    f.write(markdown_section)

        return "Scrapped Successfully"


    except Exception as e:
        print(e)

blogs_scrap()

Make sure to get URL which contains tag in it.

For Example let's consider following url

Now let's scrap some blog's related to follwoing url

As we run this script we will be able to retrieve following url data. As you can see in image

But why only six md file if we have passed in ratio of [1:8]
let me explain it to you, We are only getting blogs which are public to user's.

we can also change our blog ratio in scrap.py

If you like this blog please do give a like.

Simple FriendList App In Pyhton Tkinter

Shreyas Mohite — Fri, 09 Jun 2023 17:42:16 +0000

Create an Tkintsimple Tkinter Friendlist application in python3
We do not required any requirement as we are going to use inbuilt all depenedencies.

Let's create our database for our UI
db.py

import sqlite3

def Friendlist():
    conn=sqlite3.connect("Friend")
    cur=conn.cursor()
    cur.execute("create table if not exists friendlist(id integer primary key,name text,facebook text,instagram text,whatsapp text)")
    conn.commit()
    conn.close()

def add_friendlist(name,facebook,instagram,whatsapp):
    conn=sqlite3.connect("Friend")
    cur=conn.cursor()
    cur.execute("insert into friendlist values(null,?,?,?,?)",(name,facebook,instagram,whatsapp))
    conn.commit()
    conn.close()

def view_friendlist():
    conn=sqlite3.connect("Friend")
    cur=conn.cursor()
    cur.execute("select * from friendlist")
    row=cur.fetchall()
    conn.close()
    return row


def delete_friendlist(id):
    conn=sqlite3.connect("Friend")
    cur=conn.cursor()
    cur.execute("delete from friendlist where id=?",(id,))
    conn.commit()
    conn.close()

def update_friendlist(id,name=" ",facebook=" ",whatsapp=" ",instagram=" "):
    conn=sqlite3.connect("Contact")
    cur=conn.cursor()
    cur.execute("select * from friendlist where name=?  or facebook=? or whatsapp=? or instagram=?",(name,facebook,whatsapp,instagram,id))
    conn.commit()
    conn.close()


Friendlist()

Now once we create db.py just make sure to run it.
As we run db.py we will able to retrive an file name
friends.db

Now create our GUI. Create a new file as app.py

app.py

""" 
This is desktop application for adding friends in database 
tkinter project 
"""

from tkinter import *
from tkinter import ttk
from db import *
import tkinter.messagebox



class FriendList:

    def __init__(self,root):
        self.root=root
        self.root.title("Friend list")
        self.root.geometry("400x450")
        # self.root.iconbitmap("./favicon.ico")
        self.root.resizable(0,0)


        #  variable for taking inputs from entry 
        sid=StringVar()
        name=StringVar()
        facebook=IntVar()
        instagram=IntVar()
        whatsapp=IntVar()


        # function for button

        #  for adding data in database
        def add():

            try:
                if facebook.get()==1:
                    face="present"            
                else:
                    face="not present"

                if instagram.get()==1:
                    insta="present"
                else:
                    insta="not present"

                if whatsapp.get()==1:
                    whats="present"  
                else:
                    whats="not present"

                if name.get()!="":                    
                    add_friendlist(name.get(),face,insta,whats)
                    show()
                else:
                    tkinter.messagebox.showerror('Error',"Please Add Your Friend Name")

            except Exception as e:
                print(e)


        #  showing data from database to contact_tree
        def show():
            contact_trees.delete(*contact_trees.get_children())
            for row in view_friendlist():
                contact_trees.insert('',END,values=row)





        #  deleting data from database
        def delete():
            delete_friendlist(sid.get())
            sid.set("")
            show()


        #  update database
        def update():
            try:
                if sid.get()!="":
                    delete_friendlist(sid.get())
                    if name.get()!="":
                        if facebook.get()==1:
                            face="present"            
                        else:
                            face="not present"

                        if instagram.get()==1:
                            insta="present"
                        else:
                            insta="not present"

                        if whatsapp.get()==1:
                            whats="present"  
                        else:
                            whats="not present"                    
                        add_friendlist(name.get(),face,insta,whats)

                        show()
                else:
                    tkinter.messagebox.showerror("Error","Please Select what you have to update and also write Id")
            except:
                pass


        #   This is for hovering button
        def on_enter1(e):
            but_add['background']="black"
            but_add['foreground']="cyan"

        def on_leave1(e):
            but_add['background']="SystemButtonFace"
            but_add['foreground']="SystemButtonText"

        def on_enter2(e):
            but_update['background']="black"
            but_update['foreground']="cyan"

        def on_leave2(e):
            but_update['background']="SystemButtonFace"
            but_update['foreground']="SystemButtonText"

        def on_enter3(e):
            but_delete['background']="black"
            but_delete['foreground']="cyan"

        def on_leave3(e):
            but_delete['background']="SystemButtonFace"
            but_delete['foreground']="SystemButtonText"


        def on_enter4(e):
            but_show['background']="black"
            but_show['foreground']="cyan"

        def on_leave4(e):
            but_show['background']="SystemButtonFace"
            but_show['foreground']="SystemButtonText"




#====================frame==============================#

        mainframe=Frame(self.root,width=400,height=450,relief="ridge",bd=3)
        mainframe.place(x=0,y=0)

        firstframe=Frame(mainframe,width=394,height=230,relief="ridge",bd=3)
        firstframe.place(x=0,y=0)

        secondframe=Frame(mainframe,width=394,height=214,relief="ridge",bd=3)
        secondframe.place(x=0,y=230)


#======================firstframe========================#


        lab_name=Label(firstframe,text="Enter Name",font=('times new roman',15))
        lab_name.place(x=150,y=10)

        identry=Entry(firstframe,width=3,font=('times new roman',14),relief="ridge",bd=3,textvariable=sid)
        identry.place(x=27,y=50)

        ent_name=Entry(firstframe,width=25,font=('times new roman',15),relief='ridge',bd=3,justify="center",textvariable=name)
        ent_name.place(x=65,y=50)

        but_show=Button(firstframe,text="List",width=4,font=('times new roman',14),cursor="hand2",command=show)
        but_show.place(x=325,y=45)
        but_show.bind("<Enter>",on_enter4)
        but_show.bind("<Leave>",on_leave4)

        facebook_cbutton=Checkbutton(firstframe,text="facebook",variable=facebook,font=('times new roman',15),onvalue=1,offvalue=0)
        facebook_cbutton.place(x=15,y=110)

        instagram_cbutton=Checkbutton(firstframe,text="Instagram",variable=instagram,font=('times new roman',15),onvalue=1,offvalue=0)
        instagram_cbutton.place(x=135,y=110)

        whatsapp_cbutton=Checkbutton(firstframe,text="Whatsapp",variable=whatsapp,font=('times new roman',15),onvalue=1,offvalue=0)
        whatsapp_cbutton.place(x=255,y=110)

        but_add=Button(firstframe,text="Add",width=10,font=('times new roman',14),cursor="hand2",command=add)
        but_add.place(x=10,y=170)
        but_add.bind("<Enter>",on_enter1)
        but_add.bind("<Leave>",on_leave1)

        but_update=Button(firstframe,text="Update",width=10,font=('times new roman',14),cursor="hand2",command=update)
        but_update.place(x=135,y=170)
        but_update.bind("<Enter>",on_enter2)
        but_update.bind("<Leave>",on_leave2)

        but_delete=Button(firstframe,text="Delete",width=10,font=('times new roman',14),cursor="hand2",command=delete)
        but_delete.place(x=260,y=170)
        but_delete.bind("<Enter>",on_enter3)
        but_delete.bind("<Leave>",on_leave3)

#===================================secondframe==================================================#

        def  game(event):
            crow=contact_trees.focus()
            contents=contact_trees.item(crow)
            row=contents['values']
            sid.set(row[0])
            name.set(row[1])

        scol=Scrollbar(secondframe,orient="vertical")
        scol.place(relx=1, rely=0, relheight=1, anchor='ne')

        contact_trees=ttk.Treeview(secondframe,columns=("ID","Name","Facebook","Instagram","Whatsapp"),height=9,yscrollcommand=scol.set)
        contact_trees.heading("ID",text="ID")
        contact_trees.heading("Name",text="Name")
        contact_trees.heading("Facebook",text="Facebook")
        contact_trees.heading("Instagram",text="Instagram")
        contact_trees.heading("Whatsapp",text="Whatsapp")
        contact_trees['show']="headings"
        contact_trees.column("ID",width=30,minwidth=10)
        contact_trees.column("Name",width=110,minwidth=40)
        contact_trees.column("Facebook",width=76,minwidth=40)
        contact_trees.column("Instagram",width=76,minwidth=40)
        contact_trees.column("Whatsapp",width=76,minwidth=40)
        contact_trees.place(x=0,y=0)
        contact_trees.bind('<ButtonRelease-1>',game)



if __name__=="__main__":
    root=Tk()
    FriendList(root)
    root.mainloop()

Now just run application

python app.py

Once it run you will get an GUI given below.
Thank you !

CI/CD Github Actions For Deta.sh

Shreyas Mohite — Thu, 06 Oct 2022 09:02:35 +0000

If you don’t know about deta.sh you can find out more about deta.sh on my blog

Once you deploy your app on deta.sh. Open deta.sh and click on settings and create a new access token

Once you create a new token go to your repository and click on settings

In settings you can see secrets where you what to add your app secrets, an secrets can be any env credentials.

Add DETA_TOKEN and its value which you got from deta.sh access token

Once you add your secrets. Click on Actions of your github repo and click on set up a workflow yourself

Once you create main.yml file in your repository replace the github actions code with below code.

Now you will able to see all your deployments in github it’s self.

Top 10 Backend Framework

Shreyas Mohite — Sat, 01 Oct 2022 13:04:14 +0000

Frontend development and backend development are the two main kinds of development. Building a web application's server-side falls under the purview of a backend developer.

The backend is in charge of the internal operations while the frontend communicates with the user. A frontend might or might not function without a backend, but in order to build a web application that is fully functioning, a frontend and backend must be properly coupled.

A frontend developer's job is distinct from a backend developer's. A backend developer is more concerned with servers, databases, and how the two ends are connected and communicate with one another, whereas a frontend developer is more concerned with how the programme will look.

A backend developer needs to possess a wide range of abilities. Backend framework knowledge is one of them. There are many different backend frameworks; this article will focus on the most popular ones currently in use.

The best backend frameworks according to programming language and tech stack are included in the list. This means you don't have to master them all; instead, pick the one that best suits the programming language and technology stack you want to use.

10 Best Backend Framework for Web Development :

Here is a list of the top 10 backend frameworks for web development that programmers can learn without wasting any more of your time. While there are many options for a certain technology stack or programming language, such as MicroNaut, Quarkus, Vert.X, and other frameworks for Java developers, I have only selected the finest frameworks, such as Spring Boot, based on their level of popularity, maturity, and features.

1. Spring Framework + Spring boot for Java Developers :

Programming language experts consider Java to be timeless. It is frequently used in backend development. One of the best Java frameworks is the Spring Framework.

Since its debut in 2002, it has become one of the most widely used backend frameworks for building high-quality standalone and spring-based applications.

Because Spring Boot does not have the same configuration overheads as Spring Framework, it is simple to utilise. When you utilise Spring Framework products like Spring Boot, Spring Cloud, Spring Security, etc., you may gain a lot of important functions like caching, transaction management, configuration management, monitoring, security etch, and more.

2. Django for Python Developers :

The most widely used Python framework for web development is Django. Using the Don’t Repeat Yourself (DRY) philosophy as its foundation, Django concentrates on code reuse, accelerating development. Additionally, it is a very safe foundation.

Django is a very user-friendly and simple to learn Python framework. It is also one of the most well-known full-stack frameworks and offers a lot of functionality right out of the box, including support for REST APIs.

3. Express.js for JavaScript Developers :

The most used programming language worldwide is JavaScript. With the introduction of Node.js, JavaScript’s acceptance in the backend development community grew quickly, and over the past ten years, Node.js has established itself as one of the leading names.

Express was made available in 2010 for node developers for this reason. It is a straightforward Node.js framework used to create incredibly adaptable applications.

Expressjs can be a fantastic option for programmers given that JavaScript is the most well-liked web development language and is likely the only language that offers full-stack development from the front to the backend to mobile learning.

4. ASP.NET core for .NET developers :

The replacement for ASP.NET is ASP.NET core. It is a free, open-source framework that works with any platform and is used to create online applications for the.NET framework. It is regarded as one of the top backend frameworks in terms of performance.

It simply combines the ASP.NET MVC and ASP.NET Web API versions into one programming module to increase its capability.

5. Laravel for PHP Programmers :

One of the most popular languages for backend development is PHP. An MVC-based PHP framework called Laravel was created. It is a user-friendly framework with a huge library, excellent API support, and a beginner-friendly UI.

Laravel makes it easier to design backends for contemporary, secure web apps. This is particularly true if you are working on apps that are not simple.

6. Ruby on Rails for Ruby Programmers :

Rails, also referred to as Ruby on Rails, is a server-side framework built on the MVC paradigm. It is a simple framework that is simple to comprehend and pick up.

Although its benefits and drawbacks are still up for dispute, backend developers all over the world adore it for its user-friendliness.

7. Fiber Framework for Golang Developers :

A web development framework for Golang programmers is called Fiber. On top of Fasthttp, the quickest HTTP engine for Go, it was constructed. It is created with performance and zero memory allocation in mind to simplify things for quick development.

A slender but incredibly potent foundation, fibre. Due to Martini’s remarkable leanness, integrating outside libraries is simple. More scalability and fewer overhead issues are provided by the framework.

Similar to Express.js, it offers excellent routing, static file serving, REST API development, support for flexible middleware, a template engine, and low memory footprint.

8. CakePHP Framework for PHP Developers :

Another PHP framework, CakePHP, is built on the ideas of Ruby on Rails, a well-liked backend framework.

In addition to model-view-controller, front controller, active record, data mapping, and convention over configuration, it also adheres to the MVC design.

9. Flask Framework for Python Developers :

You can also use Flask as a Python backend development framework. Due to the absence of the use of particular tools and libraries, it is referred to as a microframework. Additionally, there is no form validation, database abstraction layer, or reliance on external sources.

It is an efficient, straightforward, and simple web framework. Flask is a simple framework to learn and comprehend since it is a lightweight framework, or micro-framework. Additionally, it is incredibly user-friendly because it is a Python framework.

10. Play Framework for Scala Developers :

PLAY is one of the most potent backend web frameworks and is written in Scala and Java. It has many capabilities, including hot code reloading, displaying programme errors, and a focus on enhancing developer profitability. It is built on the MVC architecture.

Additionally, it is non-blocking and RESTful.

That concludes our discussion of the top backend frameworks for programmers in 2022. As I previously stated, studying backend development is worthwhile if you want to work as a full-stack engineer or backend engineer. The backend frameworks are now necessary for backend development.

Many different backend frameworks exist, some of which are well-known. We covered the top backend frameworks in this article. All of these frameworks are in high demand on the job market and very well-liked in the backend community.

These frameworks are mostly created using well-known programming languages like Java, Python, and JavaScript. If you’re just starting out, use lightweight frameworks like Flask or beginner-friendly frameworks like Ruby on Rails.