DEV Community: Trideep Barik

Why do I love WebDev <3 ?

Trideep Barik — Sat, 25 Jul 2020 19:37:23 +0000

Hello Everyone,

This post is for all the beginners or the newbies, who decided to start their journey in the IT world and still unaware or confused about various fields. In my point of view, you should explore every field and choose which one is your "Soul-FieldofInterest".
And for the ones who are already at a great level in WebDev, you might find this kind of a Nostalgic post.

So here I am, discussing about my " Soul-FieldofInterest " i.e. Web Development . I will try not use any technical word and for sure no code, for better understanding. And this is all about my point of view, some of you might find it conflicting.

The Beginning

Everyday you wake up to notifications from social media like Instagram, from shopping sites like Amazon, text from your crush (never happens I know) or (in this lock down) from online classes .

Ever wondered how all these things started? One day one person built a WEBSITE ( No, not your crush. Trust me you deserve better :p ). One day a person sitting in his hostel room decided to build a website to connect to friends; One day a person decided to make a website to sell books and now they are Facebook and Amazon.

Well, for me, all of these started with a startup idea, in my 1st year of my Btech. I was never curious about Web Development. But for that idea, I thought,"Let's give it a shot". And now here I am writing a post about my "Soul-FieldofInterest".

You all might have encountered with those registration information forms for joining a social media or those survey forms you get online or (the least expected one) the Google docs for class tests. Let's use that. Let me explain how things are interesting with WebDev; not just after you become a web developer but also while you are learning to be a Web Developer.

Note: The following things have alternatives. These are just to explain things.

HTML

Some of you might get a question,"Using which language we can code HTML?".

"HyperText Markup Language is the 
standard markup language for documents
designed to be displayed in a web 
browser."- Wikipedia

Ya, bla bla bla , you will learn about those terms someday.
So the answer to the previous question is "No, you don't code HTML. You build websites using HTML."

Well, whatever you see in the internet is HTML( more like hundreds of lines of HTML). If we compare this with our body, then HTML is like our skeletal system.

Let's go to the form.

Source : Codepen

See that's how your browser treats HTML file.
You get fields to fillup and then submit.
But that's it? Real world websites doesn't look like that? It's uglier than me .

CSS

Here comes CSS into play. He is the Genie for HTML aladdin.

"Cascading Style Sheets is a style 
sheet language used for describing 
the presentation of a document written 
in a markup language like HTML."- Wikipedia

Again, a bla bla bla
In simple words, that makes our HTML file..

Source : Codepen

...beautiful? or more than that?
Well, it's like a makeup kit. A kit with few lines of code.
Trust me that's the same HTML code with CSS. This might look like a huge change and a tough thing but it is hardly 15-20 lines of code. CSS is like muscles and organs for HTML.

That's why couples say,"You are the CSS to my HTML".

And now with CSS, HTML looks like me( my mom's version of me )

Cool, now we have the website. We are done here. That's all about Webdev. But, wait a minute, how will you know the data given in the form is valid? Someone might give an incorrect email address or some invalid phone number? How will you tackle that?

JAVASCRIPT

In the present world, "look sells" .
You and I as a customer will always look for something which is interactive.
This is where Javascript comes into play.

"JavaScript is most 
well-known as the scripting 
language for Web pages, many 
non-browser environments also 
use it". - MDN

Javascript helps us to validate data, make the interface user interactive, build a web app, build servers etc. etc. All those interactive animations in the websites are a gift from JS. It is like a soul for your body. I believe that's the part you find webdev a bit rough but once it's done you will love it.

Source:Freefrontend

This is where the Front-end work ends and Back-end work joins us.

Backend

(I chose PHP for the backend works, you have options )

Using the form, you collected the data but you need to process it and store it somewhere. This is the use of Backend.
It helps you reuse data collected. It helps you make a website dynamic(means to show different results under different situations). Ever wondered, do Facebook has different code for different users? There are millions of users and millions of code?
No, It uses one code which changes according to the user.

Backend includes a lot and lot of things like storing and using data from Database(like Mysql or Mongodb), creating services, dealing with server,"not just about making websites or web apps" and the list goes on. After this, you go deeper and explore frameworks, libraries etc. There is more but I won't like to extend this post.

But at this point, I realised "WebDev it is. 
That's fun.". <3 :)

You see the fun? where you learn things step by step. Every step makes you see and feel,that you have improved. Just like a mini-success before a huge success. Adding a CSS to HTML feels like that childhood memory of try the Coloring book for the first time.

As I said earlier, try every field. I tried most of them but I never found them so connected.

Webdev is all about you and your creativity. It is all about how and what you think. This is the thing that can integrate with any other field. You can practice it in your free time and you won't feel bored. You can do freelancing and earn. That's the thing that don't need a degree. It lets you get into a huge community. And rest things are surprises or suspense things, which I don't wan't to spoil for you.

What else do you need as a developer?

I hope you get an idea about WebDevelopement. If you are a WebDeveloper, I would love to know why do you love this so much?

Download a whole YouTube Playlist at one go

Trideep Barik — Sun, 24 May 2020 08:49:16 +0000

Note:This post is just for educational purpose to show the use of Webscraping.

Have you ever been in a situation like, you are having you final exams in a week or so and you haven't attended a single lecture(I mean mindfully, of course)? Then you ask YouTube for help (happens in my case every time) and what you get is a huge(even larger than node_modules) playlist to watch and limited data/data-speed at your place.

Or like, you wanted to learn a new skill/language/framework from YouTube and you get a good "playlist" but limited storage space in your phone.

Now, there are websites/applications to download YouTube videos.
OOPS, either they download one video at a time or if they download a complete playlist at once, they are paid ones. You can download the playlist videos one-by-one(until that's a huge one). Exams are important, you can pay for them and if it is about learning, let it take some more time, we have entire life?

Wait a minute, you are a Python developer. Why pay for a service you can build with a few lines of code?
This post will be about a simple project "YoPlaDo-YouTube Playlist Downloader" built using Python. We will be writing a program that takes the YouTube playlist link and web-scraps all the video links using Selenium and download the videos using YouTube-dl.

Web Scraping

Have you ever searched and downloaded images? Or have ever Ctrl+ C and Ctrl+ V (If you know you know)? Or submitted an assignment with the solutions you get online? Basically, this is what scraping is.

Collecting data or to be more specific Extracting data from a website is Web-Scraping. Instead doing things manually, you can automate things. That's what a Web Scrapper does. You give it a list of things to extract and then it goes for shopping(scraping) from a website.
For example, you need an image, it searches for img tags.
These days Web-Scraping is being used in every fields staring from Digital Marketing to DataScience or AI.

So, various languages provide various libraries,frameworks and tools to make a Web Scraper or a Web Crawler. Python uses Selenium ,Beautiful Soup, Scrapy and a few more.

In this we will be creating a basic project with Selenium for a dynamic website.

YoPlaDo

Selenium

"Selenium is a portable framework for testing web applications." - Wikipedia

Primarily it is for automating web applications for testing purposes, but is certainly not limited to just that.
Boring web-based administration tasks can (and should) also be automated as well.
I will be explaining things in a simpler way. For detailed information, you can go through the Docs.

Youtube-dl

"Program to download videos from YouTube.com and other video sites" - Pypi

For detailed information, you can go through the Docs.

Let's get started,

We will be extracting links from a playlist(like https://www.youtube.com/playlist?list=) and download each video automatically with the program. This is just a basic project overview for beginners. You can go for documentation and make improvements.

- Import the libraries

If you want to follow-up with the post, you can make use of "https://www.youtube.com/playlist?list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-" as the input link as this is used in the below example. It will be easier to understand. Also make sure the playlist you use, should have videos which are downloadable and without any deleted videos. This problem can be solved using Exception block but in order to keep this post simple for beginners, the block hasn't been added.

Before importing make sure , you have downloaded the required libraries. You can use,

pip install selenium
pip install youtube-dl


from selenium import webdriver
import time 
import youtube_dl 
import os

- Initiate Chrome web-driver with the playlist URL


url = input("Enter the Youtube Playlist URL : ")
driver = webdriver.Chrome() 
driver.get(url)
time.sleep(5)

Used to initiate Chrome browser with input URL.
time.sleep(5) provides a time gap to start the driver.

- Scraping all the videos links from the Playlist


playlist=[]
videos=driver.find_elements_by_class_name('style-scope ytd-playlist-video-renderer')

First we create an empty list "playlist" to store all the links to be extracted.

Then webscraping comes into play.
For simpler understanding, the line
driver.find_elements_by_class_name('style-scope ytd-playlist-video-renderer') is used to extract all the contents of source file that comes under specified class division.

- Scraping(part 2)


for video in videos:
    link=video.find_element_by_xpath('.//*[@id="content"]/a').get_attribute("href")
    end=link.find("&")
    link=link[:end]
    playlist.append(link)
"""
For example, a playlist with 6 videos

Enter youtube playlist link : https://www.youtube.com/playlist?list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-
['https://www.youtube.com/watch?v=iyL9-EE3ngk&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=1', 'https://www.youtube.com/watch?v=G7E8YrOiYrQ&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=2', 'https://www.youtube.com/watch?v=79D4Y1cUK7I&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=3', 'https://www.youtube.com/watch?v=MUe0FPx8kSE&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=4', 'https://www.youtube.com/watch?v=UkpmjbHYV0Y&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=5', 'https://www.youtube.com/watch?v=WTOFLmB9ge0&list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-&index=6']


"""

A new term? Xpath?(superhero path?)

"XPath is a technique in Selenium to navigate through the HTML structure of a page."
For simpler understanding that is just a path to find specific tags.

While the loop inspects all the elements in "videos" object

video.find_element_by_xpath('.//*[@id="content"]/a').get_attribute("href")
finds all the anchor tags or links present under the division whose id="content" and extract or scrape their href.

The rest part of the code is for validation done to find only the link to the videos of the selected playlist by striping the playlist id and index number from the link.


"""
After processing it looks like:


Enter youtube playlist link : https://www.youtube.com/playlist?list=PLGzz7pyosmlJfx9ivigemSouoZR9uLT2-
['https://www.youtube.com/watch?v=iyL9-EE3ngk', 'https://www.youtube.com/watch?v=G7E8YrOiYrQ', 'https://www.youtube.com/watch?v=79D4Y1cUK7I', 'https://www.youtube.com/watch?v=MUe0FPx8kSE', 'https://www.youtube.com/watch?v=UkpmjbHYV0Y', 'https://www.youtube.com/watch?v=WTOFLmB9ge0']
"""

Wondering why those specific classes?

Link of videos are under these sections.

- Downloading Videos


os.chdir('C:/Users/Trideep/Downloads') 

for link in playlist:
        with youtube_dl.YoutubeDL(ydl_opts) as ydl:
            ydl.download([link])
driver.close()

os.chdir('C:/Users/Trideep/Downloads')
Used to change the download location to the "Downloads".

with youtube_dl.YoutubeDL(ydl_opts) as ydl:
ydl.download([link])

Here comes the Youtube-dl. Looping through the "playlist" list, each link is processed and downloaded using YoutubeDL.

"ydl.download('Url to directory')" processes the link and downloads it to the mentioned directory.
You can further add video specifications or type of video using other attributes of youtube-dl .

driver.close() used to close the driver.

And with merely 30 lines of code, you saved a few dollars and gifted yourself a good project.
Well, for proper execution you can add your own exception blocks and logic. I would suggest you to go through the documentation.

For complete code, you can visit:

https://github.com/Dstri26/YoPlaDo-Youtube-Playlist-Downloader/

Happy Scraping! Happy Coding.

DEV Community: Trideep Barik

Why do I love WebDev <3 ?

The Beginning

HTML

Source : Codepen

CSS

Source : Codepen

JAVASCRIPT

Source:Freefrontend

Backend

(I chose PHP for the backend works, you have options )

Download a whole YouTube Playlist at one go

Web Scraping

YoPlaDo

Selenium

Youtube-dl

Let's get started,

- Import the libraries

- Initiate Chrome web-driver with the playlist URL

- Scraping all the videos links from the Playlist

- Scraping(part 2)

- Downloading Videos

That's my first technical blog. Do correct me if I am wrong. <3 <3 <3