DEV Community: Mehul Mohan

Gift-a-thon by codedamn

Mehul Mohan — Thu, 23 Jun 2022 07:21:29 +0000

Every month, we at codedamn come up with new events to keep your brain cells engaged over the weekend. What’s unique about this one?

What is Gift-a-thon?

Gift-a-thon is a 2-day long initiative of learning and sharing. On the 24th of June, we’ll be rolling out a feature that will allow you to get licenses to Learning Paths and Pro Subscriptions for your buddies at discounted prices.

Only for the launch weekend commencing soon, you’ll be able to grab these vouchers at heavily slashed rates and we are low-key excited about it!

Why should you get licenses for your friends?

Because mugs are outdated! We are tired of watching people buy expensive diaries & pens their friends have enough of. We are also tired of listening to people complain about not finding the right opportunities, resources, or projects to work on.

We came up with the idea of letting you share the gift of knowledge! Simply hand over a redemption code to our platform to anyone you fancy, which can be activated at any time in the future and your friends are set!

Plus, you can actively collaborate with your friends on codedamn Playgrounds while you learn! Read more about Collaboration on Playgrounds here.

How do you participate?

If you’re interested in being a part of Gift-a-thon, simply head over to this page and fill out your details so we can send you a notification when we go live.

We have limited licenses available at the slashed rates so make sure you’re quick to grab your codes.

Once the clock strikes midnight on the 24th of June, head over to the same link and add as many licenses to your cart as you’d need. Keep your job-hungry friends in mind when you do so in order to get a higher discount and that’s pretty much it. For every license that you add to your cart, you’ll unlock an additional discount.

How does redemption work?

If you grab 10 licenses, you will get your hands on 10 unique codes. These codes can be redeemed on the same link used to get the codes.

Note that every code can be used only once, and can be used only on a valid codedamn account.

Can you gift a code to yourself?

You can, however, codes cannot be used to extend your existing subscription. If you’re already a Pro Subscriber (and we are really glad you are), you will not be able to redeem the voucher code for your account.

With that being said, if you already have access to a specific Learning Path, you can always get vouchers for different domains.

Ending Notes

We are super excited to have you invite your friends to our platform. We are sure they’ll love codedamn just as much as you do! In case of any doubts or queries, as usual, shoot an email to support@codedamn.com and we’d be more than happy to help you out!

While you wait for Gift-a-thon to unfold, learn to code like it’s 2030 with us!

How much JavaScript to learn before learning React?

Mehul Mohan — Mon, 28 Jun 2021 19:09:30 +0000

How much JS should I know before I start learning React - this is a very common question I get asked all the time. In this blog post, I will breakdown a few myths and also list some very important topics/things you should know before you start React, or any other similar library/framework (Angular, Vue, etc.)

You don't need to learn all of JavaScript

Stress on the keyword "all". A very important thing is that if your aim is to become, say a frontend developer (a complete guide on it here), you don't want to spend month after month on only JavaScript preparation.

The ideal mix that worked for me is to learn "just enough" to move on to React. This absolutely does not mean that you have learned JavaScript. This only means that you don't need to. JavaScript, like any programming language, can be vast, especially with all the tooling and ecosystem around it. Therefore, it is super important to get the core, fundamental understanding of JavaScript ready before you learn anything that involves JavaScript.

Should I learn React directly without JavaScript?

Different people might give different answers here. Don't get tricked by only one opinion (applies to this blog post too). In my opinion, the fastest way to learn and absorb React is to go through JavaScript, and not directly jumping on React.

The reason for this is context switching. When you learn React, you will parallelly learn a lot of JavaScript too, all the time. If you come with a solid understanding of JavaScript to React, you will spend a lot of time in React methodologies and how to work inside a React codebase, best practices and libraries to choose. This is only possible if you aren't asking every minor detail about JavaScript syntax/code. If you do too much context switching from React to JavaScript learning, you'll eventually make less progress on both ends.

However this also does not mean that you should over optimize your JavaScript learnings for React. You don't need to, for example, learn about Node.js or streams in Node.js to become a good React developer. You also don't need to learn about service workers and ES6 proxies for becoming a good React developer. A typical coding course might confuse you that you need those parts, but trust me, initially you don't. You can always "context switch" and come back to learn these things if needed.

Okay, so if this is not required, then what is truly required before you move on to React?

Things to learn before learning React

Here's a compiled list of topics I believe you should know. I have divided these topics into 3 categories: Diamond, Gold, Silver.

Diamond topics - absolutely mandatory and you should have a crystal clear solid understanding. They will haunt you back if you try to skip these.

Gold - recommended to learn, but you can do some context switching and learn them on the "runtime"

Silver - these might be advanced topics that are good to know under the hood things, but usually not needed in initial phase.

Diamond topics are super important. I often think of this quote when people ask me can we start directly with React and learn JS later:

It isn't the mountains ahead to climb that wear you out; it's the pebble in your shoe.- Muhammad Ali

Diamond pre-requisites for React

The points below are few things you can almost not afford to miss. If you have a weak understanding on these topics, they'll become bigger bottlenecks in your journey as and when you move ahead.

Using a computer, a browser.
Language syntax.
Programming fundamentals - variables, functions, loops, conditionals, how a program runs.
Working with a text editor like VSCode.
Understanding of HTML, CSS and how it connects to JavaScript with DOM API.
Basic operations with JavaScript - network requests, alerts, DOM changes, event listeners
Asynchronous behavior of JavaScript - Callbacks, promises, async/await.
Data structures and data types in JavaScript and how they differ from each other - string, boolean, undefined, null, objects, etc.
Ability to write simple standalone JS programs and functions (like calculating prime numbers, making 10 HTTP requests sequentially/parallelly using loops/promises, etc.)
Ability to google your problems, always.

If you have any additions to it, feel free to leave them in a comment below.

Gold pre-requisites for React

These things are not strictly required, but are recommended in order to create an even stronger steel framework of understanding.

ES6+ syntax - how ES6 module system works, why const and let were introduced, variable scoping concepts, function hoisting.
Linux command line - How to work with bash commands, creating folders, running scripts, deleting files, navigating around filesystem.
Git - How to deploy your projects to remote websites like GitHub using git from your command line.
Package manager understanding - how npm or yarn work, why are they even required, semantic versioning, lockfiles.
High level overview of additional tooling - webpack, babel, prettier, vscode.
Hosting providers you can use to deploy your static sites easily - netlify, vercel, github pages, heroku, etc.
Browser devtools like chrome devtools for debugging your code/errors or inspecting network requests.
Understanding about JSON/APIs/HTTP/REST practices in a little depth.

Silver pre-requisites for React

These things are not required initially, but can really help you in long run becoming a great developer overall with your stack. In a nutshell, this section can come "after" learning React too.

TypeScript with React - how to strongly type your code to catch more errors at development time
A clear structure of organizing your files/folders in a project.
CI/CD knowledge, for example, with GitHub actions for deploying new changes automatically.
Code testing to ship code confidently to production.
Stronger grip on best libraries in the ecosystem for their specific use case ( react-query? apollo-graphql? tailwindcss? next?)

codedamn's frontend learning path

The sad state of web development right now is that there's not a lot that needs to be learnt to become a great developer, but the content noise levels are all time high. The secret is to just a few things in depth.

If you're looking to learn frontend web development, my personal recommendation is to research for any one good complete source where you can learn from start to end, instead of being half pregnant on nine websites teaching you different things differently. We also provide a full blown full-stack learning path roadmap, that covers frontend as close to 50% of the curriculum. Feel free to try it.

Conclusion

This blog post should have given you a clarity on how much JavaScript you should be aware of before starting learning React. How to learn React properly? That's for another blog post. If the article helped you, feel free to subscribe to the developer newsletter below to receive weekly updates on more high quality articles here.

The complete guide to becoming a frontend web developer in 2021

Mehul Mohan — Sat, 26 Jun 2021 14:18:17 +0000

Today, it is clear that web development is a great skill to have. In this blog post, let us look at what do you need to do exactly to become a frontend web developer in 2021 and beyond.

The problem

The problem with the web right now is the information overload and noise. jQuery, Angular, Ember, Vue, React, Next.js, Nuxt.js, TypeScript, Sass, Webpack, NPM, Yarn, ESbuild, snowpack, ...! There are so many things, so many tools, so much you can learn - that often it becomes discouraging to even start.

Then there's the point of what is relevant today, what will be relevant 3 years from now, etc. Let us clarify a few things in this blog:

You don't need to learn everything

A common misconception is that you need to learn everything to be a great developer. This is completely false.

You need to own a "complete" tech stack. A complete tech stack is a stack using which you can create almost any project you'd like.
You need to be aware of most, if not all popular tools in the ecosystem. Note, this does not mean you need to know how to work with them. But you should know they exist.

You need to bet on a stack

Angular vs React vs Vue vs Svelte vs xyz. This discussion will never end. The truth is, every single popular framework/library out there is popular because people use it and they're good at what they do.

You can try spending time with different technologies if you'd like. But my personal recommendation is not to switch major libraries/frameworks every month or two.

Sure, you can switch from React to Angular watching a 4 hour course. However remember that there's a certain mental model you need to develop when you're working with a technology, and that could take weeks if not months to get comfortable with.

Do not switch to the hottest technology in the market, stick to a tech stack you work with.

The framework to learn frontend web development in 2021

Alright. Now the big question. What do you actually learn from all the noise in the world. Let's split the question in two parts - (What and How) to learn frontend web development. Both are equally important questions as you can save or waste a lot of time depending on the decision you take. Let's start.

What to learn for frontend web development?

This is a no bs list of things you should learn for frontend web development. Please note that this would be opinionated because my experience is heavily with these technologies (the platform you're reading this on, codedamn, has been built with all of these technologies)

Internet fundamentals

This might come as a surprise to many. Most sites will recommend starting with HTML/CSS, but I feel even before that, you should understand how the internet and web works on a high level. What are HTML documents, how they're sent from a backend server, what is DNS, what exactly happens when you type google.com in the browser, etc.

Of course, you should not need to go in depth of any of the topics, but it provides a very solid understanding of things when you're suddenly using dev-servers for development, slightly start shifting to backend, or try to debug network requests later in your journey.

HTML5/CSS3

HTML and CSS are the fundamental building blocks of web. There's no doubt that this should be the first thing you should learn as a frontend developer.

Some very important points:

Make sure you learn about modern HTML5 syntax
Try to learn about semantic tags in HTML5 instead of sticking everything inside divs
Learn about modern layout practices in CSS (flexbox, grid)
Learn about modern CSS syntax and have a clarity on selectors, media queries, properties, values, etc. - the fundamentals.
Do not stick with older and deprecated syntax ( center tag, marquee, etc.)
Do not spend a lot of time learning about old layout practices in CSS with floats, etc.
Do not try every cutting edge CSS3/4 feature. A lot of them might not be supported in browsers yet. Stick to the widely supported CSS3 features at most.

JavaScript

JavaScript is super important for interactivity and anything you do beyond a simple static HTML/CSS landing page. It can be used extensively beyond frontend too (in Node.js/Deno/TypeScript) therefore it is essential you get your fundamentals right with JavaScript.

Some very important points:

Make sure you learn about ES6+ syntax. Do not use var and even follow tutorials using them. They're not needed in most cases. One way to check that is to try following tutorials done after 2017.
JavaScript has some behaviors that might catch you off guard. For example, why 0.1 + 0.2 is not 0.3, or why null == undefined is true in JS. Remember, there are always reasons for these behaviors. You're a google search away to understand such things.
JavaScript is fundamentally an asynchronous and non-blocking I/O language. This means you should understand how some core parts of asynchronous things like Promises work in JavaScript. Learn about DOM API and understand it's an API provided by your browser to your JS code to interact with your HTML/CSS.

Developer Tools

At this time, you should also start exploring how to work with devtools of your browser. I prefer Chrome devtools because they're great for debugging and working with JavaScript and DOM. Irrespective of your choice, I would prefer building projects and debugging any issues you face using devtools, instead of staring at your code for hours.

Chrome devtools, just like any other tool in real life or a virtual product, would require a little training and you'll have to spend some time to be maximum efficient with it. However, the time you spend learning browser devtool will pay for itself in orders of magnitude as your developer life goes on.

Some important points:

I recommend sticking to Chrome devtools because they're super powerful.
Learn about DOM manipulation, debugging network requests, setting breakpoints on events, jumping through sources, searching for code in your website assets, throttling network speed, and more.
This will be an ongoing learning, just like all other technologies. You should not aim to learn devtools at once, simply learn small but important things and move on. You will learn a lot more as you're working with projects, etc. Git
You need to learn a Version Control System (VCS), and it should be git. Git is a widely used VCS used by teams and companies to manage their codebases across employees. Git can be super complex to learn, but initially your use case of learning git would be to get your code deployed on GitHub, or a similar site.

This is required because when you learn git, you can push to github. When you are able to push to github, you can integrate with many services like GitHub pages, Vercel, Netlify, etc. to create a live preview of your website/project. This live preview then can be shared with your friends, family and even added on your resumes.

As you get employed, or work with a team of 2+ developers, you'll need git knowledge anyway. But even as a solo developer, it is of utmost importance as it makes your deployment cycle to a live website a breeze.

Some important points:

Git can get complex super quick. Do not get confused with a lot of git commands. As a solo developer, you'll hardly need anything beyond 5-7 commonly used commands.
While point 1 is true, you should still understand the overall architecture of how git works, what branches are, what does one mean by remote repositories, etc. This is again, a solid framework of understanding that you should have.

NPM/Yarn

NPM and Yarn are two most popular package managers for JavaScript. They both exist for the same purpose - managing dependencies in your codebase. A dependency is a package/software usually not developed by you/your team, but you can still use and integrate it inside your app. For example, a design system like bootstrap or tailwind CSS is a dependency.

If there's any update to the dependency, a bugfix or a security patch, the maintainer of the package updates it on the NPM repo, and you can update the package with your package manager. Alongside this, it also helps with a number of things you'll find out when you learn about NPM and yarn.

Some important points:

NPM and yarn are very close concepts-wise. When you learn about npm, you'll also learn about package.json, lockfiles, why they're needed and more. All these things are transferrable to yarn v1 as package manager too.
Therefore, do not stress a lot about what package manager you should learn. Learn any one (like yarn), and you can automatically switch to another if your project requires.

React

At codedamn, we bet heavily on React. Not because we feel like it is superior to others, but this is us following our own advice given in the top of the article - stick to a tech stack.

I worked with Angular.js in the past (2015 time), switched to Angular 2 - didn't like it much, and then made the switch to React. Since then, I've been using React myself for almost any medium+ complexity frontend project.

React is an amazing library with a tremendous community support. Any problem/bug you run in with React - chances are someone already did that mistake and asked a question on 1000 websites. Therefore, if you're a good googler, you should not face any problem working with React.

Some important points:

React looks simple on the surface, but it requires developing a mental model to work at scale. It's not only about concepts and hooks, it's also about how to architect your app, make it maintainable, choosing the right libraries for the ecosystem, etc.
React does not enforce any standard practices for common tasks, like network requests or UI. But we recommend the following options - react-query for network requests, tailwind CSS or material UI for UI library.
Do not learn in-depth about class based components unless you have to work on a legacy codebase that requires you to learn about them in depth. Class based components are not the future of React. Run away from tutorials that teach you class based components. Learn about functional components and hooks instead.

Module bundlers

Webpack is one of the most popular module bundler that React uses out of the box right now. Again, you don't necessarily need to learn webpack in depth, but you should know about what webpack.config.js file is and what are the top 10 useful options in it, and what they can do.

ESBuild is also another relatively very very fast module bundler right now, but the plugin support for it isn't great yet. As a beginner, you don't need to worry about ESBuild yet.

Code Testing

Last but not least, you should also focus on code testing. I feel people should start with end to end testing because it seems easier and more intuitive as a reason for code testing, compared to snapshot testing or unit testing.

My recommended tool here is to use Cypress - which has a very friendly syntax to test your applications end to end i.e. your code will act like a user using your app and can pretty much perform any action your user can perform, but in an automated fashion saving your time and energy.

Practice! Practice! Practice!

One thing not explicitly mentioned in the blog post above is the fact that you need to practice a lot, all the time. Practice is the way to fit concepts in your brain. If you're only watching videos, you are not learning.

How to learn frontend web development?

The above list is not an exhaustive list of topics, but definitely a solid start as a guide. Now the question is how do you learn frontend web development? Here, I have two options for you:

You want a structured learning approach with support and practice modules

The purpose of codedamn platform to exist is to solve this structuring, support and practice. We have a full Frontend Web Developer curriculum of 50+ hours with over 60+ practice sessions including the following:

Practice projects built in Interactive learning (completing labs and challenges inside browser)
Video lectures by top industry experts
We always keep this curriculum up to date with industry standards with all the things mentioned in the blog post above, and some more.

You want a free learning approach

If you cannot afford codedamn's learning path above with money, you can spend your time on free resources instead. Irrespective of that, using these resources with codedamn learning path is also recommended.

Use google, a lot, for learning and doubt clarification. Watch YouTube videos, but make sure they're relatively new, especially with topics like JavaScript and React.
Practice on your own, build projects and deploy them online for people to see.
Take up free courses on codedamn (we have a lot of 100% free courses too), like this react query course.

Conclusion

Time is money, in fact, sometimes more important than money.

What you want to spend (or can spend), is upto you. But I'd recommend that even if not codedamn, spending money on your education to get an acceleration and clarity on what you're trying to achieve would definitely be worth it.

The average salary of a frontend developer is $100,000/year. In fact, a codedamn yearly subscription would pay for itself, probably in the first 10 days of your job as a frontend developer.

Web Scraping Python Tutorial – How to Scrape Data From A Website

Mehul Mohan — Fri, 25 Sep 2020 22:20:05 +0000

Python is a beautiful language to code in. It has a great package ecosystem, there's much less noise than you'll find in other languages, and it is super easy to use.

Python is used for a number of things, from data analysis to server programming. And one exciting use-case of Python is Web Scraping.

In this article, we will cover how to use Python for web scraping. We'll also work through a complete hands-on classroom guide as we proceed.

Note: We will be scraping a webpage that I host, so we can safely learn scraping on it. Many companies do not allow scraping on their websites, so this is a good way to learn. Just make sure to check before you scrape.

Introduction to Web Scraping classroom

If you want to code along, you can use this free codedamn classroom that consists of multiple labs to help you learn web scraping. This will be a practical hands-on learning exercise on codedamn, similar to how you learn on freeCodeCamp.

In this classroom, you'll be using this page to test web scraping: https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/

This classroom consists of 7 labs, and you'll solve a lab in each part of this blog post. We will be using Python 3.8 + BeautifulSoup 4 for web scraping.

Part 1: Loading Web Pages with 'request'

This is the link to this lab.

The requests module allows you to send HTTP requests using Python.

The HTTP request returns a Response Object with all the response data (content, encoding, status, and so on). One example of getting the HTML of a page:

import requests

res = requests.get('https://codedamn.com')

print(res.text)
print(res.status_code)

Passing requirements:

Get the contents of the following URL using requests module: https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/
Store the text response (as shown above) in a variable called txt
Store the status code (as shown above) in a variable called status
Print txt and status using print function
Once you understand what is happening in the code above, it is fairly simple to pass this lab. Here's the solution to this lab:

import requests

# Make a request to https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/
# Store the result in 'res' variable
res = requests.get(
    'https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/')
txt = res.text
status = res.status_code

print(txt, status)
# print the result

Let's move on to part 2 now where you'll build more on top of your existing code.

Part 2: Extracting title with BeautifulSoup

This is the link to this lab.

In this whole classroom, you’ll be using a library called BeautifulSoup in Python to do web scraping. Some features that make BeautifulSoup a powerful solution are:

It provides a lot of simple methods and Pythonic idioms for navigating, searching, and modifying a DOM tree. It doesn't take much code to write an application
Beautiful Soup sits on top of popular Python parsers like lxml and html5lib, allowing you to try out different parsing strategies or trade speed for flexibility.
Basically, BeautifulSoup can parse anything on the web you give it.

Here’s a simple example of BeautifulSoup:

from bs4 import BeautifulSoup

page = requests.get("https://codedamn.com")
soup = BeautifulSoup(page.content, 'html.parser')
title = soup.title.text # gets you the text of the <title>(...)</title>

Passing requirements:

Use the requests package to get title of the URL: https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/
Use BeautifulSoup to store the title of this page into a variable called page_title

Looking at the example above, you can see once we feed the page.content inside BeautifulSoup, you can start working with the parsed DOM tree in a very pythonic way. The solution for the lab would be:

import requests
from bs4 import BeautifulSoup

# Make a request to https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Extract title of page
page_title = soup.title.text

# print the result
print(page_title)

This was also a simple lab where we had to change the URL and print the page title. This code would pass the lab.

Part 3: Soup-ed body and head

This is the link to this lab.

In the last lab, you saw how you can extract the title from the page. It is equally easy to extract out certain sections too.

You also saw that you have to call .text on these to get the string, but you can print them without calling .text too, and it will give you the full markup. Try to run the example below:

import requests
from bs4 import BeautifulSoup

# Make a request
page = requests.get(
    "https://codedamn.com")
soup = BeautifulSoup(page.content, 'html.parser')

# Extract title of page
page_title = soup.title.text

# Extract body of page
page_body = soup.body

# Extract head of page
page_head = soup.head

# print the result
print(page_body, page_head)

Let's take a look at how you can extract out body and head sections from your pages.

Passing requirements:

Repeat the experiment with URL: https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/
Store page title (without calling .text) of URL in page_title
Store body content (without calling .text) of URL in page_body
Store head content (without calling .text) of URL in page_head

When you try to print the page_body or page_head you'll see that those are printed as strings. But in reality, when you print(type page_body) you'll see it is not a string but it works fine.

The solution of this example would be simple, based on the code above:

import requests
from bs4 import BeautifulSoup

# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Extract title of page
page_title = soup.title

# Extract body of page
page_body = soup.body

# Extract head of page
page_head = soup.head

# print the result
print(page_title, page_head)

Part 4: select with BeautifulSoup

This is the link to this lab.

Now that you have explored some parts of BeautifulSoup, let's look at how you can select DOM elements with BeautifulSoup methods.

Once you have the soup variable (like previous labs), you can work with .select on it which is a CSS selector inside BeautifulSoup. That is, you can reach down the DOM tree just like how you will select elements with CSS. Let's look at an example:

import requests
from bs4 import BeautifulSoup

# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Extract first <h1>(...)</h1> text
first_h1 = soup.select('h1')[0].text

.select returns a Python list of all the elements. This is why you selected only the first element here with the [0] index.

Passing requirements:

Create a variable all_h1_tags. Set it to empty list.
Use .select to select all the h1 tags and store the text of those h1 inside all_h1_tags list.
Create a variable seventh_p_text and store the text of the 7th p element (index 6) inside. The solution for this lab is:

import requests
from bs4 import BeautifulSoup
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Create all_h1_tags as empty list
all_h1_tags = []

# Set all_h1_tags to all h1 tags of the soup
for element in soup.select('h1'):
    all_h1_tags.append(element.text)

# Create seventh_p_text and set it to 7th p element text of the page
seventh_p_text = soup.select('p')[6].text

print(all_h1_tags, seventh_p_text)

Let's keep going.

Part 5: Top items being scraped right now

This is the link to this lab.

Let's go ahead and extract the top items scraped from the URL: https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/

If you open this page in a new tab, you’ll see some top items. In this lab, your task is to scrape out their names and store them in a list called top_items. You will also extract out the reviews for these items as well.

To pass this challenge, take care of the following things:

Use .select to extract the titles. (Hint: one selector for product titles could be a.title)
Use .select to extract the review count label for those product titles. (Hint: one selector for reviews could be div.ratings) Note: this is a complete label (i.e. 2 reviews) and not just a number.
Create a new dictionary in the format:

info = {
   "title": 'Asus AsusPro Adv...   '.strip(),
   "review": '2 reviews\n\n\n'.strip()
}

Note that you are using the strip method to remove any extra newlines/whitespaces you might have in the output. This is important to pass this lab.
Append this dictionary in a list called top_items
Print this list at the end

There are quite a few tasks to be done in this challenge. Let's take a look at the solution first and understand what is happening:

import requests
from bs4 import BeautifulSoup
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Create top_items as empty list
top_items = []

# Extract and store in top_items according to instructions on the left
products = soup.select('div.thumbnail')
for elem in products:
    title = elem.select('h4 > a.title')[0].text
    review_label = elem.select('div.ratings')[0].text
    info = {
        "title": title.strip(),
        "review": review_label.strip()
    }
    top_items.append(info)

print(top_items)

Note that this is only one of the solutions. You can attempt this in a different way too. In this solution:

First of all, you select all the div.thumbnail elements which give you a list of individual products
Then you iterate over them
Because select allows you to chain over itself, you can use select again to get the title.
Note that because you're running inside a loop for div.thumbnail already, the h4 > a.title selector would only give you one result, inside a list. You select that list's 0th element and extract out the text. Finally, you strip any extra whitespace and append it to your list. Straightforward right?

Part 6: Extracting Links

This is the link to this lab.

So far you have seen how you can extract the text, or rather innerText of elements. Let's now see how you can extract attributes by extracting links from the page.

Here’s an example of how to extract out all the image information from the page:

import requests
from bs4 import BeautifulSoup
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Create top_items as empty list
image_data = []

# Extract and store in top_items according to instructions on the left
images = soup.select('img')
for image in images:
    src = image.get('src')
    alt = image.get('alt')
    image_data.append({"src": src, "alt": alt})

print(image_data)

In this lab, your task is to extract the href attribute of links with their text as well. Make sure of the following things:

You have to create a list called all_links
In this list, store all link dict information. It should be in the following format:

info = {
   "href": "<link here>",
   "text": "<link text here>"
}

Make sure your text is stripped of any whitespace
Make sure you check if your .text is None before you call .strip() on it.
Store all these dicts in the all_links
Print this list at the end
You are extracting the attribute values just like you extract values from a dict, using the get function. Let's take a look at the solution for this lab:

import requests
from bs4 import BeautifulSoup
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Create top_items as empty list
all_links = []

# Extract and store in top_items according to instructions on the left
links = soup.select('a')
for ahref in links:
    text = ahref.text
    text = text.strip() if text is not None else ''

    href = ahref.get('href')
    href = href.strip() if href is not None else ''
    all_links.append({"href": href, "text": text})

print(all_links)

Here, you extract the href attribute just like you did in the image case. The only thing you're doing is also checking if it is None. We want to set it to an empty string, otherwise, we want to strip the whitespace.

Part 7: Generating CSV from data
This is the link to this lab.

Finally, let's understand how you can generate CSV from a set of data. You will create a CSV with the following headings:

Product Name
Price
Description
Reviews
Product Image
These products are located in the div.thumbnail. The CSV boilerplate is given below:

import requests
from bs4 import BeautifulSoup
import csv
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

all_products = []

products = soup.select('div.thumbnail')
for product in products:
    # TODO: Work
    print("Work on product here")


keys = all_products[0].keys()

with open('products.csv', 'w', newline='') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_products)

You have to extract data from the website and generate this CSV for the three products.

Passing Requirements:

Product Name is the whitespace trimmed version of the name of the item (example - Asus AsusPro Adv..)
Price is the whitespace trimmed but full price label of the product (example - $1101.83)
The description is the whitespace trimmed version of the product description (example - Asus AsusPro Advanced BU401LA-FA271G Dark Grey, 14", Core i5-4210U, 4GB, 128GB SSD, Win7 Pro)
Reviews are the whitespace trimmed version of the product (example - 7 reviews)
Product image is the URL (src attribute) of the image for a product (example - /webscraper-python-codedamn-classroom-website/cart2.png)
The name of the CSV file should be products.csv and should be stored in the same directory as your script.py file Let's see the solution to this lab:

import requests
from bs4 import BeautifulSoup
import csv
# Make a request
page = requests.get(
    "https://codedamn-classrooms.github.io/webscraper-python-codedamn-classroom-website/")
soup = BeautifulSoup(page.content, 'html.parser')

# Create top_items as empty list
all_products = []

# Extract and store in top_items according to instructions on the left
products = soup.select('div.thumbnail')
for product in products:
    name = product.select('h4 > a')[0].text.strip()
    description = product.select('p.description')[0].text.strip()
    price = product.select('h4.price')[0].text.strip()
    reviews = product.select('div.ratings')[0].text.strip()
    image = product.select('img')[0].get('src')

    all_products.append({
        "name": name,
        "description": description,
        "price": price,
        "reviews": reviews,
        "image": image
    })


keys = all_products[0].keys()

with open('products.csv', 'w', newline='') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_products)

The for block is the most interesting here. You extract all the elements and attributes from what you've learned so far in all the labs.

When you run this code, you end up with a nice CSV file. And that's about all the basics of web scraping with BeautifulSoup!

Conclusion

I hope this interactive classroom from codedamn helped you understand the basics of web scraping with Python.

If you liked this classroom and this blog, tell me about it on my twitter and Instagram. Would love to hear feedback!

Create a URL Shortener Project using MongoDB + Node - but hands-on!

Mehul Mohan — Fri, 18 Sep 2020 23:09:32 +0000

Let us learn about MongoDB, Mongoose, Node, and other tech by building a simple URL shortener project.

Have you ever wondered how you could create a quick URL shortener for yourself? Like how twitter shortens your links when you share them? Or how bit.ly works?

Sure enough, these are complicated companies, but the concept for URL shorteners is simple. Let's learn about MongoDB and other backend tools by actually building this project in 7 steps.

Introduction to Project

We will be using this free URL shortener classroom from codedamn to really, like really create and evaluate our hands-on project and see the feedback.

We will be using the following technologies:

Mongoose as the ORM
MongoDB as the backend database
Node.js as the backend
A simple embedded JS file as the frontend

Lab 1: Setting up the Express server

The link of this lab is here

It's a fairly straightforward lab. We have to just create a route /short which should respond appropriately. This code would let us pass:

// Initialize express server on PORT 1337
const express = require('express')
const app = express()

app.get('/', (req, res) => {
    res.send('Hello World! - from codedamn')
})

app.get('/short', (req, res) => {
    res.send('Hello from short')
})

app.listen(process.env.PUBLIC_PORT, () => {
    console.log('Server started')
})

Lab 2: Setting up our view engine

The link of this lab is here

We're using a single .ejs file, so let's explore that a little. Again, a very simple lab because we just have to change the name of the variable. This should get us done:

const express = require('express')
const app = express()

app.set('view engine', 'ejs')

app.get('/', (req, res) => {
    res.render('index', { myVariable: 'My name is John!' })
})

app.listen(process.env.PUBLIC_PORT, () => {
    console.log('Server started')
})

Lab 3: Setting up MongoDB

The link of this lab is here

In this lab, we'll connect to MongoDB correctly and insert a record, just for the record

This is the solution which should get us to the next lab:

app.post('/short', (req, res) => {
    const db = mongoose.connection.db
    // insert the record in 'test' collection
    db.collection('test').insertOne({ testCompleted: 1 })

    res.json({ ok: 1 })
})

// Setup your mongodb connection here
mongoose.connect('mongodb://localhost/codedamn', {
    useNewUrlParser: true,
    useUnifiedTopology: true
})
mongoose.connection.on('open', () => {
    // Wait for mongodb connection before server starts
    app.listen(process.env.PUBLIC_PORT, () => {
        console.log('Server started')
    })
})

Lab 4: Setting up a Mongoose schema

The link of this lab is here

Finally, we define a schema in the models/url.js file for proper handling with Mongoose, and here's the code for that:

const mongoose = require('mongoose')
const shortId = require('shortid')

const shortUrlSchema = new mongoose.Schema({
  full: {
    type: String,
    required: true
  },
  short: {
    type: String,
    required: true,
    default: shortId.generate
  },
  clicks: {
    type: Number,
    required: true,
    default: 0
  }
})

module.exports = mongoose.model('ShortUrl', shortUrlSchema)

Also, as part of the challenge, we update the /short route now.

app.post('/short', async (req, res) => {
    // insert the record using the model
    const record = new ShortURL({
        full: 'test'
    })
    await record.save()
    res.json({ ok: 1 })
})

Lab 5: Linking the frontend, backend, + MongoDB

This is also a simple lab. We just have to update the route to extract the URL passed and store it in the database using our schema.

app.use(express.urlencoded({ extended: false }))

app.post('/short', async (req, res) => {
    // Grab the fullUrl parameter from the req.body
    const fullUrl = req.body.fullUrl
    console.log('URL requested: ', fullUrl)

    // insert and wait for the record to be inserted using the model
    const record = new ShortURL({
        full: fullUrl
    })

    await record.save()

    res.redirect('/')
})

Lab 6: Displaying short URLs on the frontend

Now, we display the URL set on our website using the .ejs variables passed.

app.get('/', async (req, res) => {
    const allData = await ShortURL.find()
    res.render('index', { shortUrls: allData })
})

Lab 7: Making the redirection work

Finally, we link up the redirection scheme using dynamic express routes and correct status codes.

app.get('/:shortid', async (req, res) => {
    // grab the :shortid param
    const shortid = req.params.shortid

    // perform the mongoose call to find the long URL
    const rec = await ShortURL.findOne({ short: shortid })

    // if null, set status to 404 (res.sendStatus(404))
    if (!rec) return res.sendStatus(404)

    // if not null, increment the click count in database
    rec.clicks++
    await rec.save()

    // redirect the user to original link
    res.redirect(rec.full)
})

Conclusion

And we can call it a day! You just built a fully working URL shortener by yourself using Express + Node + MongoDB. Give yourself a pat on back!

The final source code is available on GitHub.

If you have any feedback on this article or codedamn classrooms, feel free to reach out to me on Twitter. Let's discuss :)