Label Encoding in ML

Nitish Kushwaha — Wed, 21 Aug 2024 15:04:50 +0000

Label Encoding is one of the most used techniques in machine learning. It is used to convert the categorial data in numerical form. So, data can be fitted into the model.

Let us understand why we use the Label Encoding. Imagine having the data, containing the essential columns in the form of string. But, you cannot fit this data in the model, because modelling only works on numerical data, what do we do? Here comes the life-saving technique which is evaluated at the preprocessing step when we ready the data for fitting, which is Label Encoding.

We will use the iris dataset from Scikit-Learn library, to understand the workings of Label Encoder. Make sure you have the following libraries installed.

pandas
scikit-learn

For installing as libraries, run the following command:

$ python install -U pandas scikit-learn

Now open Google Colab Notebook, and dive into coding and learning Label Encoder.

Let's Code

Start with importing the following libraries:

import pandas as pd
from sklearn import preprocessing

Import the iris dataset, and initialize it for usage:

from sklearn.datasets import load_iris
iris = load_iris()

Now, we need to select the data that we want Encode, we will be encoding the species names for the irises.

species = iris.target_names
print(species)

Output:

array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

Let's instantiate the class LabelEncoder from preprocessing:

label_encoder = preprocessing.LabelEncoder()

Now, we are ready to fit the data using the label encoder:

label_encoder.fit(species)

You will output similar to this:

If you get this output, you have successfully fitted the data. But, the question is how you will find out what values are assigned to each species and in which order.

The order in which Label Encoder fits the data is stored in classes_ attribute. Encoding starts from 0 to data_length-1.

label_encoder.classes_

Output:

array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

The label encoder will automatically sort the data, and start the encoding from the left side. Here:

setosa -> 0
versicolor -> 1
virginica -> 2

Now, let's test the fitted data. We will transform the iris species setosa.

label_encoder.transform(['setosa'])

Output: array([0])

Again, if you transform the specie virginica.

label_encoder.transform(['virginica'])

Output: array([2])

You can also input the list of species, such as ["setosa", "virginica"]

Scikit Learn documentation for label encoder >>>

Data Analysis 1: Scraping web pages

Nitish Kushwaha — Thu, 18 Apr 2024 20:30:34 +0000

There is no shortage of excellent datasets on the internet, but you might want to show prospective employers that you're able to find and scrape your own data as well. Plus, knowing how to scrape the web means you can find and use datasets that match your interests, regardless of not they've already been compiled.

Scraping your own dataset also gives you the ability to build your custom dataset for testing as well as for large projects.

Today we're going to scrape the famous news site "Times of India" and find all the link tags available on the page, verify those URLs, and process it.

Setting up

We will start with importing the required libraries and save the URL we want to scrape in the "URL" variable.

from bs4 import BeautifulSoup
import requests
import validators

URL = "https://timesofindia.indiatimes.com/"

Request Page

The first we are going to do is request the page using the requests library, we going to send the get request, and we will receive a response with page content, we can also say that we have downloaded the page.

...
soup = None
res = requests.get(URL)
if res.status_code == 200: ...

Here, save the response in res variables, then we check whether our request was successful, and if the status_code, if status_code is 200, then we parse the page with beautiful soup to extract information.

Parsing Content

In the above code, we have created the soup variable, which will store the parsed page.

...
soup = None
if res.status_code == 200:
    soup = BeautifulSoup(res.content, 'html.parser')

Filtering link Tags

We have successfully requested and parsed the web page, now it's time to filter all the link tags from the page.

...
allLinkTag = soup.findAll('a')
unverified = []
for link in allLinkTag:
    unverified.append(link.attrs['href'])

Here, we have filtered all the link tags and stored them in the list allLinkTag, and we have created an unverified list, which will store all the URLs, present in those link tags, we had done this by iterating over link tag and extracting the href attribute from the link tag.

Validating URLs

Now we have all the URLs from the page, it is time to validate them.

...
validUrls = []
inValidUrls = []
for url in unverified:
    if validators.url(url):
        validUrls.append(url)
    else:
        inValidUrls.append(url)

We have iterated over all the unverified URL, and validated them using the url() function of the validators library. If the URL is valid, we push it to the validUrls list, else push the URL to the inValidUrls list.

Here we have filtered all the valid and invalid URLs from the page, using web scraping.

Using the above procedure you can extract as many websites as you want, and build your custom dataset for testing or for your project.

Checkout the full code on GitHub:

Github: Web Scraping 1

DEV Community: Nitish Kushwaha