DEV Community: Tano Paul

A Beginner's Guide to Web Scraping with BeautifulSoup

Tano Paul — Mon, 16 Oct 2023 14:06:35 +0000

What is Web Scraping?

Web scraping allows you to unlock a treasure trove of data from the internet. It involves using software or programming code to access and gather data from web pages, which can then be processed, analyzed, or stored for various purposes. It allows users to retrieve specific pieces of information from websites in a structured and organized manner.

Heed of Warning

Before moving on, it's important to note that websites implement various protections to deter web scrapers and ensure fair and responsible use of their resources. A couple ways to ensure you're scraping the web ethically is by reading the robots.txt file that most websites have. Here is an example of https://www.apple.com/robots.txt:

Another way of keeping unwanted web scrapers out of one's website is with the well-known CAPTCHA verification that most people are familiar with.

Getting Started

import requests
from bs4 import BeautifulSoup

Beautiful Soup is a Python library used for web scraping purposes to pull the data out of HTML and XML files.

url = "http://books.toscrape.com/"
response = requests.get(url)

There's a variety of websites on the internet thats sole purpose is to act as practice for web scraping. I'm using toscrape.com that mimics an online store to capture the prices of the products stored on the website. The method requests.get() sends a request to the website.

soup = BeautifulSoup(response.text, "html.parser")

After sending your request, you'll need to create a BeautifulSoup object to parse the HTML of the website.

prices = soup.find_all("p", class_="price_color")
titles = soup.find_all('article')

title_list = []
price_list = []
title_text_list = []

for price in prices:
    price_list.append(price.text)

for title in titles:
    title_list.append(title.find_all('h3'))

for book_title in title_list:
    title_text_list.append(book_title[0].text)

all_books = dict(zip(title_text_list, price_list))

for title, cost in all_books.items():
    print(title, cost)

Lastly, in order to find the elements that we need, we'll need to refer to the DOM of the page we're scraping. If we want to find the the titles of the books as well as the prices, we can use the find_all() method for the 'p' element and 'article' element since that is where our information lives on the page. We then loop through all of the elements and save the texts of the elements in a list using .text().

From there, I was able to create a dictionary with the names of the books as the key and the price as the value.

Conclusion

Web scraping with Python and Beautiful Soup is a valuable skill for any software engineer. It opens the door to a wealth of data that can be harnessed for various purposes. As you explore this world, remember to respect websites' terms of service and robots.txt files, and always scrape responsibly and ethically.

Resources

https://www.geeksforgeeks.org/what-is-web-scraping-and-how-to-use-it/

A Beginner's Guide to File Input/Output (I/O) in Python

Tano Paul — Sun, 24 Sep 2023 17:48:09 +0000

Python is a versatile and powerful programming language that offers a range of functionalities, one of which is handling files. In this guide, we'll explore the basics of File Input/Output (I/O) in Python, focusing on the open method and file modes ('r', 'a', and 'w').

The 'open' Function

The open function gives you the ability to interact with files in Python. It's used to open files and create a file object, which allows you to read, write, or append content.

file = open("things_i_wrote.txt", "w")
file.write("Text I'm writing")
file.close()

The first argument to open is the file path, and the second argument is the mode in which you want to open the file. The .close() method is effectively a "save" button when writing into files.

Closing Files Automatically with a Context Manager (with)

To ensure that files are properly closed after you're done with them, it's good practice to use a context manager. This will automatically close the file when you're done with it, allowing for more compact code. The same code block above can be rewritten like this:

with open("things_i_wrote.txt", "w") as file:
    file.write("Text I'm writing")

Modes in File I/O

Writing ('w' mode)
The 'w' mode is used for writing to files. If the file exists, the new content that is to be written will be written over the already existing file. If not, a new file will be created. Be careful, as this can potentially overwrite existing content.

thing_to_write = input("What would you like to write? ")

with open("things_i_wrote.txt", 'w') as file:
    file.write(thing_to_write)

In the example above, we have written code to collect user input from the terminal and write the input into a .txt file. You'll notice that after inputing "Thing 1", "Thing 2", and "Thing 3" to the .txt file, only "Thing 3" appears in the file. This is because the file was rewritten three different times with the last rewrite being "Thing 3". In order to get around this, we will instead use the 'a' mode and append the content into the .txt file.

Appending ('a' mode)
The 'a' mode is used for appending content to the end of an existing file. It's useful when you want to add data without overwriting the original content.

thing_to_write = input("What would you like to write? ")

with open("things_i_wrote.txt", 'a') as file:
    file.write(f"{thing_to_write}\n")

As you can see, we were able to successfully write all of our user inputs into our .txt file by appending the inputs rather than writing them. If you're wondering what the "\n" is, this combination of symbols is an indicator for the computer that the end of the input starts a new line in the file. If we omitted the "\n", the file would look like this:

Reading ('r' mode)

The 'r' mode is used for reading files. It's the default mode when no mode is specified, so it is not necessary to write. When you open a file in read mode, you can only read the content and not modify it, but it allows you to collect data and save them into variables for your program.

extracted_from_file = []

with open("things_i_wrote.txt") as file:
    for line in file:
        extracted_from_file.append(line.rstrip())

print(extracted_from_file)

Say we never created and wrote the .txt file that we've been using previously. If we were to want to extract information from a file, we can use the "r" mode in Python to read and extract that data. We've omitted the "r" because the open() method is set to read by default. We are able to create a list variable, read through the .txt file, and save the content of the file in the list variable. We also used .rstrip() to strip the "\n" characters so they aren't saved into the list. If we neglected to add .rstrip(), the list would look like this:

Conclusion

File I/O is a fundamental aspect of programming. With the open method and different modes ('r', 'a', and 'w'), you can read, write, and append content to files in Python. Remember to use a context manager ("with") for safe file handling. With more knowledge on handling files, a programmer can have the ability to manipulate audio files, image files, and much more.

Sources:
https://www.w3schools.com/python/ref_func_open.asp

https://docs.python.org/3/tutorial/inputoutput.html

useState vs useRef Hooks

Tano Paul — Wed, 06 Sep 2023 13:39:35 +0000

React is a powerful tool that allows developers to create powerful websites and apps by combining the programming language of JavaScript with the power of HTML through the use of a syntax extension called JSX (JavaScript XML). In React, one of the more powerful tools is referred to as a "hook", which is a special function that allows you to "hook" onto or use special React features within functional components.

One of the most used hooks is the useState hook that allows you to store and manipulate data in your React program. Its use is essential if one is to build a website that a user is able to interact with. Something crucial to understand, which isn't necessarily a bad thing, is that useState causes re-renders every time the state is updated. In most cases, this is exactly what you want. However, there may be some cases that a developer doesn't want to re-render the component when updating a variable. This is where useRef comes in.

Getting Started with useRef

Like any other hook, the first thing you do is import it at the top of your component in order to declare it.

import { useRef } from 'react';

When initializing useRef, the hook is called differently in both appearance and functionality than useState, which is stored as an array with a variable and setter function. With useRef, it's stored as an object thats key is always "current". You can then place a reference property on a JSX element to "refer" to that element.

const inputRef = useRef(any value);
console.log(inputRef)
// logged on console:   {current: input}
return (
    <div>
      <input ref={inputRef} />
      <button onClick={handleClick}>
        Focus the input
      </button>
    </div>
)

Best Use Cases

Most of the time when working in React you'll want to use the useState hook rather than useRef, but there are some cases where useRef will come in handy.

Say you wanted to scroll an image into view. If a programmer used useState to accomplish this, after clicking a button to focus on another picture, useState will re-render the page to land on the desired image. With useRef, you can reference the element that does not need to re-render in order to reach the desired effect.

const listRef = useRef(null);

  function scrollToIndex(index) {
    const listNode = listRef.current;

    const imgNode = listNode.querySelectorAll('li > img')[index];
    imgNode.scrollIntoView({
      behavior: 'smooth',
      block: 'nearest',
      inline: 'center'
    });
  }

  return (
    <>
      <nav>
        <button onClick={() => scrollToIndex(0)}>
          Tom
        </button>
        <button onClick={() => scrollToIndex(1)}>
          Maru
        </button>
        <button onClick={() => scrollToIndex(2)}>
          Jellylorum
        </button>
      </nav>
      <div>
        <ul ref={listRef}>
          <li>
            <img
              src="https://placekitten.com/g/200/200"
              alt="Tom"
            />
          </li>
          <li>
            <img
              src="https://placekitten.com/g/300/200"
              alt="Maru"
            />
          </li>
          <li>
            <img
              src="https://placekitten.com/g/250/200"
              alt="Jellylorum"
            />
          </li>
        </ul>
      </div>
    </>
  );
}

Another use for useRef would be to apply it to a video element so that the video does not re-render every time you hit the play/pause button.

const [isPlaying, setIsPlaying] = useState(false);
  const ref = useRef(null);

  function handleClick() {
    const nextIsPlaying = !isPlaying;
    setIsPlaying(nextIsPlaying);

    if (nextIsPlaying) {
      ref.current.play();
    } else {
      ref.current.pause();
    }
  }

  return (
    <>
      <button onClick={handleClick}>
        {isPlaying ? 'Pause' : 'Play'}
      </button>
      <video
        width="250"
        ref={ref}
        onPlay={() => setIsPlaying(true)}
        onPause={() => setIsPlaying(false)}
      >
        <source
          src="https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.mp4"
          type="video/mp4"
        />
      </video>
    </>
  );
}

Some other use cases would be building a stopwatch or creating a click counter.

In conclusion, you should use useRef in React when you need to perform certain tasks that involve managing mutable references to elements or values within your components without causing re-renders.

REFERENCES:
https://react.dev/reference/react/useRef

Basics of Big O

Tano Paul — Fri, 18 Aug 2023 04:51:38 +0000

Big O notation is a mathematical shorthand used in programming to describe the worst and based case scenarios of any given algorithm's performance. Note that this is not to be confused with a mathematical formula that has values plugged into it. Instead, it helps one analyze and compare the relative efficiency of algorithms without having to delve into detailed mathematical calculations. In short, it provides a way to express how the running time of an algorithm grows relative to the size of its input.

To get started with Big O we'll go over four main notations. I'll also include a simple real life example of each notation along with code blocks to describe my examples using JavaScript.

O(1) - Constant Time

O(1) describes an algorithm's efficiency where the amount of time it takes for the algorithm to complete is constant regardless of the input size. Therefore, it reliably performs the same way whether the input is small or large.

Imagine you had a phone book and every time you wanted to search for someone's name in the phone book, you would magically turn to the exact page of the person you were trying to find. In the code block below, the function is given an array of 1,000 different names while measuring the amount of time that it took to find the name. In the case of O(1), the algorithm isn't searching through the entire array. Instead, it's going straight to the given index. Notice that the running time came in at 0.09999999962747097 ms, the fastest of the examples.

// O(1)
const start = performance.now();    // Used to test running time 


function constantTimeSearch(names, targetIndex) {
      return names[targetIndex]
  }

  const targetIndex = names.indexOf('Thomas'); 
  let found = constantTimeSearch(names, targetIndex);

  const end = performance.now();    // Used to test running time
  if (found === 'Thomas') {
    console.log('FOUND!')
  }
  console.log(`Execution time: ${end - start} ms`);
  // LOG: Execution time: 0.09999999962747097 ms

O(n) - Linear Time

O(n) describes the efficiency of an algorithm where the amount of time it takes it to complete grows linearly with the algorithm's input size. So the time that it takes to process the data is directly proportional to the size of the data.

Imagine you had the same phone book from the first example, but this time not magical. If you wanted to search for the name 'Thomas' in the phone book using O(n), you would start at page one with the A's, and flip one page at a time until you land on the page that Thomas is on. So if Thomas was on page 2500, it will take us 2500 steps to complete the task.

This process will take a long time to complete, even for a computer (relatively speaking). The code below details the example with a for loop starting from the first name and working its way one by one to Thomas. Not the most efficient with the execution time of 0.39 ms.

// O(n)

const start = performance.now();    // Used to test running time

for (let i = 0; i < names.length; i++) {
    if (names[i] === 'Thomas') {
        console.log('FOUND!')
    }
}

const end = performance.now();    // Used to test running time
console.log(`Execution time: ${end - start} ms`);
// LOG: Execution time: 0.3999999985098839 ms

O(log n) - Logarithmic Time

Algorithms using O(log n) typically use a 'divide and conquer' strategy, which breaks down a problem into smaller sub problems in order to solve the main problem.

To make things simple, we'll use the same phone book example, but this time we'll start our search from the middle of the book while looking for Thomas. Our first step lands us in the M's section of the phone book. Since we know that 'T' in Thomas is after 'M', we can dramatically rip out everything from the phone book before section 'M'.

We now have a book that starts with section 'M' and ends in 'Z'. Our second step is to turn to the middle of the remaining book and land in section 'T'. Look at that, it took us around two steps using O(log n) to find the exact same name that we tried to find using O(n), but in a fraction of the time.

//O(log n)
const start = performance.now();


function binarySearch(names, target) {
    let left = 0;
    let right = names.length - 1;

    while (left <= right) {
      const mid = Math.floor((left + right) / 2);

      if (names[mid] === target) {
        return mid; 
      } else if (names[mid] < target) {
        left = mid + 1; 
      } else {
        right = mid - 1; 
      }
    }
  }

  const targetName = "Thomas";
  const index = binarySearch(names, targetName);

const end = performance.now();

if (names[index] === targetName) {
    console.log('FOUND!')
  } 
console.log(`Execution time: ${end - start} ms`);
// LOG: Execution time: 0.10000000149011612 ms

O(n²) Quadratic Time

O(n²) describes an algorithm in which its performance time increases exponentially as the size of the input increases. These types of algorithms are extremely inefficient and generally not used in large scale processing.

With the phone book example, it would be equivalent to starting at page 0 to look for Thomas, looking at the first word and then manually comparing each letter to the letters in 'Thomas'. After completing each letter, you move on to the next name and repeat the process. In programming terms, it would be a function within a loop within a function within another loop, drastically reducing the performance of the algorithm.

function searchForMatchingNames(names, targetName) {
    const matchingNames = [];

    for (let i = 0; i < names.length; i++) {
      const currentName = names[i];
      let isMatching = true;

      if (currentName.length !== targetName.length) {
        isMatching = false;
      } else {
        for (let j = 0; j < targetName.length; j++) {
          if (currentName[j] !== targetName[j]) {
            isMatching = false;
            break;
          }
        }
      }

      if (isMatching) {
        matchingNames.push(currentName);
      }
    }

    return matchingNames;
  }


  const targetName = "Thomas";
  const matchingNames = searchForMatchingNames(names, targetName);

// LOG: Execution time: 11.399999998509884 ms

Conclusion

Big O notation provides insights into the efficiency of different algorithms. It doesn't give you precise timings or exact performance measurements. Instead, it focuses on how an algorithm's performance scales as the input size grows larger. It's a tool meant for analysis and decision making, especially when dealing with different algorithms and problem-solving strategies.

Sources

https://www.freecodecamp.org/news/big-o-notation-why-it-matters-and-why-it-doesnt-1674cfa8a23c/

https://en.wikipedia.org/wiki/Big_O_notation

https://www.educative.io/answers/what-is-big-o-notation

Learning To Code Before Joining A Coding Bootcamp

Tano Paul — Sat, 12 Aug 2023 20:17:57 +0000

Joining a coding bootcamp can be both daunting and frustrating if you don't play your cards right, especially when joining an accelerated program. The bootcamp I decided to dedicate myself to is Flatiron School's software engineering program, which helps its students become job-ready in about four months if you decide to enroll in the accelerated program.

Now, you may be thinking that four months is way too little time for someone to go from zero programming experience to being job-ready and you'd be right.

Luckily, I was in no rush to pay for the expensive tuition bill, so as I saved my money month after month, I also scoured the internet for online resources that people recommended. This blog is meant to detail some of my favorite resources.

Codecademy

Aah codecademy.. this was the first place where I learned to code. I flew through the fundamentals of HTML and CSS and thought to myself,

"Look mom, I'm a programmer now!"

Boy, was I wrong. Little did I know, programming languages and markup/sheet style languages are two completely different animals. When I stepped into the world of JavaScript, my dreams were absolutely crushed and I found myself coding less and less as I struggled to grasp each lesson. But I would always find myself coming back for more frustration.

Scrimba

I first heard about Scrimba when I was looking through YouTube videos about people who changed career paths into software engineering. I was extremely impressed by the website's ability to allow users to code things directly into the lecture video. Amazing! Here is an example of what I'm talking about:

Throughout the various videos, the instructors go over all of the fundamentals of programming with JavaScript. Following along during the coding challenges and projects in the videos allowed my mind to finally "click" when it came to programming.

Harvard CS50

Not many people know this, but Harvard offers a free class that anyone can take to truly understand the ins and outs of computer science. The instructor, David Malan, (unsurprisingly) is extremely passionate about teaching the world of computers and does an exceptional job at using real life examples to help truly understand what a computer is doing under the hood.

The first few lectures are taught in C. Not C#, not C++, but C. The reason David started with C is to help his students understand how computers store memory by manually having to allocate it yourself in code (a.k.a. dynamic memory allocation), which allowed me to have a much deeper appreciation for higher-level programming languages that do all of this for you automatically.

All of the lectures are available on YouTube, but Harvard also offers a way to participate in the assignments of the class via EDX.org. The assignments linked to the classes allow you to actually put what you learned during lectures into practice by coding the solutions to different problems and testing your code to confirm that it runs correctly.

Conclusion

Out of all of the resources that I've outlined thus far, Harvard CS50's course may be the one that I recommend the most. Between the instructor's passion for teaching, the course's thorough content, and the ability to participate in assignments, it serves as a great tool to get started in your coding journey. Plus, it feels great to pretend that you're a student at Harvard.