DEV Community: Krishnanshu Rathore

Master the Web Scraping Game: Conquer Data with Python, Beautiful Soup & Requests!

Krishnanshu Rathore — Sat, 01 Apr 2023 14:10:19 +0000

Well, well, well, who's ready to become a web scraping master! Embrace the power of Python, Beautiful Soup, and Requests as we conquer the fascinating world of data extraction together. Let's dive right in and claim your spot in the web scraping hall of fame!

In this tutorial, we'll delve into the amazing world of Python to help you dominate the web scraping arena.

Prerequisites:

A burning desire to become a web scraping virtuoso
Basic knowledge of Python
Python 3.x installed on your faithful computer
A code editor that sparks joy, such as Visual Studio Code or Sublime Text.

Step 1: Summon Beautiful Soup and Requests to Your Arsenal

Before embarking on our epic quest, let's enlist the help of Beautiful Soup and requests libraries. Open your terminal or command prompt, and install them with pip, Python's trusty package manager:

pip install beautifulsoup4 requests

Step 2: Assemble Your Web Scraping Tools

Create a new Python file (e.g., "web_scraping_quest.py") and import the powerful libraries we just installed:

import requests
from bs4 import BeautifulSoup

Step 3: Venture into the Website's Realm

Choose a website you're eager to explore. For our adventure, we'll brave the land of "https://www.space.com/news" and uncover the enthralling titles of its articles.

To fetch the HTML content, use the requests library to make an HTTP GET request:

url = "https://www.space.com/news"
response = requests.get(url)

# Check if the website welcomed us with open arms (status code 200)
if response.status_code == 200:
    print("Success! We've gained entry!")
else:
    print("Alas! Something went awry. Status code:", response.status_code)

Step 4: Decipher the HTML Treasure Map with Beautiful Soup

We've got the HTML content! Now let's make sense of it with Beautiful Soup. Create a Beautiful Soup object to interpret the HTML treasure map:

soup = BeautifulSoup(response.text, "html.parser")

Step 5: Seek the Hidden Gems

To extract the article titles, we need to identify the HTML elements that hold them. Put on your detective hat, inspect the website's source code (right-click on the webpage and select "Inspect" or "View Page Source"), and search for the HTML tags containing the titles.

On "https://www.space.com/news", the titles are nestled within 'h3' tags with the class "article-name". To find all such elements, use the find_all() method:

article_titles = soup.find_all("h3", class_="article-name")

Step 6: Revel in Your Web Scraping Triumphs

The moment of truth has arrived! Process and display the article titles we've successfully extracted:

for i, title in enumerate(article_titles, start=1):
    print(f"{i}. {title.text.strip()}")

Complete Code:

import requests
from bs4 import BeautifulSoup

url = "https://www.space.com/news"
response = requests.get(url)

if response.status_code == 200:
    print("Success! We've gained entry!")
else:
    print("Alas! Something went awry. Status code:", response.status_code)

soup = BeautifulSoup(response.text, "html.parser")

article_titles = soup.find_all("h3", class_="article-name")

for i, title in enumerate(article_titles, start=1):
    print(f"{i}. {title.text.strip()}")

Conclusion:

Bravo, web scraping champion! You've now harnessed the power of Python, Beautiful Soup, and requests to conquer the world of web scraping. With your newly acquired skills, you're ready to embark on countless data extraction adventures. Just remember to respect each website's terms of service and robots.txt file to ensure you're gathering their data ethically and responsibly.

Your journey has just begun, and the web scraping hall of fame awaits! Keep exploring, and may you continue to triumph in the web scraping realm.

For more in-depth knowledge, visit the official documentation of Beautiful Soup and requests:

Beautiful Soup Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Requests Documentation: https://docs.python-requests.org/en/latest/

Happy web scraping!

How to build an API using Flask

Krishnanshu Rathore — Tue, 28 Mar 2023 01:24:51 +0000

In this tutorial, we will learn how to create a simple RESTful API using Flask, a lightweight web framework for Python.

We will also use SQLAlchemy, an ORM (Object-Relational Mapper) that allows us to interact with a database using Python objects. We will use SQLite as our database, but you can use any other database that SQLAlchemy supports.

What is a RESTful API?

A RESTful API (Representational State Transfer) is a way of designing web services that follow some principles:

Each resource (such as a user, a product, a post, etc.) is identified by a unique URI (Uniform Resource Identifier), such as /users/1 or /products/42.
The client can perform different operations on the resources using HTTP methods, such as GET, POST, PUT, PATCH, DELETE, etc. For example, to create a new user, the client can send a POST request to /users with the user data in the request body.To update an existing user, the client can send a PUT or PATCH request to /users/1 with the updated data in the request body. To delete a user, the client can send a DELETE request to /users/1.
The server responds with the appropriate status code and data in the response body, usually in JSON (JavaScript Object Notation) format. For example, if the user creation was successful, the server can respond with a 201 (Created) status code and the created user data in the response body.

If the user update was successful, the server can respond with a 200 (OK) status code and the updated user data in the response body. If the user deletion was successful, the server can respond with a 204 (No Content) status code and no response body.

What is Flask?

Flask is a micro web framework for Python that allows us to create web applications quickly and easily.

It has minimal dependencies and provides us with the essential tools to build web services, such as routing, request and response handling, templating, etc.

Flask is also extensible and supports various extensions that add more functionality to our applications, such as SQLAlchemy for database integration, Flask-RESTful for building RESTful APIs, Flask-JWT for authentication and authorization, etc.

What is SQLAlchemy?

SQLAlchemy is an ORM that allows us to work with databases using Python objects. It abstracts away the low-level details of SQL queries and provides us with a high-level interface to manipulate data.

SQLAlchemy supports various databases, such as SQLite, PostgreSQL, MySQL, Oracle, etc. SQLAlchemy also provides us with declarative models that define our database schema using Python classes and attributes.

We can then use these models to perform CRUD (Create, Read, Update, Delete) operations on our data.

Setting up our project:

To start our project, we need to install some dependencies:

Python 3: You can download it from https://www.python.org/downloads/ or use your preferred package manager.
Pip: A tool for installing Python packages. It should come with Python 3 by default.
Virtualenv: A tool for creating isolated Python environments. You can install it using pip install virtualenv.
Flask: Our web framework. You can install it using pip install flask.
SQLAlchemy: Our ORM. You can install it using pip install sqlalchemy.
Flask-SQLAlchemy: An extension that integrates SQLAlchemy with Flask. You can install it using pip install flask-sqlalchemy.

Next, we need to create our project directory and files:

Create a directory called flask-api and navigate to it.
Create a virtual environment called venv using virtualenv venv.
Activate the virtual environment using source venv/bin/activate on Linux/Mac or venv\Scripts\activate on Windows.
Create a file called app.py that will contain our main application code.
Create a file called models.py that will contain our database models.
Create a file called config.py that will contain our configuration settings.

Configuring our application

In our config.py file, we need to define some configuration settings for our application:

import os

# Get the absolute path of the current directory
basedir = os.path.abspath(os.path.dirname(__file__))

# Define the SQLALCHEMY_DATABASE_URI variable that tells SQLAlchemy where to find our database
# We will use SQLite for simplicity and store it in a file called app.db in our project directory
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db')

# Define the SQLALCHEMY_TRACK_MODIFICATIONS variable that tells SQLAlchemy whether to track changes to the database
# We will set it to False to avoid unnecessary overhead
SQLALCHEMY_TRACK_MODIFICATIONS = False

Creating our application

In our app.py file, we need to create our Flask application and initialize it with our configuration settings.

To interact with the database using SQLAlchemy, we need to create a db object and associate it with our Flask application. This can be achieved by importing the SQLAlchemy class from the flask_sqlalchemy module and creating a db object in a separate file called db.py.

Here is an example code snippet for creating the db object:

from flask_sqlalchemy import SQLAlchemy

# Create a SQLAlchemy object
db = SQLAlchemy()

Then, in the main application file app.py, we can create a Flask application and associate it with the db object by calling the init_app() method of the db object. Here is an example code snippet:

from flask import Flask
from db import db

# Create a Flask application with the name of the current module
app = Flask(__name__)

# Load the configuration settings from the config.py file
app.config.from_pyfile('config.py')

# Initialize the SQLAlchemy object with our application
db.init_app(app)

Defining our database models

In our models.py file, we need to define our database models using SQLAlchemy’s declarative syntax. A model is a Python class that represents a table in our database and its attributes represent the columns. We also need to import the db object which we will use to interact with our database.

For this tutorial, we will create a simple model called User that has the following attributes:

id: An integer that is the primary key of the table and uniquely identifies each user.
name: A string that stores the name of the user.
email: A string that stores the email address of the user.

Now, we can define our model as follows:

from db import db

# Define a User model that inherits from db.Model
class User(db.Model):
    # Define the id attribute as an integer column that is the primary key
    id = db.Column(db.Integer, primary_key=True)
    # Define the name attribute as a string column that is not nullable
    name = db.Column(db.String(80), nullable=False)
    # Define the email attribute as a string column that is unique and not nullable
    email = db.Column(db.String(120), unique=True, nullable=False)

    # Define a __repr__ method that returns a string representation of the user object
    def __repr__(self):
        return f'<User {self.name}>'

Creating our database

After defining our database model, the next step is to create the actual database and its tables. This can be done using SQLAlchemy's create_all method. To do this, we'll first need to activate our virtual environment (if it isn't already), and then create a file called create_db.py.

Inside create_db.py, we'll include the following code snippet:

from app import app, db

# Create and push an application context
with app.app_context():
    # Now you can use the db object
    db.create_all()

Execute the file in the terminal.

We can verify that our database and its tables have been created by looking at the app.db file in our project directory. We can also use a tool like DB Browser for SQLite (https://sqlitebrowser.org/) to inspect and manipulate our database.

Building Our API

Now that we have created our database and its model, we can start building our API. We will use Flask’s built-in routing system to define different endpoints for our API and handle different HTTP methods. We will also use Flask’s request and jsonify functions to parse and return JSON data.

We will implement the following endpoints for our API:

GET /users: Return a list of all users in JSON format.
GET /users/<id>: Return a single user with the given id in JSON format. If no user with that id exists, return a 404 (Not Found) error.
POST /users: Create a new user with the data provided in the request body in JSON format. Return the created user in JSON format with a 201 (Created) status code.
PUT /users/<id>: Update an existing user with the given id with the data provided in the request body in JSON format. Return the updated user in JSON format with a 200 (OK) status code. If no user with that id exists, return a 404 (Not Found) error.
DELETE /users/<id>: Delete an existing user with the given id. Return a 204 (No Content) status code. If no user with that id exists, return a 404 (Not Found) error.
We can add the following code to our app.py file to implement these endpoints:

from flask import Flask, jsonify, request
from db import db

# Create a Flask application with the name of the current module
app = Flask(__name__)

# Load the configuration settings from the config.py file
app.config.from_pyfile('config.py')

# Initialize the SQLAlchemy object with our application
db.init_app(app)

# Import the User model from models.py
from models import User

# Define a route for the GET /users endpoint
@app.route('/users', methods=['GET'])
def get_users():
    # Query all users from the database
    users = User.query.all()
    # Convert each user object to a dictionary
    users_dict = [user.__dict__ for user in users]
    # Remove the _sa_instance_state attribute from each dictionary
    for user_dict in users_dict:
        user_dict.pop('_sa_instance_state')
    # Return a JSON response with the list of users
    return jsonify(users_dict)

# Define a route for the GET /users/<id> endpoint
@app.route('/users/<int:id>', methods=['GET'])
def get_user(id):
    # Query a user by id from the database
    user = User.query.get(id)
    # Check if the user exists
    if user is None:
        # Return a 404 error if not found
        return jsonify({'message': 'User not found'}), 404
    else:
        # Convert the user object to a dictionary
        user_dict = user.__dict__
        # Remove the _sa_instance_state attribute from the dictionary
        user_dict.pop('_sa_instance_state')
        # Return a JSON response with the user data
        return jsonify(user_dict)

# Define a route for the POST /users endpoint
@app.route('/users', methods=['POST'])
def create_user():
    # Get the data from the request body as a dictionary
    data = request.get_json()
    # Check if the data is valid
    if 'name' not in data or 'email' not in data:
        # Return a 400 error if missing name or email
        return jsonify({'message': 'Name and email are required'}), 400
    else:
        # Create a new user object with the data
        user = User(name=data['name'], email=data['email'])
        # Add and commit the user to the database
        db.session.add(user)
        db.session.commit()
        # Convert the user object to a dictionary
        user_dict = user.__dict__
        # Remove the _sa_instance_state attribute from the dictionary
        user_dict.pop('_sa_instance_state')
        # Return a JSON response with the created user data and a 201 status code
        return jsonify(user_dict), 201

# Define a route for the PUT /users/<id> endpoint
@app.route('/users/<int:id>', methods=['PUT'])
def update_user(id):
    # Query a user by id from the database
    user = User.query.get(id)
    # Check if the user exists
    if user is None:
        # Return a 404 error if not found
        return jsonify({'message': 'User not found'}), 404
    else:
        # Get the data from the request body as a dictionary
        data = request.get_json()
        # Check if the data is valid
        if 'name' not in data or 'email' not in data:
            # Return a 400 error if missing name or email
            return jsonify({'message': 'Name and email are required'}), 400
        else:
            # Update the user object with the data
            user.name = data['name']
            user.email = data['email']
            # Commit the changes to the database
            db.session.commit()
            # Convert the user object to a dictionary
            user_dict = user.__dict__
            # Remove the _sa_instance_state attribute from the dictionary
            user_dict.pop('_sa_instance_state')
            # Return a JSON response with the updated user data and a 200 status code
            return jsonify(user_dict), 200

# Define a route for the DELETE /users/<id> endpoint
@app.route('/users/<int:id>', methods=['DELETE'])
def delete_user(id):
    # Query a user by id from the database
    user = User.query.get(id)
    # Check if the user exists
    if user is None:
        # Return a 404 error if not found
        return jsonify({'message': 'User not found'}), 404
    else:
        # Delete the user from the database
        db.session.delete(user)
        db.session.commit()
        # Return a 204 status code with no response body
        return '', 204

if __name__ == '__main__':
    app.run()

Testing our API

Now that we have implemented our API endpoints, we can test them using a tool like Postman (https://www.postman.com/) or curl (https://curl.se/). We can use these tools to send different HTTP requests to our API and inspect the responses.

To test our API, we need to do the following steps:

Run our Flask application using python app.py.
Open Postman or curl and send different requests to our API endpoints.
Check the status codes and response bodies of each request.

First let's create new users using POST method:

POST /users: Create a new user with the data provided in the request body in JSON format. Return the created user in JSON format with a 201 (Created) status code.

Request:

#User 1:

curl -X POST -H "Content-Type: application/json" -d '{"name": "Alice","email":"alice@example.com"}' http://127.0.0.1:5000/users

Response:

{}

User 2:

curl -X POST -H "Content-Type: application/json" -d '{"name": "Bob", "email": "bob@example.com"}' http://127.0.0.1:5000/users

User 3:

curl -X POST -H "Content-Type: application/json" -d '{"name": "Charlie","email": "charlie@example.com"}' http://127.0.0.1:5000/users

Now we will use GET method to return the list of users in JSON format.

GET /users:

Request:

curl http://127.0.0.1:5000/users

Response:


[
  {
    "id": 1,
    "name": "Alice",
    "email": "alice@example.com"
  },
  {
    "id": 2,
    "name": "Bob",
    "email": "bob@example.com"
  },
  {
    "id": 3,
    "name": "Charlie",
    "email": "charlie@example.com"
  }
]

GET /users/<id>: Return a single user with the given id in JSON format. If no user with that id exists, return a 404 (Not Found) error.

Request:

curl http://localhost:5000/users/1

Response:

{
  "id": 1,
  "name": "Alice",
  "email": "alice@example.com"
}

Request:

curl http://localhost:5000/users/4

Response:

{
  "message": "User not found"
}

PUT /users/<id>: Update an existing user with the given id with the data provided in the request body in JSON format. Return the updated user in JSON format with a 200 (OK) status code. If no user with that id exists, return a 404 (Not Found) error.

Request:

curl -X PUT -H "Content-Type: application/json" -d '{"name": "Alice", "email": "alice@new.com"}' http://localhost:5000/users/1

Response:

{}

Use GET to check the updated details of the user

curl http://localhost:5000/users/1

Response:

{
  "id": 1,
  "name": "Alice",
  "email": "alice@new.com"
}

Request:

curl -X PUT -H "Content-Type: application/json" -d '{"name": "Eve", "email": "eve@example.com"}' http://localhost:5000/users/4

Response:

{
  "message": "User not found"
}

DELETE /users/<id>: Delete an existing user with the given id. Return a 204 (No Content) status code. If no user with that id exists, return a 404 (Not Found) error.

Request:

curl -X DELETE http://localhost:5000/users/2

Response:

No response body.

Request:

curl -X DELETE http://localhost:5000/users/5

Response:

{
  "message": "User not found"
}

To sum up, in this tutorial we have learned how to create a simple RESTful API using Flask and SQLAlchemy. We have also learned how to perform CRUD operations on our database using Python objects and how to test our API using curl.

Flask and Flask-RESTful provide many advanced features and options for building REST APIs such as connecting to databases using flask_sqlalchemy, serialization and deserialization of data using flask_marshmallow, adding authentication and authorization using flask_jwt_extended, and generating interactive documentation for the API using flask_swagger_ui.

For more information and examples, you can refer to the official documentation of Flask and Flask-RESTful available at:

I hope you enjoyed this tutorial and found it useful. Happy coding!

How to Create a Simple Web App with Python and Flask in 8 Easy Steps

Krishnanshu Rathore — Thu, 23 Mar 2023 12:25:28 +0000

Creating a web app using Python and Flask is a fun and rewarding project that can help you learn web development skills and showcase your creativity.

In this article, I will show you how to create a simple web app that displays HTML text on the browser, using Flask as the web framework and Python as the programming language.

Flask is a lightweight and extensible Python web framework that provides useful tools and features for creating web applications. It gives developers flexibility and is an accessible framework for new developers because you can build a web application quickly using only a single Python file.

Flask is also easy to install and run, and supports various extensions to add more functionality to your web app.

To create a web app using Python and Flask, you will need the following:

A local Python 3 programming environment. You can follow the tutorial for your distribution in How To Install and Set Up a Local Programming Environment for Python 3 series.
An understanding of basic Python 3 concepts, such as data types, lists, functions, and other such concepts.
An understanding of basic HTML concepts.
A text editor or an IDE (Integrated Development Environment) of your choice to write and edit your code.

The steps to create a web app using Python and Flask are as follows:

Install Flask using pip, the package installer for Python. You can do this by running the following command in your terminal:

pip install flask

Create a project directory for your web app. You can name it anything you like, but in this tutorial we will call it flask_app.

In your project directory, create a Python file and name it app.py. This will hold all the code for your web app.

In your app.py file, import the Flask class from the flask module:

from flask import Flask

Create an instance of the Flask class and pass it the name of the current module (__name__) as an argument. This will create a Flask object that represents your web app:
app = Flask(__name__)

Define a route for your web app using the @app.routedecorator. A route is a URL path that maps to a function that handles what happens when a user visits that URL.

For example, if you want to display a welcome message on the home page of your web app, you can define a route for / (the root URL) and write a function that returns the message as HTML text:

@app.route("/")
def main():
    return "<h1>Welcome to my web app</h1>"

Run your web app using the app.run() method. This will start a local server that hosts your web app on your machine. You can also pass some optional arguments to this method, such as debug=True to enable debug mode, which allows you to see error messages and reload changes automatically:

if __name__ == "__main__":
    app.run(debug=True)

Now run your program, open your browser and go to http://localhost:5000/ or http://127.0.0.1:5000/ to see your web app in action. You should see something like this:

Congratulations! You have created your first web app using Python and Flask.

You can now modify your code to add more routes and functions to handle different requests from users. You can also use templates to render dynamic HTML pages using variables, loops, conditions, and other Python concepts. You can also use databases to store and retrieve data for your web app.

To learn more about how to use Flask features and extensions, you can check out these resources:

I hope this article was helpful and informative for you.

Effective Reading of Technical Documentation: Tips and Tricks

Krishnanshu Rathore — Fri, 10 Feb 2023 00:42:05 +0000

Reading documentation can be a daunting task, especially if it is lengthy and technical. But with the right approach, you can make the process much easier and even enjoyable. Here are some tips to help you read documentation effectively.

Start with an Overview: Before diving into the details, take a look at the overview or summary of the document. This will give you a general idea of what the document is about and help you prioritize what sections to focus on.

Break it down into smaller sections: If the document is lengthy, break it down into smaller sections and tackle one section at a time. This will make it less overwhelming and easier to understand.

Use headings and subheadings: Documentation often has headings and subheadings to organize information. Use these to quickly find the information you need and get an idea of the structure of the document.

Take Notes: Taking notes while reading will help you retain information better and refer back to it later. Highlight or underline key information, jot down questions, and summarize important points.

Ask questions: If there is something you don't understand, don't be afraid to ask questions. Reach out to the author, ask on forums, or clarify with colleagues.

Practice and experimentation: The best way to understand something is to try it yourself. Practice what you have learned and experiment with different scenarios.

Get feedback: Seek feedback from peers or more experienced individuals to ensure that you have a good understanding of the information.

In conclusion, effective reading of documentation requires patience, focus, and a systematic approach. By breaking down the document into smaller sections, using headings, taking notes, asking questions, practicing and experimenting, and seeking feedback, you can make the process less overwhelming and gain a deeper understanding of the information.