DEV Community: DEVLOKER

Getting Started with Building Chrome Extensions Using React & TypeScript

DEVLOKER — Tue, 23 Jul 2024 07:00:26 +0000

Introduction

Hello, dev community!
I'm DEVLOKER, today, I want to share my recent project about a chrome extension that lets users toggle between dark and light themes on any webpage. It doesn’t matter if the page originally supports dark mode, the extension applies a CSS filter: invert to achieve the effect. This can be particularly useful for users who prefer reading in a dark mode but visit websites that don’t have built-in support for it.

You can find the complete code on my GitHub repository.

Chrome extension structure

The extension consists of several files, each playing a specific role in the overall functionality. Let's go through the main files and their purposes:

manifest.json

The manifest file is the configuration file for the Chrome extension. It includes metadata about the extension such as its name, version, description, icons, permissions, and the scripts it uses (background, content, and popup). It defines the structure and behavior of the extension, specifying how it interacts with the browser and the web pages.

manifest_version: Specifies the manifest file format version. Here, version 3 is used, which is the latest standard for Chrome extensions.
name and description: Provide the name and a brief description of the extension. These fields help users understand the purpose of the extension.
version: Indicates the current version of the extension. This is useful for version management and updates.
icons: Defines the different sizes of the extension’s icon. Icons are used in the browser toolbar and the Chrome Web Store.
permissions: Lists the permissions requested by the extension:
- tabs: Allows the extension to interact with the browser’s tabs.
- activeTab: Grants access to the currently active tab.
- storage: Enables the use of Chrome’s storage API for saving user preferences.
- scripting: Allows the extension to execute scripts in the context of web pages.
background: Configures the background service worker that handles background tasks and event listeners.
- type: "module" allows using ES6 modules in the background script.
- service_worker: Specifies the script file that acts as the background worker.
content_scripts: Defines scripts that are injected into web pages:
- matches: Determines which URLs the content script will be injected into. ensures it runs on all pages.
- js: Lists the JavaScript files to be injected. Here, content.js contains the logic to apply the dark/light mode styles.
- type: "module" indicates that the script is an ES6 module.
action: Specifies the default popup that appears when the extension’s icon is clicked. index.html is the popup’s HTML file.
host_permissions: Allows the extension to request permissions for all hosts (:///*). This is required for interacting with any webpage.

Popup

The popup is the user interface of the Chrome extension that appears when the user clicks on the extension icon in the browser toolbar. It’s typically used to provide a quick way for users to interact with the extension.

The popup’s lifecycle is short-lived compared to other components.
It is not persistent and is created anew each time it is opened.
Communication with other parts of the extension is typically done using chrome.runtime.sendMessage and chrome.runtime.onMessage.

Lifecycle

Initialization: When the user clicks the extension icon, Chrome creates a new popup window and loads the HTML, CSS, and JavaScript specified for the popup in manifest.json.
Active: While the popup is open, its scripts (JS) are active and can interact with the DOM and perform actions based on user input.
Close: The popup closes when the user clicks outside of it, or it can close programmatically. When closed, the popup’s JavaScript context is terminated.

background script

The background script runs in the background and is responsible for handling events that affect the extension, such as browser actions, alarms, or network requests. It can also manage extension-wide state and perform tasks that need to be available continuously.

It can be either a persistent background page (old model) or a service worker (new model).
It’s ideal for long-running tasks, event handling, and managing global state.
Communication is often done using chrome.runtime.sendMessage, chrome.tabs.sendMessage and chrome.runtime.onMessage, or chrome.storage for managing state.

Lifecycle

Initialization: The background script starts when the extension is first installed or when Chrome is started. It persists across browser sessions (if using persistent background pages) or is recreated each time (if using service workers).
Active: While running, it handles events and messages from other parts of the extension and can interact with the browser API.
Termination: If using a persistent background page, it remains active until Chrome is closed or the extension is uninstalled. If using a service worker, it only stays active as long as it’s needed and is terminated when idle.

content scripts

Content scripts run in the context of web pages. They can interact with the DOM of the pages they are injected into, allowing the extension to modify or read content from those pages.

They run in isolated environments and have access to the web page’s DOM but not to other parts of the extension or the Chrome API directly (except the limited chrome.runtime and chrome.storage).
They can be specified to run at different times (e.g., when the DOM is fully loaded) using run_at in manifest.json.

Lifecycle

Initialization: Content scripts are injected into web pages based on the rules specified in manifest.json (e.g., URL patterns). They start when a page matching the specified patterns is loaded.
Active: They execute their code within the page’s context and can interact with the page’s DOM. They can also communicate with the background script using chrome.runtime.sendMessage and chrome.runtime.onMessage.
Termination: Content scripts are automatically removed when the user navigates away from the page or when the page is refreshed.

Implementation

Why Choose React and TypeScript?

Using React and TypeScript for developing the Chrome extension brought numerous benefits. React’s component-based architecture and efficient rendering, combined with TypeScript’s type safety and improved developer experience, resulted in a more maintainable, scalable, and robust extension. These tools not only enhance the development process but also ensure that the extension is reliable and performant for users.

Project Folder Structure

When setting up a Chrome extension with React and TypeScript using Vite, a well-organized project folder structure is crucial. Here's an overview of the folder structure:

theme-switcher/
│
├── public/
│   ├── icons/
│   │   ├── icon-16.png
│   │   ├── icon-32.png
│   │   ├── icon-48.png
│   │   └── icon-128.png
│   └── manifest.json
│
├── src/
│   ├── scripts/
│   │   ├── service-worker/
│   │   │   └── background.ts
│   │   ├── injection/
│   │   │   └── content.ts
│   ├── types/
│   │   ├── TMessage.ts
│   │   └── TTheme.ts
│   ├── utils/
│   │   ├── helpers.ts
│   ├── constants/
│   │   └── theme.ts
│   ├── features/
│   │   └── Popup.tsx
│   ├── styles/
│   │   └── styles.css
│   ├── index.html
│   ├── index.tsx
│
├── dist/
│
├── .gitignore
├── package.json
├── tsconfig.json
└── vite.config.ts

Now let’s explain the most important chrome extension components (popup, background, content):

Popup Component (Popup.tsx)

The popup component is the user interface of the extension that appears when the extension icon is clicked. It allows users to switch between dark and light themes and is responsible for:

Initializing the Theme: Upon opening, the popup sends an "INIT_THEME" message to the background script to get the current theme and set it in the state.
Displaying the Current Theme: The popup displays the current theme (dark or light) based on the state managed in the component.
Handling User Input: When the user selects a different theme, the popup updates the state and sends an "APPLY_THEME" message to the background script to apply the new theme.

Background Script (background.ts)

The background script serves as the central hub of the extension. It runs in the background and listens for messages from the popup and content scripts. Its main responsibilities include:

Initializing the Theme: When the popup or content script sends an "INIT_THEME" message, the background script loads the stored theme from the browser's storage, updates the extension's badge, and sends the theme back to the sender.
Applying the Theme: When the user changes the theme in the popup, the background script receives the "APPLY_THEME" message. It then asks the content script to apply the new theme to the current webpage, updates the badge to reflect the new theme, and saves the theme in the browser's storage.
Updating the Badge: The badge text and color on the extension icon are updated based on the current theme (dark or light) to provide a visual indication of the active theme.

Content Script (content.ts)

The content script is injected into all web pages specified in the manifest. It interacts directly with the web pages and is responsible for:

Applying the Theme: When the content script is first loaded, it sends an "INIT_THEME" message to the background script to get the current theme. Upon receiving the theme, it applies the appropriate styles to the webpage.
Listening for Theme Changes: The content script listens for "APPLY_THEME" messages from the background script. When a theme change is detected, it updates the webpage styles accordingly.
Managing Styles: It dynamically creates and removes a style element to apply the dark mode styles to the webpage, ensuring that elements like the background, text, and images are appropriately styled for the active theme.

This schema illustrates the workflow of the extension:

Configuring vite.config.ts for Building

This section specifies the entry points for the build process. Each entry point corresponds to a different part of the Chrome extension:

popup: The HTML file that serves as the popup's entry point.
background: The TypeScript file for the background script.
content: The TypeScript file for the content script.

This section customizes the output file names based on the entry point name:

If the chunk name is "popup", it outputs as "popup.js".
If the chunk name is "background", it outputs as "background.js".
If the chunk name is "content", it outputs as "content.js".
Otherwise, it uses the default naming convention [name].js.

import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import { resolve } from "path";

// https://vitejs.dev/config/
export default defineConfig({
    plugins: [react()],
    resolve: {
        alias: {
            "@": resolve(__dirname, "src"),
        },
    },
    build: {
        rollupOptions: {
            input: {
                popup: resolve(__dirname, "index.html"),
                background: resolve(
                    __dirname,
                    "src",
                    "scripts",
                    "service-worker",
                    "background.ts"
                ),
                content: resolve(
                    __dirname,
                    "src",
                    "scripts",
                    "injection",
                    "content.ts"
                ),
            },
            output: {
                entryFileNames: (chunk) => {
                    if (chunk.name === "popup") {
                        return "popup.js";
                    }
                    if (chunk.name === "background") {
                        return "background.js";
                    }
                    if (chunk.name === "content") {
                        return "content.js";
                    }
                    return "[name].js";
                },
            },
        },
    },
});

Building the Extension

To build the extension, run the following command: npm run build
This command triggers Vite to compile and bundle the extension's code, generating the output files in the dist folder.

To test the extension in Chrome, follow these steps:

Open Chrome and go to chrome://extensions/.
Enable "Developer mode" by toggling the switch in the top right corner.
Click on the "Load unpacked" button.
Select the dist folder that was generated by the build process. This will load the unpacked extension into Chrome, allowing you to test its functionality directly in the browser.

now, let's open any website for testing (eg: https://www.google.com), and these are the results for google light/dark:

Conclusion

Creating a Chrome extension with React and TypeScript offers a robust way to build modern web tools. This guide covered developing an very simple extension that switches between dark and light modes using: React's component-based architecture and TypeScript's type-checking for enhancing code maintainability and scalability.

We also set up the project using Vite, simplifying the development and build process. The vite.config.ts configuration was essential for correctly bundling the popup, background script, and content script.

While the invert method provided a quick way to implement dark mode, it doesn't always produce perfect results, as some elements may not invert correctly. Despite this, it remains effective for most cases.

And finally, you can find the complete code on my GitHub repository. Feel free to check it out, contribute, or use it as a reference for your own projects.

Thanks for reading, and happy coding!

Mathematics secret behind AI on Digit Recognition

DEVLOKER — Sat, 15 Jun 2024 20:45:29 +0000

Introduction

Hi everyone! I’m devloker, and today I’m excited to share a project I’ve been working on: a digit recognition system implemented using pure math functions in Python. This project aims to help beginners grasp the mathematics behind AI and digit recognition without relying on high-level libraries like TensorFlow or PyTorch.
You can find the complete code on my GitHub repository.

Fundamental concepts in AI world

Artificial Intelligence (AI)

Artificial Intelligence (AI) is a broad field of computer science focused on creating systems capable of performing tasks that typically require human intelligence. These tasks include problem-solving, understanding natural language, recognizing patterns, and making decisions. AI can be categorized into several subdomains, each with its own focus and techniques.

Artificial Neural Networks (ANN)

Artificial Neural Networks are a type of machine learning model inspired by the structure and function of the human brain. They consist of layers of interconnected nodes (or neurons), each performing simple computations.

Neuron

Neurons are the basic building blocks of artificial neural networks, inspired by biological neurons in the human brain. In AI, a neuron is a mathematical function that receives one or more inputs, applies weights to these inputs, sums them up, applies an activation function, and produces an output. In the context of artificial neural networks, a neuron performs the following operations:

Input Features: The neuron takes multiple input features, each input represents a characteristic or attribute of the input data which represented as: x1 ,x2 , ..., xn.
Weights: Each input feature is associated with a weight w1, w2, ...,wn. which indicates the importance of the feature in making the prediction. During training, these weights are adjusted to learn the optimal values.
Summation Function: Each input is multiplied by its weight, the weighted inputs are summed together, often with an added bias term: z = sum(xi * wi for xi, wi in zip(x, w)) + b
Bias: The bias b is an additional parameter that allows to make adjustments that are independent of the input, which helps the model make accurate predictions.
Activation Function: This function decides whether the neuron should be fired or not based on weighted sum, introducing non-linearity to the model. Common activation functions include softmax, sigmoid, and ReLU (Rectified Linear Unit).
Output: the neuron's output is the obtained result after applying the activation function. This output can be fed as input to the next layer of neurons or can be the final output in the case of the output layer, the final output represents the decision or prediction based on the input and the weights.

These operations work together to enable a neuron to learn and make predictions, while a single neuron can only solve linearly separable problems, combining multiple neurons into layers allows the creation of more complex models capable of solving non-linear problems. This structure forms the basis of multi-layer neural networks used in deep learning.

Deep Learning (DL)

Deep Learning is a subfield of machine learning that focuses on neural networks with many layers (hence "deep" networks). These networks are capable of learning from vast amounts of data and can model complex, high-dimensional patterns. Deep learning has been particularly successful in fields like speech & image recognition, natural language processing, medical diagnosis, and game playing. These models require vast amounts of data and computational power to train effectively but can achieve remarkable accuracy and performance.

Deep learning models consist of multiple layers of neurons. The common types of layers include:

Input Layer: The first layer, which receives the initial data (e.g., pixel values of an image).
Hidden Layers: Intermediate layers that transform the input of previous layer into more abstract representations through weighted connections and activation functions.
Output Layer: The final layer, which produces the final prediction or classification (e.g., the probabilities of each digit in digit recognition).

Training deep networks involves adjusting the weights and biases of the network to minimize the error in predictions. This is done using backpropagation and optimization algorithms like gradient descent.

Forward Propagation: Calculate the output of the network for given inputs.
Loss Computation: Measure the error between the predicted output and the actual output.
Backword Propagation: Compute the gradient of the loss function with respect to each weight and bias, propagating the error backward through the network.
Weight Update: during training, the perceptron learns by adjusting its weights and bias based on the difference between the predicted output and the true output.

Types of Deep Neural Networks:

Feedforward Neural Networks (FNNs): The simplest type where connections between the nodes do not form a cycle.
Convolutional Neural Networks (CNNs): Primarily used for image processing, recognizing patterns using convolutional layers.
Recurrent Neural Networks (RNNs): Suitable for sequence data like time series or text, where outputs from previous steps are fed as inputs to the next step.
Generative Adversarial Networks (GANs): Consist of two networks (generator and discriminator) that compete against each other, useful for generating synthetic data.

Digits recognition process

Digit recognition is a classic application of neural networks where the goal is to correctly identify handwritten digits (0-9) from images. This task involves several key steps:

1. Preparing the Data

To start with digit recognition, we first need to prepare our data. We'll be using the MNIST dataset, a standard dataset consisting of 60,000 training images and 10,000 testing images of handwritten digits (0-9).

Loading the Data: Load the MNIST dataset, which contains images of handwritten digits.
Normalizing: Normalization involves scaling pixel values to a range of 0 to 1. This helps the model converge faster during training. Each pixel value, originally between 0 and 255, is divided by 255.
Reshaping: Each image in the MNIST dataset is 28x28 pixels. We'll reshape these 2D arrays into 1D vectors of 784 elements (28 * 28). This reshaped vector will serve as the input features for our model.

Here’s a sample code snippet for data preparation:

import numpy as np
from keras.datasets import mnist

# Load the MNIST dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalize the images to values between 0 and 1
X_train = X_train / 255.0
X_test = X_test / 255.0

# Reshape the images from (28, 28) to (784,)
X_train = X_train.T.reshape(-1, 784)
X_test = X_test.T.reshape(-1, 784)

X_train: A numpy array containing the training images. Each image is a 28x28 pixel grayscale image of a handwritten digit (0-9). The shape of X_train is typically (60000, 28, 28), where 60000 is the number of training images.
Y_train: A numpy array containing the labels for the training images. Each label is an integer (0-9) representing the digit shown in the corresponding training image. The shape of Y_train is typically (60000,), where 60000 is the number of training labels.
X_test: A numpy array containing the testing images. Similar to X_train, each image is a 28x28 pixel grayscale image. The shape of X_test is typically (10000, 28, 28), where 10000 is the number of testing images.
Y_test: A numpy array containing the labels for the testing images. Each label is an integer (0-9) representing the digit shown in the corresponding testing image. The shape of Y_test is typically (10000,), where 10000 is the number of testing labels.

2. Model Architecture

Our neural network will consist of an input layer, two hidden layers, and an output layer. The structure and sizes of weights and biases for each layer are as follows:

Input Layer: 784 neurons (one for each pixel in the image).
Hidden Layer 1: 10 neurons.
- Each neuron is connected with 784 input neurons from the previous layer, so have 10*784 different weights, and we can store them in a matrix W1 with size of (10, 784).
- Each neuron has own bais, in total we have 10 different biases, and we can store them in vector B1 with size of (10,1).
Hidden Layer 2: 10 neurons.
- Each neuron is connected with 784 input neurons from the previous layer, so have 10*10 different weights, and we can store them in a matrix W2 with size of (10, 10).
- Each neuron has own bais, in total we have 10 different biases, and we can store them in vector B2 with size of (10,1).
Output Layer: 10 neurons (one for each digit 0-9).
- Each digit assigns by one neuron, and each neuron represents the probability of the assigned digit.
- the predicted digit is the one corresponding to the neuron with the highest probability.

Here’s the structure of our model:

# Initialize weights and biases
W1 = np.random.rand(10, 784)
B1 = np.random.rand((10, 1))
W2 = np.random.rand(10, 10)
B2 = np.random.rand((10, 1))

3. Training the Model

Training the model involves forward propagation, loss and accuracy calculation, backward propagation, and updating weights and biases.

Initialize the neural network's weights W1, W2 and biases B1, B2 with random float between 0 and 1.
For each epoch from 1 to epochs (inclusive):
- Forward Propagation: Compute the activations A1 and A2 (output of each layer) using the current weights and biases.
- Compute error: quantifies the error in the predictions using output A2 and the true labels Y_train.
- Compute Accuracy, which is the proportion of correctly predicted labels (evaluate the performance of the model).
- Backward Propagation: compute the gradients of the error with respect to the weights and biases. Witch indicate how much each parameter needs to change to reduce the error.
- Update the model parameters: Adjust the weights W1, W2 and biases B1, B2 using the computed gradients and the learning rate.
End For Loop
Return the last version of the trained model parameters (weights W1, W2 and biases B1, B2)

Forward Propagation: Calculating the activations of each layer using the weights and biases. We'll use the ReLU (Rectified Linear Unit) activation function for the hidden layers and softmax for the output layer.

def relu(Z):
    return np.maximum(0, Z)

def softmax(Z):
 exp_z = np.exp(Z - np.max(Z))
 return exp_z / exp_z.sum(axis=0, keepdims=True)

def forward_propagation(X):
    Z1 = np.dot(W1, X) + B1
    A1 = relu(Z1)
    Z2 = np.dot(W2, A1) + B2
    A2 = softmax(Z2)
    return Z1, A1, Z2, A2

Loss and Accuracy Calculation: Using cross-entropy loss to measure the model's performance and calculating accuracy.

def compute_loss(A2, Y):
    m = Y.size
    return -np.sum(np.log(A2[Y, np.arange(m)]))/ m

def compute_accuracy(A2, Y):
    predictions = np.argmax(A2, axis=0)
    return np.sum(predictions == Y)/Y.size

Backward Propagation: Calculating gradients using derivatives to adjust the weights and biases. The chain rule is applied here.

def backward_propagation(X, Y, A1, A2):
    m = X.shape[1]
    dZ2 = A2 - Y
    dW2 = np.dot(dZ2, A1.T) / m
    dB2 = np.sum(dZ2, axis=1, keepdims=True) / m
    dZ1 = W2.T.dot(dZ2) * (A1>0)
    dW1 = np.dot(dZ1, X.T) / m
    dB1 = np.sum(dZ1, axis=1, keepdims=True) / m
    return dW1, dB1, dW2, dB2

def update_parameters(dW1, dB1, dW2, dB2, learning_rate):
    W1 -= learning_rate * dW1
    B1 -= learning_rate * dB1
    W2 -= learning_rate * dW2
    B2 -= learning_rate * dB2

Training Loop: Iteratively performing forward and backward propagation, and updating weights and biases.

learning_rate = 0.01
epochs = 1000

for epoch in range(epochs):
    A1, A2 = forward_propagation(X_train)
    loss = compute_loss(A2, y_train)
    accuracy = compute_accuracy(A2, y_train)
    dW1, dB1, dW2, dB2 = backward_propagation(X_train, y_train, A1, A2)
    W1, B1, W2, B2 = update_parameters(W1, B1, W2, B2, dW1, dB1, dW2, dB2, learning_rate)
    # print results for each 10 iterations
    if epoch % 10 == 0:
        print(f'Epoch {epoch}, Loss: {loss}, Accuracy: {accuracy}')

4. Evaluating and Reusing the Model

After training, we evaluate the model's performance on the test set and discuss how to save and reuse the model.

Saving the Model: The trained model, represented by its weights and biases, can be saved to a file for future use.

import pickle

model_parameters = {'W1': W1, 'B1': B1, 'W2': W2, 'B2': B2}
with open('digit_recognition_model.pkl', 'wb') as file:
    pickle.dump(model_parameters, file)

Loading and Evaluating the Model: Load the saved model and evaluate its performance.
In the context of digit recognition and neural networks, accuracy is a key metric used to evaluate the performance of the model. It represents the proportion of correctly predicted digits out of the total number of predictions made. High accuracy indicates that the model is effectively learning and generalizing from the training data to make correct predictions on unseen data.

with open('digit_recognition_model.pkl', 'rb') as file:
    model_parameters = pickle.load(file)

W1 = model_parameters['W1']
B1 = model_parameters['B1']
W2 = model_parameters['W2']
B2 = model_parameters['B2']

# Evaluate the model on the test set
A1, A2 = forward_propagation(X_test)
test_accuracy = compute_accuracy(A2, y_test)
print(f'Test Accuracy: {test_accuracy}')

By following these steps, we can build, train, and evaluate a neural network for digit recognition using the MNIST dataset. This process highlights the importance of data preparation, model architecture, training, and evaluation in developing effective machine learning models.

The implementation overview

In this section, we will describe the implementation of a digit recognition system using Python. The system consists of two main components:
User Interface (UI): Built using PyQt6, this provides an interactive interface for drawing digits, training the model using (epochs, target accuracy and learning rate) parameters, loading a pre-trained model, and predicting the drawn digit.
Backend Script: Contains the NeuralNetworkModel class, which handles the core functionalities of training the model and making predictions.

run main python file app.py

Configure the training parameters (epochs, target accuracy, and learning rate) and click on the "Train" button, or alternatively, load a pre-trained model using the "Load" button.

Important: For optimal results, ensure you train your model until achieving a high accuracy (e.g. greater than 95%) by setting the target accuracy to 0.95. Note that reaching high training accuracy may require a significant amount of time (several minutes or longer).

Once trained, draw a digit on the left side (e.g., 6), and click the "Recognize" button. The system will display the probability for each digit, with the highest probability indicating the most likely digit (e.g. digit: 6, probability: 97.04%).

Conclusion

By implementing digit recognition using pure math functions, we’ve demystified the math behind AI. I hope this helps you understand the fundamentals and encourages you to dive deeper into the world of machine learning.

You can find the complete code on my GitHub repository.
For further reading, check out this video But what is a neural network? | Chapter 1, Deep learning

Playwright Scraping infinite loading & pagination

DEVLOKER — Wed, 01 May 2024 10:20:08 +0000

Before to start

Web data scraping itself isn’t illegal, but it can be illegal (or in a grey area) depending on these three things:

The type of data you are scraping (personal, copyrighted…)
How you plan to use the scraped data
How you extracted the data from the website

So, before to start scraping data make sure to double-check the plan to ensure you’re conducting both legal and ethical web scraping with these checks: Am I scraping personal or copyrighted data? Am I scraping data from behind a login? Am I violating the Terms and Conditions?

If your answers to all of these questions is No, then your web scraping is legal. However, if you answer Yes to any of them, then you should take a step back and do a full legal review of your web scraping to ensure you’re not scraping the web illegally.

Check out this for more info.

Introduction

Playwright is a powerful tool developed by Microsoft, it allows developers to write reliable end-to-end tests and perform browser automation tasks with ease. What sets Playwright apart is its ability to work seamlessly across multiple browsers (Chrome, Firefox, and WebKit), it provides a consistent and efficient way to interact with web pages, extract data, and automate repetitive tasks. Moreover, it supports various programming languages such as Node.js, Python, Java, and .NET, that’s making it a versatile choice for web scraping projects.
Whether you're scraping public data for analysis, building a web crawler, or automating manual workflows, Playwright has you covered.

Scraping process

In this post, we'll explore how to harness the power of Playwright for web scraping using JavaScript, you can find the full code on GitHub repository.
In our scenario will try to extract public products from an e-commerce website, note that the scraping is not explicitly prohibit in its terms of service, and in order to avoid a lot of requests that could impact the server resources will try to extract only the first three pages of products (just for learning purposes).
Detail the steps involved in scraping process:

- Waiting for the page to load.
- Scrolling down to load more content.
- Scraping products from the current page.
- Clicking on the next page link and repeating the process.
- Save scraped data to (csv or json) file, and download photos.

Coding using playwright

Let's dive in and discover how to use playwright to extract content from multiple pages by following the next page link.

1. scrollDownUntilNoMoreContent

This function is designed to scroll down a web page until no more content is loaded. Here's a breakdown of its functionality:

Presses the "End" key to scroll to the bottom of the page.
Waits for a search container element ".search .v-lazy:nth-child(4) .search-view-item" to appear on the page.
Scrolls down the page incrementally until the bottom is reached, using a timer-based approach (setInterval) to simulate scrolling behavior.
Waits for the page to reach a stable state with no network activity "networkidle".



/**
 * Scrolls down the page until no more content is loaded.
 * @param {Page} page - The Playwright page object.
 * @param {number} nbPage - The page number for logging purposes.
 */
const scrollDownUntilNoMoreContent = async (page, nbPage) => {
    log(`scroll down until no more content in page : ${nbPage}`);

    // Presses the "End" key to scroll to the bottom of the page.
    await page.keyboard.down("End");

    // Waits for the search container element to appear on the page.
    await page.waitForSelector(searchContainerPath);

    // Scrolls down the page until the bottom is reached.
    await page.waitForFunction(async () => {
        let totalHeight = 0,
            scrollHeight = 0;
        await new Promise((resolve, reject) => {
            const distance = 100,
                delay = 100;
            const timer = setInterval(() => {
                scrollHeight = document.body.scrollHeight;
                window.scrollBy(0, distance);
                totalHeight += distance;
                if (totalHeight >= scrollHeight) {
                    clearInterval(timer);
                    resolve(true);
                }
            }, delay);
        });
    });

    // Waits for the page to reach a stable state with no network activity.
    await page.waitForLoadState("networkidle");
};

2. scrapePageContent

This function is responsible for scraping the content of a web page, particularly products, and extracting relevant information:

Calls the scrollDownUntilNoMoreContent function to ensure all content is loaded before scraping.
Utilizes Playwright's page.evaluate method to execute JavaScript code within the context of the page, extracting product data based on CSS selectors.
Constructs an array of objects containing product information, including ID, image URL, title, price,....
Returns the array of scraped product data.



/**
 * Scrapes the content of the current page.
 * @param {Page} page - The Playwright page object.
 * @param {number} nbPage - The page number for logging purposes.
 * @returns {Array} - An array containing the scraped products.
 */
const scrapePageContent = async (page, nbPage) => {
    // Scrolls down the page until no more content is loaded.
    await scrollDownUntilNoMoreContent(page, nbPage);

    // Logs the page URL being scraped.
    log(`Scraping content of page ${nbPage}: ${await page.url()}`);

    // Evaluates JavaScript in the context of the page to scrape product data.
    const currentPageProducts = await page.evaluate(
        /**
         * Extracts product information from the page.
         * @param {string} searchContainerPath - The CSS selector for the search items.
         * @returns {Array} - An array containing the scraped product data.
         */
        ({ searchContainerPath }) => {
            const items = document.querySelectorAll(searchContainerPath);
            return [...items].map((item) => {
                const product = item.querySelector(".v-card.o-announ-card");
                if (product) {
                    const imgEl = product.querySelector(
                        ".o-announ-card-image img"
                    );
                    const titleEl = product.querySelector(
                        "h3.o-announ-card-title"
                    );
                    const priceEl = product.querySelectorAll(
                        "span.price > span > div"
                    );
                    const infosEl = product.querySelectorAll(
                        "div.col.py-0.px-0.my-1 > span.v-chip"
                    );
                    const othersEl = product.querySelectorAll(
                        ".mb-1.d-flex.flex-column.flex-gap-1.line-height-1 > span"
                    );
                    return {
                        id: item.getAttribute("id"),
                        img: imgEl.getAttribute("src"),
                        title: titleEl.textContent.trim(),
                        price: {
                            value: priceEl?.[0]?.textContent.trim() ?? "",
                            unit: priceEl?.[1]?.textContent.trim() ?? "",
                        },
                        infos: Array.from(infosEl).map((info) =>
                            info.textContent.trim()
                        ),
                        state: othersEl?.[0]?.textContent.trim() ?? "",
                        timeAgo: othersEl?.[1]?.textContent.trim() ?? "",
                    };
                }
            });
        },
        { searchContainerPath }
    );

    // Logs the number of scraped results.
    log(`Scrap results: ${currentPageProducts.length}`);

    // Returns the scraped product data.
    return currentPageProducts;
};

3. navigateToNextPage

This function is responsible for navigating to the next page of search results:

Waits for the next page link to appear on the page.
Locates the next page link element “.v-paginationnext” using the provided CSS selector.
If the next page link is found:
- Clicks the link to navigate to the next page.
- Waits for the search container element to appear, ensuring the page is fully loaded.
If the next page link is not found, it throws an error indicating the inability to navigate to the next page.



/**
 * Navigates to the next page of search results.
 * @param {Page} page - The Playwright page object.
 * @param {number} nbPage - The page number for logging purposes.
 * @throws {Error} - If unable to navigate to the next page.
 */
const navigateToNextPage = async (page, nbPage) => {
    // Waits for the next page link to appear on the page.
    await page.waitForSelector(nextPageLinkPath);

    // Locates the next page link element on the page.
    const nextPageLink = await page.locator(nextPageLinkPath);

    // Checks if the next page link is found.
    if (nextPageLink) {
        // Clicks the next page link and waits for the search container element to appear.
        await Promise.all([
            nextPageLink.click(),
            page.waitForSelector(searchContainerPath),
        ]);

        // Logs the navigation to the next page.
        log(`Navigating to the next page ${nbPage}: ${await page.url()}`);
    } else {
        // Throws an error if unable to navigate to the next page.
        throw new Error(
            `Can't navigate to the next page: ${nbPage}: ${await page.url()}`
        );
    }
};

4. saveData

This function provides a convenient way to organize and store scraped data in multiple formats (JSON, CSV), facilitating further analysis and processing, it also allows optionally download images associated with the scraped products:

Creates a directory to store the results and images, ensuring it exists recursively.
Constructs the filename for the JSON and CSV files based on the product type.
Saves the scraped data to a JSON file using the saveToJsonFile function.
Saves the scraped data to a CSV file using the *saveToCsvFile * function.
If the withDownloadingImages flag is set to true:
- Constructs the path for the images directory within the results directory.
- Downloads images associated with the scraped products to the images directory using the downloadImages function.



/**
 * Saves scraped data to JSON, CSV, and optionally downloads images.
 * @param {string} productType - The type of product being saved.
 * @param {Array} products - An array containing the scraped product data.
 */
const saveData = async (productType, products) => {
    // Creates a directory to store results, including images.
    fs.mkdirSync(resultsDir, { recursive: true });
    const imagesPath = path.join(resultsDir, "images");
    fs.mkdirSync(imagesPath, { recursive: true });

    // Constructs the filename for JSON and CSV files.
    const filename = path.join(resultsDir, `${productType}`);

    // Saves scraped data to JSON file.
    saveToJsonFile(`${filename}.json`, products);

    // Saves scraped data to CSV file.
    await saveToCsvFile(path.join(`${filename}.csv`), products);

    // Optionally downloads images associated with the scraped data.
    if (withDownloadingImages) {
        await downloadImages(imagesPath, products);
    }
};

5. Main

This is the main function, it orchestrates the web scraping process by performing the following tasks:

Launches a new Chromium browser instance and opens a new page in the browser context and navigates to URL products.
Retrieves the total number of pages to scrape from the pagination element on the page.
Iterates over each page to scrape content, accumulating products in an array.
Navigates to the next page if not the last page.
Saves scraped data to JSON, CSV, and optionally downloads images.
Closes the browser and exits the process.



/**
 * Main function to orchestrate the web scraping process.
 */
(async () => {
    // Launches a new Chromium browser instance.
    const browser = await chromium.launch({ headless: false }); // Consider making headless: true for production

    // Creates a new browser context.
    const context = await browser.newContext();

    // Opens a new page in the browser context.
    const page = await context.newPage();

    try {
        // Navigates to the specified URL and waits for the page to load.
        await page.goto(url);
        await page.waitForSelector(paginationPath);

        // Retrieves the total number of pages to scrape.
        const totalPages = await page.evaluate(
            /**
             * Retrieves the total number of pages from the pagination element.
             * @param {string} paginationPath - The CSS selector for the pagination element.
             * @returns {number} - The total number of pages to scrape.
             */
            ({ paginationPath }) => {
                const paginationElement =
                    document.querySelector(paginationPath);
                return paginationElement
                    ? Number(paginationElement.getAttribute("length"))
                    : 1;
            },
            { paginationPath }
        );

        // Logs the URL, number of pages to scrape, and separator.
        log(
            `
            #######################################################
            URL: ${url}
            Pages to scrape: ${totalPages}
            #######################################################
        `,
            false
        );

        // Array to store scraped products.
        let products = [];
        const nbPages = Math.min(numberOfPagesToScrape, totalPages);
        // Iterates over each page to scrape content.
        for (let i = 1; i <= nbPages; i++) {
            const currentPageProducts = await scrapePageContent(page, i);
            products.push(...currentPageProducts);
            // Navigates to the next page if not the last page.
            if (i < nbPages) {
                await navigateToNextPage(page, i);
            }
        }

        // Logs the total number of scraped products.
        log(
            `
            #######################################################
            Total scraped products : ${products.length}
            #######################################################
        `,
            false
        );

        // Saves scraped data to JSON, CSV, and optionally downloads images.
        await saveData(productType, products);
    } catch (error) {
        // Logs any errors that occur during scraping.
        console.error("Error while scraping:", error);
    } finally {
        // Closes the browser and exits the process.
        await browser.close();
        process.exit(1);
    }
})();

You can find the full code on GitHub repository.

Conclusion

Web scraping with Playwright opens up a world of possibilities for developers seeking to extract data from the web efficiently and responsibly. By harnessing the power of Playwright's automation capabilities and cross-browser compatibility, developers can streamline their web scraping workflows and extract valuable insights from a wide range of websites, but it's essential to navigate this process with legal and ethical considerations in mind.

Implementing Authentication with JWT Tokens in React and Express.js

DEVLOKER — Sat, 20 Apr 2024 21:22:32 +0000

Hello developers!

Authentication is a crucial aspect of modern web development, ensuring that users can securely access resources and data. One popular method for implementing authentication is through the use of JSON Web Tokens (JWT). In this post, we'll delve into the concept of JWT tokens (accessToken & refreshToken), how they work, and then explore a practical implementation using React for the client-side and Express.js for the server-side.

Understanding JWT Tokens

JSON Web Tokens (JWT) are an open, industry standard RFC 7519 method for representing claims securely between two parties. They provide a compact and self-contained way to transmit information between parties as a JSON object. JWT tokens consist of three parts: a header, a payload, and a signature.

1. Header

The header typically consists of two parts: the type (typ) of token which is "JWT", and the signing algorithm (alg) being used, such as HMAC SHA256 or RSA.



{

  "typ": "JWT",

  "alg": "HS256"

}

Payload

The payload contains the claims such as the expiration-time (exp) and Issued-at (iat), which are statements about the user and additional data. Custom claims are usually also included, depending on the purpose of the token.



{

  "iat": 1713636257, // Issued-at

  "exp": 1713636557, // expiration-time

  "id": "ab86a1e1ef70dff97959067b723c5c24" // custom claim userId

}

Signature

The signature is used to verify that the sender of the JWT is who it says it is and to ensure that the message wasn't changed along the way. The signature is created by combining the encoded header, the encoded payload, a secret key, and the specified algorithm.

The three parts (header, a payload, and a signature) are encoded separately using Base64url Encoding RFC 4648, and concatenated using periods to produce the JWT, something like this jwt.io:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6ImFiODZhMWUxZWY3MGRmZjk3OTU5MDY3YjcyM2M1YzI0IiwiaWF0IjoxNzEzNjQyMDUxLCJleHAiOjE3MTM2NDIzNTF9.hfRGkwX3ASdMm-tzR5NA_vy6DrTL7i6Doxt1U99LGUE

Authentication with JWT Tokens

Access Token

In the context of authentication, the access token is a JWT token that represents the authorization granted to the user. It typically contains information such as the user's ID and any relevant permissions. Access tokens have a short lifespan to minimize the risk if they are intercepted. They are used to access protected resources on the server.

Refresh Token

Alongside the access token, a refresh token is often issued to the client during authentication. The refresh token is also a JWT token but with a longer lifespan. Its purpose is to obtain a new access token once the current one expires. Refresh tokens are securely stored on the client-side and are only sent to the server when requesting a new access token.

How they work!

When a user logs in, the server generates a JWT token and sends it back to the client. This token is then included in subsequent requests to authenticate the user. The server can verify the token's authenticity and extract the necessary information from the payload to grant access to protected resources.
When the access token expires, the client can use the refresh token to obtain a new access token without requiring the user to log in again. The server receives the refresh token, verifies its authenticity, and issues a new access token if the refresh token is valid. This process helps maintain a seamless user experience while ensuring security by regularly rotating access tokens.

So, both JWT tokens (access tokens & refresh tokens) can implement robust authentication mechanisms that protect user data while providing a seamless user experience.

Implementation with React and Express.js

In our implementation, we'll use React with Redux Toolkit on the client-side and Express.js on the server-side. Our server will consist of two parts: an authentication server and a resource server.

The authentication server: running on port 8800, will handle login, logout, and refreshing access tokens. We'll secure the login and refresh routes using express-rate-limit to prevent brute force attacks and limit the number of requests.
The resource server: running on port 8080, will expose a protected resource accessible only to authenticated users.

Code Walkthrough

Let's dive into the code to see how it all comes together. We'll start with the client-side implementation in React, where we'll manage authentication state using Redux Toolkit. Then, we'll explore the server-side implementation in Express.js, where we'll define routes for login, logout, and refresh-access-token.
You can find the complete code repository JWT-Practice, which includes detailed comments to guide you through each step.

Happy coding! 🚀

Feel free to adjust the content as needed to fit your style and the specific details of your implementation!

Future Perspectives

As we wrap up, it's essential to consider future perspectives for our authentication system. One such perspective is enabling users to logging in from different devices securely. This can be achieved by associating each JWT token with a unique device identifier and implementing session management to revoke tokens when necessary.

In conclusion, JWT tokens provide a robust and scalable solution for authentication in modern web applications. By understanding their inner workings and implementing them in our projects, we can ensure secure access to resources while providing a seamless user experience.

I hope this post provides valuable insights into authentication using JWT tokens with React and Express.js. Feel free to leave any questions or feedback in the comments below!

Deploying React Projects Located in Subdirectories to GitHub Pages Using GitHub Actions (CI/CD)

DEVLOKER — Thu, 11 Apr 2024 06:58:37 +0000

Hello developers!

Let's suppose that you have multiple React projects within the same GitHub repository, and you're looking to deploy them all to GitHub Pages while automating the deployment process for each commit, you've come to the right place!

Today, I'd like to share some of the challenges I encountered while setting up a CI/CD pipeline via GitHub Actions for recent react projects I've been working on. Specifically, I'll be focusing on a scenario where I have a GitHub repository named react-challenges, which contains multiple React projects, each residing in a subdirectory (accordion, guess-word, landing-page, tic-tac-toe, and tree).

Are you ready? Let's dive in!

1. Customize `vite.config.ts`

First you have to know that all those react projects are created using Vite, and for each of them, you need change the vite.config.ts file by adding the following configuration:

build: {
    outDir: "build",
},
base: "",

Note: The default build folder for a Vite project is dist, however in my projects I used build folder. So, If you prefer to use the default, make sure to adjust the paths accordingly in the YAML file (step 2).

2. Create `deploy.yml`

Create a deployment file named deploy.yml inside the .github/workflows directory and paste the following code:

name: Deploy static content to Pages

on:
  push:
    branches: ["main"]
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

concurrency:
  group: "pages"
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      # landing-page project
      - name: Build landing-page
        run: |
          cd landing-page
          npm install && npm run build
          cd ..
          mkdir -p deploy/landing-page
          cp -r landing-page/build/* deploy/landing-page/
      # tic-tac-toe project
      - name: Build tic-tac-toe
        run: |
          cd tic-tac-toe
          npm install && npm run build
          cd ..
          mkdir -p deploy/tic-tac-toe
          cp -r tic-tac-toe/build/* deploy/tic-tac-toe/
      # guess-word project
      - name: Build guess-word
        run: |
          cd guess-word
          npm install && npm run build
          cd ..
          mkdir -p deploy/guess-word
          cp -r guess-word/build/* deploy/guess-word/
      # accordion project
      - name: Build accordion
        run: |
          cd accordion
          npm install && npm run build
          cd ..
          mkdir -p deploy/accordion
          cp -r accordion/build/* deploy/accordion/
      # tree project
      - name: Build tree
        run: |
          cd tree
          npm install && npm run build
          cd ..
          mkdir -p deploy/tree
          cp -r tree/build/* deploy/tree/

      - name: Setup Pages
        uses: actions/configure-pages@v4
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: "./deploy"
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

What this YAML file do? after building the projects, will copy the build output of each project to a root folder named ./deploy, while the root folder will be served at the following link: https://{username}.github.io/react-challenges/.
And each project will be accessible by adding its subdirectory name to the end of the root link: https://{username}.github.io/react-challenges/{react-sub-directory}.

In my case:

3. GitHub Repository Settings

Ensure that the build and deployment source are set to GitHub Actions from Settings/Pages/Build and deployment.

Also, grant read and write permissions for Workflow permissions by checking Read and write permissions from Settings/Actions/General/Workflow permissions.

Finally, with these configurations in place, your deployment process should end successfully!

Thats all!
I really hope this post was beneficial to you and saved you some research/debug time. If you find any thing incorrect or lacking in explanation, please leave a comment and I will edit accordingly.

Feel free to fork the Github repository

Check out the deployed GitHub Pages for each project:

Happy coding and deploying!

DEV Community: DEVLOKER

Getting Started with Building Chrome Extensions Using React & TypeScript

Introduction

Chrome extension structure

manifest.json

Popup

background script

content scripts

Implementation

Why Choose React and TypeScript?

Project Folder Structure

Popup Component (Popup.tsx)

Background Script (background.ts)

Content Script (content.ts)

Configuring vite.config.ts for Building

Building the Extension

Conclusion

Mathematics secret behind AI on Digit Recognition

Introduction

Fundamental concepts in AI world

Artificial Intelligence (AI)

Artificial Neural Networks (ANN)

Neuron

Deep Learning (DL)

Digits recognition process

1. Preparing the Data

2. Model Architecture

3. Training the Model

4. Evaluating and Reusing the Model

The implementation overview

Conclusion

Playwright Scraping infinite loading & pagination

Before to start

Introduction

Scraping process

Coding using playwright

1. scrollDownUntilNoMoreContent

2. scrapePageContent

3. navigateToNextPage

4. saveData

5. Main

Conclusion

Implementing Authentication with JWT Tokens in React and Express.js

Understanding JWT Tokens

1. Header

Payload

Signature

Authentication with JWT Tokens

Access Token

Refresh Token

How they work!

Implementation with React and Express.js

Code Walkthrough

Future Perspectives

Deploying React Projects Located in Subdirectories to GitHub Pages Using GitHub Actions (CI/CD)

1. Customize vite.config.ts

2. Create deploy.yml

3. GitHub Repository Settings

1. Customize `vite.config.ts`

2. Create `deploy.yml`