DEV Community: Shashank

Setup Jellyfin On Raspberry Pi - Home Media Server

Shashank — Sun, 20 Apr 2025 16:05:00 +0000

In this guide, we’ll walk through setting up a personal media server using Jellyfin on a Raspberry Pi 5 running Ubuntu 24.04. We'll configure a USB drive as the media storage, mount it properly, auto-mount it at boot, and set up Samba so you can transfer media from your PC or laptop over the network. Great for anyone wanting a self-hosted Netflix-like experience.

Setting Up Jellyfin Media Server

📦 Step 1: Install Jellyfin

sudo apt update
curl -s https://repo.jellyfin.org/install-debuntu.sh | sudo bash

Once installed, Jellyfin runs on port 8096 by default.

Open it in your browser:

http://<your-pi-ip>:8096

Before setting up Jellyfin, lets go through other steps to add the media folders in Jellyfin server.

🔌 Step 2: Connect Your USB Drive

Plug in your USB drive.
Ubuntu may auto-mount it under:
```
/media/<your-username>/<usb-name>
```

👉 This location is fine temporarily, but it can change or fail to mount after a reboot — which is not reliable for a media server.

🔍 Step 3: Identify the USB Drive and Its Filesystem

To list connected drives:

lsblk

To get UUID and filesystem type:

sudo blkid

Look for your USB device, something like:

/dev/sda1: UUID="1234-ABCD" TYPE="exfat"

Copy the UUID and note the TYPE (e.g., exfat, ntfs, ext4, or vfat).

📂 Step 4: Create a Permanent Mount Point

sudo mkdir -p /media/usbdrive

usbdrive can be anything. You can use your actual usb drive name for this.

🛠 Step 5: Install Filesystem Support (if needed)

For exFAT:

sudo apt install exfat-fuse exfatprogs

For NTFS:

sudo apt install ntfs-3g

⚙️ Step 6: Configure Auto-Mount with `/etc/fstab` (Based on Filesystem Type)

Edit your fstab:

sudo nano /etc/fstab

✅ For `exfat` drives:

UUID=1234-ABCD /media/usbdrive exfat uid=jellyfin,gid=jellyfin,umask=0022,nofail,x-systemd.automount 0 0

✅ For `ext4` drives:

UUID=1234-ABCD /media/usbdrive ext4 defaults,nofail,x-systemd.automount 0 2

✅ For `ntfs` drives:

UUID=1234-ABCD /media/usbdrive ntfs-3g uid=jellyfin,gid=jellyfin,umask=0022,nofail,x-systemd.automount 0 0

✅ For `vfat` (FAT32) drives:

UUID=1234-ABCD /media/usbdrive vfat uid=jellyfin,gid=jellyfin,umask=0022,nofail,x-systemd.automount 0 0

After saving, mount all drives:

sudo mount -a

Check if the drive mounted:

ls /media/usbdrive

🔑 Step 7: Fix Permissions (if needed)

sudo chown -R jellyfin:jellyfin /media/usbdrive
sudo chmod -R 755 /media/usbdrive

🎞 Step 8: Add USB Folder to Jellyfin

Open http://<your-pi-ip>:8096
Go to Dashboard → Libraries
Create a new library (e.g. "Movies")
Set the path to:
```
/media/usbdrive
```
Save and scan the library

{:.blockquote}

If you’re more old-school, you can simply disconnect the USB drive from your Raspberry Pi and connect it to another laptop to transfer files. But if you’d prefer to avoid that hassle and transfer files over the network, stick around — we’ll set up a Samba server to make file sharing seamless and easy.

Transfering Files

We can install Samba on Ubuntu to access the USB drive over the network from other PCs or laptops. However, currently only the jellyfin user has full permission to modify or delete files on the USB drive.

To fix this, we’ll create a shared group, add both the jellyfin and pi users to it, and then mount the USB drive using this shared group to ensure proper access for both users.

📁 Step 9. Mount USB Drive as a Shared Group

Create and assign group:

sudo groupadd media
sudo usermod -aG media pi
sudo usermod -aG media jellyfin

FSTAB entry:

UUID=XXXX-XXXX  /media/usbdrive  exfat  defaults,uid=pi,gid=media,umask=0002,nofail,x-systemd.automount  0  0

Reload:

sudo umount /media/usbdrive
sudo mount -a

To check ownership:

ls -ld /media/usbdrive

🌐 Step 10. Set up Samba for File Transfers

Install Samba:

sudo apt install samba samba-common-bin

Create a new config block at the end of /etc/samba/smb.conf:

Run:

sudo nano /etc/samba/smb.conf

Add the below config

[usbonpi]
  path = /media/usbdrive
  writeable = yes
  browseable = yes
  public = no
  read only = no
  guest ok = no
  force user = pi
  force group = pi
  create mask = 0777
  directory mask = 0777
  delete readonly = yes

Reload Samba Service:
```
sudo systemctl restart smbd
```
Add a Samba user:
```
sudo smbpasswd -a pi
```

Access the share via another PC:

If you are on Mac:

Open Finder
Go to Go > Connect to Server
Type the following:
```
smb://<raspberrypi_ip>\usbonpi
```

If you are on Windows:

Open File Explorer
In the address bar, type the following:
```
\\<raspberrypi_ip>\usbonpi
```

📋 Summary

We installed Jellyfin and accessed it via a browser.
A USB drive was configured to auto-mount using fstab.
Permissions were assigned to allow both Jellyfin and the Samba-accessing user.
A Samba server was set up for easy file transfer.

✅ You're All Set! 🎉

Your Raspberry Pi is now a full-fledged Jellyfin server with a persistent, auto-mounted USB drive as the media library.

Deploy a Node.js Application Using MySQL and Prisma on a Raspberry Pi

Shashank — Sun, 05 Jan 2025 06:22:00 +0000

Deploying applications has become increasingly accessible, with a range of free and paid hosting options like Render, AWS, and DigitalOcean. However, for developers who want to learn, experiment, and deploy applications without recurring hosting fees, a Raspberry Pi offers an excellent alternative. This compact yet powerful device allows you to create your own Linux-based server for hosting web applications.

In this blog, we’ll explore how to deploy a TypeScript Node.js application using MySQL (MariaDB on Raspberry Pi) and Prisma ORM on a Raspberry Pi. Additionally, we’ll configure NGINX for reverse proxying and use Ngrok to expose the application to the internet. Let’s dive in!

Tools Overview

Raspberry Pi

A low-cost, single-board computer that runs a Linux-based operating system. It’s ideal for creating your own server for IoT or web applications.

Node.js & TypeScript

Node.js is a runtime environment for executing JavaScript on the server, and TypeScript adds static typing to JavaScript, making the codebase more maintainable.

MySQL (MariaDB on Raspberry Pi)

A popular relational database system, MariaDB is a compatible replacement for MySQL and is lightweight enough for a Raspberry Pi.

Prisma ORM

An Object-Relational Mapping (ORM) tool that simplifies database interactions with a type-safe query language and schema migrations.

NGINX

A high-performance HTTP server and reverse proxy server. It helps route traffic to your Node.js application.

Ngrok

A tunneling tool that exposes your locally hosted applications to the internet securely without complex network configurations.

Prerequisites

A working Raspberry Pi - Ensure that SSH is enabled on the Raspberry Pi and you can access it remotely.
Github Repository - Your Node.js TypeScript application should be hosted in a GitHub repository for easy deployment.
Ngrok Account - Create a free account on Ngrok to obtain an auth token for exposing your Raspberry Pi app to the internet.

Setting Up Raspberry Pi

Install the OS

Setup your Raspberry Pi with an OS like Raspberry Pi OS. Use the Raspberry Pi Imager to find other OS compatible to your Raspberry Pi.
Find the IP Address

Use a tool like Angry IP Scanner to discover your Raspberry Pi’s IP address. Ensure the Raspberry Pi is connected to the same network as your local machine.
Check Raspberry Pi Status
```
ping <IP_ADDRESS_OF_RPI>
```
SSH Into the Raspberry Pi
```
ssh <username>@<IP_ADDRESS_OF_RPI>
```
Replace <username> with your Raspberry Pi’s username and <IP_ADDRESS_OF_RPI> with the IP address and then enter the password.
Update the System
```
sudo apt update && sudo apt upgrade
```
Install Git

Check if git is installed. If not, run the below command to install git
```
sudo apt install git
```

Installing Node.js

To install node js, we will be using nvm (Node Version Manager). It allows you to quickly install and use different version of node via command line.

Install NVM

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

Verify Installation
```
nvm --version
```
Install the Latest LTS Version of Node.js
```
nvm install --lts
```

Verify Node.js and npm Installation

node --version
# v22.12.0

npm --version
# 10.9.0

Setting Up MySQL (MariaDB)

For Raspberry Pi OS, we will be installing MariaDB.

Install the MariaDB SQL Server
```
sudo apt install mariadb-server
```
Secure MariaDB Installation
```
sudo mysql_secure_installation
```
Follow the prompts to secure your database.
- Enter the current root password: - Press Enter when asked to enter the current password for the root user (since it hasn’t been set yet).
- Set the root password: - Type n when prompted to set the root password (we’ll set it later).
- Remove anonymous users: - Type Y to remove anonymous users and improve security. (For testing purposes, you can type n to keep anonymous users.)
- Disallow root login remotely: - Type n to allow root login remotely (optional but less secure).
- Remove the test database: - Type y to remove the test database and access to it. (Type n if you want to keep it.)
Login to MariaDB Client
```
sudo mysql
```
Setup a Root password for MariaDB

First, we need to tell the database server to reload the grant tables.
```
MariaDB [(none)]> FLUSH PRIVILEGES;
```
Change the root password with below query.
```
MariaDB [(none)]> ALTER USER 'root'@'localhost' IDENTIFIED BY '<new_password>';
```
Replace <new_password> with your own password.

Use the exit command to exit from MariaDB CLI.
```
MariaDB [(none)]> exit;
Bye
```
Login to MariaDB Client With Root User
```
sudo mysql -u root -p
```
Enter the password for the root user.

Setting up Database and User

Let's create a new database and a user. We will be granting all privileges to the new user for the new database we have created.

Create a Database

MariaDB [(none)]> CREATE DATABASE <database_name>

Create a New User With Password

MariaDB [(none)]> CREATE USER '<new_username>'@'localhost' IDENTIFIED BY '<new_password>';

Grant Privilege To New User Created

MariaDB [(none)]> GRANT ALL PRIVILEGES ON <database_name>.* TO '<new_username>'@'localhost';

Flush The Privileges Table
```
MariaDB [(none)]> FLUSH PRIVILEGES;
```
Exit from mysql client using exit command.
Login With New User
```
sudo mysql -u <new_username> -p
```
Enter the password you used while creating the user.
Verify User Can List The Database
```
MariaDB [(none)]> SHOW DATABASES;
```

That’s it! We will use this database and user in our application.

Setup your Node.js Application

Clone Your Github Repository
```
git clone <your_github_repo_url>
```
Navigate To Your Project Repository
```
cd <your_project_name>
```
Install Project Dependencies
```
npm install
```
Compile TypeScript Code
```
npm run build
```

Make sure you have configured the outDir property in your tsconfig.json file. This specifies the directory where the compiled JavaScript code will be generated. By default, it’s commonly set to dist, but you can customize it based on your project structure.

Setting up environment variables (Optional)

If your project uses environment variables, you need to set them on your Raspberry Pi. You can create a .env file in the root directory of your project to store all the environment variables.

Create .env File
```
touch .env
```
Update .env File
```
sudo nano .env
```
Enter your Environment Variables
```
PORT=5000
DATABASE_URL="mysql://<username>:<password>@localhost:3306/<database_name>"
```
Replace the <username>, <password> and <database_name> with the one you created in the previous steps.

Save the file by pressing Ctrl+O, then press Enter, and exit the editor using Ctrl+X.

Migrate Prisma Schema

If you are using Prisma, all the schema files will be located inside the prisma/schema directory. We will now deploy these schemas to the database.

Run the below command

npx prisma migrate deploy

This command will use the DATABASE_URL provided in the .env file to deploy the schemas to the database. You can verify the deployment by logging into the MySQL client and using the command SHOW TABLES; to list all the tables.

Setting Up PM2

PM2 is a production process manager for Node.js applications which helps in managing and keeping the application online. Install PM2 to manage your Node.js application.

npm install pm2 -g

Configuring NGINX

Install NGINX
```
sudo apt install nginx
```

Create a Site Configuration

sudo nano /etc/nginx/sites-available/<your_project_name>

Add the Below Code
```
server {
    listen 80;
    server_name <your_raspberrypi_IP>;

    location / {
        proxy_pass http://localhost:YOUR_NODE_JS_PORT;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
```
Here’s a breakdown of each part:

listen 80; This directive tells NGINX to listen on port 80, which is the default port for HTTP traffic.

server_name <your_raspberrypi_IP>; This specifies the domain name or IP address of your Raspberry Pi. Replace with the actual IP address of your Raspberry Pi. NGINX will respond to requests sent to this address.

location / { ... } This block defines how NGINX should handle requests to the root URL (/). Essentially, this tells NGINX that whenever a request is made to the root, it should be forwarded to the backend (your Node.js application) running on the specified port.

proxy_pass http://localhost:YOUR_NODE_JS_PORT; This is the key line that forwards incoming requests to your Node.js application. Replace YOUR_NODE_JS_PORT with the actual port where your Node.js app is running (for example, 5000). The requests will be sent to the Node.js application running on the same machine (localhost).

proxy_http_version 1.1; This sets the HTTP version to 1.1 for the proxy connection, which ensures better handling of certain features like WebSockets.

proxy_set_header Upgrade $http_upgrade; This header allows WebSocket connections to be upgraded, which is important for real-time applications.

proxy_set_header Connection 'upgrade'; This header is used alongside the Upgrade header to manage WebSocket connections, ensuring that the connection is properly upgraded from HTTP to WebSocket.

proxy_set_header Host $host; This passes the original Host header from the client request to the backend server. This is useful for applications that rely on the original Host header (e.g., for routing or virtual hosting).

proxy_cache_bypass $http_upgrade; This ensures that WebSocket connections bypass any caching mechanisms, allowing real-time communication to work without interference from caching.

Save the file by pressing Ctrl+O, then press Enter, and exit the editor using Ctrl+X.

Enable the Site Configuration

sudo ln -s /etc/nginx/sites-available/<your_project_name> /etc/nginx/sites-enabled/

Test NGINX Configuration

sudo nginx -t

If the test is successfull, you will see something like below:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Restart NGINX Server To Apply the Changes
```
sudo systemctl restart nginx
```
Check NGINX Server Status
```
sudo systemctl status nginx
```

Running the Application

Navigate to your project
Start Your Application Using PM2

If you have setup a script in package.json, use the below command:
```
pm2 start "npm run start"
```
Or, you can directly run you application using index.js file in your dist directory:
```
pm2 start dist/index.js
```
You can also check the logs using below command:
```
pm2 logs
```

Now, check your app by entering the IP address of your Raspberry Pi in the browser on your local machine. It should work. Make sure both your local machine and Raspberry Pi are connected to the same network; otherwise, it will not work.

Exposing Your App To The World Using Ngrok

Now that you have deployed your app to the Raspberry Pi, you can only access the app from the same network in which the Raspberry Pi is running. To expose it to the internet, we need to use port forwarding.

You can set up port forwarding using your router settings, but in this case, I will be using ngrok. Ngrok is useful for development, allowing us to run our apps for testing purposes for free.

Make sure to create an account by visiting https://dashboard.ngrok.com/login. You will need the auth token to configure ngrok on the Raspberry Pi.

Install Ngrok

curl -sSL https://ngrok-agent.s3.amazonaws.com/ngrok.asc \
| sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null \
&& echo "deb https://ngrok-agent.s3.amazonaws.com buster main" \
| sudo tee /etc/apt/sources.list.d/ngrok.list \
&& sudo apt update \
&& sudo apt install ngrok

Add your auth token to ngrok configuration file
```
ngrok config add-authtoken <your_auth_token>
```

Disable default nginx config file

sudo rm /etc/nginx/sites-enabled/default

Test NGINX configuration
```
sudo nginx -t
```
Restart NGINX server to apply the changes
```
sudo systemctl restart nginx
```
Deploy your app online
```
ngrok http 80
```
This should provide a URL like https://xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx.ngrok-free.app/ that forwards traffic to your Node.js app. You can navigate to this URL from any other network and access your application.

Summary

In this guide, we successfully deployed a TypeScript Node.js application with MySQL and Prisma on a Raspberry Pi. We configured NGINX as a reverse proxy and used Ngrok to make the application accessible over the internet. With this setup, you have your own cost-effective, self-hosted development server.

This approach is perfect for learning and experimenting with full-stack application deployment, all while gaining valuable experience in server management.

Let me know if you deploy your application using this guide—I’d love to hear about your experience! 🚀

How to Scrape Data From Goodreads Using Python and BeautifulSoup

Shashank — Fri, 06 Dec 2024 17:58:00 +0000

Web scraping is a powerful tool for gathering data from websites. Whether you’re collecting product reviews, tracking prices, or, in our case, scraping Goodreads books, web scraping provides endless opportunities for data-driven applications.

In this blog post, we’ll explore the fundamentals of web scraping, the power of the Python BeautifulSoup library, and break down a Python script designed to scrape Goodreads Choice Awards data. Finally, we’ll discuss how to store this data in a CSV file for further analysis or applications.

What is Goodreads?

Goodreads is the world’s largest platform for readers and book recommendations. It provides users with access to book reviews, author details, and popular rankings. Every year, Goodreads hosts the Goodreads Choice Awards, where readers vote for their favorite books across various genres like fiction, fantasy, romance, and more. This makes Goodreads an ideal target for web scraping to gather insights about trending books and authors.

What is Web Scraping?

Web scraping involves extracting data from websites in an automated manner. It allows you to collect and structure information for tasks such as:

Analyzing trends and patterns.
Aggregating content like reviews or articles.
Feeding machine learning models or databases.

Setting Up Your Environment

Before diving into the script, you need to install the necessary libraries.

Install Python

Make sure you have Python installed on your system.
Install Required Libraries

Install the required libraries using pip:
```
pip install beautifulsoup4
pip install requests
```
request: Allows us to send HTTP requests to a URL and retrieve the web page’s content.

BeautifulSoup: Simplifies HTML parsing and data extraction.

Once these installations are complete, you're ready to scraping!

Introduction to BeautifulSoup

BeautifulSoup is a Python library for parsing HTML and XML documents. It enables developers to navigate page structures, extract content, and transform raw HTML into a structured format.

Key Methods in BeautifulSoup

Here are a few essential methods that we will be using in our script:

BeautifulSoup(html, 'html.parser'): Initializes the parser and allows you to work with the HTML content.
soup.select(selector): Finds elements using CSS selectors, such as classes or tags.
soup.find(class_='class_name'): Locates the first occurrence of an element with a specified class.
soup.find_parent(class_='class_name'): Finds the parent tag of the current element.
soup.get('attribute'): Retrieves the value of an attribute from an element, like href or src.

For a complete list of methods, check out the BeautifulSoup documentation.

Setting Up the Script

Let’s begin by importing the necessary libraries and defining custom headers to mimic a browser. This helps avoid getting blocked by the website.

from bs4 import BeautifulSoup as bs
import requests
import re
import csv

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64)...",
    "Accept-Language": "en-US, en;q=0.5",
}

Scraping Categories and Books

We start by defining the URLs for Goodreads’ Choice Awards page and the main application. We will send a request to start_url and get the web page's content.

app_url = "https://www.goodreads.com"
start_url = "https://www.goodreads.com/choiceawards/best-books-2024"

res = requests.get(start_url, headers=HEADERS)
soup = bs(res.text, 'html.parser')

categories = soup.select('.category')

Each category contains a genre and a link to its respective page. Using soup.select, we extract all categories listed under the .category class.

Next, iterate through each category to get the genre name and its page URL.

for index, category in enumerate(categories):
    genre = category.select('h4.category__copy')[0].text.strip()
    url = category.select('a')[0].get('href')
    category_url = f"{app_url}{url}"

Here, we extract the category name (genre) and the category page URL for further processing.

We will send another request to each category_url and locate all the books under that category.

res = requests.get(category_url, headers=HEADERS)
soup = bs(res.text, 'html.parser')

category_books = soup.select('.resultShown a.pollAnswer__bookLink')

category_books will contain the list of all the books under the respective category.

Extracting Book Data

Once we have the list of books, we will be iterating over each books and extract the data.

Extract Votes

for book_index, book in enumerate(category_books):
    parent_tag = book.find_parent(class_='resultShown')
    votes = parent_tag.find(class_='result').text.strip()
    book_votes = clean_string(votes).split(" ")[0].replace(",", "")

If we see in the DOM, voting count is present in the parent element of the category element. So we need to use find_parent method to locate the element and extract the voting count.

Extract Book Title, Author and Image URL

book_url = book.get('href')
book_url_formatted = f"{app_url}{book_url}"
book_img = book.find('img')
book_img_url = book_img.get('src')
book_img_alt = book_img.get('alt')
book_title = clean_string(book_img_alt)
print(book_title)
book_name = book_title.split('by')[0].strip()
book_author = book_title.split('by')[1].strip()

Each book's URL, cover image URL, title and author are extracted.

The clean_string function ensures the title is neatly formatted. You can define it at the top of the script

def clean_string(string):
    cleaned = re.sub(r'\s+', ' ', string).strip()
    return cleaned

Extract More Book Details

To get more details about the book like rating, reviews, etc., we will be sending another request to book_url_formatted.

res = requests.get(book_url_formatted, headers=HEADERS)
soup = bs(res.text, 'html.parser')

book_rating = soup.find(class_="RatingStatistics__rating").text.strip()
print(book_rating)

book_ratings_reviews = soup.find(class_="RatingStatistics__meta").get('aria-label').strip()
book_ratings, book_reviews = get_ratings_reviews(book_ratings_reviews)
print(f"Ratings: {book_ratings}, Reviews: {book_reviews}")

book_description_elements = soup.select('.BookPageMetadataSection__description .Formatted')
if book_description_elements:
    book_description = book_description_elements[0].text
else:
    book_description = 'No description found'

Here get_ratings_reviews returns the ratings and reviews text well formatted.

You can define this function at the top of the script.

def get_ratings_reviews(text):
    # Find the substring for ratings
    ratings = text[:text.find(" ratings")].replace(",", "")

    # Find the substring for reviews
    reviews = text[text.find("and ") + 4:text.find(" reviews")].replace(",", "")

    return int(ratings), int(reviews)

By navigating to each book’s details page, additional information like ratings, reviews, and detailed descriptions is extracted. Here, we are also checking if book description element exists otherwise putting a default description so that the script does not fails.

author_avatar_url_element = soup.select('.PageSection .AuthorPreview a.Avatar img.Avatar__image')
if author_avatar_url_element:
    author_avatar_url = author_avatar_url_element[0].get('src')
else:
    author_avatar_url = 'No Avatar found'

author_description_element = soup.select('.PageSection > .TruncatedContent .Formatted')
if author_description_element:
    author_description = author_description_element[0].text
else:
    author_description = 'No description found'

print(author_description)

bookPages = soup.select_one(".FeaturedDetails p[data-testid='pagesFormat']")
if bookPages:
    pages_format = bookPages.text[:bookPages.text.find(" pages")]
else:
    pages_format = "No pages found"
print(pages_format)

publication_info = soup.select_one(".FeaturedDetails p[data-testid='publicationInfo']")
if publication_info:
    publication = publication_info.text[16:]
else:
    publication = "No publication found"
print(publication)

Here, we have also gathered author details, publication information and other metadata.

Create a Book Dictionary

Let's store all the data we have extracted for a book in a dictionary.

book_dict = {
    "category": genre,
    "votes": book_votes,
    "title": book_name,
    "description": book_description,
    "author_name": book_author,
    "author_about": author_description,
    "avatar_url": author_avatar_url,
    "pages": pages_format,
    "rating": book_rating,
    "ratings": book_ratings,
    "reviews": book_reviews,
    "publication_info": publication,
    "img_url": book_img_url,
    "book_url": f"{app_url}{book_url}"
}

We will use this dictionary to add the data in a csv file.

Storing Data in a CSV File

We will use the csv module which is a part of Python's standard library. So you don't need to install it separately.

First we need to check if this is the first entry. This check is required to add the header in the csv file in the first row.

csv_filename = "books.csv"

if index == 0 and book_index == 0:
    with open(csv_filename, "w", newline="", encoding="utf-8") as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=book_dict.keys())
        writer.writeheader()

We are using mode="w" which will create a new csv file with the header entry.

Now for all subsequent entries, we will append the data to the CSV file:

with open(csv_filename, mode="a", newline="", encoding="utf-8") as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=book_dict.keys())
    writer.writerow(book_dict)

mode="a" will append the data to CSV file.

Now, sit back, relax, and enjoy a cup of coffee ☕️ while the script runs.

Once it’s done, the final data will look like this:

You can find the complete source code in this github repository.

Summary

We have learned how to scrape Goodreads data using Python and BeautifulSoup. Starting from basic setup to storing data in a CSV file, we explored every aspect of the scraping process. The scraped data can be used for:

Data visualization (e.g., most popular genres or authors).
Machine learning models to predict book popularity.
Building personal book recommendation systems.

Web scraping opens up possibilities for creative data analysis and applications. With libraries like BeautifulSoup, even complex scraping tasks become manageable. Just remember to follow ethical practices and respect the website’s terms of service while scraping!

How to Run Playwright Tests in Jenkins Pipeline Using Docker

Shashank — Sun, 01 Dec 2024 14:09:00 +0000

Automation testing has become an integral part of modern software development, and tools like Playwright have made it seamless to perform end-to-end testing. However, executing these tests in a continuous integration pipeline can be challenging without proper setup. In this guide, we will go through the steps to run Playwright tests in a Jenkins pipeline using Docker, ensuring an efficient and scalable testing process. By the end of this post, you’ll have a clear understanding of setting up Jenkins with Docker, connecting agent nodes, and creating pipelines to build and run tests.

Prerequisites

Before starting, ensure you have the following:

Docker Desktop installed on your local machine.
Docker Hub account created for managing Docker images.

Jenkins Controller/Agent Architecture

Before starting, let us look at the architecture. Jenkins Controller/Agent Architecture allows distributed builds, where tasks are delegated to agents for execution. This architecture improves scalability and resource utilization.

Controller Node

Controller is the central component of the Jenkins architecture.
It handles:
1. Job scheduling: Assigns tasks to agent nodes.
2. User interface: Provides a web-based UI (port 8080) for users to interact with Jenkins.
3. Job configuration: Allows users to define build pipelines and workflows.
4. Result aggregation: Collects and displays build/test results.
It is not recommended to execute builds directly in the controller.

Agent Node

The agents are distributed machines responsible for executing Jenkins jobs.
These nodes are connected to the master via the Jenkins Remoting protocol over port 50000.
They can execute builds or tests depending on the tasks assigned by the master.
Agents must have the necessary tools installed based on the jobs they run.

Communication Flow

User interact with the Controller.
Controller delegates the tasks to Agents.
Agents execute the tasks.
Results are sent back to the Controller

Setting Up Jenkins Using Docker

Docker Compose Setup for Jenkins

To set up Jenkins using Docker, create a docker-compose.yml file. Organize your directory as follows:

jenkins-docker/
├── docker-compose.yml
└── volumes/
    ├── master/
    └── node/

Here’s the docker-compose.yml file:

services:
    jenkins:
        image: jenkins/jenkins:lts-jdk17
        user: root
        ports:
            - 8080:8080
            - 50000:50000
        volumes:
            - ./volumes/master:/var/jenkins_home
        environment:
            - JAVA_OPTS="-Dhudson.model.DirectoryBrowserSupport.CSP="

Explanation

jenkins: Define service which will be created by Docker Compose.
jenkins/jenkins:lts-jdk17: Docker image to create the container
user: We have specified user as root to run the container with root privileges. This is necessary for certain administrative tasks (like installing plugins, setting permissions, or managing files)
ports will map the ports on the host to the container. 8080:8080 maps the Jenkins web UI. 50000:50000 maps the port used for Jenkins agent communication
volumes: Mounts a host directory (./volumes/master) to the container's Jenkins home directory (/var/jenkins_home). This will persist Jenkins data even if the container is deleted.
JAVA_OPTS: Configure JVM options for Jenkins.
-Dhudson.model.DirectoryBrowserSupport.CSP=: Disables Jenkins Content Security Policy. Although not recommended in production, but useful for embedding resources like images or styles in Jenkins view.

Running Docker Compose

To start Jenkins, run:

docker-compose up

Once Jenkins service has started, you can acces Jenkins at http://localhost:8080 on your host machine.

Setting Up Jenkins Web Interface

Navigate to http://localhost:8080 in your browser.
Use the administrator password from the terminal (output during docker-compose up).
Install suggested plugins.
Create the first admin user and complete the setup wizard.

Once complete, the Jenkins files will be available in the volumes/master directory.

Connecting An Agent Node

Follow these steps to connect an agent:

On the Jenkins homepage, click on Set up an agent
Enter NODE1 as the node name, select Permanent Agent, and configure the Remote root directory with the path to the node directory on your host.
Leave other fields to default and click on Save.

Now you will see a new node NODE1 is created which will be displayed with the Built-in Node.

Click on the NODE1 and you will see the instructions to run the agent.

Navigate to node directory in your local machine from the terminal and run the commands provided by Jenkins to connect the node.

Tip: Set the number of executors for the Built-in Node to 0 to ensure jobs only run on connected agents.

Navigate to Built-in Node > Configure
Set Number of executers to 0 and click on Save.

Running Playwright Test With Jenkins

You can use the Playwright Cucumber Framework or your own Playwright project. For this guide, we will use Docker to build the test environment and Jenkins to execute the tests.

This framework is built using Playwright and Cucumber. Instead of using the default Playwright command to execute tests, a custom index.ts file has been created to run the tests using the Cucumber runner.

To execute the tests, run the command npm run cucumber <tag_name>

Predefined tags, such as smoke, regression, etc., are specified in the index.ts file. Replace <tag_name> with one of these predefined values to run the corresponding test suite.

We need to install below plugins inside Jenkins:

Docker
Docker Pipeline

To install the plugins,

Navigate to Manage Jenkins > Plugins > Available Plugins
Search for the above plugins and install
Once installed, restart jenkins

To run jenkins again, run docker-compose up again.

Creating a Dockerfile for Playwright

Create a Dockerfile in your project:

FROM mcr.microsoft.com/playwright:v1.48.2-noble

WORKDIR /app
COPY . /app

RUN npm install

ENTRYPOINT ["sh", "-c", "npm run cucumber ${TEST_TARGET}"]

Explanation

Base Image
FROM mcr.microsoft.com/playwright:v1.48.2-noble uses Playwright docker image to build our docker image.

Set Working Directory
WORKDIR /app sets the working directory inside the container to /app. All subsequent commands and operations (e.g., COPY, RUN) will be executed relative to this directory.

Copy Files
COPY . /app Copies the contents of the current directory (on the host) into the container’s /app directory.

Install Dependencies
RUN npm install Installs Node.js dependencies from the package.json file.

Entry Point
ENTRYPOINT Defines the command to execute when the container starts.

Setting Up Docker Hub Credentials In Jenkins

Navigate to Manage Jenkins > Credentials
Click on (global) under Domains column.
Select Kind as Username with password
Enter username and password of Docker Hub.
Enter ID (e.g., dockerhub-creds. This same id will be used to fetch username and password in the Jenkinsfile).
Click Create.

Jenkins Pipeline for Building Docker Image

Create a Jenkinsfile in your project:

pipeline {
    agent any

    stages {
        stage('Build Image') {
            steps {
                sh "docker build -t=<username>/playwright:latest ."
            }
        }

        stage('Push Image') {
            environment {
                DOCKER_HUB = credentials('dockerhub-creds')
            }
            steps {
                sh 'echo ${DOCKER_HUB_PSW} | docker login -u ${DOCKER_HUB_USR} --password-stdin'
                sh "docker push <username>/playwright:latest"
                sh "docker tag <username>/playwright:latest <username>/playwright:${env.BUILD_NUMBER}"
                sh "docker push <username>/playwright:${env.BUILD_NUMBER}"
            }
        }
    }

    post {
        always {
            sh "docker logout"
            sh "docker system prune -f"
        }
    }
}

Explanation

Agents

agent any

Specifies that the pipeline can run on any available Jenkins agent.

Stages

The pipeline has two primary stages:

stage('Build Image')

Builds a docker image from the Dockerfile in the current directory (.).
Tags the image as <username>/playwright:latest.

*Note: * <username> refers to your Docker Hub username. Replace <username> with your actual Docker Hub username wherever it is mentioned.

stage('Push Image')

Uses Jenkins credentials (dockerhub-creds) for authenticating with Docker Hub.
- ${DOCKER_HUB_USR}: The Docker Hub username.
- ${DOCKER_HUB_PSW}: The Docker Hub password.
docker push <username>/playwright:latest pushes the latest version of the Docker image to Docker Hub
docker tag <username>/playwright:latest <username>/playwright:${env.BUILD_NUMBER} tags the image with the jenkins build number (e.g., 1, 2, etc.)
docker push <username>/playwright:${env.BUILD_NUMBER} pushed the versioned image to Docker Hub

Post

post

docker logout logs out from Docker Hub
docker system prune -f cleans up unused docker data

Building a Docker Image in Jenkins pipeline

Create a New Pipeline

Navigate to the Jenkins homepage and click on New Item.
Enter an Item Name (e.g., Playwright-Docker-Build) and select Pipeline and click OK.
Under Pipeline Definition, select Pipeline script from SCM.
Select SCM to git.
Provide the Repository URL. (For private repositories, set up credentials for GitHub in Jenkins and select the credentials here.)
Specify Branch Specifier for your repository. (e.g., */main)
Under Additional Behaviours, choose Clean before checkout.
Set Script Path as Jenkinsfile (ensure the Jenkinsfile is located in the root directory of your repository).
Click on Save

Build the Image

Click on Build Now.
Once the build is successful, verify your Docker Hub repository. A Docker image with the name /playwright:latest should be pushed to your Docker Hub.

Setting Up Jenkins Pipeline for Running Tests

For running our tests, we will be creating a separate jenkins pipeline. Create another directory for running tests (e.g., playwright-runner) and include Jenkinsfile and Docker Compose file.

In the root directory, create a docker-compose.yml file:

services:
    playwright-test:
        image: <username>/playwright
        environment:
            - TEST_TARGET
            - BROWSER_CHOICE
        volumes:
            - ./reports:/app/reports

Explanation

Services

playwright-test: defines the service

Image

image: We will be using the prebuild image <username>/playwright which contains the Playwright test environment.

Environment Variables

environment: Passes enviroment variables to the container.
TEST_TARGET: This will be used to pass the specific tags to run the test.
BROWSER_CHOICE: This will be used to pass the specific browser.

Volumes
volumes: Mounts ./reports directory on host to /app/reports directory on container.

In the same directory, create a Jenkinsfile file:

pipeline {
    agent any

    parameters {
        choice choices: ['login', 'smoke', 'regression', 'faker'], name: 'TEST_TARGET'
        choice choices: ['chromium', 'firefox'], name: 'BROWSER_CHOICE'
    }

    stages {

        stage('Run Test') {
            steps {
                sh "docker-compose up --pull=always"
                script {
                    def rerunExists = sh(script: '[ -f reports/rerun.txt ] && [ -s reports/rerun.txt ]', returnStatus: true) == 0
                    if(rerunExists) {
                        error("Some tests failed.")
                    }
                }
            }
        }
    }

    post {
        always {
            sh "docker-compose down"
            archiveArtifacts artifacts: 'reports/*.html, reports/*.json', followSymlinks: false
        }
    }
}

Explanation

Agent

agent any: Run pipeline on any available agent.

Parameters

parameters: Allows users to provide input when triggering the pipeline.

Stages

stage('Run Test'): Defines the stage for the pipeline.
docker-compose up --pull=always: Launches the Dockerized testing environment defined in the docker-compose.yml file pulling the latest image.
script: Uses a shell script to check if the reports/rerun.txt file exists and is non-empty. If its not, the pipeline fails with an error.

Post

always: Defines actions that are always executed regardless of the pipeline's success or failure.
docker-compose down: Shuts down the Docker containers and cleans up any associated resources.
archiveArtifacts artifacts: Collects test reports (e.g., HTML and JSON files) from the reports directory and stores them in Jenkins as build artifacts.

Now setup a github repository and push the code.

Running Tests in Jenkins

Create New Pipeline

Go to the Jenkins homepage and click on New Item.
Enter an Item Name (e.g., Playwright-Docker-Runner) and select Pipeline and click OK.
Under Pipeline Definition, select Pipeline script from SCM.
Select SCM as git.
Provide the Repository URL for playwright-runner. (For private repositories, set up credentials for GitHub in Jenkins and select the credentials here.)
Specify the Branch Specifier for your repository (e.g., */main).
Under Additional Behaviours, choose Clean before checkout.
Set Script Path to Jenkinsfile. (ensure the Jenkinsfile is located in the root directory of your repository)
Click Save to create the pipeline.

Run Pipeline

Click Build Now.

Note: The first build will run with the default parameters specified in the Jenkinsfile, as you will not be provided with parameter choices initially.

Run with Parameters

After the initial pipeline execution, navigate back to your created pipeline (Playwright-Docker-Runner).
Click Build with Parameters.
Select the desired parameters and run the pipeline. This will execute all tests based on the provided parameters.

View Build Artifacts

Once the pipeline is executed, you can check the build artifacts attached to the pipeline job.

Conclusion

This guide has covered the complete setup to run Playwright tests in a Jenkins pipeline using Docker. From configuring Jenkins with Docker to creating pipelines for building and running tests, each step ensures a streamlined and scalable testing process. Using this setup, you can integrate your Playwright tests seamlessly into your CI/CD pipeline and maintain a robust testing environment.

DEV Community: Shashank

Setup Jellyfin On Raspberry Pi - Home Media Server

Setting Up Jellyfin Media Server

📦 Step 1: Install Jellyfin

🔌 Step 2: Connect Your USB Drive

🔍 Step 3: Identify the USB Drive and Its Filesystem

📂 Step 4: Create a Permanent Mount Point

🛠 Step 5: Install Filesystem Support (if needed)

⚙️ Step 6: Configure Auto-Mount with /etc/fstab (Based on Filesystem Type)

✅ For exfat drives:

✅ For ext4 drives:

✅ For ntfs drives:

✅ For vfat (FAT32) drives:

🔑 Step 7: Fix Permissions (if needed)

🎞 Step 8: Add USB Folder to Jellyfin

Transfering Files

📁 Step 9. Mount USB Drive as a Shared Group

🌐 Step 10. Set up Samba for File Transfers

📋 Summary

Deploy a Node.js Application Using MySQL and Prisma on a Raspberry Pi

Tools Overview

Prerequisites

Setting Up Raspberry Pi

Installing Node.js

Setting Up MySQL (MariaDB)

Setting up Database and User

Setup your Node.js Application

Setting up environment variables (Optional)

Migrate Prisma Schema

Setting Up PM2

Configuring NGINX

Running the Application

Exposing Your App To The World Using Ngrok

Summary

How to Scrape Data From Goodreads Using Python and BeautifulSoup

What is Web Scraping?

Setting Up Your Environment

Introduction to BeautifulSoup

Setting Up the Script

Scraping Categories and Books

Extracting Book Data

Storing Data in a CSV File

Summary

How to Run Playwright Tests in Jenkins Pipeline Using Docker

Prerequisites

Jenkins Controller/Agent Architecture

Setting Up Jenkins Using Docker

Docker Compose Setup for Jenkins

Running Docker Compose

Setting Up Jenkins Web Interface

Connecting An Agent Node

Running Playwright Test With Jenkins

Creating a Dockerfile for Playwright

Setting Up Docker Hub Credentials In Jenkins

Jenkins Pipeline for Building Docker Image

Building a Docker Image in Jenkins pipeline

Setting Up Jenkins Pipeline for Running Tests

Running Tests in Jenkins

Conclusion

⚙️ Step 6: Configure Auto-Mount with `/etc/fstab` (Based on Filesystem Type)

✅ For `exfat` drives:

✅ For `ext4` drives:

✅ For `ntfs` drives:

✅ For `vfat` (FAT32) drives: