DEV Community: abbazs

pyxclip: Cross-Platform Clipboard for Python, Backed by Rust

abbazs — Sun, 22 Mar 2026 09:41:22 +0000

pyxclip: Cross-Platform Clipboard for Python, Backed by Rust

Most Python clipboard libraries function as thin wrappers for system-level command-line utilities. For instance, pyperclip relies on external binaries—invoking xclip on Linux and pbcopy on macOS—while using ctypes for Windows. Similarly, xerox (last updated in 2018) follows this pattern. Because these libraries depend on external processes, they are limited to text-only transfers and fail if the underlying system tool is missing.

The typical error looks like this:

PyperclipException: Pyperclip could not find a copy/paste mechanism for your system.

On Linux that means installing xclip or xsel manually. On modern desktops running Wayland, even with those tools installed, pyperclip silently fails to copy. A freelancer gig was posted specifically to fix this error. A Python Forum thread from April 2025 documents someone who gave up on pyperclip entirely on Ubuntu 24.04 and wrote their own xclip wrapper in Python.

What pyxclip does differently

pyxclip compiles a Rust extension module using PyO3 and the arboard crate. The compiled binary talks to platform APIs directly: X11 or Wayland on Linux, NSPasteboard on macOS, Win32 on Windows. No subprocess calls, no xclip, no xsel, no pbcopy.

It supports text, images, and file paths in a single package.

import pyxclip

# Text
pyxclip.copy("Hello, world!")
print(pyxclip.paste())

# Clear
pyxclip.clear()

The copy() function dispatches based on argument type. A str copies as text. A (width, height, bytes) tuple copies as an RGBA image. A Path or list of Path objects copies file references. The paste() function returns whatever the clipboard contains.

from pathlib import Path
from PIL import Image

# Images (RGBA pixel data)
img = Image.open("photo.png").convert("RGBA")
pyxclip.copy((img.width, img.height, img.tobytes()))

# File paths
pyxclip.copy(Path("/tmp/report.pdf"))
files = [Path("/tmp/a.pdf"), Path("/tmp/b.pdf")]
pyxclip.copy(files)

# Paste returns the appropriate type
result = pyxclip.paste()  # str, dict, or list[str]

A str is always treated as text, never as a file path. If you need to copy a file reference, wrap it in Path. All paths are canonicalized to absolute before copying.

Known limitation

File path copying works for programmatic access (copy and paste back in Python or terminal). Pasting into desktop file managers is a separate problem. GNOME's Nautilus expects x-special/gnome-copied-files alongside text/uri-list, which the arboard crate doesn't set. This is tracked in issue #2.

Install

pip install pyxclip

Python 3.8 through 3.14. Prebuilt wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64), and Windows (x86_64). No C compiler or system packages needed. MIT licensed.

Source: github.com/abbazs/pyxclip

If you find this useful, consider giving it a star on GitHub. Issues and PRs are welcome.

A Hidden Character Mystery

abbazs — Sat, 22 Mar 2025 05:40:51 +0000

This morning I was set out to refresh my geldata knowledge, I attempted to set up Geldata using Docker, following the official documentation. My initial Docker command was:

docker run --name gel -p 5650:5650 -e GEL_SERVER_PORT=5650 -e GEL_SERVER_PASSWORD=learn123 -e GEL_SERVER_TLS_CERT_MODE=generate_self_signed -e GEL_SERVER_USER=learn -e GEL_SERVER_DEFAULT_BRANCH=learn geldata/gel

However, I consistently encountered an authentication error:

ERROR gel_tokio::raw::connection] Connection error: AuthenticationError: authentication failed
gel error: Connection error: AuthenticationError: authentication failed

After troubleshooting for 90 minutes, I experimented with a modified command, replacing GEL_SERVER with EDGEDB_SERVER:

docker run --name gel -p 5650:5650 -e EDGEDB_SERVER_PORT=5650 -e EDGEDB_SERVER_PASSWORD=learn123 -e EDGEDB_SERVER_TLS_CERT_MODE=generate_self_signed -e EDGEDB_SERVER_USER=learn -e EDGEDB_SERVER_DEFAULT_BRANCH=learn geldata/gel

This command unexpectedly succeeded. I then reverted back to using GEL_SERVER with the same values:

docker run --name gel -p 5650:5650 -e GEL_SERVER_PORT=5650 -e GEL_SERVER_PASSWORD=learn123 -e GEL_SERVER_TLS_CERT_MODE=generate_self_signed -e GEL_SERVER_USER=learn -e GEL_SERVER_DEFAULT_BRANCH=learn geldata/gel

This also worked, which was puzzling.

To understand the mystery, I asked ChatGPT. It identified a potential issue: a zero-width space character in the GEL_SERVER_USER environment variable of my initial command.

To visualize this, consider the following lines:

GEL_SERVER_USER=learn
GEL_SERVER_USER=learn

ChatGPT remommended using a extension like the Gremlins in vscode, the zero-width space in the second line becomes apparent. This hidden character was causing the authentication failure.

I knew that copying text from internet can have such issues, but it has been a very while since I last encountered such an issue, however zero-width space character is a new learning.

Managing MQTT User Information with PostgreSQL in a Separate Container

abbazs — Thu, 09 Jan 2025 07:27:52 +0000

This guide will walk you through setting up an MQTT broker with PostgreSQL for managing users. By using Docker, you'll create a modular and scalable environment that integrates well with other systems.

Prerequisites

Installing Docker and Docker-Compose on Ubuntu

To install Docker and Docker-Compose from their official Git repositories on Ubuntu, follow these steps:

Update the System Packages

   sudo apt update
   sudo apt upgrade -y

Install Required Dependencies

   sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

Add Docker's Official GPG Key

   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Add Docker's Stable Repository

   echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

   sudo apt update
   sudo apt install -y docker-ce docker-ce-cli containerd.io

Verify Docker Installation

   docker --version

Install Docker-Compose

   sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   sudo chmod +x /usr/local/bin/docker-compose

Verify Docker-Compose Installation

   docker-compose --version

You are now ready to proceed with the setup.

Before starting, ensure you have the following installed:

Docker
docker-compose

If you're new to Docker, consider reading Docker's official documentation to familiarize yourself with its basics.

Step-by-Step Guide

1. Create a `docker-compose.yml` File

Begin by creating a docker-compose.yml file in a new project directory. This file will define the services for the MQTT broker, PostgreSQL, and a FastAPI application to manage users.

Here’s the content of the docker-compose.yml file:

version: '3.8'
networks:
  mqtt-net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.100.10.0/24
services:
  mqtt-broker:
    image: eclipse-mosquitto:latest
    user: mosquitto
    volumes:
      - type: bind
        source: ./config/
        target: /mosquitto/config/
        read_only: false
      - type: bind
        source: ./log/
        target: /mosquitto/log/
        read_only: false
      - type: volume
        source: data
        target: /mosquitto/data/
    ports:
      - target: 1883
        published: 1883
        protocol: tcp
        mode: host
      - target: 9001
        published: 9001
        protocol: tcp
        mode: host
    networks:
      mqtt-net:
        ipv4_address: 172.100.10.10
    environment:
      - MOSQUITTO_AUTH_PLUGIN=auth_plugin_pgsql.so
      - PGSQL_HOST=pgsql
      - PGSQL_USER=mqtt_user
      - PGSQL_PASSWORD=mqtt_password
      - PGSQL_DATABASE=mqtt_users

  pgsql:
    image: postgres:latest
    container_name: postgres-container
    environment:
      POSTGRES_USER: mqtt_user
      POSTGRES_PASSWORD: mqtt_password
      POSTGRES_DB: mqtt_users
    volumes:
      - pg_data:/var/lib/postgresql/data
    networks:
      mqtt-net:
        ipv4_address: 172.100.10.20

  fastapi-service:
    build:
      context: ./fastapi
    container_name: fastapi-container
    ports:
      - "8000:8000"
    networks:
      mqtt-net:
        ipv4_address: 172.100.10.30
    environment:
      - DATABASE_URL=postgresql+psycopg2://mqtt_user:mqtt_password@pgsql:5432/mqtt_users
volumes:
  data:
    name: "mqtt-broker-data"
  pg_data:
    name: "pg-data"

2. Create a FastAPI Service

a. Create a Directory for FastAPI

In your project directory, create a folder named fastapi. Inside this folder, you'll define the FastAPI application and its Docker configuration.

b. Create a Dockerfile for FastAPI

In the fastapi directory, create a Dockerfile with the following content:

FROM python:3.10-slim

# Set the working directory
WORKDIR /app

# Copy FastAPI app files
COPY . /app

# Install dependencies
RUN pip install --no-cache-dir fastapi uvicorn sqlalchemy psycopg2-binary pydantic

# Expose FastAPI default port
EXPOSE 8000

# Command to run FastAPI
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

3. Write the FastAPI Application

Create a file named main.py in the fastapi directory. This file will define the FastAPI application. Here’s an example:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from sqlalchemy import create_engine, Column, String, Table, MetaData
from sqlalchemy.orm import sessionmaker

app = FastAPI()

# Database configuration
DATABASE_URL = "postgresql+psycopg2://mqtt_user:mqtt_password@pgsql:5432/mqtt_users"
engine = create_engine(DATABASE_URL)
metadata = MetaData()

mqtt_users = Table(
    'mqtt_users', metadata,
    Column('username', String, primary_key=True),
    Column('password', String, nullable=False)
)

metadata.create_all(engine)
Session = sessionmaker(bind=engine)

# Pydantic model for user input
class User(BaseModel):
    username: str
    password: str

@app.get("/users")
def list_users():
    with Session() as session:
        result = session.query(mqtt_users).all()
        users = [{"username": row.username} for row in result]
        return {"users": users}

@app.post("/users")
def add_user(user: User):
    with Session() as session:
        existing_user = session.query(mqtt_users).filter_by(username=user.username).first()
        if existing_user:
            raise HTTPException(status_code=400, detail="User already exists")
        session.execute(mqtt_users.insert().values(username=user.username, password=user.password))
        session.commit()
        return {"message": "User added successfully"}

@app.delete("/users/{username}")
def remove_user(username: str):
    with Session() as session:
        result = session.query(mqtt_users).filter_by(username=username).delete()
        session.commit()
        if not result:
            raise HTTPException(status_code=404, detail="User not found")
        return {"message": "User removed successfully"}

@app.put("/users/{username}/reset-password")
def reset_password(username: str, new_password: str):
    with Session() as session:
        result = session.query(mqtt_users).filter_by(username=username).update({"password": new_password})
        session.commit()
        if not result:
            raise HTTPException(status_code=404, detail="User not found")
        return {"message": "Password reset successfully"}

4. Start the Services

Run the following command in your project directory to build and start the containers:

docker-compose up --build

5. Test the FastAPI Service

Open your browser or a tool like Postman and navigate to http://localhost:8000. Use the provided endpoints to manage users:

GET /users: List all users.
POST /users: Add a new user.
DELETE /users/{username}: Remove a user.
PUT /users/{username}/reset-password: Reset a user’s password.

Essential FFmpeg Recipes for Video Manipulation

abbazs — Thu, 09 Jan 2025 07:04:31 +0000

FFmpeg is a powerful command-line tool for handling multimedia data. Whether you're trimming, splitting, or editing videos, FFmpeg provides an efficient way to achieve your goals. In this article, we’ll explore common FFmpeg recipes for video manipulation.

Installing FFmpeg

Before diving into the recipes, ensure you have FFmpeg installed on your system. Here’s how you can install it on Ubuntu and Windows:

On Ubuntu:

Update the package list:

   sudo apt update

Install FFmpeg:

   sudo apt install ffmpeg

Verify the installation:

   ffmpeg -version

On Windows:

Download the FFmpeg executable from the official website.
Extract the downloaded ZIP file to a directory (e.g., C:\ffmpeg).
Add FFmpeg to your system PATH:
- Go to System Properties > Advanced > Environment Variables.
- Under System Variables, select Path and click Edit.
- Add the path to the FFmpeg bin directory (e.g., C:\ffmpeg\bin).
Verify the installation by opening a command prompt and running:

   ffmpeg -version

Now that you have FFmpeg installed, let’s explore some essential recipes for video manipulation.

1. Removing the First 5 Minutes

To remove the first 5 minutes of a video:

ffmpeg -i input.mp4 -ss 00:05:00 -c copy output.mp4

-ss 00:05:00: Skips the first 5 minutes.
-c copy: Copies the streams without re-encoding for speed.

2. Removing the Last 5 Minutes

To remove the last 5 minutes, you need to calculate the duration of the video and extract everything except the last 5 minutes:

ffmpeg -i input.mp4 -to $(($(ffmpeg -i input.mp4 2>&1 | grep Duration | awk '{print $2}' | tr -d , | cut -d: -f1)*3600 + $(ffmpeg -i input.mp4 2>&1 | grep Duration | awk '{print $2}' | tr -d , | cut -d: -f2)*60 + $(ffmpeg -i input.mp4 2>&1 | grep Duration | awk '{print $2}' | tr -d , | cut -d: -f3) - 300)) -c copy output.mp4

Here, 300 represents the last 5 minutes in seconds.

3. Copying the First 5 Minutes

To extract only the first 5 minutes of a video:

ffmpeg -i input.mp4 -t 00:05:00 -c copy output.mp4

-t 00:05:00: Specifies the duration to copy.

4. Copying the Last 5 Minutes

To copy the last 5 minutes of a video, calculate the starting time:

ffmpeg -i input.mp4 -ss <start_time> -c copy output.mp4

Replace <start_time> with the total duration minus 5 minutes. Use the following command to calculate the duration:

ffmpeg -i input.mp4

5. Removing a Section Between 5 Minutes and 8 Minutes

To remove a specific section of the video:

Extract the part before 5 minutes:

ffmpeg -i input.mp4 -to 00:05:00 -c copy part1.mp4

Extract the part after 8 minutes:

ffmpeg -i input.mp4 -ss 00:08:00 -c copy part2.mp4

Concatenate the parts:

Create a text file (file_list.txt) with the following content:

file 'part1.mp4'
file 'part2.mp4'

Run:

ffmpeg -f concat -safe 0 -i file_list.txt -c copy output.mp4

6. Copying Video Between 5 Minutes and 8 Minutes

To copy only a specific section:

ffmpeg -i input.mp4 -ss 00:05:00 -to 00:08:00 -c copy output.mp4

7. Splitting a Large Video into 20MB Chunks

To split a video into chunks of approximately 20MB:

ffmpeg -i input.mp4 -f segment -segment_size 20M -c copy output%03d.mp4

-segment_size 20M: Specifies the maximum size of each segment.

8. Splitting a Large Video into 5-Minute Chunks

To split a video into 5-minute segments:

ffmpeg -i input.mp4 -f segment -segment_time 300 -c copy output%03d.mp4

-segment_time 300: Specifies the duration of each segment in seconds (5 minutes = 300 seconds).

Final Thoughts

These FFmpeg recipes provide a foundation for common video editing tasks. By combining options and filters, you can customize these commands for more advanced use cases. Explore the FFmpeg documentation to dive deeper into its capabilities.

Happy editing!

Merge PDFs Recursively - Python

abbazs — Tue, 24 Dec 2024 06:17:01 +0000

Introduction

Merging multiple PDF files into a single document can be a tedious task, especially if the files are spread across multiple directories. With Python, this task becomes seamless and automated. In this tutorial, we’ll create a command-line interface (CLI) tool using PyPDF2 and click to merge all PDF files in a directory (including its subdirectories), while excluding specific directories like .venv and .git.

Prerequisites

Before getting started, ensure you have the following:

Python: Version 3.7 or above.
pip: Python's package manager.
Required Libraries:
- Install PyPDF2 for PDF manipulation:
```
 pip install PyPDF2
```

Install click for creating the CLI:
```
 pip install click
```

Code Walkthrough

Here’s the complete code for our CLI tool:

import click
from pathlib import Path
from PyPDF2 import PdfMerger
import os

EXCLUDED_DIRS = {".venv", ".git"}

@click.command()
@click.argument("directory", type=click.Path(exists=True, file_okay=False, path_type=Path))
@click.argument("output_file", type=click.Path(dir_okay=False, writable=True, path_type=Path))
def merge_pdfs(directory: Path, output_file: Path):
    """
    Merge all PDF files from DIRECTORY and its subdirectories into OUTPUT_FILE,
    excluding specified directories like .venv and .git.
    """
    # Initialize the PdfMerger
    merger = PdfMerger()

    # Walk through the directory tree, including the base directory
    for root, dirs, files in os.walk(directory):
        # Exclude specific directories
        dirs[:] = [d for d in dirs if d not in EXCLUDED_DIRS]

        # Convert the root to a Path object
        current_dir = Path(root)

        click.echo(f"Processing directory: {current_dir}")

        # Collect PDF files in the current directory
        pdf_files = sorted(current_dir.glob("*.pdf"))

        if not pdf_files:
            click.echo(f"No PDF files found in {current_dir}")
            continue

        # Add PDF files from the current directory
        for pdf in pdf_files:
            click.echo(f"Adding {pdf}...")
            merger.append(str(pdf))

    # Write the merged output file
    output_file.parent.mkdir(parents=True, exist_ok=True)
    merger.write(str(output_file))
    merger.close()

    click.echo(f"All PDFs merged into {output_file}")

if __name__ == "__main__":
    merge_pdfs()

How It Works

Directory Traversal:
- The os.walk() function recursively traverses the specified directory.
- Specific directories (e.g., .venv, .git) are excluded using a filter on dirs.
PDF File Collection:
- The current_dir.glob("*.pdf") collects all PDF files in the current directory.
Merging PDFs:
- PdfMerger from PyPDF2 is used to append all PDFs.
- The merged output is written to the specified file.
CLI Integration:
- The click library makes it easy to provide directory and output file paths as arguments.

Running the Tool

Save the code to a file, e.g., merge_pdfs.py. Run it from the terminal as follows:

python merge_pdfs.py /path/to/directory /path/to/output.pdf

Example

Suppose you have the following directory structure:

/documents
├── file1.pdf
├── subdir1
│   ├── file2.pdf
├── subdir2
│   ├── file3.pdf
├── .git
│   ├── ignored_file.pdf

Run the tool as follows:

python merge_pdfs.py /documents /merged.pdf

This will merge file1.pdf, file2.pdf, and file3.pdf into merged.pdf, skipping .git.

Features

Recursive Merging:
- The tool automatically includes PDFs from all subdirectories.
Directory Exclusions:
- Excludes directories like .venv and .git to avoid irrelevant files.
Sorted Merging:
- Ensures PDFs are added in sorted order for consistent results.
CLI Simplicity:
- Provides an intuitive interface for users to specify input and output paths.

Notes and Limitations

Large Files:
- Merging a large number of PDFs might consume significant memory. Test with smaller datasets first.
PDF Compatibility:
- Ensure all input PDFs are valid and not corrupted.
Custom Exclusions:
- Modify the EXCLUDED_DIRS set to exclude additional directories as needed.

Conclusion

This tutorial demonstrates how to automate the merging of PDFs from a directory structure using Python. The provided CLI tool is flexible and can be adapted for more complex workflows. Try it out and let us know how it works for you!

Happy coding! 🚀

Replace Text in PDFs Using Python

abbazs — Tue, 24 Dec 2024 06:12:21 +0000

Introduction

Manipulating PDFs can be a challenging task due to their complex structure, but with Python and the PyMuPDF library, you can perform tasks like searching for text, replacing it, and saving the modified PDF. In this tutorial, we’ll create a Python CLI tool that allows you to find and replace text in a PDF, while preserving the original font, size, and style as closely as possible.

Prerequisites

Before you begin, ensure you have the following installed:

Python: Version 3.7 or above.
pip: Python's package manager.
PyMuPDF: A Python library for working with PDFs.

Install PyMuPDF using pip:

pip install pymupdf

Additionally, we’ll use the click library to create a user-friendly command-line interface (CLI):

pip install click

Code Walkthrough

Here’s the complete code for our CLI tool:

import click
from pathlib import Path
import fitz  # PyMuPDF

@click.command()
@click.argument("input_pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.argument("output_pdf", type=click.Path(dir_okay=False, writable=True, path_type=Path))
@click.argument("find_text", type=str)
@click.argument("replace_text", type=str)
def replace_text_in_pdf(input_pdf: Path, output_pdf: Path, find_text: str, replace_text: str):
    """
    Replace FIND_TEXT with REPLACE_TEXT in INPUT_PDF and save the result to OUTPUT_PDF.
    """
    # Open the input PDF
    doc = fitz.open(str(input_pdf))

    for page_num, page in enumerate(doc, start=1):
        # Search for occurrences of find_text
        instances = page.search_for(find_text)

        if not instances:
            click.echo(f"No occurrences of '{find_text}' found on page {page_num}.")
            continue

        click.echo(f"Found {len(instances)} occurrences on page {page_num}. Replacing...")

        for rect in instances:
            # First, redact (remove) the original text
            page.add_redact_annot(rect)
            page.apply_redactions()

            # Default values for text properties
            font = "helv"  # Default to Helvetica
            font_size = 7.0  # Default size
            color = (0, 0, 0)  # Default to black

            # Normalize the color values to range 0 to 1
            normalized_color = tuple(c / 255 for c in color) if isinstance(color, tuple) else (0, 0, 0)

            # Calculate the baseline position for text insertion
            baseline = fitz.Point(rect.x0, rect.y1 - 2.2)  # Adjust the -2 offset as needed

            # Insert the new text at the adjusted position
            page.insert_text(
                baseline,
                replace_text,
                fontsize=font_size,
                fontname=font,
                color=normalized_color,
            )
            click.echo(f"Replaced '{find_text}' with '{replace_text}' on page {page_num}.")

    # Save the modified PDF
    doc.save(str(output_pdf))
    click.echo(f"Modified PDF saved to {output_pdf}.")

if __name__ == "__main__":
    replace_text_in_pdf()

How It Works

Searching for Text:
The page.search_for(find_text) method identifies all occurrences of the specified text and returns their bounding rectangles.
Redacting Original Text:
The page.add_redact_annot(rect) and page.apply_redactions() methods remove the original text from the PDF without leaving artifacts.
Inserting Replacement Text:
Using page.insert_text(), we add the replacement text at the same location as the original, maintaining as much visual similarity as possible.
Saving the PDF:
Finally, the modified document is saved to the specified output file.

Running the Tool

Save the code to a file, e.g., replace_text_pdf.py. Then, run it from the terminal as follows:

python replace_text_pdf.py input.pdf output.pdf "find_text" "replace_text"

Example

Suppose you have a PDF named example.pdf with the word Python in it, and you want to replace it with PyMuPDF. Run:

python replace_text_pdf.py example.pdf modified_example.pdf "Python" "PyMuPDF"

Important Notes

Font and Style:
- The tool assumes Helvetica (helv) as the default font.
- You can customize the font and style by extracting properties from the PDF, though it’s not guaranteed to perfectly match due to PDF limitations.
PDF Structure:
- PDFs are not inherently designed for text editing. This tool works best with text-based PDFs, not scanned images or PDFs with embedded fonts.
Testing:
- Always back up your original PDF before using this tool.

Conclusion

With Python and PyMuPDF, replacing text in a PDF is straightforward and powerful. This tutorial covered a CLI tool that can be extended further to suit specific needs. Try it out, and let us know how it works for you in the comments!

Happy coding! 🚀

Transform Poetry created `pyproject.toml` to UV-Compatible Format with a Bash Script

abbazs — Wed, 04 Sep 2024 19:24:29 +0000

If you have landed here looking for a solution - a better solution available at https://github.com/mkniewallner/migrate-to-uv

To learn more about poetry visit https://python-poetry.org/docs/
To learn more about uv visit https://docs.astral.sh/uv/

#!/bin/bash

# Function to display usage
usage() {
    echo "Usage: $0 [input_file]"
    echo "If input_file is not provided, the script will look for pyproject.toml in the current directory."
    exit 1
}

# Set input file
if [ "$#" -eq 0 ]; then
    input_file="pyproject.toml"
elif [ "$#" -eq 1 ]; then
    input_file="$1"
else
    usage
fi

# Check if input file exists
if [ ! -f "$input_file" ]; then
    echo "Error: Input file '$input_file' not found."
    exit 1
fi

# Check if the file contains [tool.poetry.dependencies]
if ! grep -q '\[tool\.poetry\.dependencies\]' "$input_file"; then
    echo "Exiting: The input file is not created using Poetry. [tool.poetry.dependencies] section not found."
    exit 1
fi

# Set output file name
full_path=$(realpath "$input_file")
dir_path=$(dirname "$full_path")
output_file="${dir_path}/pyproject.toml"
backup_file="${dir_path}/pyproject.toml.old"

# Create a backup of the input file
rsync -av "$input_file" "$backup_file"
echo "Backup created: $backup_file"

# Read the content of pyproject.toml
content=$(cat "$input_file")

# Extract the name, version, description, and author information
name=$(echo "$content" | grep 'name =' | cut -d '"' -f 2)
version=$(echo "$content" | grep '^version =' | cut -d '"' -f 2)
description=$(echo "$content" | grep 'description =' | cut -d '"' -f 2)
author_name=$(echo "$content" | grep 'authors =' | sed 's/.*\["\(.*\) <.*/\1/')
author_email=$(echo "$content" | grep 'authors =' | sed 's/.*<\(.*\)>.*/\1/')

# Extract Python version
# python_version=$(echo "$content" | grep '^python =' | cut -d '"' -f 2 | sed 's/\^/>=/')
python_version=$(echo "$content" | grep '^python =' | awk '{
  output = $3  
  gsub(/^"/, "", output)
  gsub(/"$/, "", output)
  if (match(output, /^[[:alnum:]]/)) {
    output = "==" output 
  } else {
    # Extract alphanumeric characters and add >=
    gsub(/^[^[:alnum:]]+/, "", output)
    output = ">=" output
  }
  print output
}')

# Remove the python version line
content=$(echo "$content" | sed -e '/^python\s=/d')
# Extract dependencies
dependencies=$(
    echo "$content" |
        sed -n '/\[tool.poetry.dependencies\]/,/\[tool\.poetry\.group\.dev\.dependencies\]/p' |
        sed -e '1d; $d' | \
        grep -v '^$' | \
        sort 
)

# Extract dev dependencies
dev_dependencies=$(
    echo "$content" |
        sed -n '/\[tool.poetry.group.dev.dependencies\]/,/\[build-system\]/p' |
        sed -e '1d; $d' | \
        grep -v '^$' | \
        sort
)

# Function to clean up dependency lines
clean_dependency() {
    if echo "$1" | grep -q 'path\s*=\s*'; then
        echo "$1" | sed -e 's/^.*\s=\s{/# {/g'
    elif echo "$1" | grep -q 'extras'; then
        echo "$1" | sed -e 's/\s=\s//g' \
            -e 's/{.*extras.*\(\[.*\]\).*version"^\(.*\)}/\1>=\2/' \
            -e 's/\"//g' \
            -e 's/^/"/;s/$/"/' # Add double quotes at start and end
    else
        echo "$1" | sed -e 's/\s=\s\"^/>=/g' \
            -e 's/\"//' \
            -e 's/^/"/;s/$/"/' # Add double quotes at start and end
    fi
}

# Create the new pyproject.toml content
cat <<EOF >"$output_file"
[project]
name = "$name"
version = "$version"
requires-python = "$python_version"
description = "$description"
readme = "README.md"
authors = [{name = "$author_name", email = "$author_email"}]
dependencies = [
$(echo "$dependencies" | while read -r line; do
    cleaned=$(clean_dependency "$line")
    [ ! -z "$cleaned" ] && echo "    $cleaned,"
done)
]
[dependency-groups]
dev = [
$(echo "$dev_dependencies" | while read -r line; do
    cleaned=$(clean_dependency "$line")
    [ ! -z "$cleaned" ] && echo "    $cleaned,"
done)
]
EOF

echo "Converted file created: $output_file"

After running the script, execute uv sync to update your project.

EdgeDB bare metal Installation and Configuration Script

abbazs — Sat, 31 Aug 2024 14:03:42 +0000

Here is a edgedb bare metal installation script that you can use!

#!/bin/bash

# EdgeDB bare metal Installation and Configuration Script
# This script automates the process of installing EdgeDB, configuring its service file,
# setting up environment variables, and linking the instance with the CLI.

set -euo pipefail

# Function definitions
error() { echo "Error: $1" >&2; exit 1; }
section() { echo -e "\n==== $1 ====\n"; }
prompt_with_default() {
    local prompt="$1" default="$2"
    read -p "$prompt [$default]: " response
    echo "${response:-$default}"
}

# Check for root privileges
[[ $EUID -ne 0 ]] && error "This script must be run as root (use sudo)."

section "Starting EdgeDB Installation and Configuration"

# Prompt for PostgreSQL connection details
PG_USER=$(prompt_with_default "Enter PostgreSQL username" "postgres")
PG_PASSWORD=$(prompt_with_default "Enter PostgreSQL password" "password")
PG_SERVER=$(prompt_with_default "Enter PostgreSQL server address" "localhost")
PG_PORT=$(prompt_with_default "Enter PostgreSQL port" "5432")
PG_DATABASE=$(prompt_with_default "Enter PostgreSQL database name" "postgres")

# Step 1: Import the EdgeDB packaging key
section "Importing EdgeDB Packaging Key"
mkdir -p /usr/local/share/keyrings
curl --proto '=https' --tlsv1.2 -sSf \
    -o /usr/local/share/keyrings/edgedb-keyring.gpg \
    https://packages.edgedb.com/keys/edgedb-keyring.gpg || error "Failed to download EdgeDB keyring"

# Step 2: Add the EdgeDB package repository
section "Adding EdgeDB Package Repository"
echo "deb [signed-by=/usr/local/share/keyrings/edgedb-keyring.gpg] https://packages.edgedb.com/apt $(grep "VERSION_CODENAME=" /etc/os-release | cut -d= -f2) main" | 
    tee /etc/apt/sources.list.d/edgedb.list || error "Failed to add EdgeDB repository"

# Step 3: Install the EdgeDB package
section "Installing EdgeDB"
apt-get update && apt-get install -y edgedb-5 || error "Failed to install EdgeDB"

# Step 4: Create the environment file
section "Setting Up Environment Variables"
ENV_FILE="/opt/edgedb/edgedb.env"
mkdir -p /opt/edgedb
EDGEDB_PASSWORD=$(uuidgen)

cat > "$ENV_FILE" << EOL
EDGEDB_SERVER_TLS_CERT_MODE=generate_self_signed
EDGEDB_SERVER_ADMIN_UI=enabled
EDGEDB_SERVER_BACKEND_DSN=postgres://${PG_USER}:${PG_PASSWORD}@${PG_SERVER}:${PG_PORT}/${PG_DATABASE}
EDGEDB_SERVER_PASSWORD=${EDGEDB_PASSWORD}
EOL

chown edgedb:edgedb "$ENV_FILE"
chmod 600 "$ENV_FILE"

# Step 5: Create run directory
RUNSTATE_DIR=/run/edgedb
mkdir -p $RUNSTATE_DIR
chmod 755 $RUNSTATE_DIR
chown edgedb:edgedb $RUNSTATE_DIR

# Step 6: Configure EdgeDB service
section "Configuring EdgeDB Service"
SERVICE_FILE="/etc/systemd/system/edgedb-server-5.service"
cat > "$SERVICE_FILE" << EOL
[Unit]
Description=EdgeDB database server
After=network.target

[Service]
Type=notify
User=edgedb
Group=edgedb
EnvironmentFile=/opt/edgedb/edgedb.env
ExecStart=/usr/lib/x86_64-linux-gnu/edgedb-server-5/bin/edgedb-server --runstate-dir=%t/edgedb --tls-cert-mode=generate_self_signed
ExecReload=/bin/kill -HUP \$MAINPID
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=0

[Install]
WantedBy=multi-user.target
EOL

# Step 7: Enable, start EdgeDB service, and configure instance
section "Enabling EdgeDB Service and Configuring Instance"
systemctl enable edgedb-server-5 || error "Failed to enable EdgeDB service"
systemctl start edgedb-server-5 || error "Failed to start EdgeDB service"

systemctl daemon-reload
systemctl restart edgedb-server-5 || error "Failed to restart EdgeDB service"

# Set password and configure listen address
edgedb --port 5656 --tls-security insecure --admin --unix-path $RUNSTATE_DIR \
    query "ALTER ROLE edgedb SET password := '${EDGEDB_PASSWORD}'" || error "Failed to set password"

echo "${EDGEDB_PASSWORD}" | edgedb --port 5656 --tls-security insecure --password-from-stdin \
    query "CONFIGURE INSTANCE SET listen_addresses := {'0.0.0.0'};" || error "Failed to change listen address"

systemctl restart edgedb-server-5 || error "Failed to restart EdgeDB service"

# Link EdgeDB instance with CLI
section "Linking EdgeDB Instance with CLI"
EDB_DATABASE=$(prompt_with_default "Enter EdgeDB database name" "bare_metal_db")

# Run the instance link command as the sudo user
su - "${SUDO_USER}" << EOF
echo "${EDGEDB_PASSWORD}" | edgedb instance link \
    --host localhost \
    --port 5656 \
    --user edgedb \
    --branch main \
    --trust-tls-cert \
    --password-from-stdin \
    "${EDB_DATABASE}" || echo "Failed to link EdgeDB instance"
EOF

# Check if the link was successful
if ! su - "${SUDO_USER}" -c "edgedb instance list | grep -q ${EDB_DATABASE}"; then
    error "Failed to link EdgeDB instance"
fi

section "Installation Complete"
echo "EdgeDB has been successfully installed, configured, and linked!"
echo
echo "Summary of actions:"
echo "1. EdgeDB version 5 has been installed."
echo "2. A systemd service file has been created at ${SERVICE_FILE}."
echo "3. An environment file has been created at ${ENV_FILE} with a UUID as the password."
echo "4. The EdgeDB service has been enabled and started."
echo "5. EdgeDB listen address has been set to 0.0.0.0 to allow external connections."
echo "6. The EdgeDB instance has been linked with the CLI as '${EDB_DATABASE}'."
echo
echo "Next steps:"
echo "1. Review and update the ${ENV_FILE} file if needed."
echo "2. Configure your firewall to restrict access to port 5656 as needed."
echo "3. Use strong authentication methods when connecting to EdgeDB remotely."
echo "4. You can now use '${EDB_DATABASE}' when running EdgeDB CLI commands."
echo
echo "For more information, visit https://docs.edgedb.com/guides/deployment/bare_metal#bare-metal"

A One-Liner `sed` Command to Format SSH Config File

abbazs — Mon, 26 Aug 2024 08:25:32 +0000

SED Command

sed -i -e '/^\s*#/!{/^\s*Host /!s/^\s*/\t/}' \
    -e '/^\s*Host /s/^\s*//' \
    ~/.ssh/config

With backup of old config file having extension .old

sed -i.old -e '/^\s*#/!{/^\s*Host /!s/^\s*/\t/}' \
    -e '/^\s*Host /s/^\s*//' \
    ~/.ssh/config

Breakdown

Skip Commented Lines:
- Ignores lines starting with #, possibly preceded by whitespace.

  /^\s*#/!{}

Indent Non-Host and Non-Comment Lines:

  {/^\s*Host /!s/^\s*/\t/}

Matches lines that do not start with Host.

  /^\s*Host /!

Replaces leading spaces or tabs with a single tab.

  s/^\s*/\t/

Remove Leading Spaces from Host Lines:

  -e '/^\s*Host /s/^\s*//'

Matches lines that start with Host.

  /^\s*Host /

Removes all leading spaces or tabs from those lines.

  s/^\s*//

Example

Input:

Host myserver
HostName ssh.example.com
User myuser
 Port 22
 IdentityFile ~/.ssh/mykey

   Host another-server
    HostName another.example.com
   User anotheruser
Port 2022
IdentityFile ~/.ssh/anotherkey

Output:

Host myserver
    HostName ssh.example.com
    User myuser
    Port 22
    IdentityFile ~/.ssh/mykey

Host another-server
    HostName another.example.com
    User anotheruser
    Port 2022
    IdentityFile ~/.ssh/anotherkey

This formats the file in place (-i flag), ensuring consistent indentation and removing leading spaces.

Reconstructing Shopping Bill Dataset from OCR Data using Bash

abbazs — Fri, 23 Aug 2024 08:25:47 +0000

Introduction

In this tutorial, we'll explore a Bash script that reconstructs a Shopping Bill dataset from OCR (Optical Character Recognition) data files. This is a common problem in data processing where we need to merge and clean data from multiple sources to create a coherent dataset. We'll dive into the problem, the solution, and explain the various Bash tools and techniques used.

Problem Statement

We have a set of OCR data files containing information about shopping bills. These files include details such as shop names, customer names, items purchased, categories, quantities, prices, and costs. Our task is to reconstruct this data into individual shopping bill files, each containing the complete information for a single transaction.

The main challenges are:

Extracting relevant information from multiple files
Dealing with potential OCR errors
Formatting the data correctly
Generating individual files for each shopping bill

Inputs are available at IITM Google Drive

The Solution

Let's break down the solution into its main components and explain the tools and techniques used.

1. The `extract_data` Function

extract_data() {
    local seq=$1
    local file=$2
    local ps=".*${seq}"
    local pe=".*$((seq + 1))"

    sed -n "
    /^\[File: ${ps}\.jpg\]/{
        n;n;
        :a;
        /^\[File: ${pe}\.jpg\]/q;
        p;n;
        ba;
    }
" "$file"
}

This function uses sed to extract data for a specific record from a file. Here's how it works:

It takes two parameters: the sequence number of the record and the file to search.
It uses sed with the -n option, which suppresses automatic printing of pattern space.
The sed script searches for a pattern matching the start of the desired section.
It then enters a loop, printing lines until it finds the start of the next section.

Key sed commands used:

/pattern/: Searches for a pattern
n: Moves to the next line
:label: Defines a label
p: Prints the current line
q: Quits the script
ba: Branches (jumps) to a label

2. The `clean_empty_lines` Function

clean_empty_lines() {
    sed '/^\s*$/d' $1
}

This simple function uses sed to remove empty lines from a file:

/^\s*$/d: Matches lines that contain only whitespace and deletes them.

3. The Main `script` Function

script() {
    local input_path=$1
    local max=$(sed -n '$s/[^0-9]*//gp' $input_path/ocr_seqno.txt)
    local names=($(sed -e 's/^\[.*\]//g' -e '/^\s*$/d' -e 's/.*\s//g' $input_path/ocr_names.txt))
    mapfile -t stores < <(sed -e 's/^\[.*\]//g' -e '/^\s*$/d' $input_path/ocr_shopname.txt)
    mapfile -t items < <(sed -e '/^\[File:/d' -e '/^Item$/!{H;d}' -e '/^Item$/{x;s/^\n//;s/\n/; /g;s/;[[:space:]]*;/;/g;s/^[[:space:]]*//;s/^;//;s/[[:space:]]*$//;s/^[[:space:]]*//;/./p;s/.*//;h}' -e '$ {x;s/^\n//;s/\n/; /g;s/;[[:space:]]*;/;/g;s/^[[:space:]]*//;s/^;//;s/[[:space:]]*$//;s/^[[:space:]]*//;/^[[:space:]]*$/d;/./p}' -e '/^[[:space:]]*$/d' $input_path/ocr_items.txt)
    local categories=()

    # Data cleanup
    names[9]=Rajesh
    names[22]=Julia

    # Print the formatted data
    for ((i = 0; i < $max; i++)); do
        printf "%s:%s:%d\n" "${stores[$i]}" "${names[$i]}" $((i + 1))
    done

    echo ${#items[@]}
    for ((i = 0; i < ${#items[@]}; i++)); do
        printf "%02d:%s\n" $i "${items[$i]}"
    done
}

This function is the heart of our script. Let's break it down:

It uses sed to extract the maximum sequence number from ocr_seqno.txt.
It processes ocr_names.txt and ocr_shopname.txt to extract names and store names.
It uses a complex sed command to process ocr_items.txt and extract items.
It performs some data cleanup to correct OCR errors.
Finally, it formats and prints the data.

Key techniques used:

sed for text processing
mapfile to read lines into an array
Bash parameter expansion for default values
Bash arrays for storing multiple values
Bash for loops for iterating over data

Explanation of Tools and Options

sed (Stream Editor)

sed is a powerful text processing tool. In this script, we use several sed options and commands:

-n: Suppresses automatic printing of pattern space.
-e: Allows multiple editing commands.
s///: Substitution command.
/pattern/: Pattern matching.
d: Delete command.
p: Print command.
q: Quit command.
H: Append pattern space to hold space.
x: Exchange contents of pattern and hold space.

mapfile

mapfile (also known as readarray) is used to read lines from standard input into an array. We use it with process substitution (< <(...)) to populate arrays directly from command output.

Bash Parameter Expansion

In the last line of the script, we use ${1:-.} and ${2:-.}. This is parameter expansion with a default value. If the parameter is unset or null, the default value (in this case, . for the current directory) is used.

Conclusion

This script demonstrates how to use bash for text processing and data manipulation tasks. By combining tools like sed with Bash's built-in features, we can create robust solutions for complex data processing problems.

The key takeaways from this tutorial are:

Use sed for complex text processing tasks.
Utilize Bash arrays and loops for handling multiple data items.
Employ functions to modularize your code and improve readability.
Use Bash parameter expansion for flexible script arguments.

Remember, when dealing with OCR data, always account for potential errors and include data cleaning steps in your scripts.

5 text processing tools `grep, sed, awk, cut, and tr` to score some marks in System Commands OPPE

abbazs — Fri, 23 Aug 2024 08:14:06 +0000

Introduction

This tutorial covers five essential Linux text processing tools: grep, sed, awk, cut, and tr.

1. grep (Global Regular Expression Print)

grep is used for searching text using patterns.

Example 1: Basic text search

$ echo -e "This is an error\nThis is a warning\nThis is info" | grep "error"
This is an error

Example 2: Case-insensitive search

$ echo -e "Warning: System overload\nwarning: Low memory" | grep -i "warning"
Warning: System overload
warning: Low memory

Example 3: Display line numbers

$ echo -e "Line 1\nLine 2\nCritical error\nLine 4" | grep -n "Critical"
3:Critical error

Example 4: Recursive search in directories

Assume we have a directory structure:

project/
├── file1.txt (contains "TODO: Fix this")
└── subdir/
    └── file2.txt (contains "TODO: Implement feature")

$ grep -r "TODO" project/
project/file1.txt:TODO: Fix this
project/subdir/file2.txt:TODO: Implement feature

Example 5: Invert match

$ echo -e "Success: Task 1\nError: Task 2\nSuccess: Task 3" | grep -v "Success"
Error: Task 2

Example 6: Use regular expressions

$ echo -e "Phone: 123-456-7890\nInvalid: 123-45-6789" | grep -E "[0-9]{3}-[0-9]{3}-[0-9]{4}"
Phone: 123-456-7890

Example 7: Print only matched parts (-o option)

$ echo "The IP address is 192.168.1.1 and 10.0.0.1" | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"
192.168.1.1
10.0.0.1

2. sed (Stream Editor)

sed is used for text transformation.

Example 1: Basic substitution

$ echo "The color is red" | sed 's/red/blue/'
The color is blue

Example 2: Global substitution

$ echo "one two one two" | sed 's/one/three/g'
three two three two

Example 3: In-place editing

Assume we have a file named "colors.txt" with content "The color is red"

$ sed -i 's/red/green/' colors.txt
$ cat colors.txt
The color is green

Example 4: Delete lines

$ echo -e "Keep this\nDelete me\nKeep this too" | sed '/Delete/d'
Keep this
Keep this too

Example 5: Print specific lines

$ seq 10 | sed -n '3,7p'
3
4
5
6
7

Example 6: Multiple operations

$ echo -e "foo bar\nbaz foo" | sed -e 's/foo/qux/g' -e '/baz/d'
qux bar

3. awk (Aho, Weinberger, and Kernighan)

awk is a powerful text-processing tool for working with structured data.

Example 1: Print specific fields

$ echo -e "John Doe 25\nJane Smith 30" | awk '{print $1, $3}'
John 25
Jane 30

Example 2: Use custom field separator

$ echo "name:John:age:30" | awk -F':' '{print $2}'
John

Example 3: Sum values in a column

$ echo -e "10\n20\n30" | awk '{sum += $1} END {print sum}'
60

Example 4: Conditional printing

$ echo -e "John 25\nJane 35\nMike 40" | awk '$2 > 30 {print $0}'
Jane 35
Mike 40

Example 5: Calculate average

$ echo -e "10\n20\n30\n40\n50" | awk '{sum += $1; count++} END {print sum/count}'
30

Example 6: Use regular expressions

$ echo -e "apple\nbanana\ncherry" | awk '/^b/ {print $0}'
banana

4. cut

cut is used for extracting sections from each line of files.

Example 1: Extract specific characters

$ echo "Hello, World!" | cut -c 1-5
Hello

Example 2: Extract specific fields with delimiter

$ echo "John,Doe,30,New York" | cut -d',' -f 2,4
Doe,New York

Example 3: Extract range of fields

$ echo "field1:field2:field3:field4:field5" | cut -d':' -f 1-3
field1:field2:field3

Example 4: Extract to end of line

$ echo "Name: John Doe, Age: 30, City: New York" | cut -d':' -f 2-
 John Doe, Age: 30, City: New York

Example 5: Exclude specific fields

$ echo "1,2,3,4,5" | cut -d',' --complement -f 2,4
1,3,5

5. tr (Translate)

tr is used for translating or deleting characters.

Example 1: Convert lowercase to uppercase

$ echo "hello world" | tr 'a-z' 'A-Z'
HELLO WORLD

Example 2: Delete specific characters

$ echo "hello 123 world" | tr -d '0-9'
hello  world

Example 3: Squeeze repeating characters

$ echo "hello      world" | tr -s ' '
hello world

Example 4: Translate characters

$ echo "hello world" | tr 'hw' 'HW'
Hello World

Example 5: Delete all except specified characters

$ echo "hello 123 world" | tr -cd '0-9\n'
123

Conclusion

These five tools - grep, sed, awk, cut, and tr - are powerful utilities for text processing in Linux. The examples provided demonstrate their basic usage and some advanced features. Practice with these examples and explore their man pages for more advanced usage using man command.

array, dictionary, for and while loops to score 25 marks in System Commands OPPE

abbazs — Fri, 23 Aug 2024 07:52:10 +0000

Introduction

This guide covers concepts focusing on arrays, loops, file operations, arithmetic operations, and string manipulation.

1. Arrays in Bash

Arrays in Bash are zero-indexed lists of values. There are several ways to declare and initialize arrays.

Declaring and Initializing Arrays

Using declare: ```bash

declare -a my_array


2. Implicit declaration with initialization:
```bash


my_array=(apple banana cherry)

Empty array declaration: ```bash

my_array=()


4. Declaring with specific indices:
```bash


my_array=([0]="apple" [1]="banana" [5]="cherry")

Declaring empty and adding elements: ```bash

my_array=()
my_array+=("apple")
my_array+=("banana")


6. Declaring a read-only array:
```bash


declare -ar my_readonly_array=(apple banana cherry)

Accessing and Manipulating Arrays



# Access elements
echo ${my_array[0]}  # Output: apple

# Get all elements
echo ${my_array[@]}  # Output: apple banana cherry

# Get array length
echo ${#my_array[@]}  # Output: 3

# Slicing an array
echo ${my_array[@]:1:2}  # Output: banana cherry (from index 1, take 2 elements)

# Finding the indices of an array
echo ${!my_array[@]}  # Output: 0 1 2 (or 0 1 5 for the array with specific indices)

2. Loops in Bash

Bash provides several types of loops for repetitive tasks.

For Loops

Traditional for loop: ```bash

for i in 1 2 3 4 5
do
echo $i
done


2. C-style for loop:
```bash


for ((i=0; i<5; i++))
do
    echo $i
done

For loop with array: ```bash

fruits=(apple banana cherry)
for fruit in "${fruits[@]}"
do
echo $fruit
done


4. For loop with command substitution:
```bash


for file in $(ls)
do
    echo $file
done

While Loops

Basic while loop syntax:



count=1
while [ $count -le 5 ]
do
    echo "Count is: $count"
    ((count++))
done

3. File Operations

Reading from and writing to files is a common task in Bash scripting.

Reading a File Line by Line



while IFS= read -r line
do
    echo "Line: $line"
done < "input.txt"

Writing to a File



echo "Hello, World!" > output.txt
echo "Appending this line" >> output.txt

4. Arithmetic Operations in Bash

Bash offers various methods for arithmetic operations:



# Using let
let result=5+3
echo $result  # Output: 8

# Using (( ))
((result = 5 + 3))
echo $result  # Output: 8

# Using $[ ]
result=$[5 + 3]
echo $result  # Output: 8

# Using expr (note the spaces)
result=$(expr 5 + 3)
echo $result  # Output: 8

5. String Manipulation

String Case Checking

This example checks if given strings start with uppercase or lowercase letters:



#!/bin/bash

declare -a upper
declare -a lower

words=("Hello" "world" "BASH" "script" "ARRAY" "example")

for word in "${words[@]}"
do
    if [[ $word == [[:upper:]]* ]]
    then
        upper+=("$word")
    elif [[ $word == [[:lower:]]* ]]
    then
        lower+=("$word")
    fi
done

echo "Uppercase words: ${upper[@]}"
echo "Lowercase words: ${lower[@]}"

Advanced Pattern Matching

This script demonstrates number detection and pattern matching:



#!/bin/bash

declare -a num_var
declare -a userid_var
declare -a other_var

strings=("123" "abc" "456" "1234abcd56" "789xyz" "2023user01" "hello123")

for str in "${strings[@]}"
do
    if [[ $str =~ ^[0-9]+$ ]]
    then
        # String is a number
        num_var+=("$str")
    elif [[ $str =~ ^[0-9]{4}[a-z]{4}[0-9]{2}$ ]]
    then
        # String matches pattern \d{4}[a-z]{4}\d{2}
        userid_var+=("$str")
    else
        # String doesn't match any specific category
        other_var+=("$str")
    fi
done

echo "Numbers: ${num_var[@]}"
echo "User IDs: ${userid_var[@]}"
echo "Other strings: ${other_var[@]}"

6. Comprehensive Example: File Processing and Data Categorization

This example combines file reading, loops, and string manipulation to categorize data from a file:



#!/bin/bash

# Initialize arrays for different types of data
declare -a names
declare -a numbers
declare -a emails
declare -a mixed

# Read the file line by line
while IFS= read -r line
do
    # Check if line is a name (contains only letters and spaces)
    if [[ $line =~ ^[[:alpha:][:space:]]+$ ]]
    then
        names+=("$line")
    # Check if line is a number
    elif [[ $line =~ ^[0-9]+$ ]]
    then
        numbers+=("$line")
    # Check if line is an email
    elif [[ $line =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}$ ]]
    then
        emails+=("$line")
    # If it doesn't match any of the above, it's mixed content
    else
        mixed+=("$line")
    fi
done < "data.txt"

# Print the categorized data
echo "Names: ${names[@]}"
echo "Numbers: ${numbers[@]}"
echo "Emails: ${emails[@]}"
echo "Mixed Content: ${mixed[@]}"

This script reads a file named data.txt, categorizes each line based on its content, and stores it in the appropriate array.

Conclusion

This tutorial has covered essential Bash scripting concepts including:

Array declaration and manipulation
For and while loops
File reading and writing operations
Arithmetic operations
String manipulation and pattern matching
A comprehensive example combining multiple concepts

DEV Community: abbazs

pyxclip: Cross-Platform Clipboard for Python, Backed by Rust

pyxclip: Cross-Platform Clipboard for Python, Backed by Rust

What pyxclip does differently

Known limitation

Install

A Hidden Character Mystery

Managing MQTT User Information with PostgreSQL in a Separate Container

Prerequisites

Installing Docker and Docker-Compose on Ubuntu

Step-by-Step Guide

1. Create a docker-compose.yml File

2. Create a FastAPI Service

a. Create a Directory for FastAPI

b. Create a Dockerfile for FastAPI

3. Write the FastAPI Application

4. Start the Services

5. Test the FastAPI Service

Essential FFmpeg Recipes for Video Manipulation

Installing FFmpeg

On Ubuntu:

On Windows:

1. Removing the First 5 Minutes

2. Removing the Last 5 Minutes

3. Copying the First 5 Minutes

4. Copying the Last 5 Minutes

5. Removing a Section Between 5 Minutes and 8 Minutes

6. Copying Video Between 5 Minutes and 8 Minutes

7. Splitting a Large Video into 20MB Chunks

8. Splitting a Large Video into 5-Minute Chunks

Final Thoughts

Merge PDFs Recursively - Python

Introduction

Prerequisites

Code Walkthrough

How It Works

Running the Tool

Example

Features

Notes and Limitations

Conclusion

Replace Text in PDFs Using Python

Introduction

Prerequisites

Code Walkthrough

How It Works

Running the Tool

Example

Important Notes

Conclusion

Transform Poetry created `pyproject.toml` to UV-Compatible Format with a Bash Script

EdgeDB bare metal Installation and Configuration Script

A One-Liner `sed` Command to Format SSH Config File

SED Command

With backup of old config file having extension .old

Breakdown

Example

Input:

Output:

Reconstructing Shopping Bill Dataset from OCR Data using Bash

Introduction

Problem Statement

The Solution

1. The extract_data Function

2. The clean_empty_lines Function

3. The Main script Function

Explanation of Tools and Options

sed (Stream Editor)

mapfile

Bash Parameter Expansion

Conclusion

5 text processing tools `grep, sed, awk, cut, and tr` to score some marks in System Commands OPPE

Introduction

1. grep (Global Regular Expression Print)

Example 1: Basic text search

Example 2: Case-insensitive search

Example 3: Display line numbers

Example 4: Recursive search in directories

Example 5: Invert match

Example 6: Use regular expressions

1. Create a `docker-compose.yml` File

1. The `extract_data` Function

2. The `clean_empty_lines` Function

3. The Main `script` Function