DEV Community: GraphPe

Essential Linux Text Manipulation Tools: cut, sort, uniq, tr 🛠️

Oliver Bennet — Wed, 20 Nov 2024 11:23:34 +0000

Linux's text processing commands are the Swiss Army knives for developers and sysadmins. In this post, I break down four essential tools—cut, sort, uniq, and tr—that can simplify your workflows:

🔥 Key Features
🔥 40+ Practical Examples
🔥 Real-World Use Cases

💡 Whether you're a beginner or an advanced user, this guide will help you unlock new efficiencies.

👉 Check it out here: 4 Most used text manipulation tools

4 Most Essential Text Manipulation Tools in Linux: cut, sort, uniq, tr | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 20, 2024 ・
graphpe.Medium

Have any creative use cases of your own? Let's discuss in the comments!

Linux #TextProcessing #CLI #OpenSource #DevOps #BashScripting

Prometheus 3.0: A Quantum Leap in Monitoring

Oliver Bennet — Mon, 18 Nov 2024 07:06:01 +0000

Prometheus, the open-source systems monitoring and alerting toolkit, has long been a favorite among DevOps engineers and system administrators. Its powerful query language, PromQL, and flexible data model have made it a go-to tool for monitoring complex systems. With the recent release of Prometheus 3.0, the platform has taken a significant leap forward, introducing a host of new features and improvements that elevate its capabilities to new heights.

Key Features and Enhancements

Native Histograms:

Prometheus 3.0 introduces native support for histograms, a data type that efficiently captures distribution information. This is particularly useful for metrics like response times, request durations, and latency.
By leveraging native histograms, Prometheus can provide more accurate and granular insights into the performance of your systems.

OpenTelemetry Support:

Prometheus 3.0 strengthens its integration with OpenTelemetry, a vendor-neutral observability framework. This enables seamless collection and analysis of telemetry data from various sources, including applications, infrastructure, and cloud services.
By adopting OpenTelemetry, you can simplify your monitoring setup and gain a more comprehensive view of your entire technology stack.
Performance Optimizations:
Prometheus 3.0 incorporates several performance optimizations, including:
Faster query execution, especially for complex queries.
Reduced memory usage, leading to improved scalability.
Enhanced data ingestion and storage efficiency.
These optimizations ensure that Prometheus can handle larger workloads and provide real-time insights without compromising performance.

Enhanced PromQL:

PromQL, the powerful query language for Prometheus, has been further refined in 3.0 with the addition of new functions and operators:
delta: Calculates the difference between two time series. changes: Counts the number of changes in a time series.
rate_interval: Calculates the rate of change over a specific interval. These new capabilities empower you to perform more sophisticated analysis and troubleshooting tasks.

Improved Alerting:

The alerting system in Prometheus 3.0 has been enhanced with:

Enhanced filtering: More flexible filtering options for alerts.
Improved grouping: Better grouping of alerts based on specific criteria.
Enhanced notification channels: Support for additional notification channels, including email, SMS, and PagerDuty.

These improvements streamline the alert management process and ensure timely notifications of critical issues.

New Web UI:

Prometheus 3.0 boasts a redesigned web UI with a modern look and feel. The new UI offers:

-Improved navigation and search capabilities.
-More intuitive query building and visualization tools.
-Enhanced performance and responsiveness.

The new UI makes it easier to explore and analyze your monitoring data.

Why Upgrade to Prometheus 3.0?

Upgrading to Prometheus 3.0 is highly recommended for several reasons:

Enhanced Performance: The performance optimizations in 3.0 ensure that Prometheus can handle larger workloads and provide real-time insights.
Improved Scalability: Native histograms and other enhancements make Prometheus more scalable, allowing it to monitor larger and more complex systems.
Increased Functionality: The new features in 3.0, such as OpenTelemetry support and enhanced PromQL, provide greater flexibility and power.
Better User Experience: The redesigned web UI offers a more intuitive and efficient user experience.

Breaking Changes

While most of the changes in Prometheus 3.0 are backward-compatible, there are a few breaking changes that you should be aware of. These include:

Changes to the rate function: The rate function now calculates the rate of change over a specific interval, rather than the instantaneous rate of change.
Changes to the irate function: The irate function now uses a more accurate algorithm to calculate the instantaneous rate of change.
Changes to the increase function: The increase function now returns the total increase over a specific interval, rather than the instantaneous rate of increase.
It's essential to review the migration guide provided by the Prometheus team to ensure a smooth upgrade process.

Conclusion

Prometheus 3.0 is a significant milestone in the evolution of this powerful monitoring solution. The new features and improvements make it even more robust, efficient, and user-friendly. By upgrading to 3.0, you can gain deeper insights into your systems, improve performance, and enhance your overall monitoring strategy.

Effortless File Transfers: A Guide to Using scp and rsync | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 15, 2024 ・
graphpe.Medium

25 Advanced Bash Commands Every Linux User Should Know | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 8, 2024 ・
graphpe.Medium

Top 10 SED Use Cases Every Linux User Should Know | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 5, 2024 ・
graphpe.Medium

Linux Aliases You Didn’t Know You Needed: Practical Shortcuts for Every User | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 15, 2024 ・
graphpe.Medium

🎉 Dive into Python 3.13: New Features, JIT Compiler, Free threading & Installation Guide! 🎉

Oliver Bennet — Tue, 12 Nov 2024 13:00:44 +0000

Introduction:

Hey, Python enthusiasts! The long-awaited Python 3.13 release has arrived, and it brings a host of exciting changes, from a Just-In-Time (JIT) compiler and free-threading support to a more interactive interpreter. In my latest video, I go over all the essential new features and also walk through the installation process for both Windows and Ubuntu.

Check out the video here: Python 3.13 Features & Installation Guide

What’s New in Python 3.13?

Python 3.13 introduces several game-changing updates:

Free-threading Support (PEP 703) – Experimental support to run Python without the Global Interpreter Lock (GIL).
Just-In-Time (JIT) Compiler (PEP 744) – A basic JIT compiler, currently disabled by default, but laying the groundwork for future performance boosts.
Enhanced Error Messages – Python’s interactive interpreter and error handling have leveled up, now with colorized tracebacks for easy debugging.
Library Improvements – Expect a streamlined standard library, with several legacy modules officially removed (PEP 594).
Improved Interactive Interpreter – Inspired by the PyPy project, the REPL now includes multiline editing, history browsing, and colorized prompts

Step-by-Step Installation for Windows & Ubuntu
If you’re eager to try Python 3.13, this video includes a complete installation guide for both Windows and Ubuntu. Whether you're upgrading or installing Python for the first time, follow along to set up the latest version on your system.

Watch the Full Breakdown Medium Post:

Python 3.13 is here and It could be a Game Changer | Medium

Oliver Bennet ・ Nov 12, 2024 ・
graphpe.Medium

Python 3.13 Features & Installation Guide

Key Highlights Covered in the Video

Error Message Improvements – Helpful error messages for common mistakes like naming conflicts with standard modules.
New Security Enhancements – Updates in ssl and other libraries to keep your applications secure.
Typing Improvements – Expanded typing functionality including default values for type parameters.
Deprecations and Removals – Say goodbye to legacy modules like cgi, crypt, telnetlib, and more.

Whether you’re developing on Windows or Ubuntu, this video has all the insights you need to get started with Python 3.13! Check it out and let me know your thoughts in the comments below. Happy coding!

Automate Database Backup with Bash, Python, and PowerShell

Oliver Bennet — Sat, 09 Nov 2024 07:00:00 +0000

Backing up databases regularly is essential for data recovery and continuity. In this guide, we’ll automate backups for MySQL and PostgreSQL using Bash and Python, and for MSSQL with PowerShell. Each script will include scheduling options and a way to manage backup retention.

Prerequisites

Database Clients: Ensure mysql, pg_dump, and sqlcmd (or SQL Server Management Objects) are installed.
Permissions: Ensure the script user has permissions to read databases and write backups.
Scheduling: We’ll cover how to schedule these scripts using cron for Linux and Task Scheduler for Windows.

Part 1: MySQL Backup Automation

Method 1: Bash Script
Bash Script for MySQL Backup:

#!/bin/bash

# Configuration
DB_NAME="your_database"
DB_USER="your_user"
DB_PASSWORD="your_password"
BACKUP_DIR="/path/to/backup/dir"
TIMESTAMP=$(date +"%F")
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}_backup_$TIMESTAMP.sql"

# MySQL Dump Command
mysqldump -u $DB_USER -p$DB_PASSWORD $DB_NAME > $BACKUP_FILE

# Compression (optional)
gzip $BACKUP_FILE

# Log completion
echo "MySQL Backup for $DB_NAME completed: $BACKUP_FILE.gz"

Save this script as mysql_backup.sh.
Make it executable: chmod +x mysql_backup.sh.
Add to cron for daily backups at midnight:

0 0 * * * /path/to/mysql_backup.sh >> /path/to/backup/logs/backup.log 2>&1

Method 2: Python Script
Python Script for MySQL Backup:

import os
from datetime import datetime
import subprocess

# Configuration
DB_NAME = "your_database"
DB_USER = "your_user"
DB_PASSWORD = "your_password"
BACKUP_DIR = "/path/to/backup/dir"
TIMESTAMP = datetime.now().strftime("%Y-%m-%d")
BACKUP_FILE = os.path.join(BACKUP_DIR, f"{DB_NAME}_backup_{TIMESTAMP}.sql")

# MySQL Dump Command
command = f"mysqldump -u {DB_USER} -p{DB_PASSWORD} {DB_NAME} > {BACKUP_FILE}"
subprocess.call(command, shell=True)

# Optional compression
subprocess.call(["gzip", BACKUP_FILE])

print(f"MySQL backup for {DB_NAME} completed: {BACKUP_FILE}.gz")

- Run as a cron job similar to the Bash script:

0 0 * * * python3 /path/to/mysql_backup.py >> /path/to/backup/backup.log 2>&1

Part 2: PostgreSQL Backup Automation

Method 1: Bash Script
Bash Script for PostgreSQL Backup:

#!/bin/bash

# Configuration
DB_NAME="your_database"
DB_USER="your_user"
BACKUP_DIR="/path/to/backup/dir"
TIMESTAMP=$(date +"%F")
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}_backup_$TIMESTAMP.sql"

# PostgreSQL Dump Command
PGPASSWORD="your_password" pg_dump -U $DB_USER $DB_NAME > $BACKUP_FILE

# Compression (optional)
gzip $BACKUP_FILE

# Log completion
echo "PostgreSQL Backup for $DB_NAME completed: $BACKUP_FILE.gz"

Save as postgres_backup.sh.
Set cron job:

0 0 * * * /path/to/postgres_backup.sh >> /path/to/backup/backup.log 2>&1

Method 2: Python Script
Python Script for PostgreSQL Backup:

import os
from datetime import datetime
import subprocess

# Configuration
DB_NAME = "your_database"
DB_USER = "your_user"
BACKUP_DIR = "/path/to/backup/dir"
TIMESTAMP = datetime.now().strftime("%Y-%m-%d")
BACKUP_FILE = os.path.join(BACKUP_DIR, f"{DB_NAME}_backup_{TIMESTAMP}.sql")

# PostgreSQL Dump Command
command = f"PGPASSWORD='your_password' pg_dump -U {DB_USER} {DB_NAME} > {BACKUP_FILE}"
subprocess.call(command, shell=True)

# Optional compression
subprocess.call(["gzip", BACKUP_FILE])

print(f"PostgreSQL backup for {DB_NAME} completed: {BACKUP_FILE}.gz")

Schedule with cron:

0 0 * * * python3 /path/to/postgres_bkp.py >> /path/to/backup/backup.log 2>&1

Part 3: MSSQL Backup Automation with PowerShell

PowerShell Script for MSSQL Backup:

# Configuration
$serverName = "your_server"
$dbName = "your_database"
$backupDir = "C:\path\to\backup\dir"
$timestamp = Get-Date -Format "yyyy-MM-dd"
$backupFile = "$backupDir\$dbName`_backup_$timestamp.bak"

# SQL Backup Command
Invoke-Sqlcmd -ServerInstance $serverName -Query "BACKUP DATABASE [$dbName] TO DISK = N'$backupFile' WITH NOFORMAT, NOINIT, NAME = '$dbName-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10"

# Optional compression with 7-Zip (if installed)
& "C:\Program Files\7-Zip\7z.exe" a "$backupFile.zip" "$backupFile"
Remove-Item $backupFile

Write-Output "MSSQL Backup for $dbName completed: $backupFile.zip"

Save as mssql_backup.ps1.
Schedule with Windows Task Scheduler: Create a new task, set the action to run powershell.exe with the script path as an argument. Set the schedule (e.g., daily at midnight).

Backup Retention Policy

To avoid running out of storage, consider setting a retention policy to delete backups older than a specified number of days.

Bash Command:

find /path/to/backup/dir -type f -mtime +30 -name "*.gz" -exec rm {} \;

PowerShell Command:

$backupDir = "C:\path\to\backup\dir"
Get-ChildItem -Path $backupDir -Filter "*.zip" | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } | Remove-Item

Add these commands to the end of each script or as separate scheduled tasks to keep backup directories clean and organized.

Conclusion

With these scripts, you’ll have a reliable way to automatically back up MySQL, PostgreSQL, and MSSQL databases, reducing the risk of data loss. Remember to test the backups regularly to ensure data integrity and be ready to restore when needed!

🔗 Support my Work

▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee

Learn Complete Bash CLI Below:

I Started Learning Docker a Year Ago, Now I make my Own Containers.

Oliver Bennet — Sat, 02 Nov 2024 05:35:15 +0000

In today’s tech-driven world, Docker has become a must-know technology for developers, system administrators, and anyone interested in modern software development and DevOps practices. Docker helps streamline application development by packaging code, runtime, and dependencies into portable containers, making it easier to build, test, and deploy applications across different environments.

This article will cover the basics of Docker, its key concepts, and how it can transform the way you approach software development. At the end, I’ll also link to my in-depth, beginner-friendly YouTube tutorial on Docker so you can get hands-on experience with Docker's essentials. If you would like to skip and watch more like a video tutorial, Here is a video of the same. I personally liked this video very much as it covers Docker from an absolute layman perspective.

What is Docker?

Docker is an open-source platform designed to automate the deployment of applications in lightweight containers. Containers allow you to bundle an application along with its dependencies, ensuring it runs consistently regardless of the environment. This solves many traditional challenges of software deployment, especially those related to compatibility and configuration issues across different systems.

Key Benefits of Docker:

Consistency: Run your app seamlessly across various environments.
Isolation: Separate apps from each other and from the underlying system.
Efficiency: Lightweight, requiring less overhead than traditional virtual machines.
Scalability: Easily replicate and scale services as demand grows.

Core Docker Concepts

Before diving in, let's get familiar with a few core Docker concepts:

Images: Pre-built packages containing everything your application needs to run. Think of images as blueprints for containers.
Containers: Running instances of Docker images. Containers execute the actual code, but each container is isolated, ensuring stability and security.
Dockerfile: A file containing instructions to build Docker images automatically.
Docker Hub: A public repository for sharing and finding Docker images. It hosts official and community-built images, making it easy to find images for popular applications.

Why Learn Docker?

Docker has become essential in DevOps and microservices architecture. Whether you’re a developer working on web applications, a data scientist deploying models, or an operations engineer ensuring stability, Docker simplifies your work by encapsulating your applications and dependencies.

Docker skills are increasingly in demand, as they open the door to more complex, scalable architectures like Kubernetes and other cloud-native technologies.

Getting Started with Docker: Step-by-Step Guide

To get started, you’ll need to install Docker on your system. Here are the basic steps to follow:

Install Docker: Visit Docker’s official website and download the Docker Desktop version for your operating system.
Run Your First Container: Use the command:

docker run hello-world

This will pull a sample image from Docker Hub and run a basic container on your system.
Explore Docker Commands: Familiarize yourself with commands like docker pull, docker build, docker ps, and docker stop to start managing your own containers.
Create a Dockerfile: This simple text file will allow you to automate the creation of Docker images and define how your applications should be built.

Watch the Full Tutorial
If you’re eager to dive deeper and get hands-on, check out my full Docker tutorial on YouTube. In this one-hour video, I’ll walk you through Docker installation, setting up images and containers, and understanding Docker’s core features from scratch – no prior experience required.

🎥 Watch the video here: Docker Tutorial for Absolute Beginners

Conclusion

Docker is an invaluable tool for anyone looking to modernize their development practices and streamline application deployment. By isolating applications into containers, Docker offers a flexible, scalable, and consistent way to build and deploy software across different environments. Whether you’re just starting out or looking to enhance your DevOps skill set, Docker is a technology worth mastering.

Complete DevOps Roadmap: https://devops.graphpe.com/roadmap

🔗 Support my Work

▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee

10 Essential Linux Commands Every Site Reliability Engineer (SRE) Should Know

Oliver Bennet — Tue, 29 Oct 2024 10:22:25 +0000

Introduction

In the role of a Site Reliability Engineer (SRE), knowing the right Linux commands is key to maintaining, monitoring, and troubleshooting complex systems. These commands can make everyday tasks smoother and ensure reliable system performance. Here’s a look at ten must-know commands that every SRE should keep handy.

1. `uptime`- System Uptime and Load

Description: Shows how long the system has been running and displays the average load.

Why It’s Important: Quick way to gauge system load and identify if the system is under stress.

Basic Usage: uptime

2. `journalctl` - System Logs Access

Description: Displays logs collected by systemd.

Why It’s Important: Essential for troubleshooting errors or investigating performance issues, particularly on systemd-based Linux systems.

Basic Usage: journalctl -u nginx.service (Show logs for a specific service)

3. `free`- Memory Usage

Description: Displays memory usage including free, used, and cached memory.

Why It’s Important: Crucial for detecting memory leaks or planning memory upgrades.

Basic Usage: free -h (Displays in human-readable format)

4. `iostat`- Input/Output Statistics

Description: Reports CPU and I/O statistics for devices and partitions.

Why It’s Important: Helps to identify I/O bottlenecks and assess disk performance.

Basic Usage: iostat -xz 1 (Shows extended statistics with per-second updates)

5. `lsof` - List Open Files

Description: Lists open files and the processes that opened them.

Why It’s Important: Useful for identifying resource usage and finding potential file descriptor leaks.

Basic Usage: lsof -i :80 (Lists processes using port 80)

6. `dstat` - System Resource Statistics

Description: Combines multiple monitoring tools in one command to display CPU, disk, network, memory, and process stats.

Why It’s Important: Provides a real-time overview of system health, combining various metrics in one view.

Basic Usage: dstat -cdnm (Displays CPU, disk, network, and memory usage)

7. `curl`- Test HTTP Services

Description: Transfers data from or to a server, often used for API and HTTP testing.

Why It’s Important: Allows you to troubleshoot and test endpoints and web services quickly.

Basic Usage: curl -I http://example.com (Fetches HTTP headers)

8. `ping` and `traceroute`- Network Connectivity and Path Analysis

Description: ping checks connectivity to a remote host, while traceroute shows the path packets take.

Why It’s Important: Essential for diagnosing network issues and pinpointing connection problems.

Basic Usage: ping google.com / traceroute google.com

9. `sar` - System Activity Report

Description: Collects and displays CPU, memory, network, and disk statistics.

Why It’s Important: Helps identify patterns and potential issues over time, especially useful for trend analysis.

Basic Usage: sar -u 1 5 (Displays CPU usage over 5 seconds)

10. `systemctl`- Manage Services

Description: Controls system services, allowing you to start, stop, enable, and check service status.

Why It’s Important: Essential for service reliability, as it enables quick service restarts or status checks during incidents.

Basic Usage: systemctl restart nginx (Restart the Nginx service)

Conclusion

These commands are invaluable tools for SREs to monitor, manage, and troubleshoot systems effectively. Mastering them can enhance both the reliability and resilience of your infrastructure, ultimately leading to a smoother and more efficient production environment.

If you would like to understand more Bash CLI and master yourself. Here is a video

🔗 Support my Work

▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee

Automating Backup and Restore Processes with Python, Bash, and PowerShell

Oliver Bennet — Sat, 26 Oct 2024 12:30:00 +0000

In the world of IT, backups are a lifeline. Losing critical data can bring entire systems down, leading to financial loss, customer dissatisfaction, and operational disruptions. Automated backup and restore processes are essential for protecting your data and ensuring that you can recover quickly when something goes wrong.

In this Scripting Saturdays post, we’ll explore how to automate backup and restore processes using three powerful scripting languages: Python, Bash, and PowerShell. You’ll learn how to schedule backups, create custom restore points, and automate the entire process to ensure your data remains secure and recoverable.

Why Automate Backups and Restores?

Manual backup procedures are error-prone and can easily be overlooked, especially in environments with large amounts of data or complex infrastructure.

Here are the key benefits of automating these processes:

Data Protection and Reliability
Regularly scheduled backups ensure that you have a recent copy of your data available at all times. Automation reduces the risk of human error, ensuring that backups happen consistently, and you’re not relying on memory or manual intervention.
Fast Recovery
Automated restore scripts make recovery faster and more predictable. In the event of a system crash, hardware failure, or data corruption, an automated restore process allows you to get back up and running with minimal downtime.
Compliance
Many industries, including finance and healthcare, have strict data retention and backup requirements. Automating backups ensures that you meet these compliance standards without the need for manual oversight.
Scalability
As your infrastructure grows, so does the complexity of managing backups. Automated solutions can scale with your environment, ensuring that each server or database is backed up regularly without extra administrative effort.

Backup Strategy: Full vs Incremental

Before diving into the automation scripts, it’s important to understand the two main types of backups: Full **and **Incremental.

Full Backup: A complete copy of all data. This method is more time-consuming and requires more storage space, but it provides a comprehensive restore point.
Incremental Backup: Only the data that has changed since the last backup is saved. This method is faster and uses less storage space but requires all previous incremental backups to restore fully.

Depending on your needs, you can combine both strategies—running full backups weekly and incremental backups daily, for example—to balance storage and speed.

Automating Backups with Python

Python’s versatility and ease of integration with APIs make it a fantastic choice for automating backups in a wide range of environments, from simple file systems to cloud storage like AWS S3, Azure Blob, or Google Cloud Storage.

1. Full File Backup with Python
Let’s start with a simple example of a full file system backup, using Python to copy all files from a directory to a backup location.

import os
import shutil
import datetime

def backup_files(source_dir, backup_dir):
    # Create a timestamped directory for the backup
    timestamp = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    dest_dir = os.path.join(backup_dir, f"backup_{timestamp}")

    # Copy files to the backup directory
    try:
        shutil.copytree(source_dir, dest_dir)
        print(f"Backup completed successfully. Files copied to {dest_dir}")
    except Exception as e:
        print(f"Backup failed: {e}")

source_directory = "/path/to/source"
backup_directory = "/path/to/backup"

backup_files(source_directory, backup_directory)

2. Incremental Backup Using Python
Incremental backups in Python can be achieved by comparing timestamps of files and only copying those that have been modified since the last backup.

import os
import shutil
import time

def incremental_backup(source_dir, backup_dir, last_backup_time):
    # Create a timestamped backup directory
    timestamp = time.strftime("%Y-%m-%d_%H-%M-%S")
    dest_dir = os.path.join(backup_dir, f"incremental_backup_{timestamp}")

    if not os.path.exists(dest_dir):
        os.makedirs(dest_dir)

    # Walk through the source directory
    for root, dirs, files in os.walk(source_dir):
        for file in files:
            file_path = os.path.join(root, file)
            # Check if the file was modified after the last backup
            if os.path.getmtime(file_path) > last_backup_time:
                # Copy only modified files
                dest_file_path = os.path.join(dest_dir, os.path.relpath(file_path, source_dir))
                dest_file_dir = os.path.dirname(dest_file_path)
                if not os.path.exists(dest_file_dir):
                    os.makedirs(dest_file_dir)
                shutil.copy2(file_path, dest_file_path)
                print(f"Copied: {file_path}")

    print(f"Incremental backup completed to {dest_dir}")

last_backup_timestamp = time.time() - 86400  # 24 hours ago
incremental_backup("/path/to/source", "/path/to/backup", last_backup_timestamp)

Cloud Backup with Python
Python’s boto3 library allows you to back up files directly to AWS S3. Here's an example of uploading files to S3 after performing a backup.

import boto3
import os

def backup_to_s3(source_dir, bucket_name):
    s3_client = boto3.client('s3')

    for root, dirs, files in os.walk(source_dir):
        for file in files:
            file_path = os.path.join(root, file)
            s3_client.upload_file(file_path, bucket_name, file)
            print(f"Uploaded {file} to S3")

backup_to_s3("/path/to/backup", "my-s3-bucket")

Automating Backups with Bash

For Linux users, Bash scripts offer an efficient and lightweight way to automate backup tasks. Bash is perfect for scheduling backups through cron jobs and handling system-level tasks.

1. Full System Backup with Bash
Here’s a simple Bash script that creates a tarball of a specified directory and stores it in a backup folder with a timestamp.

#!/bin/bash

SOURCE_DIR="/path/to/source"
BACKUP_DIR="/path/to/backup"
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
BACKUP_FILE="$BACKUP_DIR/backup_$TIMESTAMP.tar.gz"

# Create a tarball of the source directory
tar -czf $BACKUP_FILE $SOURCE_DIR

echo "Backup completed. File saved to $BACKUP_FILE"

You can schedule this script to run automatically using a cron job:

0 2 * * * /path/to/backup_script.sh

This cron job runs the backup script every day at 2 AM.

2. Incremental Backup with Bash
Bash’s rsync utility is commonly used for incremental backups, as it only copies changed files and minimizes the time and space required.

Example Bash Incremental Backup Using rsync:

#!/bin/bash

SOURCE_DIR="/path/to/source"
BACKUP_DIR="/path/to/backup"

# Perform incremental backup using rsync
rsync -av --delete $SOURCE_DIR $BACKUP_DIR

echo "Incremental backup completed."

Automating Backups with PowerShell

For Windows administrators, PowerShell provides a robust environment for automating backup tasks. With deep integration into the Windows operating system, PowerShell can handle both file system and system state backups.

1. Full File Backup with PowerShell
PowerShell can be used to automate the process of creating backups and scheduling them with Task Scheduler.

Example PowerShell Script for Full Backup:

$source = "C:\path\to\source"
$backup = "C:\path\to\backup\backup_$(Get-Date -Format 'yyyy-MM-dd_HH-mm-ss').zip"

# Create a compressed ZIP file backup
Compress-Archive -Path $source -DestinationPath $backup

Write-Output "Backup completed. File saved to $backup"

2. Incremental Backup with PowerShell
Incremental backups can be automated using PowerShell’s ability to compare file modification times and selectively copy changed files.

Example PowerShell Incremental Backup Script:

$source = "C:\path\to\source"
$backup = "C:\path\to\backup"
$lastBackup = (Get-Date).AddDays(-1)  # Check files modified in the last 24 hours

# Perform incremental backup
Get-ChildItem $source -Recurse | Where-Object { $_.LastWriteTime -gt $lastBackup } | ForEach-Object {
    $dest = Join-Path $backup $_.FullName.Substring($source.Length)
    if (!(Test-Path (Split-Path $dest))) {
        New-Item -ItemType Directory -Path (Split-Path $dest)
    }
    Copy-Item $_.FullName -Destination $dest
}

Write-Output "Incremental backup completed."

Restoring Backups

Backing up data is only half the story. Being able to restore that data quickly is equally important.

Here are scripts for restoring from the backups created in each language:

Restoring with Python

import shutil

def restore_backup(backup_dir, restore_dir):
    shutil.copytree(backup_dir, restore_dir, dirs_exist_ok=True)
    print(f"Restore completed from {backup_dir} to {restore_dir}")

restore_backup("/path/to/backup", "/path/to/restore")

Restoring with Bash

#!/bin/bash

BACKUP_FILE="/path/to/backup.tar.gz"
RESTORE_DIR="/path/to/restore"

# Extract the backup tarball
tar -xzf $BACKUP_FILE -C $RESTORE_DIR

echo "Restore completed to $RESTORE_DIR"

Restoring with PowerShell

$backup = "C:\path\to\backup.zip"
$restore = "C:\path\to\restore"

# Extract the backup ZIP file
Expand-Archive -Path $backup -DestinationPath $restore

Write-Output "Restore completed to $restore"

Conclusion

Automating backup and restore processes is crucial for ensuring the safety and availability of your data. By leveraging Python, Bash, and PowerShell, you can create flexible, scalable backup solutions that protect your critical files and infrastructure. Whether you're working in a cloud environment, managing a small server, or operating a large enterprise system, automating backups will save you time, reduce risks, and help you recover from failures quickly and efficiently.

Stay tuned for the next Scripting Saturdays post, where we'll cover automating server deployment with scripts!

🔗 Support my Work

▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee

Mastering Bash: Essential Commands for Everyday Use

Oliver Bennet — Wed, 23 Oct 2024 10:32:54 +0000

Bash, the Bourne Again Shell, is a powerful tool for interacting with your Linux system. Whether you're a beginner or an experienced user, mastering essential Bash commands can significantly boost your productivity. In this article, we’ll cover some of the most useful and frequently used Bash commands, with examples to help you get started.

1. Navigating File System with `cd`, `pwd`, and `ls`

cd(Change Directory)
The cdcommand is used to navigate through the file system. You can move between directories easily with this command:

cd /path/to/directory

To go back to the previous directory:

cd -

pwd(Print Working Directory)
If you ever get lost in the terminal and want to know where you are, use pwd:

pwd

It prints the full path of your current directory.

ls(List Directory Contents)
Want to see what files and directories are inside the current directory? Use ls:

ls

To display more details, such as file permissions, size, and modification date, use the -l flag:

ls -l

2. Managing Files with `cp`, `mv`, and `rm`

cp(Copy Files or Directories)
The cpcommand is used to copy files or directories from one location to another:

cp file1.txt /path/to/destination/

To copy a directory and its contents, use the -r (recursive) option:

cp -r /path/to/source/ /path/to/destination/

mv(Move or Rename Files)
The mv command can either move files to a new location or rename them:

mv oldfile.txt newfile.txt  # Renames file
mv file.txt /new/location/  # Moves file to new directory

rm(Remove Files or Directories)
To delete files, use the rmcommand:

rm file.txt

For directories and their contents, use the -r option:

rm -r /path/to/directory/

Be careful when using rm, as deleted files are not moved to a trash bin.

3. Viewing and Editing Files with `cat`, `less`, and `nano`

cat(Concatenate and Display Files)
The cat command displays the content of a file directly in the terminal:

cat file.txt

To concatenate multiple files and display them, list all files:

cat file1.txt file2.txt

less(View File Content One Page at a Time)
The less command lets you view large files page by page, making navigation easier:

less largefile.txt

You can scroll through the file using arrow keys and press q to quit.

nano (Simple Text Editor)
For quick file editing, nanois a lightweight and user-friendly command-line text editor:

nano file.txt

Once you're done editing, press Ctrl + X to exit, Y to save changes, and Enter to confirm.

4. Searching with `grep`and `find`

grep(Search Within Files)
The grepcommand searches for a specific pattern within files. For example, to search for the word "error" in a log file:

grep "error" logfile.txt

To ignore case sensitivity, add the -i option:

grep -i "error" logfile.txt

find(Search for Files or Directories)
To locate files or directories on your system, use the findcommand. For example, to search for a file named document.txt:

find / -name "document.txt"

The findcommand is powerful and can be combined with other options to search by date, size, or file type.

5. Viewing System Information with `df`, `du`, and `top`

df (Disk Free Space)
The df command displays the amount of available disk space on file systems:

df -h

The -h option shows the output in a human-readable format (KB, MB, GB).

du (Disk Usage)
If you need to know how much space a directory is using, ducan help:

du -sh /path/to/directory/

This command provides a summary of the directory’s size.

top(Monitor System Processes)
The topcommand gives you a real-time view of running processes and their resource usage:

top

You can monitor CPU, memory usage, and even kill processes directly from this interface.

Conclusion

These essential Bash commands are your gateway to mastering the Linux command line. By practicing these commands and incorporating them into your daily workflow, you’ll become more efficient and capable of handling system tasks with ease. Bash is vast and powerful, and this guide is just the start of your journey into command-line mastery.

🔗 Support my Work

▶️ Support by Subscribing my YouTube
▶️ Explore more open-source tutorials on my website
▶️ Follow me on X
☕ Buy me a Coffee

DevOps Fun Facts That Will Make You LOL (or at Least Smirk!) 😂

Oliver Bennet — Fri, 18 Oct 2024 13:22:24 +0000

DevOps: Where Everything is ‘Automated’… Until It’s Not! 🤖💥
Sure, automation makes life easier—until you forget that one little script, and suddenly everything breaks at 3 a.m.
99% Uptime? More Like 99% Keeping Fingers Crossed 🤞
Ever notice that “99% uptime” feels more like 99% praying that nothing crashes right before the weekend?
The Real DevOps Pipeline: Coffee In, Code Out ☕➡️💻
Let’s be real—coffee is the true CI/CD pipeline. Without it, nothing gets deployed!

“It Works on My Machine!” - The DevOps Battle Cry 🛠️🔥
The number one excuse for every bug ever. Too bad production didn’t get the memo about your machine.
DevOps: Bridging the Gap Between “It Should Work” and “Why Isn’t It Working?” 🌉
DevOps is like being the peacemaker between developers and operations, only with more firefighting involved. 🧑‍🚒
Automation: Making It Easier to Break Things Faster! 💨💻💣
Automation’s great until you accidentally automate a mistake across every server in record time. Oops!
Server's Down? Just Turn It Off and On Again… Virtually 🖥️🔄
The most advanced DevOps solutions are surprisingly similar to what your grandma does with her Wi-Fi router.

▶️ Check out my Videos on Youtube:Youtube

▶️ Explore more open-source tutorials on my website:https://graphpe.com

▶️ Follow me on X for updates:X

▶️ Check out my articles on dev.to:Dev.to

☕ Buy me a Coffee Here: Coffee

Python's Integer Precision is Infinite

Oliver Bennet — Tue, 01 Oct 2024 09:34:31 +0000

In most programming languages, integers are limited by the system architecture (e.g., 32-bit or 64-bit), meaning they can only hold numbers up to a certain size. However, in Python, integers have unlimited precision. This means you can handle incredibly large numbers without worrying about overflow.

For example, you can calculate huge numbers like factorials or perform astronomical calculations without hitting a limit:

>>> 999999999999999999999999999999999 ** 100

Python handles this gracefully, returning the massive result as expected. This flexibility makes Python ideal for fields requiring large-scale numerical computations, like cryptography or scientific computing!

Oliver | www.graphpe.com

Step by Step Guide: How to Install Grafana on Ubuntu 24.04

Oliver Bennet — Mon, 30 Sep 2024 13:43:31 +0000

Hello devs, I have created a step by Guide on How to Install Grafana on Ubuntu 24.04. Please have a look at it and let me know your feedback.

Also, a simple video tutorial here!!!

DEV Community: GraphPe

Essential Linux Text Manipulation Tools: cut, sort, uniq, tr 🛠️

4 Most Essential Text Manipulation Tools in Linux: cut, sort, uniq, tr | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 20, 2024 ・ graphpe.Medium

Linux #TextProcessing #CLI #OpenSource #DevOps #BashScripting

Prometheus 3.0: A Quantum Leap in Monitoring

Key Features and Enhancements

Native Histograms:

OpenTelemetry Support:

Enhanced PromQL:

Improved Alerting:

New Web UI:

Why Upgrade to Prometheus 3.0?

Breaking Changes

Conclusion

Recent Articles

Effortless File Transfers: A Guide to Using scp and rsync | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 15, 2024 ・ graphpe.Medium

25 Advanced Bash Commands Every Linux User Should Know | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 8, 2024 ・ graphpe.Medium

Top 10 SED Use Cases Every Linux User Should Know | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 5, 2024 ・ graphpe.Medium

Linux Aliases You Didn’t Know You Needed: Practical Shortcuts for Every User | by Oliver Bennet | Nov, 2024 | Medium

Oliver Bennet ・ Nov 15, 2024 ・ graphpe.Medium

Recent Videos

🎉 Dive into Python 3.13: New Features, JIT Compiler, Free threading & Installation Guide! 🎉

Introduction:

What’s New in Python 3.13?

Python 3.13 is here and It could be a Game Changer | Medium

Oliver Bennet ・ Nov 12, 2024 ・ graphpe.Medium

Python 3.13 Features & Installation Guide

Automate Database Backup with Bash, Python, and PowerShell

Prerequisites

Part 1: MySQL Backup Automation

Part 2: PostgreSQL Backup Automation

Part 3: MSSQL Backup Automation with PowerShell

Backup Retention Policy

Conclusion

I Started Learning Docker a Year Ago, Now I make my Own Containers.

What is Docker?

Key Benefits of Docker:

Core Docker Concepts

Why Learn Docker?

Getting Started with Docker: Step-by-Step Guide

Conclusion

10 Essential Linux Commands Every Site Reliability Engineer (SRE) Should Know

Introduction

1. uptime- System Uptime and Load

2. journalctl - System Logs Access

3. free- Memory Usage

4. iostat- Input/Output Statistics

5. lsof - List Open Files

6. dstat - System Resource Statistics

7. curl- Test HTTP Services

8. ping and traceroute- Network Connectivity and Path Analysis

9. sar - System Activity Report

10. systemctl- Manage Services

Conclusion

Automating Backup and Restore Processes with Python, Bash, and PowerShell

Why Automate Backups and Restores?

Backup Strategy: Full vs Incremental

Automating Backups with Python

Automating Backups with Bash

Automating Backups with PowerShell

Restoring Backups

Conclusion

Mastering Bash: Essential Commands for Everyday Use

1. Navigating File System with cd, pwd, and ls

2. Managing Files with cp, mv, and rm

3. Viewing and Editing Files with cat, less, and nano

4. Searching with grepand find

5. Viewing System Information with df, du, and top

Conclusion

DevOps Fun Facts That Will Make You LOL (or at Least Smirk!) 😂

Python's Integer Precision is Infinite

Step by Step Guide: How to Install Grafana on Ubuntu 24.04

Oliver Bennet ・ Nov 20, 2024 ・
graphpe.Medium

Oliver Bennet ・ Nov 15, 2024 ・
graphpe.Medium

Oliver Bennet ・ Nov 8, 2024 ・
graphpe.Medium

Oliver Bennet ・ Nov 5, 2024 ・
graphpe.Medium

Oliver Bennet ・ Nov 15, 2024 ・
graphpe.Medium

Oliver Bennet ・ Nov 12, 2024 ・
graphpe.Medium

1. `uptime`- System Uptime and Load

2. `journalctl` - System Logs Access

3. `free`- Memory Usage

4. `iostat`- Input/Output Statistics

5. `lsof` - List Open Files

6. `dstat` - System Resource Statistics

7. `curl`- Test HTTP Services

8. `ping` and `traceroute`- Network Connectivity and Path Analysis

9. `sar` - System Activity Report

10. `systemctl`- Manage Services

1. Navigating File System with `cd`, `pwd`, and `ls`

2. Managing Files with `cp`, `mv`, and `rm`

3. Viewing and Editing Files with `cat`, `less`, and `nano`

4. Searching with `grep`and `find`

5. Viewing System Information with `df`, `du`, and `top`