DEV Community: James Ackah

Streamlining Data Security and Accessibility Through Automation

James Ackah — Thu, 05 Dec 2024 23:06:11 +0000

Overview

This project automates the process of scanning a specified directory for files that haven't been accessed in a configurable period (default: 90 days). These files are moved to an archival folder and uploaded to an AWS S3 bucket for long-term storage. A cron job is set up to run the script periodically, ensuring consistent archival and backup.

Features

Automated File Scanning: Detects and moves files not accessed for a specified period to an archive directory.
AWS S3 Integration: Syncs archived files to a designated S3 bucket for reliable cloud storage.
Configurable Retention Period: Allows customization of the unused file retention period.
Logging: Maintains a log file for monitoring and troubleshooting the archival process.
Error Handling: Ensures robust operation with proper error handling and notifications.

Requirements

AWS CLI: Installed and configured with appropriate IAM credentials for S3 access.
- AWS CLI Installation Guide
Bash Shell: The script is designed to run in a Unix/Linux environment.
Cron: For scheduling automated execution of the script.
Sudo/Root Privileges: Necessary for file management in system directories.

Usage

1. Clone the Repository

git clone https://github.com/your-username/automated-log-archival.git
cd automated-log-archival

2. Update Configuration

Edit the script (archive_and_backup.sh) to customize the following variables:

SCAN_DIR: The directory to scan for unused files.
RETENTION_DAYS: The period (in days) after which files are considered unused.
ARCHIVES_DIR: The directory where files will be archived locally.
S3_BUCKET: The AWS S3 bucket name for cloud backup.

3. Grant Execution Permissions

chmod +x archive_and_backup.sh

4. Test the Script

Run the script manually to ensure it works as expected:

sudo ./archive_and_backup.sh

5. Automate with Cron

Schedule the script using a cron job:

sudo crontab -e

Add the following line to run the script daily at midnight:

0 0 * * * /path/to/archive_and_backup.sh

Logging

The script logs its operations to:

/var/log/archive_cleanup.log

Check this file for information on completed tasks or troubleshooting errors.

Contributions

Contributions, issues, and feature requests are welcome! Feel free to submit a pull request or open an issue.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Author

James Ackah-Blay

LinkedIn Profile Contributions, issues, and feature requests are welcome! Feel free to submit a pull request or open an issue.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Author

James Ackah-Blay

LinkedIn Profile

Automated Backup to S3

James Ackah — Thu, 05 Dec 2024 22:51:05 +0000

Automated Backup to S3

This project is a Bash script designed to automate the process of creating backups, uploading them to an Amazon S3 bucket, and sending email notifications to keep administrators informed about the status of the backup process.

Features

Automated Backup: Creates a compressed archive of the specified directory.
S3 Integration: Uploads the backup file to a designated Amazon S3 bucket.
Email Notifications: Sends email alerts for each stage of the process (start, success, or failure).
File Cleanup: Deletes local backup files after successful upload to S3.

Prerequisites

Before using the script, ensure the following requirements are met:

AWS CLI: Installed and configured with appropriate credentials.
Email Client: mutt installed and configured to send emails.
Bash Shell: Running on a Linux/Unix system with Bash.
Directories:
- Source directory to be backed up: /home/ubuntu/System_Data
- Backup destination: /home/ubuntu/system_backup/

Usage

Clone or Create the Script:
Save the provided Bash script as backup.sh on your system.
Modify Configuration:
Update the script variables as needed:
- BACKUP_SOURCE: Path to the directory to be backed up.
- BACKUP_DEST: Path where the backup file will be stored temporarily.
- S3_BUCKET: S3 bucket where the backup will be uploaded.
- ADMIN_EMAIL: Email address to receive notifications.
Set Execute Permissions:
Make the script executable:

   chmod +x backup.sh

Run the Script: Execute the script manually or schedule it as a cron job:

   ./backup.sh

Set Up Cron Job (Optional): Automate the backup by adding a cron job:

   crontab -e

Add the following line to run the script daily at midnight:

   0 0 * * * /path/to/backup.sh

Script Workflow

Send an email notification that the backup process has started.
Creates a compressed archive (.tar.gz) of the specified directory.
Check if the backup creation was successful:
- If successful, send a success email.
- If failed, send a failure email and terminate the script.
Upload the backup file to the specified S3 bucket.
Check if the upload was successful:
- If successful, send a success email and delete the local backup file.
- If failed, send a failure email and terminate the script.

Example Configuration

BACKUP_SOURCE="/home/ubuntu/System_Data"
BACKUP_DEST="/home/ubuntu/system_backup/"
BACKUP_FILE="backup_$(date +%Y%m%d).tar.gz"
S3_BUCKET="s3://linuxacademy-awscli-backup"
ADMIN_EMAIL="jamesblay80@gmail.com"

Dependencies

tar: Used to compress the backup files.
aws-cli: Used to upload files to S3.
mutt: Used to send email notifications.

Error Handling

The script uses $? to check the success of each operation.
If any operation fails, the script sends a failure notification and exits with a non-zero status.

Notes

Ensure the AWS CLI is configured with credentials that have appropriate permissions to upload to the specified S3 bucket.
Email notifications require mutt to be properly configured on the server.

Contact

For questions or assistance, please get in touch with James Ackah-Blay at jamesblay80@gmail.com.``