DEV Community: Niran

Your Current Backup Automation Is Missing the Key: Ansible, AWS S3, and Cron 🔑

Niran — Sun, 20 Oct 2024 06:50:41 +0000

Your current backup automation is missing the key: Ansible, AWS S3, and Cron. Data backups are a crucial part of any system or application, ensuring that you don't lose important files when things go wrong. While you may already have an automation process in place, manual backups can still be time-consuming and prone to human error.

That's where enhancing your automation comes in! In this blog, I'll show you how to supercharge your existing backup process by automating the backup of critical files and application data to AWS S3 using Ansible. We'll also set up Cron Jobs to handle the scheduling, ensuring your backups run smoothly and on time. Additionally, we'll cover how to restore those backups in case you ever need to retrieve them.

To set up the automated backup and restore process using Ansible, AWS S3, and Cron Jobs, you need to install and configure the following components:

AWS Account:

Ensure you create an AWS account if you don't have one.

AWS CLI:

Install the AWS Command Line Interface (CLI) to manage your AWS services from the command line.
After installing, configure the AWS CLI with your AWS credentials.

aws configure

You’ll need your AWS Access Key ID, Secret Access Key, default region, and output format.

Ansible:

Install Ansible on your Linux server.

Python and Boto3:

Ansible requires Python to run, so ensure it's installed.
Install Boto3 (the AWS SDK for Python), which Ansible uses for AWS-related tasks.

pip3 install boto3

If you encounter the "externally-managed-environment" error when trying to install boto3 using pip3, follow these steps to resolve the issue:

Install the Virtual Environment Package: First, ensure you have the necessary package to create virtual environments. Run the following command:

sudo apt install python3-venv

Create a Virtual Environment: Use the following command to create a new virtual environment. You can name the folder anything you like; in this example, we'll use myenv:

python3 -m venv myenv

Activate the Virtual Environment: Activate the virtual environment with this command:

source myenv/bin/activate

Your terminal prompt will change to indicate that you are now working inside the virtual environment.

Install boto3 Using pip3: Now, you can install boto3 without encountering the previous error. Run:

pip3 install boto3

Verify the Installation: After installation, you can check if boto3 was installed correctly by running:

pip3 show boto3

Deactivate the Virtual Environment: Once you're done working, you can exit the virtual environment by running:

deactivate

Ansible Collections:

Install the necessary Ansible collections for AWS to ensure smooth integration with your cloud environment:

ansible-galaxy collection install amazon.aws

Cron:

Cron is typically pre-installed on Linux systems. You can check if it's running with:

systemctl status cron

If it's not installed, you can install it:

sudo apt install cron -y

Additional Tools (Optional):

You might consider installing tools like gzip or tar if they're not already installed, as these are often used for compressing files.

With all the necessary components installed and configured, here’s your next step:

Create S3 Bucket via AWS CLI:

aws s3 mb s3://my-app-backups --region us-east-1

S3 Bucket Permissions:

Read Access: The IAM user or role used in your Ansible playbook must have permission to list and read objects from the specified S3 bucket. This is typically done through IAM policies. For example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::my-app-backups"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::my-app-backups/*"
        }
    ]
}

Replace my-app-backups with your desired bucket name, and adjust the region if needed.

Once the bucket is created, you can begin uploading files automatically using Ansible playbooks. Let’s start by creating the playbooks:
You’ll need two playbooks:

One for backing up files to AWS S3.
One for restoring the files from AWS S3.

Example: Backup Playbook (backup.yml)
This playbook compresses a directory and uploads the tarball to your S3 bucket.

---
# Playbook to back up the /etc/myapp directory to AWS S3.

- name: Backup /etc/myapp to AWS S3
  # Defines the play name, indicating its purpose.
  hosts: localhost
  # Specifies that the play will run on the local machine (localhost).
  tasks:
    # The list of tasks to be executed.

    - name: Compress /etc/myapp directory
      # Task to compress the directory into a tarball.
      archive:
        path: /etc/myapp
        # Source directory that will be compressed.
        dest: /tmp/myapp_backup.tar.gz
        # Destination path for the compressed tarball in the /tmp directory.

    - name: Upload tarball to AWS S3
      # Task to upload the tarball to an S3 bucket.
      aws_s3:
        bucket: my-app-backups
        # Name of the S3 bucket where the backup will be stored.
        object: backups/myapp_backup_{{ ansible_date_time.iso8601 }}.tar.gz
        # Specifies the object key (file path) in the S3 bucket. It includes the current date and time in ISO 8601 format to make the filename unique.
        src: /tmp/myapp_backup.tar.gz
        # The source file (the tarball) to be uploaded.
        mode: put
        # Specifies that this operation is a 'put' (upload) to the S3 bucket.
      delegate_to: localhost
      # Ensures the upload is performed from the localhost.

Example: Restore Playbook (restore.yml)
This playbook downloads the latest backup and restores it to the original location.

---
- name: Restore files from S3 bucket to local
  hosts: localhost
  gather_facts: no

  tasks:
    - name: Ensure the restore directory exists
      file:
        path: /home/niran/restore  
        # Specify the path for the restore directory
        state: directory  
        # Ensure the directory is created if it doesn't exist

    - name: List objects in the S3 bucket
      amazon.aws.s3_object:
        bucket: my-app-backups  
        # Specify the S3 bucket to list objects from
        mode: list  
        # Set the mode to list objects
      register: s3_objects  
      # Register the output to use in later tasks

    - name: Debug full S3 objects output
      debug:
        var: s3_objects  
        # Display the full output of the S3 objects for debugging purposes

    - name: Download files from S3 bucket to local
      amazon.aws.s3_object:
        bucket: my-app-backups  
        # Specify the S3 bucket to download files from
        object: "{{ item }}"  
        # Use the item from the loop to specify the object to download
        dest: "/home/niran/restore/{{ item | basename }}"  
        # Set the destination for downloaded files
        mode: get  
        # Set the mode to download files
      loop: "{{ s3_objects.s3_keys }}"  
      # Loop through the keys of the S3 objects
      when: s3_objects.s3_keys is defined and s3_objects.s3_keys | length > 0  
      # Only run if there are keys available

Before setting up the Cron job, manually run the playbooks to ensure they work as expected:
Backup Test
Run the backup playbook:

ansible-playbook /path/to/backup.yml

Verify that the compressed file is created and uploaded to S3.

Restore Test
Run the restore playbook:

ansible-playbook /path/to/restore.yml

Verify whether the restored files have been downloaded.

Note: Ensure that the necessary permissions are set for both the S3 bucket and the local file system to avoid any access issues during the restore process.

Set Up the Cron Job:

Open the Crontab file for editing:

crontab -e

Add the Cron job line at the end of the file, for example:

30 11 * * * ansible-playbook /path/to/backup.yml

Save and exit the editor.
You can verify that your Cron job has been set up correctly by listing the current Cron jobs:

crontab -l

Note:Make sure to replace /path/to/backup.yml with the actual path to your Ansible backup playbook. Also, ensure that the user executing the Cron job has the necessary permissions to run Ansible and access the required files and directories.

By automating backup and restore processes with Ansible, AWS S3, and Cron jobs, we’ve ensured reliable and consistent management of critical data. This approach not only simplifies complex tasks but also provides an efficient, scalable, and repeatable solution that can be adapted for various use cases. With daily backups scheduled via Cron and seamless restores handled through Ansible playbooks, your data management processes can be fully automated, saving both time and reducing the risk of human error.

For those looking to implement this solution, you can find the full project repository on GitHub here.

~!@#$%^&*()_+
Happy automating!

What Really Happens When You Type "amazon.com" in Your Browser? 🤔

Niran — Mon, 30 Sep 2024 14:05:26 +0000

Ever clicked on a website and wondered what goes on in the background? Take "amazon.com," for instance. The truth is, no matter which website you visit, the magic behind it works in pretty much the same way. Think about it: from locating the right server to loading all the images and content on your screen, there’s a lot happening in the blink of an eye! Let's dive in and explore what really happens, step by step.

While we're using Amazon as an example, these steps apply to any website you visit!

Step1 🚦 : When you type a URL like "amazon.com" into your browser, the first thing it needs to do is figure out where to send your request. Think of websites as having addresses, much like houses; they don’t just exist by name on the internet. Each website has a unique IP address that points to its location.

To find this address, your browser reaches out to a DNS server (Domain Name System), which acts like the internet’s phonebook. It looks up "amazon.com" and finds the corresponding IP address. If your browser can’t find the address stored locally or in your Internet Service Provider's cache, it will ask other DNS servers out there to help resolve the address. This DNS lookup is the crucial first step in loading any webpage, whether it's Amazon or any other site you visit.

Now that your browser knows the IP address, it needs to connect to Amazon's server (or any website you’re trying to visit). This is done using something called TCP (Transmission Control Protocol), which is basically a reliable way for computers to chat with each other.

Step2 🤝 : To establish this connection, your device and Amazon’s server go through a process called a three-way handshake. It’s like a quick greeting where they exchange three signals back and forth to make sure they’re on the same page and ready to communicate. This handshake is really important because it sets up a reliable connection before any actual data starts flowing—no matter what website you’re accessing.

Step3 🔐 : Since websites like Amazon use HTTPS (which is the secure version of HTTP), your browser has to set up something called an SSL/TLS handshake. This process makes sure that the connection between your device and the website is encrypted and safe.

First, your browser checks Amazon’s security certificate to confirm it’s legitimate and not a scam. Once everything checks out, an encrypted tunnel is created. This means that any data you share—like passwords or credit card information—is protected from anyone trying to intercept it.

These days, most websites use HTTPS, so this secure handshake happens almost every time you visit a site, especially if it’s an online store.

Step4 📩 : Once your connection is all secure, your browser is set to send an HTTP request! This is basically how your browser tells Amazon’s server (or any website’s server) what you want, whether it’s loading the homepage or searching for products.

The browser sends what’s called a GET request, and it includes some extra details like cookies, user-agent info, and even your location. This helps the server personalize your experience, making it feel just right for you!

Think of it like placing an order for a webpage—you ask for what you want, and the server does its thing and sends it right back to you!

Step5 💻 : When the server gets your request, it jumps into action to process it! For instance, Amazon’s servers gather everything they need to deliver the page you asked for:

HTML: This is the backbone, the basic structure of the page.
CSS: Here’s where the styling comes in—think colors, fonts, and layouts that make everything look good.
JavaScript: This is what brings the page to life with interactive features like drop-down menus and search bars.
Images and multimedia: And let’s not forget about the visuals—product photos, videos, and all the other media that make the site engaging! This whole process happens for any website you visit, whether it’s Amazon, Google, or any other place on the web!

Step6 📦 : Once the server has processed your request, it sends an HTTP response back to your browser, packed with all the data needed to display the page you asked for!

This response includes everything: the HTML for structure, CSS for styling, JavaScript for interactivity, and any media files like images and videos.

Then, your browser gets to work assembling the page. It starts by interpreting the HTML and applying the CSS to make everything look nice. This whole process is pretty much the same for every website you visit!

Step7 🏗️ : Now it’s time for your browser to really bring the webpage to life! It does this by building the DOM (Document Object Model), which organizes all that HTML into a structured format.

Next, the browser applies the CSS to make everything look visually appealing—this is where the colors, fonts, and layouts come into play! It also runs JavaScript to add those cool interactive features, like search functions, animations, and personalized product suggestions that make your browsing experience so much better.

While all this is happening, your browser might send out a few more requests to load images, fonts, or other external scripts. Whether you’re on Amazon or any other website, this rendering step is where everything really comes together and you see the final product!

Step8 🌐 : Once the rendering is all done, your browser shows you the fully-loaded webpage! Now you can dive into Amazon’s homepage, search for products, or click through different categories—everything you need is right at your fingertips!

Step9 ⚡️ : Your browser usually keeps some data, like images and scripts, saved in its cache. This means that the next time you visit amazon.com (or any other site), it can load certain resources much faster!

Final Thoughts ✨ : Next time you hop onto a website—whether it’s Amazon, Google, or anywhere else—you’ll have a better idea of the complex yet super-fast process that makes it all happen. In just a matter of milliseconds, your browser handles all the heavy lifting, from locating the website’s server to showing you the fully rendered page. Pretty awesome, right? 🔥

!@#$%^&*()_+

Happy browsing!🚀💖

Unlocking Serverless: Build Your First Python AWS Lambda Function

Niran — Fri, 20 Sep 2024 13:32:10 +0000

Setting Up Your AWS Lambda Function 🚀

Let’s build something awesome!

Log in to AWS Management Console.
Just type “Lambda” in the AWS search bar and click on the Lambda service.
On the AWS Lambda dashboard, click “Create a function”.
Choose the “Author from scratch” option.

Why Choose “Author from Scratch” for Your AWS Lambda Function? 🤔

Build it Your Way!: Authoring from scratch gives you complete control over your Lambda function’s code, settings, and environment. Customize everything to fit your exact needs, from the code itself to how it works with other AWS services and handles different inputs.

Pick Your Language: When you author from scratch, you can choose the runtime that best suits your programming language preference, such as Python, Node.js, or Java. This gives you the power to write code exactly how you like it!

Full Control Over Permissions: Starting from scratch allows you to define or attach IAM roles and policies according to the function’s needs. AWS provides basic Lambda execution roles by default, but you can customize them for more specific permissions as your function evolves.

No Unnecessary Overhead: Unlike pre-built templates or blueprints, authoring from scratch ensures that no unnecessary code, libraries, or configurations️ are included in your function. This helps keep the function lightweight and optimized for your specific task.

Deep Dive into Serverless: Authoring from scratch is an excellent way to deepen your understanding of AWS Lambda and serverless architecture. By setting up everything manually, you gain insights into how different components interact and how to optimize them for performance and cost.

Give your function a name (e.g., ImageMetadataExtractor, as this is the function we are creating).
Select Python 3.9 (or the latest supported version) as the runtime.
Select the architecture for your Lambda function. Choose** x86_64 **(widely compatible with existing libraries and binaries).
Locate the Execution Role and choose “Create a new role with basic Lambda permissions” option.

Why Choose “Create a new role with basic Lambda permissions” ? 🤔

Choosing the “Create a new role with basic Lambda permissions” option simplifies the setup process for your AWS Lambda function by automatically granting the essential permissions needed for effective operation. This option is ideal for beginners or for those who need a quick setup without complex permission configurations. It allows your Lambda function to log execution results to Amazon CloudWatch, facilitating easy monitoring and troubleshooting. This approach minimizes overhead and lets developers focus on coding, with the flexibility to update permissions later as the application’s requirements evolve.

Advanced Settings: Optional Tweaks for Your Function You can see the Advanced Settings option when creating your Lambda function. While these settings provide valuable features, they are not mandatory for the basic operation of your function. You can choose to enable them based on your specific needs:

Enable Code Signing: Optional, but recommended for enhancing security and code integrity.
Enable Function URL: Useful if you want to expose your function as an HTTP endpoint, but not required if external access is unnecessary.
Enable Tags: While not required, using tags is a best practice for resource management and cost tracking.
Enable VPC: Necessary only if your function needs to access resources within a VPC; otherwise, it can be skipped.

Click “Create Function”. AWS will create the Lambda function and take you to the function’s configuration page.

Time to Code! 💻

Writing the Python Code

Scroll Down to the Code Source Section:

You’ll see a basic code editor where you can modify the Lambda function code.

AWS Lambda doesn’t come pre-equipped with all the tools you might need. It’s like trying to build a house without a toolbox. Similarly, Lambda requires you to bring along specialized equipment, like the Pillow library, for the function we’re creating.

On your local computer, create a new folder to hold your Lambda function code and dependencies. This will be your project directory. For example, you can create a folder named “lambda_function”.

Open your terminal or command prompt and navigate to your project directory. Use the following command to install the Pillow library into your project:

pip install Pillow -t .

Create a new Python file named lambda_function.py in your project directory and also in the Lambda function editor, replace the default code with the following metadata extraction function(lambda_function.py):

from PIL import Image
import json
import io
import base64

def lambda_handler(event, context):
    try:
        # Decode the base64 image data from the event
        image_data = base64.b64decode(event['image_data'])
        image = Image.open(io.BytesIO(image_data))

        # Extract image metadata
        metadata = {
            "format": image.format,
            "size": image.size,
            "mode": image.mode,
            "info": image.info
        }

        return {
            'statusCode': 200,
            'body': json.dumps(metadata)
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': str(e)
        }

This function expertly handles base64-encoded image data. It seamlessly decodes the image, leverages Pillow’s capabilities to extract crucial metadata like format, size, and mode, and returns a JSON response for easy consumption.

After editing the code, click the Deploy button to save and deploy the function.

Ensure your working directory has these essential files and folders:

PIL/
lambda_function.py
pillow-10.4.0.dist-info/
pillow.libs/

Navigate to the directory containing these essential files and folders. Run the following command to create a deployment package:

zip -r9 lambda_function.zip PIL lambda_function.py pillow-10.4.0.dist-info pillow.libs

(This will create a lambda_function.zip file that includes all necessary files and directories.)

In the Code source section, navigate to the Upload from option.

Select the .zip file you carefully created earlier (likely named lambda_function.zip).

Click the Upload from(.zip files) button and browse to select the lambda_function.zip file.

Finally, click Save to update your Lambda function with this powerful new deployment package!

Configuring and Testing a Lambda Function 🚀
Let’s get your Lambda function up and running!

Verify the Runtime Settings.

Check the Handler setting. It should be set to:

lambda_function.lambda_handler

If it’s not set correctly, click Edit(if available), enter lambda_function.lambda_handler, and then click Save.

Click on the Test button at the function editor.

Create a new test event with a name (e.g., TestImageMetadata).

Select Private to keep the test event available only to you. This ensures that only you can access and use this test event.

Use this sample JSON data to simulate an image:

{
    "image_data": "base64-encoded-image-string"
}

Click Save changes to save the test event.

Click Test to execute the Lambda function with the test event.

Review the execution results.

Manage Environment Variables ⚙️
Scroll down to the Environment variables section on the Lambda function configuration page.

Click on Edit to modify the environment variables.

Add a new environment variable:

Key: DEFAULT_MODE
Value: RGB

Click Save to apply the changes.

Modify your lambda_function.py code to use the environment variable for the default mode if not specified.

Ensure your code includes error handling with a try-except block:

import os
from PIL import Image
import json
import io
import base64

def lambda_handler(event, context):
    try:
        # Decode the base64 image data from the event
        image_data = base64.b64decode(event['image_data'])
        image = Image.open(io.BytesIO(image_data))

        # Use environment variable for default mode if not specified
        mode = os.getenv('DEFAULT_MODE', image.mode)

        # Extract image metadata
        metadata = {
            "format": image.format,
            "size": image.size,
            "mode": mode,
            "info": image.info
        }

        return {
            'statusCode': 200,
            'body': json.dumps(metadata)
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': str(e)
        }

This code will flexibly use the value of the DEFAULT_MODE environment variable if a specific mode isn't provided in the input data.

Go to the Test tab of the Lambda function.

Click Test to invoke your Lambda function with the new code.

Ensure that the test event is set up correctly and reflects the changes made.

View Logs in CloudWatch 👀
Click on the Monitor tab. Click on View logs in CloudWatch to see the execution logs. Review the logs for any errors or successful executions.

Note: For example, I’ve used an image metadata extractor code, but you can easily replace it with any Python code you need. Whether you’re building AI models, automating tasks, or creating custom tools, this setup with AWS Lambda and S3 is incredibly flexible.

Enjoy the process! 😊

Easy Web App Deployment: Python Flask, MongoDB, and Nginx with Docker Compose 🚀🐍

Niran — Sun, 08 Sep 2024 05:29:57 +0000

Welcome to this blog!💻🚀 The primary goal here isn’t to delve deeply into the code but to demonstrate how to efficiently use Docker Compose🐳 to deploy a web application, including the deployment process, using Flask🐍, MongoDB🗄️, and Nginx🌐.

Why use Docker Compose?🐳

Docker Compose is a game-changer for managing complex Docker applications with multiple containers🛠.️ Instead of juggling multiple Docker commands, you can define and configure all your services in a single docker-compose.yml file📝. This makes it easy to create, start, stop, and scale your entire application with just one command💥.

Takeaways from this Blog📝

Simplified management: Docker Compose makes it easy to manage multiple containers as a single unit⚙️
Consistent environments: Ensure your application runs consistently across different environments📦
Efficient deployment: Deploy your full-stack application with a few simple commands🚀
Isolated components: Docker containers provide isolation, preventing conflicts between services🔒
Declarative configuration: Define your application’s desired state in a YAML file📝
Scalability: Easily scale up or down individual services based on demand📈

Prerequisites🛠️

Before you start, make sure you have Docker and Docker Compose installed on your system. These tools are essential for building and running the containers for your application.

Dive In!🚀

The application is all set up and ready to go. Just clone the repository, build the Docker images, and start the containers. You’re minutes away from getting everything up and running with Docker Compose!⚙️🐳

Clone the repository to your machine⬇️:



git clone https://github.com/niranyadav03/dockerized-python-application.git

Project Structure 📂:

Project structure should look like this:



dockerized-python-application/
├── app
│   ├── app.py
│   ├── Dockerfile
│   ├── requirements.txt
│   └── templates
│       └── index.html
├── docker-compose.yml
└── nginx
    ├── default.conf
    └── Dockerfile

Navigate to the Project Directory📂:



cd dockerized-python-application

Build and Run the Containers🚢:



docker-compose up --build

Note⚠️: Make sure to run the build commands from the dockerized-python-application directory📂. This is important because the Docker Compose file (docker-compose.yml) and other essential files are located there🔍. Running the build from this directory ensures that Docker Compose can access everything it needs to set up the services properly🛠️.

Access the Application🌐:

Once the containers are up and running, you can access the application.

Web UI🌐: Open http://localhost in your browser to view the To-Do List app.

After using the To-Do List application through the UI, make sure everything is working properly by checking the MongoDB database to confirm it’s storing your data correctly😊

Understand Docker Compose🐳

Enjoy exploring your new setup with Docker Compose! 🚀

Happy Deploying! 🎉