DEV Community: Shivam Balwani

Setting Up an S3 Bucket in LocalStack

Shivam Balwani — Fri, 25 Oct 2024 09:17:04 +0000

Introduction

Working with Amazon S3 is common for cloud storage solutions, but for local testing, interacting with AWS can be inefficient and costly. LocalStack is a fully functional local AWS cloud stack that emulates AWS services. In this guide, we’ll walk through how to set up an S3 bucket in LocalStack on macOS, discuss the benefits of using this setup, and provide a full code example.

Why Use LocalStack for S3?

Using LocalStack to simulate S3 provides key benefits:

Cost Efficiency: You avoid charges from AWS.
Speed: Tests are faster since they run locally.
Offline Testing: No internet connection required.
Isolation: Reduces risks of accidentally affecting real AWS resources.

Prerequisites

Ensure the following are installed on your respective OS:

Docker (required for LocalStack) - Download here.
Python & pip (needed for AWS CLI and boto3).
LocalStack via pip or Docker.

Step 1: Install and Start LocalStack

Install LocalStack:

brew install localstack

Run LocalStack as a Docker container:

localstack start

Note: If you face permissions issues, prepend sudo to the command.

Step 2: Set Up AWS CLI for LocalStack

Install AWS CLI:

brew install awscli

Note: The above command is for macOS. Find a complete documentation on how to install awscli.

Configure AWS CLI (necessary for LocalStack usage):

aws configure

Use placeholder values:

AWS Access Key ID: test
AWS Secret Access Key: test
Region: us-east-1
Output format: json
Set LocalStack Endpoint URL:

export LOCALSTACK_ENDPOINT=http://localhost:4566

Step 3: Create an S3 Bucket in LocalStack

To create a new S3 bucket:

aws --endpoint-url=$LOCALSTACK_ENDPOINT s3 mb s3://my-local-bucket

Step 4: Verify the Bucket

Check your bucket by listing all buckets:

aws --endpoint-url=$LOCALSTACK_ENDPOINT s3 ls

Step 5: Upload and Download Files

Create a sample file:

echo "Hello LocalStack!" > testfile.txt

Upload the file to your bucket:

aws --endpoint-url=$LOCALSTACK_ENDPOINT s3 cp testfile.txt s3://my-local-bucket

Download the file:

aws --endpoint-url=$LOCALSTACK_ENDPOINT s3 cp s3://my-local-bucket/testfile.txt downloaded_testfile.txt

Step 6: Use Python and Boto3 for S3 Operations

Install Boto3

pip install boto3

Python Code for Bucket Operations The following Python script demonstrates creating a bucket, uploading a file, listing objects, and downloading a file using Boto3:

import boto3
from botocore.config import Config

# Configuration for LocalStack
localstack_config = Config(
    region_name='us-east-1',
    retries={'max_attempts': 10, 'mode': 'standard'}
)

# Initialize the S3 client with LocalStack endpoint
s3_client = boto3.client(
    's3',
    endpoint_url="http://localhost:4566",
    aws_access_key_id="test",
    aws_secret_access_key="test",
    config=localstack_config
)

bucket_name = "my-local-bucket"

# Create the bucket
s3_client.create_bucket(Bucket=bucket_name)
print(f"Bucket '{bucket_name}' created.")

# Upload a file
s3_client.upload_file("testfile.txt", bucket_name, "testfile.txt")
print("File uploaded.")

# List objects in the bucket
objects = s3_client.list_objects_v2(Bucket=bucket_name)
for obj in objects.get('Contents', []):
    print("Found file:", obj['Key'])

# Download the file
s3_client.download_file(bucket_name, "testfile.txt", "downloaded_testfile.txt")
print("File downloaded.")

Run the script:

python your_script.py

Step 7: Clean Up Resources

To delete the bucket and its contents:

aws --endpoint-url=$LOCALSTACK_ENDPOINT s3 rb s3://my-local-bucket --force

Conclusion

This article provided a step-by-step walkthrough for setting up an S3 bucket in LocalStack. This setup is ideal for local development, allowing you to test AWS S3 functionality safely without incurring costs or needing an internet connection.

Data Migration from Monolith to Microservice in Django

Shivam Balwani — Sat, 04 Sep 2021 18:37:11 +0000

Introduction

Monolith Architecture is a design model for a software program. Under this design model, we follow a traditional approach of a unified program i.e. whole code is composed in one single piece.

Due to a unified approach monolith architecture does not provide flexibility in terms of size and changes. If the application is too large then changes or updates are difficult to make. Continuous development is difficult due to a unified architecture and also fault detection is difficult. Due to a unified architecture, it provides a restriction both in size and complexity of maintenance.

To overcome the shortcomings of monolith architecture nowadays microservices are used. The main idea of microservices is to break the unified architecture into smaller segments and interconnect them, using such a design we would reduce complete failure of the system due to breaking at one point and also help in the maintenance, testing, and continuous integration.

To break a monolith into microservices we need to make microservices in such a way that it is interconnected to others but also be able to stand alone in times when an interconnected microservices breaks off or if one microservice face downtime or some bug, then it should not lead to whole system failure.

In this article, we would learn about how to separate the data of a microservice already present from a monolith and populate it into the database linked with the microservice using Django management commands. We would learn about Django Management commands and how data migration can be done using these commands.

What is Django?

Django is a python high level web framework that helps in rapid and clean development. It is free and open source. Django provides features like fast, secure and scalable thus making it one of the most popular choices for web development.

What are Django Management commands?

Django management commands are a part of Django apps that provide us the ability to fulfill repetitive or complex tasks through a one keyword command line instruction. Every management command has a proper script that defines how the command has to be executed in order to perform tasks

How to migrate data from monolith to microservice?

In order to perform the migration we need to follow the below steps to get and populate the data of a microservice present in a monolith to the microservice database:

A management command at monolith to filter out the data related to the microservice present in monolith in a specific format like JSON or CSV(here I will be using JSON)
Use this JSON or CSV file and extract the data and populate them in Microservice Database.

Step 1: Extracting data from monolith

This will be a management command in python so we need to write this under the following directory {django-app-name}/management/commands/fetch_data(file_name).py
Code:

import json
import sys
from django.core.management.base import BaseCommand
from django.core.serializers.json import DjangoJSONEncoder
class Command(BaseCommand):
    help = "Extracting user data to JSON format"

    def handle(self, *args, **options):
        # Get User Data from User Model in monolith        
        user_microservice_data = User.objects.all()
        for user_data in user_microservice_data:
            data = {
                "model": "User",
                "id": user_data.id,
                "username": user_data.flip_collection.id,
                "mobile_number": user_data.flip_image.id,
                "created_at": user_data.created_at,
            }
            # Dumping Data into JSON Format
            json.dump(data, sys.stdout, cls=DjangoJSONEncoder)
            sys.stdout.write("\n")

After saving this command extract using the following shell command:

python manage.py fetch_data > {data_file_name}.json

Step 2: Populating the Microservice database using this JSON File

For this we will again need to write a management command on microservice project and upload the json file in microservice project.
Directory :{django-app-name}/management/commands/populate_data(file_name).py
Code:

import json
import sys
import logging
from dateutil import parser
from django.core.management.base import BaseCommand

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    help = "Populating User data obtained in JSON from Monolith."

    def handle(self, *args, **options):
        for line in sys.stdin:
            data = json.loads(line)

            # Populating User Model
            if data["model"] == "User":
                user= User(
                    user_id=data["id"],
                    username=data["username"],
                    mobile_number=data["mobile_number"],
                    created_at=parser.parse(data["created_at"]),
                )
                user.save()
                logger.debug("User populated:{}".format(user.user_id))

After saving this command populate using the following shell command:

cat {data_file_name}.json | python manage.py populate_data

Conclusion:

The above process is a sample code about how you can extract data from monolith and populate in microservice. The above code can be used to get a JSON containing data of multiple tables of monolith in a same file. We can use other file types too like CSV etc. Bulk create could also be used in Django to save all entries in one go.