DEV Community: Aarush Luthra

Recovering Locked S3 Buckets in AWS Organizations using Assume Root

Aarush Luthra — Sun, 17 Aug 2025 09:05:45 +0000

📍 Scenario

Imagine you’re the one managing all AWS accounts under your organization. One day, a developer while trying to tighten security, applies a policy so restrictive that the he blocks out everyone including himself in the process. The policy? Something like the one below

{
  "Sid": "DenyAllExceptPipeline",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::project-prod-1",
    "arn:aws:s3:::project-prod-1/*"
  ],
  "Condition": {
    "StringNotEquals": {
      "aws:PrincipalArn": "arn:aws:iam::150641481443:role/PipelineAccessS3"
    }
  }
}

The intent?
The developer wanted to allow only the pipeline to update or upload to the bucket. No manual changes allowed.

What Went Wrong?
Mistakes like these often happen in fast-paced environments with multiple contributors. In this case:

No administrator or even the developers own arn was included in the policy
IAM users, even with full admin permissions can’t access the bucket to update configurations
The pipeline can access the bucket, but you can’t invoke it to delete/fix the policy

In standalone accounts, someone with root credentials would have to log in and fix it.

But this account was created under your organization using secure defaults. That means there is no root login profile. And now, you’re locked out of the critical bucket, and IAM can’t help.

This is where AssumeRoot comes to the fore.

How will the journey look like?

Understanding Assume Root

Specifically designed for AWS Organizations, AssumeRoot provides a set of temporary credentials that help perform certain privileged tasks on a member account, such as removing a bucket policy.

While it may look similar to AssumeRole, but it’s different. AssumeRoot doesn’t rely on trust policies or IAM delegation and is much more limited in scope. It’s intended for specific and critical tasks.

Importantly, it isn’t a substitute for root access; instead, it supports a limited set of critical operations, such as:

Can We Use Custom Policies? The short answer is No. But there's more to it, and the details are just a scroll ahead! Before we dive in, make sure your setup meets the prerequisites below. These are essential to ensure everything runs smoothly.

Prerequisites

AWS Organizations is enabled
You have access to the Management account
A misconfigured S3 bucket policy in the member account.

Testing What Doesn’t Work

Before we look at how AssumeRoot helps, let’s first explore what might seem like valid approaches but fall short.

Let’s looks at the key parameters required for making an AssumeRoot request:

Action: This is the operation to perform
Version: STS API version
TargetPrincipal: Account ID of the member account
TaskPolicyArn.arn: ARN of the policy
DurationSeconds: Session duration (up to 900 seconds)(Optional)

Now, let’s look at some of the attempts.

API Request via cURL

Can we hit the endpoint like a REST API using a custom policy?

curl "https://sts.amazonaws.com/?Version=2011-06-15&Action=AssumeRoot&TargetPrincipal=156041421443&TaskPolicyArn.arn=arn:aws:iam::aws:policy/root-task/MyPolicy"

What response do we get?

We get “Request is missing Authentication Token”. This indicates that the requests must be signed using AWS Signature Version 4 (SigV4).

Signed Request with Custom Policies

To sign the request, let’s use Postman

Once request parameters are set, head over to authorization tab and set the Auth Type as AWS Signature.

Fill in the following details:

AccessKey and SecretKey: Use credentials from an IAM user in the management account (Do not use root credentials).
AWS Region: e.g., eu-north-1, ap-south-1
Service Name: sts

Let’s test it now! What do we see in the request body?

Custom Policies Not Allowed

We receive a response stating that only AWS Managed Policies are allowed! So let’s correct it by replacing the custom policy with an AWS Managed Policy

Let’s resent the call, and see what happens

Root Credentials Management not enabled

This time, we get error code 403 as Root credentials management feature wasn't enabled. To fix it, let’s make some changes in the management account.

Navigate to Account settings under IAM. We’ll enable Root credentials management. And enable Privileged root actions in member accounts, as well. This will help us to take actions in the member account.

Making a Signed AssumeRoot Request

After making the necessary changes, let’s send the request

And it was successful !! We get a response that has the AccessKey, SecretKey and thenSessionToken.

🔑Using The Temporary Credentials

Next, we’ll set the AccessKey, SecretKey and the SessionToken in our AWS CLI.

We can temporarily set them via:

export AWS_ACCESS_KEY_ID=<Your access key>
export AWS_SECRET_ACCESS_KEY=<Your secret key>
export AWS_SESSION_TOKEN=<Your token>

To verify that we have successfully accessed the account, run

aws sts get-caller-identity

Deleting the Misconfigured Bucket Policy

Now that you've configured the temporary credentials in your terminal, let's proceed to delete the misconfigured bucket policy.

Run the following command to delete the bucket policy:

aws s3api delete-bucket-policy --bucket project-prod-1

And that’s it we’ve successfully assumed root and taken action.

References: AssumeRoot

Thank you
Aarush Luthra

Unpacking Containers: CloudWatch’s Role in Docker Monitoring

Aarush Luthra — Wed, 06 Aug 2025 05:12:02 +0000

Introduction

Docker has revolutionized the way we run applications. It has enabled lightweight, portable, and scalable environments. Docker enables one to separate applications from infrastructure so as to ensure quick delivery of software.

What is Docker?

Docker provides the ability to package and run an application in a loosely isolated environment called a container. These containers package your application's code, libraries, and dependencies into a single, portable unit.

Why Monitor Docker Logs?

Running containers in production isn’t just about deploying them; monitoring logs is critical for:

🛠️ Debugging issues

📊 Gaining real-time visibility into application behavior.

🔒 Enhancing security.

In this blog, we'll explore how to manage monitor Docker logs using CloudWatch, with focus on using the CloudWatch Agent and Docker's 'awslogs' log driver.

Monitoring Logs with CloudWatch

Applications aren’t "set and forget." Although Docker streamlines deployment, monitoring logs is essential. Without effective monitoring, you could encounter issues such as undetected performance bottlenecks and hard-to-trace bugs ⚠️

Why CloudWatch?

Container logs, being ephemeral don’t persists in case the container crashes or is removed. This is where CloudWatch plays a crucial role.

CloudWatch provides centralized log management, with logs from multiple containers being aggregated at a single place. With logs in CloudWatch, one can set up alerts for specific error messages, filter, search and analyze log data. CloudWatch seamlessly integrates with other AWS services, like AWS Lambda, SNS, SES and what not!

Prerequisites

Access to an EC2 Instance: Ensure an active EC2 instance is running
Docker Installed: Docker should be installed and properly configured on the EC2 instance.
IAM Role: Instance should have an IAM role attached with the necessary permissions to write logs to CloudWatch

Monitoring Logs

Using Amazon CloudWatch Agent

CloudWatch Agent is used to monitor system-level metrics and logs from the EC2 instance or from Docker containers.

Setting up Cloud Watch Agent

1. On your EC2 instance, install the CloudWatch agent

sudo yum install amazon-cloudwatch-agent -y

2. Configure the agent using the wizard

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Or simply create a ‘config.json’ file with the below mentioned configurations

nano /opt/aws/amazon-cloudwatch-agent/bin/config.json  # Opens the file to enter configurations

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/lib/docker/containers/*/*.log",
            "log_group_name": "DockerLogsEC2",
            "log_stream_name": "{instance_id}",
            "timezone": "Local"
          }
        ]
      }
    }
  }
}

3. Now, we apply the configuration to the CloudWatch agent and start it. To do so, hit the following commands

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json \
  -s

4. CloudWatch agent will be successfully started.

5. Now we can test it by running a container and checking for its logs in CloudWatch

docker run hello-world   # Simple hello-world container

6. The container runs successfully and its logs can be seen in CloudWatch

This approach has certain drawbacks; Logs from all containers are written to the same log stream making it challenging to differentiate logs for individual containers. Additionally, the CloudWatch Agent runs as a separate process, consuming resources and requiring high availability to ensure logs are forwarded without interruption. Any failure or misconfiguration in the agent can result in missing or delayed log data.

Using ‘awslogs’ Driver

‘awslogs’ driver, is a native docker logging driver and eliminates the need for an additional process like the CloudWatch Agent. It creates unique log streams for each container, making it easier to track and analyze logs

Setting up ‘awslogs’ driver

1. Firstly, we create a log group in CloudWatch, let’s name it ‘DockerLogsEC2’

Now, configure docker logging by adding the following script in daemon.json

sudo nano /etc/docker/daemon.json

{
  "log-driver": "awslogs",
  "log-opts": {
    "awslogs-region": "eu-north-1",
    "awslogs-group": "DockerLogsEC2",
    "tag": "{{.Name}}-{{.ID}}"
  }
}

Next, restart docker

sudo systemctl restart docker

Now, we run a docker container and see if logs are available in CloudWatch.

docker run hello-world  #Run a hello-world container

As seen above, log stream has been created with container name and its ID. We can also test whether another stream will be created or not for another container

docker run alpine echo "Hello Aarush"   #Runs alpine container and prints the message

A new log stream will be created for every container!

Conclusion

Sending docker logs to Amazon CloudWatch is an effective way to centralize log management, improve system monitoring and simplify troubleshooting. Additionally, CloudWatch offers CloudWatch Insights, a powerful tool for querying logs and gaining deeper insights into your data. Both the Amazon CloudWatch Agent and the awslogs driver serve distinct roles

CloudWatch agent is best suited for environments where log collection from various sources such as system-level metrics and application logs is required.

On the other hand, the ‘awslogs’ driver offers a simpler, more streamlined setup for basic logging needs. It’s suitable for use cases where one wants to push container logs to CloudWatch without needing additional customization

Ultimately, integrating CloudWatch into your Docker workflows can significantly streamline operations, improve reliability, and help maintain better control over your infrastructure.

Happy Learning

Aarush Luthra

Connect with me on LinkedIn : Aarush Luthra

Cosmic Navigator: Unveiling Constellations from CSV to Web

Aarush Luthra — Tue, 29 Jul 2025 19:54:39 +0000

Introduction

This blog guides one through an approach of using AWS services that automates data flow from a CSV file to DynamoDB table, and then rendering it on a webpage, with user interaction to retrieve information about a specific Name.

This is possible with help of key AWS Services like EC2, Lambda, S3, API Gateway, and DynamoDB, along with essential features like AWS CloudWatch and IAM.

Let’s begin as we navigate through the clouds!!

Diagram

Features used:

Amazon S3: For secure and scalable object storage of your CSV file and website assets

Lambda Function: For executing server-less functions in response to events.

Amazon EC2: For hosting your frontend code

Amazon API Gateway: For building REST API, that is used for accepting user input and interacting with Lambda functions.

AWS CloudWatch: Monitoring, logging and troubleshooting

AWS IAM: For managing access and security permissions.

Step by Step Guide:

Create a S3 Bucket:

Create a S3 bucket, this acts as the storage for the CSV dataset
Set up DynamoDB

Create a DynamoDB table, where partition key is Name

Configure a Lambda function

a) Create a lambda function and create a trigger for S3 bucket, ensuring the Lambda function gets invoked upon file upload.

b) Implement the function to copy data from the CSV file into a DynamoDB table.

Lambda Function:

import json
import boto3
import io
import csv 
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('tablename')

s3Client = boto3.client('s3')
def lambda_handler(event, context):
    print(f"Event: {json.dumps(event)}")
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    print(f"Processing file: {key}")

    response = s3Client.get_object(Bucket=bucket, Key=key)
    data = response['Body'].read().decode('utf-8')
    reader = csv.reader(io.StringIO(data))
    next(reader) 

    for row in reader:
         item = {
                    'Name': row[0],
                    'Description':(row[1]),
                    'Major Stars': (row[2]),
                    'Asterisms':(row[3]),
                    'Season of Visibility':(row[4]),
                    'Interesting Facts':(row[5])
                    'S3Key':(row[6]),
                    'ImageURL':(row[7])

                }              
         table.put_item(Item=item)

    print("Data stored in DynamoDB.")

    return {
        'statusCode': 200,
        'body': 'Data processed and stored in DynamoDB.'
    }

Configure an API Gateway

Create a REST API with a resource, method(POST or GET) and enable CORS

Configure a Lambda function

Create a new Lambda Function, this will respond to API requests and fetch data from backend database

import boto3

def lambda_handler(event, context):
    dynamodb = boto3.resource('dynamodb')
    table_name = 'tablename' 
    table = dynamodb.Table('tablename')

    try:
        name = event['name']
        except Exception as e:
        return {
            'statusCode': 400,
            'headers': {
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Methods': 'OPTIONS,POST'
            },
            'body': 'Error parsing request body: {}'.format(str(e))
        }

    try:
            response = table.query(
            KeyConditionExpression='#name = :name',
            ExpressionAttributeNames={'#name': 'Name'},
            ExpressionAttributeValues={':name': name}
        )

        items = response.get('Items', [])

        if items:

            result = {}
            for item in items:
                result['Name'] = item['Name']
                result['Description'] = item['Descrption']
                result['Major Stars'] = item['Major Stars']
                result['Asterisms'] = item['Asterisms']
                result['Season of Visibility'] = item['Season of Visibility']
                result['Interesting Facts'] = item['Interesting Facts']
                result['ImageURL'] = item.get('ImageURL') 

            return {
                'statusCode': 200,
                'headers': {
                    'Access-Control-Allow-Origin': '*',
                    'Access-Control-Allow-Methods': 'OPTIONS,POST'
                },
                'body': result
            }
        else:
            return {
                'statusCode': 404,
                'headers': {
                    'Access-Control-Allow-Origin': '*',
                    'Access-Control-Allow-Methods': 'OPTIONS,POST'
                },
                'body': {'Error!': 'No details found for the provided name.'}
            }

Create a S3 bucket

a) Create another S3 bucket, this will be used to store the frontend codes

b) Modify the bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::bucket-name/*"
        }
    ]
}

Launch an EC2 instance

a) Launch an EC2 instance with the required keypairs, security groups and IAM roles

b) Click on connect, in order to configure a server and copy the frontend files
```
sudo yum update -y
sudo yum install nginx -y
sudo systemctl start nginx
sudo systemctl enable nginx
aws s3 cp s3://bucket-name/file-name /usr/share/nginx/html/
sudo systemctl restart nginx
```
c) Now, click on the IP Address and your site is up and running !!

Conclusion

As we reach the culmination, it is evident that this architecture unlocks a powerful toolkit for dynamic data management and engaging web experiences.

With automated workflows that seamlessly integrate data ingestion, processing, and presentation, businesses can adapt and innovate at the speed of thought, where they can automate transfer of a CSV dataset to a table and then render it on a webpage.

Please, feel free to drop suggestions in comments below

Thank you for reading!

Aarush Luthra

AWS Cloud Quest : Where Avatars Build Cities and You Build Skills

Aarush Luthra — Thu, 24 Jul 2025 08:06:03 +0000

“What if learning AWS could be done while playing a game🎮?” AWS Cloud Quest is the answer if you’ve wondered if it’s possible! AWS Cloud Quest brings a gamified learning experience where you don’t just learn about cloud computing—you live it.

I dived into AWS Cloud Quest, taking on the role of Solution Architect in an exciting adventure, to enhance my skills and gain more hands-on experience. Through a series of missions, I tackled numerous problems in a fun and challenging way, all while building my cloud expertise☁️

AWS Cloud Quest offers numerous roles and areas of focus including Cloud Practitioner, Serverless Developer, Solution Architect, Networking🛜, Machine Learning, Generative AI and more !!

Diving in 🤿

Getting started with AWS Cloud Quest is simple! Simply log into AWS Skill Builder and choose the role you want to explore !!

If you are starting with Cloud and want to explore the free version, Cloud Practitioner is great to begin with and for those who seek advanced challenges Cloud Quest offers you have got a plethora of missions ! I chose to begin my journey with the Solutions Architect role

Jumping into the Action 🎬

1. Customizing Your Avatar: Before you start your missions, you create your own avatar! Choose your outfit, accessories, and overall look to make your in-game character feel like “you.”

2. A City Like No Other: The stage is set with a city-like UI, with checkpoints scattered across map the map! These checkpoints represent various real-world problems, that need to be solved for citizens to thrive. Each challenge brings you closer to mastering AWS services, with every building you add, the city grows reflecting your success in overcoming challenges and showcasing your progress.

3. Hands-on Lab

Labs form the core of Cloud Quest. Each mission starts with:

a) Introduction to the problem: Understand the problems citizens are facing

b) Learn concepts: Explanation of the concepts that will be used

c) Practice steps: Walkthroughs that help ensure we understand how to apply the concepts

d) DIY challenges: Test your skills by solving the problem independently

You can choose between classic and modern UI modes. The labs simulate real-world scenarios, making the experience highly practical.

4. Build Your City with Rewards: Once you complete the DIY tasks, you get rewarded with buildings, infrastructure components that take your city to the next level! You also get customizing points that can help you customize buildings to your liking.

5. Drone Challenges and Reward Cards: This was the most exciting part of Cloud Quest for me! The mission is to capture drones hovering around the city, which earns you reward cards!

These cards represent AWS services that can be used to complete cloud architectures.

6. Pets and Vehicles: Cloud Quest doesn’t stop at learning! It provides a playful touch with pets and vehicles! Take your pets on for a walk or unlock unique modes of transport from hoverboard to paper plane!! Yes, a paper plane! The fun doesn’t end !

My Takeaway

It was truly fun playing AWS Cloud Quest! While the modern UI lags a bit, the adventurous journey, hands-on challenges, and rewarding experience more than make up for it. Put on your avatar, hop into the city, and take on the cloud like never before !!

Subscribe, Like and Share and Let’s connect at Linkedin

Thank you for reading😊

Aarush Luthra

Deploying Machine Learning Models with AWS SageMaker

Aarush Luthra — Wed, 25 Jun 2025 04:37:41 +0000

Introduction

Machine Learning models are rapidly transitioning from experimental stages to real-world applications. The journey from a fully trained model to functioning application involves several steps, the most significant being deployment. AWS SageMaker simplifies this process by offering a managed platform for building, training, and deploying models. It supports real-time predictions, manages dynamic workloads, and integrates seamlessly with other services.

This article guides you through the process of deploying a machine learning model, specifically a .pkl (Pickle) file onto AWS SageMaker and connecting it to a frontend application for real-time predictions.

By the end, you'll understand how to deploy a trained model on SageMaker and integrate it with a frontend application.

Diagrammatic Representation

Prerequisites

AWS Management Console
A trained model

Before we dive into the deployment process, make sure your model is saved as a .pkl file and then compressed into a '.tar.gz' format. If you don’t have your model in this format yet, use the following code snippet in your Jupyter notebook to save and compress your model:

import joblib
import tarfile
model = model_name  #Enter your model name here
joblib.dump(model, 'model.pkl') #Saving it as .pkl file

with tarfile.open('model.tar.gz', 'w:gz') as tar:    #Compressing into tar.gz
    tar.add('model.pkl')
files.download('model.tar.gz')     #Downloading the file locally

Setup Steps

1. Sign-in to the AWS Management Console and navigate to S3

2. Create a S3 bucket

a) Click create bucket

b) Enter a unique bucket name (e.g. 'mymlbucket') and then create it!

c) Go to your bucket and upload your 'model.tar.gz' file

3. Go to SageMaker

Open SageMaker console and set up a SageMaker domain.

Follow the steps below to create one

4. Launch SageMaker Studio

We'll use SageMaker Studio to create our own JupyterLab workspace.

Click on Studio and then click Open Studio.

This will take you to the SageMaker Studio console.

Navigate to JupyterLab and create your own workspace.

Now you're all set to begin the deployment process in SageMaker Studio!

Deploying the model

1. Install the SageMaker Python SDK
We start by installing SageMaker Python SDK in our notebook and configuring it with an IAM role to access S3. Follow these steps:

!pip install sagemaker   #This command installs SageMaker Python SDK

2. Import SageMaker and Configure the Session
Next, import the SageMaker library and setup the session using IAM role

import sagemaker
from sagemaker.sklearn.model import SKLearnModel
sagemaker_session = sagemaker.Session()
role = 'arn:aws:iam::account_id:role/service-role/AmazonSageMaker- ExecutionRole'

Now, we come to one of the most essential parts— the 'inference.py' script. This script defines how the input data will be processed during inference. It comprises several key components such as loading the model, preprocessing inputs (like converting JSON inputs to formats such as DataFrames), predicting the data, formatting the output, and most importantly, handling errors.

import json
import joblib
import os
import numpy as np
import pandas as pd

def model_fn(model_dir):
    model = joblib.load(f"{model_dir}/model.pkl")  # Loading the model
    return model

def input_fn(request_body, request_content_type):
    if request_content_type == 'application/json':
        input_data = json.loads(request_body)         

        # Ensuring correct data types for the input
        input_data['Store'] = np.int64(input_data['Store'])
        input_data['Holiday_Flag'] = np.int64(input_data['Holiday_Flag'])
        input_data['Month'] = np.int32(input_data['Month'])
        input_data['DayOfWeek'] = np.int32(input_data['DayOfWeek'])
        input_data['Temperature'] = np.float64(input_data['Temperature'])
        input_data['Fuel_Price'] = np.float64(input_data['Fuel_Price'])
        input_data['CPI'] = np.float64(input_data['CPI'])
        input_data['Unemployment'] = np.float64(input_data['Unemployment'])

        input_df = pd.DataFrame([input_data])  # Converting input to DataFrame

        expected_features = ['Store', 'Holiday_Flag', 'Temperature', 'Fuel_Price', 'CPI', 'Unemployment', 'Month', 'DayOfWeek']
        input_df = input_df[expected_features]

        return input_df
    else:
        return None

def predict_fn(input_data, model):
    prediction = model.predict(input_data)  # Making predictions
    return prediction

def output_fn(prediction, response_content_type):
    result = {'Predicted Weekly Sales': prediction[0]}
    return json.dumps(result)

3. The above steps helped us in setting up the environment and configuring the role. Now it is time to DEPLOY!

I am using the following code snippet for this. This code loads the model from S3 and we specify entry point script for inference

from sagemaker.sklearn.model import SKLearnModel

model_uri = 's3://bucket_name/model.tar.gz'  # Bucket that has the model
entry_point = 'inference.py'                 # Entry point script

model = SKLearnModel(
    model_data=model_uri,
    role=role,
    entry_point=entry_point,
    framework_version='1.2-1',               # scikit-learn version
    sagemaker_session=sagemaker_session
)

predictor = model.deploy(
    instance_count=1,
    instance_type='ml.m5.large'
)

4. After successfully completing the deployment, endpoint URL can be retrieved from SageMaker Studio console. This endpoint URL is essential for testing your model and integrating it with your frontend.

5. It's time to TEST !! We'll us this endpoint URL to test our model. Use the following code snippet. Make sure to change the input data attributes as per your model

import boto3
import json

# Initialize the SageMaker runtime client
sagemaker_runtime = boto3.client('sagemaker-runtime')

# Input data
input_data = {
    'Store': 41,
    'Month': 12,
    'DayOfWeek': 17,
    'Holiday_Flag': 0,
    'Temperature': 24.31,
    'Fuel_Price': 44.72,
    'CPI': 119.0964,
    'Unemployment': 5.106
}

# Convert input data to JSON string
payload = json.dumps(input_data)

# Invoke the SageMaker endpoint
response = sagemaker_runtime.invoke_endpoint(
    EndpointName='sagemaker-scikit-learn-2024-08-18-09',  # Replace with your actual endpoint name
    ContentType='application/json',
    Body=payload
)

# Parse and print the response
result = json.loads(response['Body'].read().decode())
print(f"Predicted Weekly Sales: {result['Predicted Weekly Sales']}")

6.And just like that, your model has been successfully deployed ! Now we can take it further by connecting it with AWS Lambda and API gateway to display predictions onto the frontend applications!

Connecting SageMaker to Lambda

1. Navigate to AWS Lambda

Click 'Create function' and use Python as runtime language

2. Set up IAM role:

It is necessary to provide right set of permissions to Lambda, these include access to AWS SageMaker and API gateway

3. After creating the function, use the following code snippet to connect it with Lambda

```python
import boto3
import json

def lambda_handler(event, context):
    print("Event:", event)

    if 'body' not in event:
        return {
            'statusCode': 400,
            'body': json.dumps({'error': "'body' missing in the event"})
        }

    sagemaker_runtime = boto3.client('runtime.sagemaker')

    try:
        response = sagemaker_runtime.invoke_endpoint(
            EndpointName='sagemaker-scikit-learn-2024-08-18-09',
            ContentType='application/json',
            Body=json.dumps(input_data)
        )
        result = json.loads(response['Body'].read().decode())
        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'predicted_sales': result})
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({'error': str(e)})
        }
```

4. Deploy the function ! You have successfully connected AWS SageMaker and Lambda!

Now, with the connection established, proceed with connecting it to API Gateway and integrating it with your frontend application as your next steps.

Frontend in Action

After successful connection, you'll be able to see your frontend receiving the predictions! Here's mine!

Conclusion

Deploying a machine learning model on AWS SageMaker bridges the gap between development and production. The above steps should help you navigate your way to deploying your first model !

If you encounter any issues or need additional insights during the deployment process, CloudWatch is your go-to tool for monitoring and troubleshooting!

Redis on AWS Made Easy: Compare, Choose, and Launch with ElastiCache for Free

Aarush Luthra — Fri, 20 Jun 2025 04:07:41 +0000

📘Introduction: Getting Started with Redis on AWS

Most of you have probably heard about Redis, the famous in-memory database that’s used almost everywhere, from small startup projects to big tech giants. Widely know for being fast⚡, reliable🔒 and flexible, it helps in caching, session storage, real-time analytics and a lot more.

Running Redis on your own means managing scaling, replication, failovers, and availability. Any failure, especially in production can be catastrophic.

AWS Managed Redis Options: ElastiCache and MemoryDB

To avoid all the hassle of setting up and managing in-memory data stores, AWS offers two managed Redis services.

⚡AWS ElastiCache

ElastiCache is a fully managed in-memory data store and cache service that supports Redis, Memcached and Valkey. It’s ideal for scenarios when we want fast access to frequently-used data like user sessions, page views, and queues. It offers two flexible deployment types namely, Serverless(provides effortless autoscaling) and Cluster-based where you design your own setup.

🛡️MemoryDB

MemoryDB combines low-latency performance of Redis with added benefits of high availability and durability. it comes with Multi-AZ support, automatic failover, and persistent storage via a transactional log, making it suitable for using it as a primary database, not just a cache.

However, if you're just starting out or want to test it for learning purposes, cost can be a barrier. While MemoryDB offers 750 free hours for 2 months, ElastiCache gives you 750 hours per month for the entire Free Tier period (12 months)

📝What you’ll explore and learn in this blog

🆚Compare Redis OSS, Memcached, and the newer Valkey.
🛠️Show you how to spin up a Redis OSS cluster for free using AWS Free Tier 🆓 using CloudShell
📊And demonstrate a real-world use case at the end.

⚖️Redis OSS vs Memcached vs Valkey

Let’s go over the key differences between Redis OSS, Memcached, and Valkey. All three are in-memory data stores, but they for slightly different purposes.

Redis OSS: It’s an open source in-memory key-value store that supports a wide range of data types such as lists, sets, bit arrays. It’s ideal for caching, queues, pub/sub. It offers memory persistence through regular database snapshots (RDB) or via the append-only file (AOF) which is a log, capturing all the performed operations. Redis has built-in replication (Master-Slave Redis Sentinel and Cluster mode).

Memcached: It’s one of the simplest open source key-value stores you can find. It’s super fast, lightweight and a great choice when you need a quick caching layer without any advanced features. This simplicity comes with some tradeoffs such as it handles only strings, doesn’t support persistence and lacks native replication support.

Valkey: Sounds unfamiliar? Launched in 2024 as fork of Redis 7.2 after it was announced that the future versions of Redis would no longer remain open source. It is a community driven key-value store providing features of Redis 7.2 without any licensing fee.

Redis vs Memcached: Which Fits Your Caching Needs?

☁️Setting Up Redis OSS on AWS Elasticache using CloudShell

Now it’s time we set our infrastructure up! Using CloudShell, we’ll spin everything up. No need to worry about having CLI and configuring it on your system

1. Open CloudShell from your console

2. Once CloudShell is running, we’ll run a series of commands.

3. Redis OSS Cluster is launched inside a VPC Cluster. To keep things simple, let’s use the default VPC. Firstly. we’ll find the VPC ID and then list the subnets associated with the VPC. To create the cluster, we’ll first need the subnet ID from this VPC

# Find the default VPC ID
aws ec2 describe-vpcs \
  --filters "Name=isDefault,Values=true" \
  --query "Vpcs[0].VpcId" \
  --output text

#Finding the Subnet IDs associated with the VPC.
#Make sure to replace <vpc-id> with your VPC ID
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=<vpc-id>" \
  --query "Subnets[*].SubnetId" \
  --output text

# Sample command to get the Subnet IDs
aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=vpc-06ae0857425ccf225" \
  --query "Subnets[*].SubnetId" \
  --output text

4. Next, we’ll create a cache subnet group. Add the subnets you received after running the above commands.

aws elasticache create-cache-subnet-group \
--cache-subnet-group-name pagecount-subnet-gp \
--cache-subnet-group-description "Subnet group for pagecount" \
--subnet-ids <subnet1> <subnet2> \
--region eu-north-1

Let’s see how do things look in the CloudShell

We can confirm the creation of subnet-group via console as well

5. Before we begin the cluster creation. We need two more things, i.e the security group and an auth-token.

We’ll create a security group that allows ingress on port 6379

#Creating a security group
aws ec2 create-security-group \
  --group-name pagecount-sg \
  --description "Security group for Redis" \
  --vpc-id <your-default-vpc-id> \  #Make sure to enter your VPC ID
  --region eu-north-1

#After running the above command, you'll get Security group ID
#Use it in the below command, where we'll add ingress rule for port 6379
aws ec2 authorize-security-group-ingress \
  --group-id <your-sg-id> \
  --protocol tcp \
  --port 6379 \
  --cidr 0.0.0.0/0    #For Demo Only, it's better to use your IP for better security

As the cluster will have TLS encryption enabled, we also need to set up a secure authentication token. This token will help in ensuring only authorized clients can connect to the Redis.

Generate the token using

openssl rand -base64 32

Now, it’s time to create the replication cluster. We’ll use cache.t3.micro node

aws elasticache create-replication-group \
  --replication-group-id pagecount \
  --replication-group-description "Redis cluster for pagecount with TLS" \
  --engine redis \
  --engine-version 7.0 \
  --cache-node-type cache.t3.micro \
  --num-node-groups 1 \
  --replicas-per-node-group 0 \
  --security-group-ids <your-security-group-id> \
  --cache-subnet-group-name pagecount-subnet-group \
  --auth-token <GeneratedRedisTokenBase64> \
  --transit-encryption-enabled \
  --at-rest-encryption-enabled \
  --region eu-north-1

Your CloudShell will look something like

Let’s wait for the cluster to spin up. Run the following command, this displays the current status of your cluster.

#Running this command will pro
aws elasticache describe-replication-groups \
  --replication-group-id pagecount \
  --region eu-north-1 \
  --query "ReplicationGroups[0].Status"

It took me approximately 10mins⌛and to have cluster up and running

👀 See Who’s Visiting — Tracking Page Views with Redis (A sample use-case)

Before we begin, it is important that the EC2 instance be in the same VPC as your ElastiCache cluster

Create an EC2 instance. Choose Amazon Linux as your AMI. Importantly, choose the default VPC and the security group we created earlier.
SSH into your instance. Install the required dependencies such as express, redis

Use the following Nodejs code snippet to connect to Redis

const redisClient = redis.createClient({
  url: process.env.REDIS_URL,
  socket: {
    tls: true, 
  },
});
redisClient.on('error', (err) => console.error('Redis error:', err));
(async () => {
  try {
    await redisClient.connect();
    console.log('Connected to Redis');
  } catch (err) {
    console.error('Redis connection failed:', err);
  }
})();

app.get('/views', async (req, res) => {
  try {
    const count = await redis.incr('page:views');
    res.json({ count });
  } catch (err) {
    console.error('Redis error:', err);
    res.status(500).json({ error: 'Failed to connect to Redis' });
  }
});

Ensure that you either export your REDIS_URL or setup a .env file. The best way? Use Parameter Store

The URL looks something like:

rediss://default:<AUTH_TOKEN>@<ENDPOINT>:6379

🧾Conclusion

Caching, today, is must for building applications and pages that are scalable and load fast. With AWS ElastiCache, setting up Redis OSS instance is fast and completely free using the AWS Free Tier.

Common real-world use cases include Leaderboard systems, Location-based features like "near me" searches, Persisting user sessions, Page-count and much more!

If you're building something that needs low latency and high throughput, AWS Managed services like ElastiCache, MemoryDB might be the one you need.

Let’s keep building and keep exploring

Thank you

Aarush Luthra

Developing with Prompts: Building a Game Using Q CLI

Aarush Luthra — Sun, 01 Jun 2025 12:39:50 +0000

Have you heard about Q Developer CLI? If you haven’t — it’s a generative AI-powered assistant. From generating bash commands that can be executed directly on your system, querying your AWS account for data, creating code modules, or even building entire games — Q can do it all!

Introduction

Building games can be fun — but also a bit overwhelming. Setting up files, linking levels and what not! It takes time before the fun part arrives..

In this blog, I’ll show you how easy it is to use Q CLI and share the outputs you get from a simple prompt. I built an interactive Dart Game using Q CLI and Pygame which helped me explore Q’s capabilities, all while participating in a fun competition!!

If you build a game using Q Developer CLI, you stand a chance to win a free T-shirt !! Check out the challenge ➡️ Q Dev CLI Challenge

Setting it up

To build my game, I used WSL (it integrates super easily into your windows system)

To install WSL, run the following command

   wsl --install -d Ubuntu

Install Q CLI, to do so refer ➡️ Installing Q CLI
Hit the following command and you are good to go!

q chat

🕹️ What I Built – The Dart Game

As part of the competition, I created a simple but addictive Dart Game using Python and Pygame, powered by Q Developer.

🧩 Game Highlights:

🎯 Level 1: Static dartboard – Learn basic aiming

🎯 Level 2: Moving dartboard – Test timing and accuracy

🎯 Level 3: Moving board + obstacles – The Test!

You get 3 darts per level, and must score enough points to move forward. The dartboard reacts to hit zones (bullseye, middle ring, outer ring), each awarding different scores.

All this was made super easy with help of Q CLI.

My prompts

Q CLI took my prompts and generated exactly what I needed. No setup stress, just describe and the magic begins✨

Whether it was setting up the initial project, linking level logic — Q CLI handled it with ease.

Some of the prompts were:

Add a dart object on the left side of the screen. Make the dart small and sharp, with a red and white tail.

Add controls; Left arrow rotates the dart counter-clockwise slightly while right arrow rotates the dart clockwise slightly. Spacebar or "Throw" button releases the dart in the direction it's pointing.

When thrown the dart travels in a straight line based on its current angle, if it hits the board, stop its motion and show a hit effect.

If some corrections are required, all it takes is another prompt

Below is an example, as to how beautifully Q summarizes all the changes:

Need more updates and detailed insights as to what was changed? Q CLI covers that too!

Check out the code here: GitHub Repo

Check out some snapshots of my game 🎯

My thoughts🤔

The Dart Game might seem simple, but it’s proof of how quickly and creatively you can bring an idea to life with Q Developer CLI. It helps you with everything from debugging to code generation!

Give Q CLI a try — and who knows? You might walk away with a cool game, new skills, and maybe even a T-shirt. 😄

Thank you

Aarush Luthra

Automating My Web App Deployment with AWS CI/CD

Aarush Luthra — Wed, 19 Mar 2025 21:58:22 +0000

Manually deploying my web application wasn’t difficult, but it quickly became annoying. Every update meant SSH-ing into my EC2 instance, copying files, installing dependencies, and restarting the server. I knew it was time to shift to a better, automated way —so I explored AWS CI/CD tools to make my life easier.

At least that was what I thought…

My first few attempts were filled with with failed builds, missing permissions and unexpected deployment failures. It took me time to get it up and running but once it was up, the advantages were clear. CodePipeline handled the entire workflow from getting the latest changes from GitHub to deploying it. CodeBuild ensured that the dependencies were installed correctly while CodeDeploy made sure the deployment on EC2 was a success!

Then came security! Out of curiosity, I tested my site using Gobuster to check for exposed directories and files. And behold—my whole logic was , up for the taking🤯! That was the moment I realized—security🔐 and deployment🚀 need to go hand in hand.

Here’s how I did it, the mistakes I made, and the steps I took to secure my site !

Why automate with AWS CodePipeline?

The next question after deciding to integrate CI/CD into my project was: Which tool to use? . There were plenty of options —GitHub actions, Jenkins, GitLab CI/CD—but AWS CodePipeline was the bestfit for my need. It checked all the right boxes:

🔐Security? Covered. IAM roles, encrypted SSM parameters.

🗄No CI/CD server. Unlike Jenkins, there’s no need to manage the underlying infrastructure.

☁️Seamless AWS Integration. Works effortlessly with EC2, SSM, DocumentDB, and more.

Prerequisites

Before we begin, ensure you have the following:

✅ Access to AWS Management Console

✅ A GitHub repository containing your application

Besides this you’ll need appropriate permissions to allow services to interact seamlessly. The diagram below gives an overview of how everything connects.

Setting up the Pipeline

It's time to build the CI/CD pipeline step by step.

Step1: Connect GitHub to CodePipeline

We create a pipeline from AWS Management console and select GitHub as source. We authorize AWS to access the repository and choose the branch. And let the magic begin 🪄

Step2: Configuring AWS CodeBuild

AWS CodeBuild compiles and packages the code before deployment. It ensures that every change is built and tested before moving to the deployment stage. To configure CodeBuild, firstly setup the environment and then define build steps in a buildspec.yml file. This file defines how to install dependencies, prepare artifacts.

My buildspec.yml

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 16   #Nodejs version
  build:
    commands:
      - cd Backend      #navigate to your directory
      - npm install     #install the dependencies
artifacts:
  files:
    - '**/*'

Make sure to integrate it with CodePipeline

Step3: Creating an EC2 instance and integrating it with CodeDeploy

Next, we setup an EC2 instance. AWS CodeDeploy automates the complete deployment process ensuring updates are rolled out efficiently and seamlessly

In this setup, I have choose Amazon Linux instance with permissions to access Systems Manager (Parameter Store), CodeDeploy and DocumentDB. Systems Manager allows us to securely retrieve environment variables while DocumentDB serves as our database.

Additionally, it is important to ensure that appspec.yml and configure.sh are placed in the root directory of the repository. appspec.yml defines the entire deployment lifecycle while configure.sh gets executed to configure the instance with necessary dependencies such as Apache server.

My appspec.yml file:

version: 0.0
os: linux
files:
  - source: /
    destination: /var/www/html
permissions:
  - object: /var/www/html
    pattern: "**"
    owner: ec2-user
    group: ec2-user
hooks:
  AfterInstall:
    - location: configure.sh
      runas: root

Myconfigure.sh file:

#!/bin/bash
sudo yum update -y
sudo yum install -y httpd
sudo systemctl start httpd
sudo systemctl enable httpd
cd /var/www/html/Frontend
sudo echo "CONFIG = { PUBLIC_IP: '$(curl -s ifconfig.me)' };" > config.js
cd /var/www/html/backend
wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
aws ssm get-parameters --names "MONGO_URL" "PORT" --with-decryption --query "Parameters[*].[Name,Value]" --output text | awk '{print $1 "=" $2}' 
sudo yum install -y nodejs
sudo npm install -g pm2
sudo pm2 start index.js
echo "Deployment complete and server restarted successfully."

Make sure to create an application in AWS CodeDeploy and set up a deployment group. I chose the In-Place deployment type for now and used a single instance (the one we created earlier). Don't forget to associate an IAM role with the deployment group —it is crucial for CodeDeploy to manage deployment on your instance.

Navigate back to CodePipeline page and ensure that you have correctly added the the CodeDeploy application name and the deployment group. This ensures that the deployment stage knows where to deploy the application

Step4: Testing the pipeline

Everything is configured and ready to be run !! Push a commit to your repository and see the magic happen🪄🪄

AWS CodePipeline executes the following steps to deploy your application with zero manual efforts

1️⃣Detect the change and trigger the pipeline

2️⃣Use CodeBuild to compile and package the code

3️⃣Deploy it to EC2 via CodeDeploy

Pro Tip📝

Don't tell me you pushed your .env file to GitHub… 😬 Make sure to store your DocumentDB credentials in SSM Parameter Store. It keep your secrets safe 🔏 and ensures your application retrieves it at runtime.

Here’s how mine looks

Troubleshooting Errors

If you do run into some failed builds, deployment issues don’t worry, we have all been there ! My Pipeline at one point my pipeline looked like this

And here’s how it looks once everything works

One trick to debug your errors is to check logs from the CodeDeploy Agent on EC2. It will provide you insights into potential misconfigurations. Check it out by running:

sudo tail -f /var/log/aws/codedeploy-agent/codedeploy-agent.log

If BeforeInstall, AfterInstall, or ApplicationStart fail check the logs by running the following command:

sudo tail -n 50 /opt/codedeploy-agent/deployment-root/deployment-logs/codedeploy-agent-deployments.log

GoBuster vs My Deployment: Exposing My Own Mistakes

After deploying my application and feeling I ran Gobuster against the IP… and let’s just say, it was an eye-opener.

Directories and all my files were sitting there in plain sight!

I ran the following command:

gobuster dir -u http://<ip> -w lists/directory-list-2.3-medium.txt

And this is how one can easily see my code files present

Mitigation Steps:

It was time to make things secure. I SSH-ed into my EC2 instance and modified the apache configuration files. Check the commands below:

sudo nano /etc/httpd/conf/httpd.conf

Now, replace the directory block (for the root directory) with the following

<Directory /var/www/html>
    Options -Indexes
    AllowOverride All
    Require all granted
</Directory>

To secure your other files (if there) use the following code block

<FilesMatch "^(\.env|\.git|config\.php|\.htaccess)">
    Order allow,deny
    Deny from all
</FilesMatch>

Wrapping It Up 🎯

This journey was a mix of automation and a little bit of shock (thanks to Gobuster!). The biggest lesson? Ensuring that the application is SECURE🔐 is as important as getting it running.

A small misconfiguration exposes more than what you anticipate! Keep it secure, keep it running !!🚀☁️

Thank your for reading !!

Aarush Luthra