DEV Community: Rajeshwari Vakharia

Build a Serverless File to Link Converter using AWS Services✨👩‍💻

Rajeshwari Vakharia — Tue, 20 May 2025 14:29:19 +0000

Build your own file to link converter with the magic of URL Shortener🪄

While working with Ubuntu VM, the Drag and Drop functionality was not working as intended and was showing glitches, and it made transferring files from Ubuntu to Windows, or vice-versa, a nightmare.

Hence, I found a solution which I created myself, that too on the Cloud.

File to Link Converter

Yes! Rather than moving the whole file why not just convert it into a link and make the sharing more better.

Tech Stack Used

Cloud Services Used
API Gateway (for better API handling)
Lambda (for making the app serverless)
S3 (for storing files)
DynamoDB (for storing Short keys to Original URL)

Languages Used

Python

Architecture of the App

What is happening in the diagram?

The API Gateway has two endpoints created one for POST and another to GET, and they both are integrated to one lambda function which handles both the request.

When a user wants to convert a file into link, he/she will go to POST /shorten and the lambda will put that object into the S3, with the key name being the filename given by the user.

Then a shortKey and Pre Signed URL of that object will be created, with a TimeToLive attribute of 5 mins only. So the user gets total 5 mins of time to use the URL to share or copy paste it into the browser to download the file. All this data will be stored in DynamoDB to create a mapping between the three keys.
In return to the POST request he gets the generated shortened URL.

Now with this URL, the user would like to copy it and share it or paste it in some other browser to GET the download of the file.

This is done by actually redirecting the location to the corresponding Pre Signed URL, and the download will start automatically.

Setting up the Infrastructure

This is the whole Terraform code to automate the setup w/o manual interference.

#create a bucket resource
resource "aws_s3_bucket" "s3bucket"{
  bucket = "your-bucket-name"
}

#lambda iam role
resource "aws_iam_role" "lambda_exec_role" {
    name = "lambda_exec_role"

    assume_role_policy = jsonencode({
      Version = "2012-10-17",
      Statement = [{
      Action    = "sts:AssumeRole",
      Effect    = "Allow",
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

#give that iam role permissions to access dynamodb and s3
resource "aws_iam_policy" "lambda_policy" {
  name = "lambda_policy"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = ["logs:*"],
        Resource = "arn:aws:logs:*:*:*",
      },
      {
        Effect = "Allow",
        Action = ["s3:GetObject", "s3:PutObject"],
        Resource = "${aws_s3_bucket.s3bucket.arn}/*",
      },
      {
        Effect = "Allow",
        Action = ["dynamodb:PutItem", "dynamodb:GetItem"],
        Resource = aws_dynamodb_table.db.arn,
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role = aws_iam_role.lambda_exec_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn

}
#create a zip file for go
data "archive_file" "zip_file_lambda" {
  type = "zip"
  source_dir = "/path/to/lambda/lambda_function.py"
  output_path = "/path/to/lambda/function.zip"
}

#create lambda function
resource "aws_lambda_function" "lambda_func" {
  function_name = "MyProjectFunction"
  role          = aws_iam_role.lambda_exec_role.arn
  handler       = "lambda_function.lambda_handler" 
  runtime       = "python3.9"                         
  filename         = "/path/to/lambda/function.zip"
  source_code_hash = filebase64sha256("/path/to/lambda/function.zip")

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.db.name
      API_ID = aws_api_gateway_rest_api.rest_api.id
      BUCKET_NAME = aws_s3_bucket.s3bucket.bucket
    }
  }
}

#creating a rest api
resource "aws_api_gateway_rest_api" "rest_api" {
  name = "myprojectAPI"
  description = "MyProject Rest API"
  binary_media_types = ["multipart/form-data"]
}

#creating the resource for shorten to post files
resource "aws_api_gateway_resource" "shorten" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  parent_id = aws_api_gateway_rest_api.rest_api.root_resource_id
  path_part = "shorten"
}

#creating a method for the /shorten endpoint
resource "aws_api_gateway_method" "shorten_method" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id =  aws_api_gateway_resource.shorten.id
  http_method = "POST"
  authorization = "NONE"

  request_parameters = {
    "method.request.header.Content-Type" = true
    "method.request.header.Accept" = true
  }
}

#creating integration
resource "aws_api_gateway_integration" "integration_shorten" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id =  aws_api_gateway_resource.shorten.id
  http_method = aws_api_gateway_method.shorten_method.http_method
  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = aws_lambda_function.lambda_func.invoke_arn

  request_parameters = {
    "integration.request.header.Content-Type" = "method.request.header.Content-Type"
    "integration.request.header.Accept" = "method.request.header.Accept"
  }
}

resource "aws_api_gateway_resource" "short" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  parent_id = aws_api_gateway_rest_api.rest_api.root_resource_id
  path_part = "short"
}

#alloting method to the path
resource "aws_api_gateway_method" "method" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id = aws_api_gateway_resource.short.id
  http_method = "GET"
  authorization = "NONE"
}

#integrating the created api with lambda function
resource "aws_api_gateway_integration" "integration_short" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id = aws_api_gateway_resource.short.id
  http_method = aws_api_gateway_method.method.http_method
  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = aws_lambda_function.lambda_func.invoke_arn
}

#make a deployment
resource "aws_api_gateway_deployment" "api_deployment" {
  depends_on = [ 
    aws_api_gateway_integration.integration_shorten,
    aws_api_gateway_integration.integration_short,
   ]
   rest_api_id = aws_api_gateway_rest_api.rest_api.id  
}

#create a stage
resource "aws_api_gateway_stage" "stage" {
  stage_name = "project"
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  deployment_id = aws_api_gateway_deployment.api_deployment.id

}

#permission for lambda for apigw
resource "aws_lambda_permission" "api_gw" {
  statement_id  = "AllowExecutionFromAPIGateway"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda_func.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_rest_api.rest_api.execution_arn}/*/*"
}

#create a dynamodb table with ttl enabled to store mapping of short code with the originalURL
resource "aws_dynamodb_table" "db" {
    name = "URLShortner"
    billing_mode = "PAY_PER_REQUEST"
    hash_key = "shortKey"
    attribute {
      name = "shortKey"
      type = "S"
    }

    ttl {
      attribute_name = "TimeToExist"
      enabled = true
    }
}

output "myprojectlink" {
  value = "https://${aws_api_gateway_rest_api.rest_api.id}.execute-api.us-east-1.amazonaws.com/${aws_api_gateway_stage.stage.stage_name}/shorten"

}

Let us go through it resource by resource.

Resource 1: Creating an S3 Bucket

We, at very first, named and created an S3 Bucket where actual files will be stored of whose Pre Signed URLs will be generated, using which the file will be automatically downloaded.

Resource 2: Creating a Lambda function

Next, we need to create an AWS Lambda function that will Put objects into S3, creates the Pre Signed URL, generates short key, creates a shortened URL and finally Redirects to the Pre Signed URL

Steps:

#lambda iam role
resource "aws_iam_role" "lambda_exec_role" {
    name = "lambda_exec_role"

    assume_role_policy = jsonencode({
      Version = "2012-10-17",
      Statement = [{
      Action    = "sts:AssumeRole",
      Effect    = "Allow",
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

#give that iam role permissions to access dynamodb and s3
resource "aws_iam_policy" "lambda_policy" {
  name = "lambda_policy"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = ["logs:*"],
        Resource = "arn:aws:logs:*:*:*",
      },
      {
        Effect = "Allow",
        Action = ["s3:GetObject", "s3:PutObject"],
        Resource = "${aws_s3_bucket.s3bucket.arn}/*",
      },
      {
        Effect = "Allow",
        Action = ["dynamodb:PutItem", "dynamodb:GetItem"],
        Resource = aws_dynamodb_table.db.arn,
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role = aws_iam_role.lambda_exec_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn

}

The above code is just used to create a role for Lambda named lambda_exec_role, create a policy to give it permissions to access S3, DynamoDB and CloudWatch.
This policy is then attached to the lambda function role using its ARN.

Creating the Lambda Function

#create a zip file for go
data "archive_file" "zip_file_lambda" {
  type = "zip"
  source_dir = "/path/to/lambda/lambda_function.py"
  output_path = "/path/to/lambda/function.zip"
}

#create lambda function
resource "aws_lambda_function" "lambda_func" {
  function_name = "MyProjectFunction"
  role          = aws_iam_role.lambda_exec_role.arn
  handler       = "lambda_function.lambda_handler" 
  runtime       = "python3.9"                         
  filename         = "/path/to/lambda_code/function.zip"
  source_code_hash = filebase64sha256("/home/rashu/myproject/function.zip")

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.db.name
      API_ID = aws_api_gateway_rest_api.rest_api.id
      BUCKET_NAME = aws_s3_bucket.s3bucket.bucket
    }
  }
}

Here, we have first started by creating a zip file.
Then we created the lambda function by specifying the handler ({filename_of_lambda}.{main_entry_point_in_the_code}), the runtime as python3.9, the role for lambda that we just created above and the zip file.
One of the best feature I find of Terraform is the environment attribute. As lambda runs inside a container, so I used this code block to declare the environment variables in that container so that I can use those values without hardcoding them in the code.

Also, here's the Python code that for the Lambda that we created, so now lets go through it:

import json
import boto3
import os
import io
import base64
import random
import string
import time
import re
import cgi

s3_client = boto3.client('s3')
dynamodb = boto3.client('dynamodb')

BUCKET_NAME = os.environ.get('BUCKET_NAME')
TABLE_NAME = os.environ.get('TABLE_NAME')
API_ID = os.environ.get('API_ID')

def generate_short_key(length=6):
    return ''.join(random.choices(string.ascii_letters + string.digits, k=length))

def sanitize_filename(name):
    return re.sub(r'[^a-zA-Z0-9._-]', '_', name)

def upload_file_to_s3(file_content, key):
    s3_client.put_object(
        Bucket=BUCKET_NAME,
        Key=key,
        Body=file_content
    )

def create_presigned_url(key):
    return s3_client.generate_presigned_url(
        'get_object',
        Params={'Bucket': BUCKET_NAME, 'Key': key},
        ExpiresIn=300
    )

def store_url_in_dynamodb(short_key, url):
    ttl = int(time.time()) + 300
    dynamodb.put_item(
        TableName=TABLE_NAME,
        Item={
            'shortKey': {'S': short_key},
            'originalUrl': {'S': url},
            'TimeToExist': {'N': str(ttl)}
        }
    )

def get_url_from_dynamodb(short_key):
    response = dynamodb.get_item(
        TableName=TABLE_NAME,
        Key={'shortKey': {'S': short_key}}
    )
    if 'Item' not in response:
        raise Exception("Invalid Key")
    return response['Item']['originalUrl']['S']

def lambda_handler(event, context):
    method = event.get("httpMethod")

    if method == "POST":
        try:
            # Decode base64-encoded body (multipart/form-data comes in like this)
            content_type = event["headers"].get("Content-Type") or event["headers"].get("content-type")
            body = base64.b64decode(event["body"])

            # Use cgi to parse multipart data
            environ = {'REQUEST_METHOD': 'POST'}
            headers = {'content-type': content_type}
            fs = cgi.FieldStorage(fp=io.BytesIO(body), environ=environ, headers=headers)

            # Get the file
            file_field = fs['file']
            file_bytes = file_field.file.read()

            # Get custom filename
            custom_name = fs.getvalue('filename')
            filename = sanitize_filename(custom_name) if custom_name else file_field.filename

            # Detect if it's a binary file (e.g., based on extension)
            binary_extensions = ['.pdf', '.jpg', '.jpeg', '.png', '.xlsx']
            is_binary = any(filename.endswith(ext) for ext in binary_extensions)

            content = file_bytes if is_binary else file_bytes.decode("utf-8")

            upload_file_to_s3(content, filename)
            presigned_url = create_presigned_url(filename)
            short_key = generate_short_key()
            store_url_in_dynamodb(short_key, presigned_url)

            short_url = f"https://{API_ID}.execute-api.us-east-1.amazonaws.com/project/short?shortKey={short_key}"

            return {
                "statusCode": 200,
                "headers": {"Content-Type": "application/json"},
                "body": json.dumps({"msg": short_url})
            }

        except Exception as e:
            return {
                "statusCode": 500,
                "headers": {"Content-Type": "application/json"},
                "body": json.dumps({"msg": "Error: " + str(e)})
            }

    elif method == "GET":
        try:
            short_key = event.get("queryStringParameters", {}).get("shortKey")
            original_url = get_url_from_dynamodb(short_key)

            return {
                "statusCode": 301,
                "headers": {'Location': original_url},
            }

        except Exception as e:
            return {
                "statusCode": 500,
                "headers": {"Content-Type": "application/json"},
                "body": json.dumps({"msg": "Error: " + str(e)})
            }

    else:
        return {
            "statusCode": 405,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps({"msg": "Method not allowed"})
        }

Resource 3: Configuring the API Gateway

#creating a rest api
resource "aws_api_gateway_rest_api" "rest_api" {
  name = "myprojectAPI"
  description = "MyProject Rest API"
  binary_media_types = ["multipart/form-data"]
}

#creating the resource for shorten to post files
resource "aws_api_gateway_resource" "shorten" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  parent_id = aws_api_gateway_rest_api.rest_api.root_resource_id
  path_part = "shorten"
}

#creating a method for the /shorten endpoint
resource "aws_api_gateway_method" "shorten_method" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id =  aws_api_gateway_resource.shorten.id
  http_method = "POST"
  authorization = "NONE"

  request_parameters = {
    "method.request.header.Content-Type" = true
    "method.request.header.Accept" = true
  }
}

#creating integration
resource "aws_api_gateway_integration" "integration_shorten" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id =  aws_api_gateway_resource.shorten.id
  http_method = aws_api_gateway_method.shorten_method.http_method
  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = aws_lambda_function.lambda_func.invoke_arn

  request_parameters = {
    "integration.request.header.Content-Type" = "method.request.header.Content-Type"
    "integration.request.header.Accept" = "method.request.header.Accept"
  }
}

resource "aws_api_gateway_resource" "short" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  parent_id = aws_api_gateway_rest_api.rest_api.root_resource_id
  path_part = "short"
}

#alloting method to the path
resource "aws_api_gateway_method" "method" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id = aws_api_gateway_resource.short.id
  http_method = "GET"
  authorization = "NONE"
}

#integrating the created api with lambda function
resource "aws_api_gateway_integration" "integration_short" {
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  resource_id = aws_api_gateway_resource.short.id
  http_method = aws_api_gateway_method.method.http_method
  type = "AWS_PROXY"
  integration_http_method = "POST"
  uri = aws_lambda_function.lambda_func.invoke_arn
}

#make a deployment
resource "aws_api_gateway_deployment" "api_deployment" {
  depends_on = [ 
    aws_api_gateway_integration.integration_shorten,
    aws_api_gateway_integration.integration_short,
   ]
   rest_api_id = aws_api_gateway_rest_api.rest_api.id  
}

#create a stage
resource "aws_api_gateway_stage" "stage" {
  stage_name = "project"
  rest_api_id = aws_api_gateway_rest_api.rest_api.id
  deployment_id = aws_api_gateway_deployment.api_deployment.id

}

Here, we created a rest api.

As my app would be transferring file contents by using mutipart/form-data, I configured the api to accept binary_media_types, it is a mandatory step.
I created 2 endpoints,
POST /shorten: This route will handle the operation of put_objects in the S3 and creation of short URLs.
GET /short?shortKey={shortKey}: This route will redirect users to the original URL based on the short code provided.
Integrate Routes with Lambda:
For each route, select the Lambda function that we created.
API Gateway will automatically link the function to the routes.
Deploy the API:
After setting up the routes, you need to deploy the API through their lambda integrations.
And finally attached them to a stage named project.

Resource 5: Creating a DB in DynamoDB to store the mapping.

#create a dynamodb table with ttl enabled to store mapping of short code with the originalURL
resource "aws_dynamodb_table" "db" {
    name = "URLShortner"
    billing_mode = "PAY_PER_REQUEST"
    hash_key = "shortKey"
    attribute {
      name = "shortKey"
      type = "S"
    }

    ttl {
      attribute_name = "TimeToExist"
      enabled = true
    }
}

This was really easy to setup. All you need to do is:

Create a dynamodb table named as URLShortner, with the billing mode PAY_PER_REQUEST, if you use PROVISIONED_MODE then you are at your own risk as that will start billing you as the table is created, so be careful.
Then I gave shortKey as the attribute of HASH_KEY or PRIMARY_KEY as that has to be unique for each case.
I also used TTL as in my app the time limit to use the URL and copy paste it to download the file is of only 5 mins.

Resource 5: Generating the Output

output "myprojectlink" {
  value = "https://${aws_api_gateway_rest_api.rest_api.id}.execute-api.us-east-1.amazonaws.com/${aws_api_gateway_stage.stage.stage_name}/shorten" 
}

In the end it's always a good choice to output the api link so that its easy to use.

Drawbacks

One of the major drawback that I found with this project is that the setup is a monolith. As, the POST method is handling everything from creating the bucket to generating the shortkeys to getting the presigned urls and at last putting it all into the dynamodb.

And that being said, a better architecture has to be build to make services loosely coupled to each other.

And of course, other drawbacks are that it still doesn't include NOT creating a different short Key even for the same file. Hence, it lacks uniqueness.
The S3 object still stays even after the deletion of its URL mapping in the DynamoDB.
And of course, a custom Domain Name, as this API Gateway can't be used in real life application.

Thanks for reading till here.🙌
Leave down your thoughts in the comment to discuss how to improve this app more.⭐

🐧Penguin Scripting Suite: v.1.0

Rajeshwari Vakharia — Thu, 01 May 2025 13:18:42 +0000

Bash scripts are a Linux developer’s secret weapon — they help automate repetitive tasks, simplify complex workflows, and keep environments consistent. Whether you're managing development environments, deploying to servers, or working with containers and cloud services, bash scripting makes life easier.

Linux users — especially developers, DevOps engineers, and sysadmins — often find themselves writing small, handy scripts to boost their productivity. So from today, I’m kicking off a series of such small but useful Bash scripts that can save time and reduce effort in everyday development.

🚀 Version v.1.0: Quick-Select a Python Virtual Environment

In this first post of the series, let’s create a simple script that helps you easily choose and activate a Python virtual environment — without navigating into folders every time.

Older way

Every time we work with Python, a virtual environment is usually essential. Like many of you, I have multiple virtual environments — each tailored for a different project.

To activate one, we typically do:

source ENVIRONMENT_NAME/bin/activate

But this can be improved — especially if you want a cleaner, more organized, and faster workflow.

📁 Step 1: Organize Your Environments

First, keep all your virtual environments in a single directory — say, venv. This way:

Your workspace stays clean and manageable.
You always know where to look for your environments.

🛠️ Step 2: A Bash Script to Help

Now, here’s a basic script to list and activate a virtual environment from your organized folder:

#!/bin/bash

list="$(ls venv)"
echo $list
echo Which environment do you want?
read env
response="$(source venv/$env/bin/activate 2>&1)"

if  [[ $? -eq 0 ]] ; then
        echo "Activating Virtual Environment..."
        source venv/$env/bin/activate

else
        echo "Error: $response"

fi

This is actually just a basic script. What it is doing is that:

list="$(ls venv)"
echo $list

This line just lists out all the virtual envs in the folder and prints it to the console.

echo Which environment do you want?
read env
response="$(source venv/$env/bin/activate 2>&1)"

This will ask "which environment we want to activate".
The line read env will read the output from the console, it actually holds the value of the environment we have selected and is used in the line response="$(source venv/$env/bin/activate 2>&1)as a variable to be used to activate the environment and store that value in another variable known as response.
response="$(source venv/$env/bin/activate 2>&1)
This line is actually used for error handling. What happens is that when we do source venv/$env/bin/activate, if the selected env is valid that is with no spelling mistakes or does actually exits, it will result as an exit code of 0 with no errors and will continue further.
But if the value is wrong then it will result in an error causing the script to exit abruptly.
Hence to prevent this Redirecting of Output is used. 2 stands for stderr and 1 for stdout.
The statement '2>&1, says that if error occurs redirect that to thestdout` only. And that adds the the result to the response variable.

if [[ $? -eq 0 ]] ; then echo "Activating Virtual Environment..." source venv/$env/bin/activate else echo "Error: $response" fi

(Don't know why this couldn't get into a code block)

This is an if statement, if the exit code(or the $?) is equal to (-eq) 0, then activate the virtual env.
Or if the exit code is not equal to 0 then give the error message in the output.

Running the script

This is an interesting part. As some of you might think that this can run easily as a executable script. But it will not.
Using ./pyenv.sh will run the script in a new shell- a subshell is created. This will result into activation of no venv as we want it to be in the current shell where we are working.

So, to remove this we use either source pyenv.sh or . pyenv.sh1 to tell the os that run the script in the current shell.

METRICS: STREAMLINE YOUR SYSTEM INSIGHTS

Rajeshwari Vakharia — Wed, 16 Apr 2025 12:29:39 +0000

INTRODUCTION

METRICS is a lightweight, user-friendly, CLI-based system monitoring tool designed to provide clear and accessible insights into system performance.

My inspiration for this project came from when I was using WSL to work with my Ubuntu Distro. In such a system it is really hard to monitor the system when there are no GUI or better, lightweight application is provided.
It comes with a choice to either use traditional monitoring commands as mentioned before such as htop or top, but they don’t provide much clear info, or we can use high-fi tools such as Node Exporter, Grafana or Prometheus, which again requires some prior learning of how to use it and also gets somewhere complex to use.

Hence, I wanted to make something that can provide a better understanding of the System’s metrics for day-to-day life. It promotes to not to use any fancy tools which can actually be much overwhelming for a normal user and may also come with steep learning curve.

For simple system such as using a WSL, or a small server setup in Linux, it is more beneficial to make small and lightweight applications to monitor the system.

Key Highlights:

Lightweight & Minimal Setup: No complex configurations or dependencies.
Clear Visualizations: Offers resource insights through tables, concise text output, and simple CLI-based graphs.
Beginner-Friendly: Intuitive to use without sacrificing functionality.

Whether you're managing a small server or working within WSL, METRICS is your go-to tool for efficient, maintainable system monitoring—without the bloat.

How to run it?

To get METRICS in your system, you need to follow the given steps:

For ease of use, create a directory:

mkdir METRICS
cd METRICS

Then type:
git clone https://github.com/vak-rashu/METRICS.git
After doing the git clone, follow the given steps:

cd METRICS
cd packages
source packages.sh

This will activate the virtual environment and will install the necessary pip requirements.

Now go to the parent directory by typing cd .. and run the following commands:

source metrics_cli.sh

Now your METRICS CLI will start working and you can start tracking your system resources. 🌟

Important Note:

Make sure you have python3 downloaded, if not download it from here:

  sudo apt update
  sudo apt install python3

Make sure you have required setup downloaded for the graphs to run smoothly, if not run the following command:

   sudo apt-get install tcl-dev tk-dev python3-tk

Result Analysis

It would be great to see you all also using METRICS to monitor your system resources.
Do also share your thoughts or reviews about this project. 😄

Visualize System Metrics with Grafana & Prometheus in Docker

Rajeshwari Vakharia — Sun, 06 Apr 2025 06:09:05 +0000

Follow the link to the Github Repo:- https://github.com/vak-rashu/Docker/tree/main/Prometheus-and-Grafana-with-Docker

Prometheus and Grafana are two main tools that are used to monitor, analyse and visualise the metrics, logs or event traces from applications and services.

Prometheus is a tool to get the metrics and logs from the application.
While Grafana is used to convert those metrics into beautiful dashboards to visualise them.

Node Exporter (Node means a system) is a tool to get the metrics, logs and related observations from the system it has been configured for. It provides the data in Prometheus format to Prometheus so that it can interfere it easily.

In this article, we will create a dashboard using Grafana and Prometheus by taking from the Node Exporter, which will be downloaded in the local machine only.

Download Node Exporter

Follow the steps to download and run the node_exporter.

mkdir node_exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz

tar xvfz node_exporter-1.9.0.linux-amd64

cd node_exporter-1.9.0.linux-amd64

./node_exporter #this is will execute the service

Configure the docker-compose.yml file to run Prometheus and Grafana.

Create a docker-compose.yml

Prometheus runs on port 9090 and Grafana on 3000. You can copy paste the same code to run in your machine.

Note: Prometheus requires a prometheus.yml file that will be used to describe the jobs that prometheus has to get metrics for. This file is configured in /etc/prometheus/prometheus.yml .

services:
    prometheus:
        image: prom/prometheus:latest
        ports:
          - "9090:9090"
        volumes:
          - ./prometheus.yml:/etc/prometheus/prometheus.yml
          - prom_data:/prometheus 
        command: "--config.file=/etc/prometheus/prometheus.yml"

    grafana:
      image: grafana/grafana
      ports:
        - "3000:3000"
      environment:
        - GF_SECURITY_ADMIN_USER=admin 
        - GF_SECURITY_ADMIN_PASSWORD=grafana 
      volumes:
        - grafana_data:/var/lib/grafana

volumes:
   prom_data: 
   grafana_data:

The above code will run the container but we also have to create the prometheus.yml file, which is as below:-

---
global:
  scrape_interval: 15s 

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'rashu-ubuntu'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['172.16.0.4:9100']

This is the prometheus.yml file. Here, I want to get the metrics of the target set as "targets: ['172.16.0.4:9100']", which is actually where the Node Exporter is running on my service. You can configure it according to your ip address.

Though you can also run Node Exporter as one of the container, but I think running it locally is much better.

Now, run the docker compose up -d command and the node exporter, you will see that the Prometheus is running in port 9090 and Grafana in 3000.

Then in Grafana go to, Data Sources> Select Prometheus> Enter the name whatever you want > in Connection, type http://prometheus:9090, (remember only enter the name of the domain name which should be correspond to the name of the service name you have given to prometheus in the docker-compose.yml file, like I have given it prometheus) > Click on Save and Test.

Then go to Build a Dashboard> Add Visualisation> Select Data Source and start making the visualises as you want.

How I Used Pulumi ESC to Securely Manage Secrets and Configs in My Infrastructure

Rajeshwari Vakharia — Fri, 04 Apr 2025 07:04:45 +0000

This is a submission for the Pulumi Deploy and Document Challenge: Shhh, It's a Secret!

What I Built💯

The project deploys an AWS Lambda function that fetches weather data from weatherstack.com and stores the current temperature in AWS Systems Manager (SSM) under the parameter path:
/weather/temperature_value

Project Repo🗃️

You can check out my project at
Pulumi-Dev-Challenge.🔗

My Journey🚀

This is my very first time to participate in a Dev Challenge and I loved the experience. The whole journey was filled with a lot of learning new things and also finding out the best practices to use Pulumi.

Using Pulumi ESC🔐

I used Pulumi ESC to store the API key.
Pulumi ESC made it extremely easy to manage secrets like API keys without hardcoding them or using environment variables. The integration into Pulumi's configuration system was seamless and secure.It was really easy to setup the whole thing.

I really found Pulumi ESC to be really helpful. As first time using it, the documentation made it really quick and easy to navigate me on how to use Pulumi ESC effectively.

But sadly, I couldn't use ESC SDK as the documentation was not quite easy to understand. I tried to ask Pulumi AI too but couldn't get any help.

I wanted to use the configuration values set in esc environment in my lambda code (lambda.py file) but was not able to find a correct approach to it.
It wiuld be really great if someone could help me out of this.

But overall experience was good and I genuinely think that Pulumi ESC is really helpful as a Secrets Manager.

Next, I will use this to create esc environment to store values for my Localstack configurations:)

Host a Static Website in LocalStack using Terraform with S3 and Route 53

Rajeshwari Vakharia — Thu, 13 Mar 2025 18:12:10 +0000

Link to the Github Repo

LocalStack is an emulator for AWS Services. It let's AWS like environment to run on our localmachine.
LocalStack is really helpful for beginners to not just understand AWS but to also work with it.

In this blog, we will discover how to host a static S3 website using Route53 in LocalStack with the help of another tool that is Terraform.
Terraform is a IaC tool that helps in defining the infrastrusture.

To use Terraform with LocalStack, we have to install tflocal, which is a wrapper for Terraform.

So, let's get started.

Configuring S3

Create a bucket

provider "aws" {
    secret_key = "test" 
    region = "us-east-1"
    access_key = "test"
}

resource "aws_s3_bucket" "tf_bucket" {
    bucket = "websitebucket"
}

First, we will define the provider block and will put the required configurations.
And in the resource block we have defined the S3 bucket with the name websitebucket. This bucket will contain the files of our website.

Define the object and Remove Public access blocks from the bucket

resource "aws_s3_object" "tf_bucket_object" {
    bucket = aws_s3_bucket.tf_bucket.id
    key = "index.html"
    source = "./index.html"
    content_type = "text/html"
    acl = "public-read"
}

resource "aws_s3_bucket_public_access_block" "tf_pab"{
    bucket = aws_s3_bucket.tf_bucket.id 

    block_public_acls = false
    block_public_policy = false 
    ignore_public_acls = false
    restrict_public_buckets = false 
}

The first resource is used to create an object in the bucket which contains the souorce file path i.e. ./folder name.
And the other resource is used to remove the public access block from the bucket so that it is not restricted to the internet.

Configure the Bucket Policy

resource "aws_s3_bucket_policy" "tf_bucketpolicy" {
    bucket = aws_s3_bucket.tf_bucket.id

    policy = jsonencode({
        Version = "2012-10-17"
        Statement = [{
                Sid = "PublicReadGetObject"
                Effect = "Allow"
                Principal = "*"
                Action = "s3:GetObject"
                Resource = "${aws_s3_bucket.tf_bucket.arn}/*" 
        }]
    })
}

This is a very important configuration to make. By defining this bucket policy, it makes the objects in the bucket to be accessible by other people.
Even after removing the acl, it is important to apply the bucket policy only then the objects will be accessible or else we will get error 403.

Set the bucket Website Configuration

resource "aws_s3_bucket_website_configuration" "tf_bucket_config" {
    bucket = aws_s3_bucket.tf_bucket.id

    index_document {
        suffix = "index.html"
    }
}

AWS provides a great way to host the bucket as a website through this configuration.

Configure the Route53

⚠️DNS Service in LocalStack comes with their Pro Version, which I don't have and hence the we will not be able to fetch the website using the domain name.

Create a Hosted Zone

resource "aws_route53_zone" "tf_zone" {
    name = "mylocalwebsite.test"  
}

This block is used in creating a Hosted Zone, which will host our domain name.

Create an Alias Record with the S3 bucket endpoint.

resource "aws_route53_record" "tf_record"{
    zone_id = aws_route53_zone.tf_zone.zone_id
    name = "mylocalwebsite.test"
    type = "A"
    alias {
        name = aws_s3_bucket_website_configuration.tf_bucket_config.website_endpoint
        zone_id = aws_s3_bucket.tf_bucket.hosted_zone_id
        evaluate_target_health = false 
    }
}

This block will create the alias with our S3 bucekt endpoint.

And with this at the end we have to do three things to run the infrastructure code:-

tflocal init

This syntax initiates the resources in the file.

tflocal plan

It helps in viewing the plan of the resources and all its configuration that we can check before applying.

tflocal apply

And finally applying the code after all the checks.

And when we access the S3 endpoint using the path :-
http://localhost:4566/websitebucket/index.html