DEV Community: Ashraf Minhaj

For added data security, you can use IAM authentication for RDS for any aws services or for any IAM users! This is a bit overhead to configure at first, but I find it very secure instead of sharing the master password.

Ashraf Minhaj — Mon, 22 Dec 2025 02:28:05 +0000

Ashraf Minhaj

Dec 21 '25

Access AWS RDS without Password - use IAM

#devops #aws #awsrds #databasesecurity

Comments

5 min read

Access AWS RDS without Password - use IAM

Ashraf Minhaj — Sun, 21 Dec 2025 17:00:11 +0000

Backstory

Part of setting up or debugging a database requires to get into the db itself. In general we use password to access into it. But we faced a situation where the db authentication was set to use AWS Secrets Manager secret and we did not have access to it. So, if you are in a situation like we were, or you are a DevOps who just does not feel comfortable sharing the password directly to the developers, you can use IAM based access to AWS RDS DB. Let's see how we can do it.

Step1: Create RDS DB

While creating RDS Database, under Choose a database creation method select Full Configuration. I will create a postgres db today. You can see the list of supported regions and dbs for IAM auth from this list.

I am keeping the default settings, but go ahead as per your need. Under Connectivity section, check Public Access to `Yes' since we will access the db from a local machine. If the db will be accessed from by an ec2 from the same VPC, then the db can be kept as private. But for our case, we need it public.

Now we need a security group (firewall rule to allow db port from any machine). Go to ec2 -> Security Groups -> Create Security Group and allow the db port 5432 to be accessed by any IP -

And under the same Connectivity section, attach the Security Group to the db.

Scroll down and Under Database authentication section and select Password and IAM database authentication option -

That's about it. Now We can just create the database.

Wait a while for a it to start.

Step2: Setup IAM role/policy

First get the ARN and resource ID of the databse, click on the database and under Configuration tab you will find the ARN and resource ID of the database -

Since we are doing this for developers (users), we will create a policy and attach to the user. Let's create a an IAM policy.

Please mind the db db-resource-id and db-username section, it's written as arn:aws:rds-db:<region>:<account-id>:dbuser:<db-resource-id>/<db-username> . In my case the db_user is postgres` . So the policy looks like this -

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds-db:connect"
      ],
      "Resource": [
        "arn:aws:rds-db:ap-southeast-1:980305500084:dbuser:db-GMICPKT7J4JMSEYSHVWUFEN7VQ/postgres"
      ]
    }
  ]
}

Now click next and name the policy, then create it. I gave it this name - rds-iam-access .

Now from iam -> Users select your user and click add permissions to add the policy.

Select the policy and click add permission to attach the policy to the user. And that's about it.

If you want to give access to other services such as to your bastion host or k8s pods, create a role selecting respecting service.

Step3: Connect with db using IAM

Prerequisite

Get the DB endpoint from the Connectivity and Security tab -

We need to allow the db user to support IAM auth first. So first time, we will login with password as usual (or using secrets manager if that's your case).

# psql "postgresql://<db-user>:<db-password>@<db-endpoint>:<port>/<db-name>"

psql "postgresql://postgres:YourMasterPasswordHere@database-1.cdkgmeaiqn6n.ap-southeast-1.rds.amazonaws.com:5432/postgres"

and run this -

#GRANT rds_iam TO <user>;

GRANT rds_iam TO postgres;

Generate token using the db endpoint from the cli. Run this from your terminal, change db endpoint, port and user as per your need -

# aws rds generate-db-auth-token --hostname Endpoint --port Port_number --username DB_username

# get token on terminal 
# aws rds generate-db-auth-token --hostname database-1.cdkgmeaiqn6n.ap-southeast-1.rds.amazonaws.com --port 5432 --username postgres

# get the token in an env (preferred)
TOKEN=$(aws rds generate-db-auth-token \
  --hostname database-1.cdkgmeaiqn6n.ap-southeast-1.rds.amazonaws.com \
  --port 5432 \
  --region ap-southeast-1 \
  --username postgres)

# echo $TOKEN

This returns a token that is valid for 15 minutes. Meaning you can connect to the DB within minutes, but it won't affect the db session duration.

Option 1: Use psql

Use psql tool to connect to the db from the terminal -

#psql "host=<rds-endpoint> port=5432 dbname=<db-name> user=<db-user> sslmode=require password=<IAM_AUTH_TOKEN>"

PGPASSWORD="$TOKEN" psql "postgresql://postgres@database-1.cdkgmeaiqn6n.ap-southeast-1.rds.amazonaws.com:5432/postgres?sslmode=require"

BAM! I'm in -

Now, some of you are not happy with terminal so ...

Option 2: Use GUI client (i.e. PgAdmin)

Simple, just like you used to access any other db, just paste the token in place of the password.

Go to Parameters tab and set Enable ssl to require -

Save it. Now use it. As simple as that. And just like that I'm in from the PgAdmin tool too -

Last thought

Remember, PgAdmin can't generate token. So, after a session times out you have to generate another token to get access to the db. A token is valid for only 15 minutes, that's the window to connect. But after the connection is established, it won't hamper the session, meaning, you can use a token to talk to the db forever.. until the session ends somehow.
It may seem annoying to some, but I see the secure feature here. I know the next time what I will do when a developer asks for db credentials, do you?

[Boost]

Ashraf Minhaj — Wed, 22 Oct 2025 06:57:32 +0000

Build and push Docker images to Amazon ECR using Terraform

Ashraf Minhaj ・ Oct 22

#devops #aws #terraform #docker

Build and push Docker images to Amazon ECR using Terraform

Ashraf Minhaj — Wed, 22 Oct 2025 06:19:16 +0000

Introduction

In general you would want to deploy infrastructure using terraform, build and push docker images in CI/CD phase. But let's say, just in case, someone pointed a gun at your head asking you to build and push docker images from within Terraform, how would you do it? Fear not, I am here, let me save you today.

Action Plan

It's simple, we will just wait check for file change - where a specific Dockerfile is kept, if any changes are detected we will build the image and push it.

Prerequisites

You have to have aws cli configured (v2 is prefered but v1 will just do)
Terraform installed (I am using 1.5.5)

Get into action, we don't have much time

The directory Structure

So this will be our directory structure -

.
├── src
│   └── frontend
│       ├── Dockerfile
│       └── index.html
└── Terraform
    ├── ecr.tf
    ├── main.tf
    └── variables.tf

Here, we have a very basic nginx app (code below) serving a html page. And inside the Terraform directory we will store the Terraform specific files.

The sample app

You can skip this if you already have an app and Dockerfile.

├── src
│   └── frontend
│       ├── Dockerfile
│       └── index.html

For this demo I built this src/frontend/index.html -

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Service Connect Test</title>
  <style>
    body {
      font-family: Arial, sans-serif;
      max-width: 600px;
      margin: 50px auto;
      padding: 20px;
    }
    input, button {
      padding: 10px;
      width: 100%;
      margin: 5px 0;
      font-size: 1rem;
    }
    pre {
      background: #f4f4f4;
      padding: 10px;
      border-radius: 5px;
      overflow-x: auto;
    }
  </style>
</head>
<body>
  <h2>ECS Service Connect Tester</h2>
  <input type="text" id="urlInput" placeholder="Enter internal service URL (e.g., http://service-name:8080)">
  <button id="connectBtn">Connect</button>
  <h3>Response:</h3>
  <pre id="responseBox">No data yet.</pre>

  <script>
    document.getElementById('connectBtn').addEventListener('click', async () => {
      const url = document.getElementById('urlInput').value;
      const responseBox = document.getElementById('responseBox');
      responseBox.textContent = "Connecting...";
      try {
        const res = await fetch(url);
        const text = await res.text();
        responseBox.textContent = text;
      } catch (err) {
        responseBox.textContent = "Error: " + err.message;
      }
    });
  </script>
</body>
</html>

Like I mentioned, this is not important, use any basic html or your complex app, your wish.

If you are interested, this is to check other APIs.

Now the Dockerfile - src/frontend/Dockerfile -

FROM nginx:alpine-slim

COPY ./index.html /usr/share/nginx/html/index.html

EXPOSE 80

Terraform Scripts

└── Terraform
    ├── ecr.tf
    ├── main.tf
    └── variables.tf

You you love bad practice, you can put all files in one directory. But I am the opposite so let's maintain multiple files -

1. variables.tf

variable "aws_region" {
  default = "ap-southeast-1"
}

variable "frontend_dir" {
  default = "../src/frontend"
}

variable "frontend_repo_name" {
  default = "frontend"
}

2. main.tf

provider "aws" {
  region = var.aws_region
}

3. ecr.tf
Here's the main game (full code below). We will create a repository first, where our image will be pushed -

resource "aws_ecr_repository" "frontend_repo" {
  name                 = var.frontend_repo_name
  image_tag_mutability = "MUTABLE" # Or "IMMUTABLE" based on your needs

  image_scanning_configuration {
    scan_on_push = false # Set to true if you want to scan images as they are pushed
  }
}

Now to track changes, we will generate hash of the Dockerfile (or any other file you wish to track -

locals {
  frontend_hash = filebase64sha256("${path.module}/${var.frontend_dir}/Dockerfile")
}

Based on that hash, we will generate a random tag for the image. byte_length = 2 generates 4 char long random string, increase to 4,8 or 12 as you please.

resource "random_id" "frontend_rand_tag" {
  keepers = {
    # Generate a new tag each time there's change in Dockerfile hash
    frontend_img_tag = local.frontend_hash
  }

  byte_length = 2
}

output "image_tag" {
  value = {
    frontend_tag = random_id.frontend_rand_tag.hex
  }
}

Finally, we build and push the image. Just like we would do using the CLI. We are utilizing terraform null_resource and local-exec provisioner to run the commands -

resource "null_resource" "build_and_push_frontend" {
  triggers = {
    # Trigger a rebuild if the Dockerfile or source code changes
    dockerfile_hash = local.frontend_hash
  }

  provisioner "local-exec" {
    command = <<-EOT
      aws ecr get-login-password --region ${var.aws_region} | docker login --username AWS --password-stdin ${aws_ecr_repository.frontend_repo.repository_url}
      docker build -t ${aws_ecr_repository.frontend_repo.repository_url}:${random_id.frontend_rand_tag.hex} ${path.module}/${var.frontend_dir}
      docker push ${aws_ecr_repository.frontend_repo.repository_url}:${random_id.frontend_rand_tag.hex}
    EOT
  }
}

Now, combining all -

Full ecr.tf

resource "aws_ecr_repository" "frontend_repo" {
  name                 = var.frontend_repo_name
  image_tag_mutability = "MUTABLE" # Or "IMMUTABLE" based on your needs

  image_scanning_configuration {
    scan_on_push = false # Set to true if you want to scan images as they are pushed
  }
}

locals {
  frontend_hash = filebase64sha256("${path.module}/${var.frontend_dir}/Dockerfile")
}

resource "random_id" "frontend_rand_tag" {
  keepers = {
    # Generate a new tag each time there's change in Dockerfile hash
    frontend_img_tag = local.frontend_hash
  }

  byte_length = 2
}

resource "null_resource" "build_and_push_frontend" {
  triggers = {
    # Trigger a rebuild if the Dockerfile or source code changes
    dockerfile_hash = local.frontend_hash
  }

  provisioner "local-exec" {
    command = <<-EOT
      aws ecr get-login-password --region ${var.aws_region} | docker login --username AWS --password-stdin ${aws_ecr_repository.frontend_repo.repository_url}
      docker build -t ${aws_ecr_repository.frontend_repo.repository_url}:${random_id.frontend_rand_tag.hex} ${path.module}/${var.frontend_dir}
      docker push ${aws_ecr_repository.frontend_repo.repository_url}:${random_id.frontend_rand_tag.hex}
    EOT
  }
}

output "image_tag" {
  value = {
    frontend_tag = random_id.frontend_rand_tag.hex
  }
}

You can download the full code and script from this git repo.

Now if we init, plan, apply -

terraform init

terraform plan

terraform apply --auto-approve

You will see that after creating the repository it ran docker-login command and started building the image -

After the build process is complete, it pushed the image to ECR -

Let's verify. Based on the output the image tag should be 87ed . Let's see on the console.

Yeah looks like we have successfully done it!

Conclusion

Today we learnt how we can automate Docker image building and pushing using Terraform. But it's not production ready, use this only when using minor tasks, or specific needs - like dodging a bullet. Congratulations! you are saved today.

: what do we say to the god of manual labour?
: Not today, we have Terraform.

Happy Coding!

Securing AWS EC2: Change SSH Port 22 to a Custom Port

Ashraf Minhaj — Thu, 28 Aug 2025 03:50:52 +0000

Introduction

When you deploy an ec2 instance or any server instance, it defaults to SSH in port 22. You already know it, so do the whole world and all of the hackers. So, most of the automated attacks start by scanning the known ports, say 22 for SSH.

One of the many ways to mitigate that is to change the port to a different port. Yes it won't make it bulletproof, but it will surely reduce the chances of brute force attacks.
Let's start.

This method is applicable for all kinds of servers, including your Virtual Machines, but since I have an aws cloud account, I will use that, you can just use GCP Compute instance, Azure VM as well.

what is Security by Obscurity

In simple words, it means hiding something from the attackers, believing if the attackers can't see they can't exploit.

Kind of removing your door to an unexpected location so that thieves don't find a way to enter from known places... say you removed the front door completely.

Deploy an ec2 instance

We will deploy an ec2 instance like how we usually do. In the security group, open port 22 for ssh and add a key (or generate one, save it), my key name is test-min-keys.pem.

Change the port

1. Connect to the server/instance

Ok for that we need to access the VM first, let's ssh into it normally -

Change key file permission and ssh -

chmod 400 "test-min-keys.pem"

# your key, sever IP or hostname will be different, use that
ssh -i "test-min-keys.pem" ec2-user@ec2-13-228-72-54.ap-southeast-1.compute.amazonaws.com

2. Update SSH configuration

Open the ssh config file -

sudo nano /etc/ssh/sshd_config

And look for this portion -

We will change it from 22 to 2222 - that's our door change -

Now we can just restart the ssh service and use the new port to ssh.

sudo systemctl restart sshd

But.. if you are using Selinux enabled systems like RHEL/CentOS/Amazon Linux machines then you have to run this command to tell Selinux about the change as well.

sudo semanage port -a -t ssh_port_t -p tcp 2222

If you see this error then use a different port (headache? remove the head) -

Now restart ssh service -

sudo systemctl restart sshd

Now, update your firewall, or in aws, the security group inbound rule and open port 2222 instead of 22.

3. Access the server with new ssh port

Hold your horses, just in case, test the new port before closing the old session, open another terminal tab and try to ssh with the port tag -p <port> -

# ssh -p 2222 ec2-user@<your-public-ip>

ssh -p 2222 -i "test-min-keys.pem" ec2-user@ec2-13-228-72-54.ap-southeast-1.compute.amazonaws.com

And voila! We are in!

Let's close all the sessions and try again -

Conclusion

In practice, you should never put instances in public subnet when you don't need to. Also harden the security by not allowing password based SSH access.
For ec2 instances, there's numerous ways to get access to private ec2 instances which is more secure.

Happy Coding!

Time to ditch public IP in ec2 instances

Ashraf Minhaj — Sun, 17 Aug 2025 12:59:19 +0000

Access Private ec2 instances without Public IP using Instance Connect Endpoint

Ashraf Minhaj ・ Aug 7

#aws #devops #networking #cloud

Access Private ec2 instances without Public IP using Instance Connect Endpoint

Ashraf Minhaj — Thu, 07 Aug 2025 03:59:50 +0000

Introduction

So far you know that you need to attach a public IP address to an ec2 instance to SSH into it or say port forward (access ec2 port from local machine).
But, there are multiple use cases where we might need a private ec2 instance. It can be the application we deployed running on a private subnet (saves cost and best for security too). Or we have a bastion host for mission critical applications but we are worried about that bastion with public IP addresses.
What if I tell you that you don't actually need to have public IP addresses to access the ec2 instances?

Idea

In general, our private instances are routed to public internet using a NAT gateway. We just need to add a ec2 instance connect endpoint to our vpc to access to the VM.

So it's gonna look like this -

In this blog we will see it with Terraform.

Create ec2 instance connect endpoint

When creating a VPC, we can define instance connect endpoint and attach the subnet where the ec2 instance will be deployed, also allow the security groups of the desired instances you want to connect to.

resource "aws_ec2_instance_connect_endpoint" "instance_connect_endpoint" {
  subnet_id          = local.instance_connect_subnet
  security_group_ids = [module.bastion_sg.id, module.jenkins_sg.id]
}

The full vpc.tf now looks like this -


# select the private subnet 
locals {
  instance_connect_subnet = aws_subnet.private_subnets[0].id
}

resource "aws_vpc" "vpc" {
  cidr_block           = var.vpc_cidr
  instance_tenancy     = "default"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name = local.vpc_name
  }
}

resource "aws_internet_gateway" "vpc_int_gw" {
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = local.igw_name
  }
}

# public subnets
resource "aws_subnet" "public_subnets" {
  count                   = length(var.public_subnet_cidr_blocks)
  vpc_id                  = aws_vpc.vpc.id
  cidr_block              = var.public_subnet_cidr_blocks[count.index]
  availability_zone       = element(var.availability_zones, count.index)
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.component_prefix}-pubsub-${count.index}-${terraform.workspace}"
  }
}

# private subnets
resource "aws_subnet" "private_subnets" {
  count                   = length(var.private_subnet_cidr_blocks)
  vpc_id                  = aws_vpc.vpc.id
  cidr_block              = var.private_subnet_cidr_blocks[count.index]
  availability_zone       = element(var.availability_zones, count.index)
  map_public_ip_on_launch = false

  tags = {
    Name = "${var.component_prefix}-prvsub-${count.index}-${terraform.workspace}"
  }
}

resource "aws_nat_gateway" "nat_gateway" {
  depends_on    = [aws_internet_gateway.vpc_int_gw]
  count         = length(var.private_subnet_cidr_blocks) > 0 ? 1 : 0
  allocation_id = aws_eip.nat_eip[0].id
  subnet_id     = aws_subnet.public_subnets[0].id

  tags = {
    Name = "${var.component_prefix}-nat-${count.index}-${terraform.workspace}"
  }
}

resource "aws_route_table" "public_route_table" {
  vpc_id = aws_vpc.vpc.id
  route {
    cidr_block = "0.0.0.0/0" # all IPs
    gateway_id = aws_internet_gateway.vpc_int_gw.id
  }

  tags = {
    Name = local.pub_rt_name
  }
}

resource "aws_route_table" "private_route_table" {
  depends_on = [aws_nat_gateway.nat_gateway]
  count      = length(var.private_subnet_cidr_blocks) > 0 ? 1 : 0 # create only if there is more than 0 private subnets
  vpc_id     = aws_vpc.vpc.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway[0].id
  }

  tags = {
    Name = "${var.component_prefix}-prvrt-${count.index}-${terraform.workspace}"
  }
}

# routing association between routetable and subnets
resource "aws_route_table_association" "public_subnets_association" {
  count          = length(var.public_subnet_cidr_blocks)
  subnet_id      = aws_subnet.public_subnets[count.index].id
  route_table_id = aws_route_table.public_route_table.id
}

# Routing association between private route table and private subnets
resource "aws_route_table_association" "private_subnets_association" {
  count          = length(var.private_subnet_cidr_blocks)
  subnet_id      = aws_subnet.private_subnets[count.index].id
  route_table_id = aws_route_table.private_route_table[0].id
}

resource "aws_ec2_instance_connect_endpoint" "instance_connect_endpoint" {
  subnet_id          = local.instance_connect_subnet
  security_group_ids = [module.bastion_sg.id, module.jenkins_sg.id]
}

Now we can create the ec2 instance, remember to add that specific private instance we defined using local instance_connect_subnet. So the ec2.tf will be -

module "bastion_server" {
  count              = var.create_bastion == true ? 1 : 0
  source             = "./modules/ec2_instance"
  name               = local.bastion_name
  ami_id             = var.bastion_ami_id
  instance_type      = var.bastion_instance_class
  subnet_id          = local.instance_connect_subnet
  security_group_ids = [module.bastion_sg.id]
  key_name           = data.aws_key_pair.bastion_key.key_name 
  volume_size        = var.bastion_disk_size
  volume_name        = local.bastion_volume_name

  tags = local.default_tags
}

Connect to the private instance

Now the fun part, we will connect to the instances from CLI, don't worry, if you have something running on the instance, you can also access it via port forwarding. Let me show you -

1. SSH into private ec2 instance -

a. Connect using ec2 instance id

Use your ec2 instance id and the private key file .pem to connect to it. Remember to change the region as per your region -

INSTANCE_ID=<add your ec2 instance id here>

ssh -i bastion-private-key.pem ubuntu@$INSTANCE_ID \
  -o ProxyCommand="aws ec2-instance-connect open-tunnel --instance-id $INSTANCE_ID \
  --region ap-southeast-1"

But finding the instance id is a manual process and I hate manual processes, I can remember ec2 names, let's access using only the name -

b. Connect using ec2 instance name

Or, find the ec2 instance id by name, that's what I did to make my life easier -

INSTANCE_NAME=my-bastion-instance

INSTANCE_ID=$(aws ec2 describe-instances \
--region ap-southeast-1 \
--filters "Name=tag:Name,Values=$INSTANCE_NAME, running" \
  "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].InstanceId" --output text)

ssh -i bastion-private-key.pem ubuntu@$INSTANCE_ID \
  -o ProxyCommand="aws ec2-instance-connect open-tunnel --instance-id $INSTANCE_ID \
  --region ap-southeast-1"

2. Access services running on private ec2 instance -

Say you have Jenkins running on port 8080, or nginx on 80. Let's see how we can open not only 1, but also multiple ports and access using localhost -

a. Port forward single port

Let's port forward jenkins port 8080 to our localhost port 8080 -

INSTANCE_NAME=my-bastion-instance

INSTANCE_ID=$(aws ec2 describe-instances \
--region ap-southeast-1 \
--filters "Name=tag:Name,Values=$INSTANCE_NAME, running" \
  "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].InstanceId" --output text)

ssh -i bastion-private-key.pem ubuntu@$INSTANCE_ID \
  -o ProxyCommand="aws ec2-instance-connect open-tunnel --instance-id $INSTANCE_ID \
  --region ap-southeast-1" \
  -L 8080:localhost:8080

Here's what each part means - -L [local_port]:[remote_host]:[remote_port], forget about it and remember local_port:localhost:ec2_instance_port .

Now from the browser, if we type localhost:8080 , we will be able to access jenkins.

b. Access multiple ports

The same way we can tunnel multiple ports just by adding another -L portion, let's access port 80 and 8080 -

INSTANCE_NAME=my-bastion-instance

INSTANCE_ID=$(aws ec2 describe-instances \
--region ap-southeast-1 \
--filters "Name=tag:Name,Values=$INSTANCE_NAME, running" \
  "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].InstanceId" --output text)

ssh -i bastion-private-key.pem ubuntu@$INSTANCE_ID \
  -o ProxyCommand="aws ec2-instance-connect open-tunnel --instance-id $INSTANCE_ID \
  --region ap-southeast-1" \
  -L 8080:localhost:8080 \
  -L 80:localhost:80 \

Now you can access ec2 instance's 80 port using localhost:80 . It's that simple.

Remarks

Keeping your instance in private subnet is a very good choice since public IPv4 addresses cost nowadays. Also, traffic flowing out of the VPC also costs. So if the instances run using private subnets, the data never leaves the VPC itself, and that saves a lot.
Obviously there might be a case when the ec2 instance has to reside on public IP, that's a different case. Till then, keep your private resources in private subnets and save costs smartly, access securely.

Terraform Tip: Avoiding Resource Ownership Conflicts

Ashraf Minhaj — Thu, 17 Jul 2025 06:33:52 +0000

Ashraf Minhaj

Jul 17 '25

Terraform: Silent Resource Ownership Conflict

#terraform #aws #devops #infrastructureascode

Comments 2

3 min read

Terraform: Silent Resource Ownership Conflict

Ashraf Minhaj — Thu, 17 Jul 2025 05:31:14 +0000

Introduction

Today I'll talk about a more interesting and tricky problem, the Resource Ownership Conflict occurs when Terraform tries to manage a resource that is already managed by another Terraform configuration, or not managed by it at all.
In simple scenarios, Terraform might show an EntityAlreadyExists error, this is easy to resolve by renaming the resource or using a data source.
What about the silent ones? The one that does not show any error messages?

Background of my issue

Situation

I have two repositories.
On Repo 1: creates ECS cluster, Aurora postgres cluster and through a security group permits the ECS cluster to access the db. So we have some ingress rules -

  ingress {
    description     = "Allow Mobile API ECS to connect to Aurora"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [module.mobile_api_sg.id]
  }

  ingress {
    description     = "Allow bastion to connect to Aurora"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [module.bastion_sg.id]
  }

Repo 2: is for ETL, here a Glue job needs to access the RDS db that is managed by repo 1. The simplest thing we can do is, inherit the db security group using a data source by name and add another rule to it. Just like this -

 # inherit rds sg
data "aws_security_group" "rds_sg" {
  filter {
    name   = "tag:Name"
    values = [local.db_sg_name]
  }
}

# add ingress rule
# Add ingress rule to RDS sg
resource "aws_security_group_rule" "postgres_ingress" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  description              = "Allow Glue to access RDS"
  source_security_group_id = module.glue_sg.id                
  security_group_id        = data.aws_security_group.rds_sg.id
}

What went wrong?

Now unlucky for me, I did not see any error message. And to be honest there should not be any.
Since Repo1 owns the security group, it considers any rules added outside its configuration (e.g., from Repo2) as drift—and removes them on apply.

The obvious reason is, the state files. State file on the repo1 owns the RDS security group, and while creating the security group resource we added the rules inside it. So it will see any other changes as drift and will remove them. Drift - what you want vs what is actually present.
So it's a conflict created by proud me!

Solution to Silent Resource Ownership Conflict

Terraform keeps track of resources. So the solution would be simple. Create a security group resource, no rules/configs to it. Add rules in a separate resource block.

So, for Repo1 we will create an empty security group and add rules using aws_security_group_rule block as such -

resource "aws_security_group" "db_sg" {
  name        = local.db_sg_name
  description = "Aurora DB Security Group"
  vpc_id      = aws_vpc.vpc.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = local.db_sg_name
  }
}

# add the rules outside, using sg rule block -
resource "aws_security_group_rule" "admin_api_db_access" {
  type              = "ingress"
  description       = "Allow backend ECS to connect to Aurora"
  from_port         = 5432
  to_port           = 5432
  protocol          = "tcp"
  security_group_id = aws_security_group.db_sg.id
  source_security_group_id = module.admin_api_sg.id
}

# other rules just like that...

Repo2 remains as is.

How this solves the conflict/bug/error? When we define rules inside the aws_security_group block, it takes it as source of truth, now any rules not defined inside the block is a drift for it, so it will destroy those.
If we define the rules from outside the resource using aws_security_group_rule, terraform only cares about the aws_security_group_rule block, if that's there then the beast will remain silent and won't bother you.

It proved my suspicion when I ran a plan -

It said three resources to add, but did not say anything about the rules I removed from the aws_security_group block. It only cares about what it sees.

It's like convincing a child that no one is playing with it's toy.

Have fun practicing DevOps!!
See you in another one.

AWS Multi Region Failover Infrastructure

Ashraf Minhaj — Sat, 24 May 2025 16:54:57 +0000

Introduction

So you have selected a region closer to your target customer you want to make it Highly-Available. For that typically we create different subnets spanning over 2 or more AZs (Availability Zone) and call it a day.
But hey, a Region Consists of multiple AZs, what if the whole region gets down? Also, what if your target customers are from different regions?
Both the cases we can simply use AWS Global Accelerator which does the following things -

Routes user traffic through the AWS global network (not the public internet), reducing latency and jitter.
Continuously monitors endpoint health and routes traffic away from unhealthy ones.
Supports intelligent traffic routing to multiple AWS regions or endpoints based on health, geography, or weights.

Why are we here?

Well, when talking about a multi-region failover architecture, it's not that straight forward. A typical application consists of user facing Frontend, Backend, DataBase layer etc. all needs to be designed in a way that at any time the application serves the same thing from any of the selected regions.

This blog will discuss about those decisions.

1. Stateless Frontend & Backend: The First Step Toward Scalability

The first thing to do is to make your application stateless, meaning it won't save any data in itself but will fetch the data from a DB.
For a better performance and faster deployment time we can containerise the applications, because, why not!

And the same application will be deployed across multiple regions. Writing the infra using an IaC tool like Terraform simplifies things here.

2. Shared & Synced Data Layer Across Regions

Now comes the most challenging part. We will have multiple db instances across multiple regions but they need active syncing else not all the users across the regions will get the same results. The solution is to deploy db across 2 regions considering one primary and other secondary (read replica).
AWS aurora global DB can be used here, but it does not support failover automatically (promotion of database), and the db endpoint will change. For that a route53 (it's global) record can be used to to provide a custom db url like (db.my_app.com).

But since as of May24,2025 AWS does not support db promotion upon a failover (we still have to do it manually, a lambda function can be used to promote the db of another region to become the master (read, write).

3. Static Contents with cross region replication

For static assets we usually use an s3 bucket along with cloudfront to serve the contents. Since the buckets are also regional, we have to have multiple buckets across different regions and we can just use s3 cross region replication, the data will be synced across multiple regions automatically, and on cloudfront we can define a primary and secondary region, it will automatically serve from the healthy region.

Final Infrastructure

Finally here's a sample infrastructure where the
a. Frontend is a static website served from s3 bucket using cloudfront
b. The other static contents are served from again - s3 bucket
c. The business logic layer (Backend) is an ecs cluster spanned across multiple AZs and Regions at the same time
d. The DB has 1 master in one AZ, others are read replicas. Will be promoted by a lambda function which can run periodically or manually if needed.

It's not perfect, no solution is. Looking forward to your suggestions down in the comment section.

References

S3 Cross-Region Replication with Terraform - s3 High Availability

Ashraf Minhaj — Sat, 12 Apr 2025 19:12:57 +0000

Introduction

We keep our data in the cloud, it's stored in a region, but what if that region gets compromised? What if a natural disaster hits the datacenter and now our application is gone because it has data outage.
To avoid such scenarios, we should create a replica of our data in another region. The chance of both regions going down at once is extremely low. And that system has to be automatic, we can't copy paste everything every time. That's where AWS Cross Region Replication comes into play.

What is aws CRR?

In simple words, s3 Cross-Region Replication (CRR) lets you replicate data automatically from one bucket to another bucket. Here, Cross-Region means the bucket can be on different AWS regions.
In a simpler way, if we put one file in a bucket a copy will be created (replicated) in another bucket in another region.

Here's how CRR helps -

enhances data durability and availability, ensuring data is protected across geographically separate locations
useful for disaster recovery (DR) and
compliance requirements

The Game Plan

To set up CRR using Terraform, we need to:

Create two buckets in different regions (source and destination)
Enable versioning on both buckets
Set proper bucket policies
Create an IAM role for replication
Define replication rules

s3 Cross Region Replication

This will be the directory structure for the project -

├── main.tf
├── s3.tf
├── s3_replication.tf

1. Create two buckets in different regions

1.1. Create two providers

We need to create two provider for each regions. Alias is used so that we can identify them easily, you can name anything. So, the main.tf file is -

# Providers for different AWS regions
provider "aws" {
  alias  = "primary"
  region = "us-east-2" # Source region
}

provider "aws" {
  alias  = "secondary"
  region = "eu-north-1" # Destination region
}

1.2. Create Source and Destination buckets

Let's write the s3.tf file to create two buckets and enable versioning -

# Source S3 Bucket (Primary Region)
resource "aws_s3_bucket" "source_bucket" {
  provider      = aws.primary
  bucket        = "primary-bucket-for-replication"
}

resource "aws_s3_bucket_versioning" "source_versioning" {
  provider = aws.primary
  bucket   = aws_s3_bucket.source_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Destination S3 Bucket (Secondary Region)
resource "aws_s3_bucket" "destination_bucket" {
  provider      = aws.secondary
  bucket        = "secondary-bucket-for-replication"
}

resource "aws_s3_bucket_versioning" "destination_versioning" {
  provider = aws.secondary
  bucket   = aws_s3_bucket.destination_bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

2. Add replication Policy

Now we will create a replication policy which will contain which actions can be done by one bucket on another like ReplicateObject, ReplicateDelete etc. and finally will bind it to a role that can be attached to our buckets using aws_s3_bucket_replication_configuration.
Notice using

filter {
      prefix = "" # Replicate all objects or *.pdf *.mp4
    }

You can filter out what kind of files to replicate. Here I kept it on for all, because - why not it's a demo!
So, the s3_replication.tf file is -

# IAM Role for Replication
resource "aws_iam_role" "replication_role" {
  name = "s3-replication-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "s3.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

# IAM Policy for Replication
resource "aws_iam_policy" "replication_policy" {
  name        = "s3-replication-policy"
  description = "Allows S3 replication between primary and secondary buckets"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:ReplicateObject", "s3:ReplicateDelete", "s3:GetObjectVersion", "s3:GetObjectVersionAcl"]
        Resource = "arn:aws:s3:::${aws_s3_bucket.source_bucket.bucket}/*"
      },
      {
        Effect   = "Allow"
        Action   = ["s3:ReplicateObject", "s3:ReplicateDelete"]
        Resource = "arn:aws:s3:::${aws_s3_bucket.destination_bucket.bucket}/*"
      }
    ]
  })
}

# Attach Policy to Role
resource "aws_iam_role_policy_attachment" "replication_policy_attach" {
  role       = aws_iam_role.replication_role.name
  policy_arn = aws_iam_policy.replication_policy.arn
}

# S3 Replication Configuration
resource "aws_s3_bucket_replication_configuration" "replication" {
  provider = aws.primary
  bucket   = aws_s3_bucket.source_bucket.id
  role     = aws_iam_role.replication_role.arn

  rule {
    id     = "cross-region-replication"
    status = "Enabled"

    filter {
      prefix = "" # Replicate all objects or *.pdf *.mp4
    }

    destination {
      bucket        = aws_s3_bucket.destination_bucket.arn
      storage_class = "STANDARD"
    }
    delete_marker_replication {
      status = "Enabled"
    }
  }
}

Now, if we put one file in the source bucket (primary-bucket-for-replication) the object will very soon be created automatically in the secondary bucket as well (secondary-bucket-for-replication`.

Final thoughts

For a production use case, use variables file to handle variables and use strict permission on IAM policies, grant only what’s necessary.
Happy Coding!

Run aws Lambda locally with LocalStack

Ashraf Minhaj — Mon, 10 Feb 2025 17:56:29 +0000

Introduction

How do we usually debug a Lambda function? We deploy it to aws using the Web Console or Terraform then run it, see the output and repeat until we are satisfied. But hey, every lambda execution costs, not to mention the tiresome process to run it manually.
What if I tell you that you can deploy an aws-lambda function in locally simulated aws environment and actually run it with fruitful results! No more extra costs and boring process.

Imagine deploying your full micro services architecture in local! And yeah you don't have to have an aws account too. I can't wait to show you.

The Game Plan

We need LocalStack which simulates aws cloud inside our local machine for deploying, testing, debugging aws cloud infrastructure. It exposes an API (default: http://localhost:4566) using which we can deploy and manage aws resources.
There's another tool named tflocal, which is a wrapper for Terraform for LocalStack.
We just have to define some mock credentials in the provider section and that's it.

We will deploy the infrastructure (lambda here) and invoke it then save the output in a .json file for the sake of debugging. Let's go!

Build the infrastructure

We will write a basic lambda function. This will be the directory structure for the project -

├── README.md
├── Terraform
│   ├── lambda.tf
│   └── provider.tf
└── app
    └── lambda_function.py

1. Write the lambda function

First let's write the basic lambda function using python app/lambda_function.py -

import json

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps('Hello, I ran using LocalStack!')
    }

2. Write the Terraform code

Now let's write the provider Terraform/provider.tf -

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {

  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"

  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
}

We can write the lambda manifest now Terraform/lambda.tf. This automatically zips the lambda function, so chill.

# Automatically zip the Lambda code from the ../app directory
resource "archive_file" "lambda_zip" {
  type        = "zip"
  source_dir  = "../app"
  output_path = "../app/lambda_function.zip"
}

resource "aws_lambda_function" "my_lambda" {
  function_name = "my-test-lambda"
  role          = aws_iam_role.lambda_exec.arn
  handler       = "lambda_function.lambda_handler"
  runtime       = "python3.8"

  # Specify the location of the Lambda function code (the zip file)
  # the script takes it automatically
  filename      = "../app/lambda_function.zip"
}

resource "aws_iam_role" "lambda_exec" {
  name               = "lambda_exec_role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role_policy.json
}

data "aws_iam_policy_document" "lambda_assume_role_policy" {
  statement {
    actions   = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

You can now run terraform init, terraform plan to see if things went fine, but that's optional. We will use tflocal in the next step anyway.

All the codes can be downloaded from here.

3. Setup LocalStack

3.1. Install and run docker on your machine.
3.2. Install localstack - see instructions. I just installed using homebrew -

brew install localstack/tap/localstack-cli

3.3. Install tflocal - a wrapper for localstack to work with Terraform

pip3 install terraform-local

3.4. Run LocalStack -

localstack start -d
localstack status

# to stop
# localstack stop

3.5. You have AWS CLI installed, don't you?

4. Deploy and Run Lambda

4.1. Deploy the Lambda

Now on your project's Terraform/ directory run

tflocal init

Plan to see the draft output -

tflocal plan

Now we can apply the resources (localstack must be running) -

tflocal apply --auto-approve

Now let's see if there are any functions deployes. Let's use the aws cli command to list the lambda functions. Note, we have to pass the endpoint-url, that's the only change.

aws --endpoint-url=http://localhost:4566 --region us-east-1 lambda list-functions

There you have it!! We have successfully deployed a lambda function locally using localstack!!

4.2. Invoke the lambda

Let's invoke (run) the lambda to test, again using aws CLI commands but with the endpoint url passed -

aws --endpoint-url=http://localhost:4566 --region us-east-1 lambda invoke --function-name my-test-lambda output.json

So this will invoke the lambda function and save it's output in a file called output.json

Congratulations! Now you know how you can run aws lambda functions without putting pressure on the bills and automate any way possible.

What's next?

LocalStack opens up a whole new window to explore and enhance your DevOps project management capability. Without deploying on the actual cloud you can now play around on the local which gives you enough flexibility to explore and no worries about costs and errors.

The next projects where we can explore in future -

Connecting Lambda with API Gateway
Testing with event payloads
Running multiple microservices locally

May be I can write another blog on it with API GateWay. Till then, best wishes. Happy learning.