DEV Community: Md Shamim

AWS Security Series: AWS Access Key is Compromised. Now What? An Incident Response Playbook.

Md Shamim — Mon, 08 Dec 2025 17:18:07 +0000

In the world of AWS, it's always best practice to: use IAM roles whenever possible. IAM roles provide temporary, automatically rotated credentials that are the gold standard for security. But reality is not the same. What happens when we need to integrate with a third-party service that only supports static, long-term credentials? This is where the IAM user with an access key becomes a necessary...

So, the critical moment arrives: an access key is exposed. What to do?

Phase 1: Assess the situation

Our first instinct might be to hit the delete button—and fast. But hold on. A rash deletion could instantly break our production environment. A smarter, safer first move is to deactivate the key. Think of it as putting the key on 'pause.' It immediately stops working, giving you a crucial window to assess the impact on the applications.

Often, AWS detects this kind of exposure. It might send a notification and even proactively attach the AWSCompromisedKeyQuarantineV3 policy to the user to help limit the damage.

Phase 2: The Hidden Threat

We've deactivated the key, created a new set, and updated our applications. But then, when we check CloudTrail, and shocked!!! Unauthorized API calls are still happening.

How is this possible?

The attacker was one step ahead. Before we deactivated the main key, they used it to generate temporary session tokens using a command like this:

aws sts get-session-token --duration-seconds 129600 # (36 hours)

These tokens are independent of the original key and can remain valid for up to 36 hours, giving the attacker a persistent backdoor long after the original key is dead.

Phase 3: Add Deny Policy to restrict Temporary Credentials

We can create a "time-based fence" with an IAM policy to block new temporary tokens; Therefore, we can use a policy like the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyAccessWithTemporaryCredentialsIssuedAfterCompromise",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "DateLessThan": {
                    "aws:TokenIssueTime": "2025-12-08T16:00:00Z"
                }
            }
        }
    ]
}

Deny all temporary credentials that were created before the exposure of access keys.

Phase 4: The Recovery - Restoring Access Safely

It's time to rebuild. The goal is to restore access without re-introducing the same risk.

Create New Credentials: The cleanest approach is to create brand new access keys. And review the policy and apply the principle of least privilege, granting only the permissions necessary for its job.
Update All Systems: Update every application, script, configuration file, and secret store (like AWS Secrets Manager) with the new credentials.

Phase 5: The Post-Mortem - Hardening Our Defenses

The incident is over, but the work isn't. Now it's time to learn from the attack and strengthen our defenses.

Forensic Analysis with CloudTrail: Dive into CloudTrail logs. Filter events by the compromised user's name to see every API call they made. What data did they access? Did they try to create new users, roles, or backdoors? This log is our forensic evidence of the attack's scope.
Threat Detection with GuardDuty: We can use Amazon GuardDuty is an intelligent threat detection service that was likely monitoring our account during the attack. Review its findings for indicators of compromise, such as unusual API behavior or attempts to escalate privileges. GuardDuty can help us uncover other malicious activity we might have missed.

📌 Connect With Me
🔗 LinkedIn: https://www.linkedin.com/in/shamimice03/

Cross-Account VPC Associations with Route53 Private Hosted Zone and Addressing Terraform State Update Issue

Md Shamim — Sat, 31 Aug 2024 12:18:55 +0000

Background:

Let assume, we have a private hosted zone in Account A and a VPC associated with it from the same account. Now, we need to associate another VPC from Account B (which is a Cross-Account) to the private hosted zone residing in Account A.

However, this cannot be done via the AWS console. To accomplish this requirement, we'll need to use the programmatic approach. In this tutorial, we will be using AWS CLI to perform the necessary operations.

The following commands need to be run on Account A:
Account A needs to create a VPC association authorization to authorize the association of a VPC from Account B.

Create vpc association authorization:

aws route53 create-vpc-association-authorization \
    --hosted-zone-id <hosted-zone-id> \
    --vpc VPCRegion=<region>,VPCId=<vpc-id> \
    --region <your-region>

Check if VPC is authorized:

aws route53 list-vpc-association-authorizations \
    --hosted-zone-id Z03168043HMQYLM46KQBL

Expected Outcome:

{
    "VPCs": [
        {
            "VPCRegion": "region",
            "VPCId": "< target-vpc-id >"
        }
    ],
    "HostedZoneId": "< hosted-zone-id >"
}

The following commands need to be run on Account B:

Account B needs to associate-vpc-with-hosted-zone using the following command:

aws route53 associate-vpc-with-hosted-zone \
    --hosted-zone-id <hosted-zone-id> \
    --vpc VPCRegion=<region>,VPCId=<vpc-id> \
    --region <your-region>

Now, from the console, we can verify the associated VPC:

Addressing Terraform State Update Challenges

After associating cross-account VPC with a private hosted zone using CLI. In terraform, we might see terraform will delete the cross-account VPC from the hosted zone:

  # aws_route53_zone.private will be updated in-place
  ~ resource "aws_route53_zone" "private" {
        id                  = "Z03168043HMQYLAGDGAL"
        name                = "example.com"
        tags                = {}
        # (7 unchanged attributes hidden)

      - vpc {
          - vpc_id     = "vpc-072877fb4e12c2427" -> null
          - vpc_region = "us-east-1" -> null
        }

        # (1 unchanged block hidden)
    }

To resolve this issue we can use the lifecycle block inside the aws_route53_zone resource code:

resource "aws_route53_zone" "private" {
  name = "example.com"

  vpc {
    vpc_id = "vpc-0f76856d99df4csbf"
  }
  # Like this 
  lifecycle {
    ignore_changes = [vpc]
  }
}

That's all for now. Please let me know your feedback and if you have any questions.

Thanks!!
Md Shamim

Kubeconfig for EKS Cluster

Md Shamim — Fri, 09 Jun 2023 10:21:28 +0000

How to create a `kubeconfig` file for an existing EKS Cluster?

Step-1:

Install aws-cli on your machine using the following script:

#!/bin/bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

Step-2:

Run aws configure command to set up aws credentials:

$ aws configure

AWS Access Key ID [None]: <AccessID>
AWS Secret Access Key [None]: <AccessKey>
Default region name [None]: <Region>
Default output format [None]: <Format, e.g:json>

Step-3:

Create kubeconfig file on ~/.kube/config location:

$ aws eks update-kubeconfig --name <EKS_CLUSTER_NAME> \
                            --region <REGION_CODE>

Run:

$ aws eks update-kubeconfig --name cluster-1 --region ap-northeast-1 

$ cat ~/.kube/config

Bonus: How to list all existing EKS Cluster ?

$ aws eks list-clusters --region <AWS-REGION> 

$ aws eks list-clusters --region ap-northeast-1

Output:

{
    "clusters": [
        "Cluster-1",
        "Cluster-2"
    ]
}

Query Cluster Name:

$ EKS_CLUSTER_NAME=$(aws eks list-clusters --region ap-northeast-1 --query clusters[0] --output text)

$ echo $EKS_CLUSTER_NAME

Output:

Cluster-1

A Guide to Docker Multi-Stage Builds

Md Shamim — Thu, 22 Dec 2022 05:36:15 +0000

A Docker image is built up from a series of layers. Each layer represents an instruction in the image’s Dockerfile. Each layer except the very last one is read-only.

One of the most challenging things about building images is decreasing image size. In this article, we will discuss how we can optimize a docker image size.

Let’s create a custom docker image for a simple golang application.

# app.go

package main

import (
    "fmt"
    "time"
    "os/user"
)

func main () {
    user, err := user.Current()
    if err != nil {
        panic(err)
    }

    for {
        fmt.Println("user: " + user.Username + " id: " + user.Uid)
        time.Sleep(1 * time.Second)
    }
}

Now, let’s write a **Dockerfile **to package the golang application :

# Dockerfile

FROM ubuntu   # Base image 
ARG DEBIAN_FRONTEND=noninteractive   
RUN apt-get update && apt-get install -y golang-go    # Install golang
COPY app.go .                                         # Copy source code                 
RUN CGO_ENABLED=0 go build app.go                     
CMD ["./app"]

Next, Create a **docker image **and run a container from that image :

# Create image from the Dockerfile
>>  docker build -t goapp .
...
Successfully built 0f51e92fe409
Successfully tagged goapp:latest

# Run a container from the image created above
>> docker run -d goapp

04eb7e2f8dd2ade3723af386f80c61bdf6f5d9afe6671011b60f3a61756bdab6

Now, ‘exec’ into the container we created earlier :

# exec into the container
>> docker exec -it 04eb7e2f8dd sh

# list the files
~ ls
app  app.go  bin  boot  dev  etc  home  ...

# run the application 
~ ./app
user: root id: 0
user: root id: 0
user: root id: 0
user: root id: 0
user: root id: 0
...

We can see that after building the application we have appartifact inside the container. If we check the image size which helped us to build our application artifact :

>> docker images goapp

REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
goapp                       latest    0f51e92fe409   16 hours ago   870MB

The image size is ‘870MB’, but we can slim this down using multi-stage builds. With multi-stage builds, we will use multiple **FROM **statements in our Dockerfile. Each **FROM **instruction can use a different base, and each of them begins a new stage of the build. We can selectively copy artifacts from one stage to another by leaving everything that we don’t want in the final image. To show how this works, let’s adapt the **Dockerfile **from the previous section to use multi-stage build.

We will divide our Dockerfile into two stages. One will be the build stage , which will help us to build our application and generate the artifact. And then we will only copy the artifact from the build stage to another stage and create a tiny production image.

# Dockerfile
# named this stage as builder ----------------------
FROM ubuntu AS builder         
ARG DEBIAN_FRONTEND=noninteractive   
# Install golang
RUN apt-get update && apt-get install -y golang-go   
# Copy source code
COPY app.go .                                             
RUN CGO_ENABLED=0 go build app.go

# new stage -------------------
FROM alpine
# Copy artifact from builder stage                   
COPY --from=builder /app .   
CMD ["./app"]

Now, build the image and check the image size :

>> docker build -t goapp-prod  .

Successfully built 61627d74f8b8
Successfully tagged goapp-prod:latest

>> docker images goapp-prod

REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
goapp-prod   latest    61627d74f8b8   5 minutes ago   8.92MB  # <---

As we can see image size has been reduced significantly. It’s time to check if we can run a container from the image we created.

# create docker container
>> docker run goapp-prod

user: root id: 0
user: root id: 0

Great! We were able to use the tiny production image we created and it is working perfectly.

👉All Articles on Linux

All Articles on Linux

👉All Articles on Kubernetes

All Articles on Kubernetes

Helm — Create a Private Repository Using Apache Webserver

Md Shamim — Mon, 28 Nov 2022 02:45:36 +0000

Suppose, we have a scenario where we want to distribute our helm chart internally or within the organization. One of the ways to achieve that is to create a private repository using the Apache Server and the server will serve as a helm repository.

Read Full Article

Git Clone using Init-container | Kubernetes

Md Shamim — Sun, 27 Nov 2022 14:37:05 +0000

An Init-container is a special kind of container, which will run before the actual application. Init containers always run to completion.

A pod can have several init-containers. Init containers execute in a sequential manner. Each init-container must be completed successfully before the next one starts to execute.

Read Full Article

Helm — Dependencies

Md Shamim — Wed, 09 Nov 2022 05:24:06 +0000

In helm, one chart can be dependent on another chart. For instance, a WordPress application requires a database to start functioning. In helm, we can deploy WordPress as part of the parent chart and MySQL or any other required application as a dependency of the parent chart.

Helm charts store their dependencies in ‘charts/’. There are two different ways to add dependency charts to parent charts:

Listing all the dependencies inside the Chart.yaml file and helm will download the dependencies and store them in the ‘charts/’ Directory.

Manually populate the dependency chart inside the ‘charts/’ directory.

List dependencies inside the Chart.yaml file

Before listing dependencies in the Chart.yaml file, By default Chart.yaml file looks like this :

# Chart.yaml
apiVersion: v2
name: webserver
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"

Now, let's see how we can populate the dependencies into the Chart.yaml file:

dependencies:

- name: mysql
  version: "9.3.4"
  repository: "https://charts.bitnami.com/bitnami"

After adding dependencies to the Chart.yaml file :

# Chart.yaml

apiVersion: v2
name: webserver
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"
dependencies:
- name: mysql
  version: "9.3.4"
  repository: "https://charts.bitnami.com/bitnami"

Using a simple helm command we can pull the chart from the repository defined in the dependencies field:

   helm dependency update [CHART] 
>> helm dependency update ~/webserver

The above-defined command will generate .webserver/Chart.lock file as well as download all dependencies into the .webserver/charts directory

The Chart.lock file lists the exact versions of immediate dependencies and their dependencies and so on.

Read full article...

A Series on Bash Scripting

Md Shamim — Fri, 04 Nov 2022 00:44:30 +0000

What is Bash?

Bash is a type of shell that allows running commands on Linux.

What is Bash Script?

A Bash script is a plain text file that contains a series of commands.

Use Cases of Bash Scripting

Bash scripts can be used for various purposes, such as executing a shell command, running multiple commands together, customizing administrative tasks, performing task automation, etc.

How to learn?

You can start your bash scripting journey today. Following are the lists of articles :

● Start Your Scripting Journey Today | Bash Script — Part 1

● Start Your Scripting Journey Today | Bash Script — Part 2

● Start Your Scripting Journey Today | Bash Script — Part 3

Follow me to get more articles like this.

Start Your Scripting Journey Today | Bash Script

Md Shamim — Mon, 31 Oct 2022 05:37:55 +0000

A Bash script is a plain text file that contains a series of commands. Anything we can normally run from the command line can be used to create a script.

Article Links:

Helm—Named Templates

Md Shamim — Wed, 26 Oct 2022 06:00:49 +0000

We can create partial or sub-templates in helm, mainly known as “named templates”. A named template is simply a template defined inside a file and given a name.

Named templates can be considered just like functions. Named templates will allow us to reuse syntax or logic throughout the helm chart.

Article Link - Helm—Named Templates