DEV Community: Manasseh M

CI/CD Pipeline: DevSecOps for Application Deployment on Kubernetes: GitHub Actions and ArgoCD

Manasseh M — Mon, 14 Oct 2024 06:49:43 +0000

Terraform Configuration for AWS EKS Cluster
Github Link

Kubernetes DevSecOps CICD Project Using Github Actions and ArgoCD
Github Link

Part one: Setting up the environment

ssh exchange between my local computer and my github account

ssh-keygen 
export GIT_SSH_COMMAND="ssh -i ~/.ssh/key"

This command tells Git to use the specified SSH key for authentication during operations like git clone, git pull, and git push.

Steps to Create an IAM User and Generate Access Key

In the IAM dashboard, click on Users on the left-hand menu.
Click the Add users button.

Attach existing policies directly. Search for and select AdministratorAccess to give the user full access.

On the success screen, you will see an Access key ID and Secret access key for the user. Make sure to download the CSV or copy these to a safe place as they will not be retrievable again.

Set up an S3 bucket on AWS to store Terraform state files.

This is a common practice in Infrastructure-as-Code (IaC) deployments to ensure the state is stored remotely, securely, and is accessible for team collaboration.

The bucket will eventually populate once we run terraform init and store the file shown.

Store the AWS credentials

(Access Key, Secret Key, and BUCKET NAME) securely in GitHub Secrets for use in a CI/CD pipeline. These credentials will be used by GitHub Actions to authenticate with AWS services, such as deploying infrastructure or uploading state files to an S3 bucket.

The Terraform files define AWS infrastructure resources like S3, DynamoDB, and IAM roles, automating deployment through code.

*Deploying AWS resources using Terraform Configuration *Github Link

On push, the CI/CD pipeline in GitHub triggers automatic builds and deployments based on the latest code changes.

The pipeline ran successfully, completing all tasks such as building and deploying without failures.

Part two: Configuring the environment

The jump host server, often used for secure access to private network resources, has been successfully deployed. Lets SSH into it.
Confirm the tools such as docker, terraform, aws cli, kubectl, trivy and eksctl are installed.

Create an eks cluster
eksctl create cluster --name quizapp-eks-cluster --region us-east-1 --node-type t2.large --nodes-min 2 --nodes-max 4

Part three: Setting up mongodb, SonarQube, snyk and docker hub

Create a MongoDB cluster and a user.

Click "Build a Cluster" and choose a cloud provider, region, and configuration (shared/ free plan).
Click Create Cluster and wait for deployment.

Go to Database Access under the Security tab.
Click "Add New Database User".
Set a username and password, and assign the role (e.g., Atlas Admin or read/write).
Choose where the user can connect from (allow access from your IP or anywhere).

After the application is fully deployed using argocd, the database is actively handling requests and successfully connected to the application.

Set up SonarQube variables for a CI/CD pipeline

SONAR_ORGANIZATION: Specify your organization name in SonarQube.
SONAR_PROJECT_KEY: Define a unique key for your project within the organization.
SONAR_TOKEN: Generate a secure token from your SonarQube account to authenticate API requests.
SONAR_URL: Set the base URL (https://sonarcloud.io).

These variables help integrate SonarQube for code quality analysis within your CI/CD pipeline, ensuring secure authentication and project identification.

SonarQube is an open-source platform used to inspect the quality of code by analyzing codebases for bugs, vulnerabilities and technical debt.
It helps development teams ensure that their code adheres to best practices, maintains high standards, and meets specific criteria before it's deployed. SonarQube provides an easy-to-read dashboard that visualizes key metrics and offers detailed feedback on areas that need improvement.

Key Features of SonarQube:

Code Analysis: SonarQube performs static code analysis on multiple programming languages, identifying bugs, security vulnerabilities, and maintainability issues.
Quality Gates: A set of conditions that your project must meet before it is considered to pass. If the quality gate is not met (e.g., due to failing code coverage, high bug count, or poor maintainability), it will result in a "Quality Gate Failed" message on the dashboard.
Security Hotspots: SonarQube highlights security issues that require developer review, ensuring that code is secure before it goes into production.
Technical Debt Measurement: It calculates the time and effort required to fix the issues in your codebase (measured as technical debt).
Integration with CI/CD Pipelines: SonarQube integrates with popular build tools and CI/CD pipelines (e.g., Jenkins, GitHub Actions) to automate code quality checks.

GitHub Personal Access Token (PAT)

GitHub Personal Access Token is crucial for enabling secure and controlled interactions between the CI/CD pipeline and the GitHub repository, ensuring both functionality and security.

Authenticating with Snyk

SNYK_TOKEN is essential for securely authenticating with Snyk, automating vulnerability scanning, and ensuring controlled access within your CI/CD pipeline.

Snyk is a popular security tool that helps developers find and fix vulnerabilities in their code, open-source dependencies, container images, and infrastructure as code.

This indicates that the container image built on Node 18 has 168 known vulnerabilities. These could include issues in Node.js itself, or in any open-source libraries included in the image.
Upgrading from Node 18 to Node 20.18 only reduces the number of vulnerabilities slightly.

The zlib/zlibg Integer Overflow or Wrap Around refers to a critical vulnerability in the zlib library, which is a popular compression library used across many applications.

This vulnerability allows an attacker to exploit integer arithmetic by causing an overflow or wrap-around, potentially leading to arbitrary code execution, crashes, or other unpredictable behavior. It could allow malicious actors to exploit systems that use this vulnerable library, making it critical to patch.
Since this vulnerability is categorized as critical, it means it could have a significant impact on the security of your application, and patching it should be prioritized.

Setting up Docker Hub token

Docker Hub serves as a centralized repository to store and manage your Docker images. This makes it easy to version control your images and share them across different environments.

These keys and tokens are essential for securely authenticating and authorizing access to various services and tools in a CI/CD pipeline. They enable automation, enhance security, and facilitate collaboration across different environments and teams.

Part Four: Deploying React Application

On push, the CI/CD pipeline in GitHub triggers automatic builds and deployments based on the latest code changes.

CI/CD pipeline Github Link

The pipeline ran successfully, completing all tasks such as building and deploying without failures.

Create an eks cluster using the below commands.

The command allows connection to the EKS cluster created allowing Kubernetes operations on that cluster.

aws eks update-kubeconfig --region us-east-1 --name quizapp-eks-cluster

validate whether nodes are ready
kubectl get nodes
Configure the Load Balancer on our EKS because our application will have an ingress controller. Download the policy for the LoadBalancer prerequisite.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

Create the IAM policy

aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json

Create OIDC Provider
To allows the cluster to integrate with AWS IAM for assigning IAM roles to Kubernetes service accounts, enhancing security and management.

eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=quizapp-eks-cluster --approve

Create Service Account

eksctl create iamserviceaccount --cluster=quizapp-eks-cluster --namespace=kube-system --name=aws-load-balancer-controller --role-name AmazonEKSLoadBalancerControllerRole --attach-policy-arn=arn:aws:iam::<ACCOUNT-ID>:policy/AWSLoadBalancerControllerIAMPolicy --approve --region=us-east-1

Deploy the AWS Load Balancer Controller using Helm

sudo snap install helm --classic
helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=quizapp-eks-cluster --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

check whether aws-load-balancer-controller pods are running or not.
kubectl get deployment -n kube-system aws-load-balancer-controller

Configure ArgoCD

Create the namespace for the EKS Cluster.

kubectl create namespace quiz
kubectl get namespaces

Create a separate namespace for it and apply the argocd configuration for installation.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Confirm argoCD pods are running
kubectl get pods -n argocd

Expose the argoCD server as LoadBalancer

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Get the password for our argoCD server to perform the deployment.
sudo apt install jq -y

export ARGOCD_SERVER=`kubectl get svc argocd-server -n argocd -o json | jq --raw-output '.status.loadBalancer.ingress[0].hostname'`
export ARGO_PWD=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
echo $ARGO_PWD

Set up the Monitoring for our EKS Cluster using Prometheus and Grafana

Helm Add all the helm repos, the prometheus, grafana repos

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Install the prometheus

helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Install the Grafana

helm install grafana grafana/grafana -n monitoring --create-namespace

Get Grafana admin user password

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Confirm the services and validate from AWS LB console.
kubectl get svc -n monitoring

Access your Prometheus Dashboard Paste the Prometheus-LB-DNS:9090 in your browser. Click on Status and select Target. You will see a lot of Targets.

Click on Data Source, Select Prometheus and in the Connection, paste your Prometheus-LB-DNS:9090

Create a dashboard to visualize our Kubernetes Cluster Logs.
Import a type of Kubernetes Dashboard. 6417 ID

Deploy Quiz Application using ArgoCD

Configure the app_code github repository in ArgoCD

Create our application which will deploy the frontend, backend, database and ingress.

Deployment is synced and healthy

Once your Ingress application is deployed. It will create an Application Load Balancer, You can check out the load balancer named with k8s-ingress.

Troubleshooting
In my case the backend nodes were not healthy and this is what I did to solve the issue.
One needs to change the health path to (/api/questions)

Copy the ALB-DNS and go to Cloud Front and set up a distribution to route traffic from the load balancer.

This ensures that Cloud Front pulls content from my load balancer and delivers it efficiently to users.

To make the application accessible via a custom domain, I used Amazon Route 53, which is Amazon's DNS service.
I created a CNAME record in my Route 53 hosted zone and pointed it to the CloudFront distribution's DNS. This setup allows users to access the application through a friendly, custom domain name while benefiting from CloudFront's caching and distribution capabilities.

I used the MongoDB connection string to interact with the database manually through my console. This approach allowed me to directly upload the data to MongoDB Atlas without relying on the automated pipeline, ensuring that my application had access to the necessary data for proper functionality.

The data has been successfully been inserted.

After deploying the application, it’s essential to confirm that all pods are running correctly within the designated namespace. They run successfully!

kubectl get nodes -n quiz
kubectl logs <name of the pod> -n quiz

Final Part: App Demo

Throughout this journey, we started by automating the deployment process using GitHub Actions and Terraform, setting up an EKS cluster as the backbone of our infrastructure. From there, we integrated essential security tools like Snyk and SonarQube, ensuring that our code remained secure and of high quality. We also connected external services such as MongoDB and Docker Hub to streamline our data and container management.

Next, we configured a load balancer to manage traffic efficiently, linked CloudFront for content delivery, and used Amazon Route 53 to route our DNS. Finally, we set up Grafana and Prometheus for monitoring and observability, giving us a comprehensive view of the system’s health and performance.

This end-to-end process has given us a scalable, secure, and well-monitored application infrastructure—ready for production use. Thank you for following along!

More Grafana dashboard IDs to try:

Dashboard ID
k8s-addons-prometheus.json 19105
k8s-system-api-server.json 15761
k8s-system-coredns.json 15762
k8s-views-global.json 15757
k8s-views-namespaces.json 15758
k8s-views-nodes.json 15759
k8s-views-pods.json 15760

Reference

Master Three-Tier Application | A Complete DevSecOps Guide on AWS with Kubernetes, GitOps & ArgoCD

Event-Driven Architecture with DynamoDB, Kinesis Data Streams, Amazon Data Firehose, Lambda, and S3

Manasseh M — Sun, 13 Oct 2024 12:26:31 +0000

Event-driven architecture have become increasingly popular in modern application design making it's possible to create scalable and efficient event-driven systems that react to data changes in real time.
In this article, I will explain how to build an event-driven pipeline using Amazon DynamoDB, Kinesis Data Streams, Kinesis Data Firehose, AWS Lambda, and Amazon S3.

Overview of the Pipeline

The goal of this pipeline is to react to changes in a DynamoDB table, process the data, and store it in either S3 or trigger a Lambda function for further processing. Here's a high-level look at the flow:

DynamoDB – Your source of data.

Kinesis Data Streams – Streams the change events from DynamoDB.
Kinesis Data Firehose – Transforms and routes the streaming data.
AWS Lambda (optional) – Processes data for real-time events or complex transformations.
S3 – Stores processed data in a scalable, durable storage.

Part 1: Setting up the environment

Create a dynamo table

Amazon DynamoDB is a serverless, NoSQL, fully managed database with single-digit millisecond performance at any scale. It is well-suited for high-performance applications that require consistent and fast data access. e.g in Financial service applications, Gaming applications and Streaming applications

Setup Amazon Kinesis Data Streams

Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that makes it easy to elastically ingest and store logs, events, clickstreams, and other forms of streaming data in real time.
Kinesis Data Streams is particularly helpful when you need to handle massive volumes of streaming data. It allows multiple consumers to read from the stream simultaneously, making it possible to apply multiple transformations or route the data to different destinations.

Setup Amazon Data Firehose
Source - Amazon Kinesis Data Streams
Destination - Amazon S3

Amazon Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, Amazon OpenSearch Serverless, Splunk, Apache Iceberg Tables, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Coralogix, and Elastic.

In the Data Firehose choose Amazon Kinesis Data Streams as the source and
Amazon S3 as the Destination.

Create an s3 Bucket

Amazon S3 is a highly scalable and durable object storage service. S3 allows you to store structured, semi-structured, and unstructured data at virtually unlimited scale.
S3 is ideal for long-term data storage or batch processing use cases. Data in S3 can then be used for analytics, reporting, or further batch processing via services like AWS Glue or Amazon Athena.

Part 2: Insert and Manage the data

Insert data into the Dynamo Db table.

Four Items have been inserted into the table

Create a lambda function and use the blueprint option.
Use the Process records sent to an Amazon Data Firehose Stream. Runtime python3.10

Lambda is a serverless compute service that triggers based on events. In this case, it processes each data batch from Firehose and can perform more complex transformations or trigger alerts/notifications.

After you have created the function, go to the configuration settings and edit the Timeout to 3 mins instead of seconds.
This is useful if since the function involves longer processing tasks, like dealing with large data sets.

The function handles the incoming event, which contains records from Firehose. Each record has a recordId and data. The function processes each record and returns the result back to Firehose for further downstream operations.

import base64

print('Loading function')


def lambda_handler(event, context):
    output = []

    for record in event['records']:
        print(record['recordId'])
        payload = base64.b64decode(record['data']).decode('utf-8')

        # Do custom processing on the payload here
        print(f"the actual data is : {payload}")

        output_record = {
            'recordId': record['recordId'],
            'result': 'Ok',
            'data': base64.b64encode(payload.encode('utf-8')).decode('utf-8')
        }
        output.append(output_record)

    print('Successfully processed {} records.'.format(len(event['records'])))

    return {'records': output}

In the Amazon Data Firehose, configuration tab there is Transform and convert records option.
Select the lambda function we had created. This function is part of a real-time processing pipeline where you can make on-the-fly changes to the data before it is permanently stored or consumed by other services.

Insert more records in the dynamodb table

After Lambda processes the data, it is delivered to the S3 bucket via Firehose. S3 acts as the final storage layer in this pipeline.

All items initially created in the DynamoDB table are now stored as objects (files) in the S3 bucket.
These stored files can be used for data analysis, archiving, or further processing, such as feeding into machine learning models, querying with Amazon Athena, or visualizing with Amazon QuickSight.

Summary:

This architecture demonstrates how AWS services can work together in an event-driven, serverless pipeline. The flow begins with changes in DynamoDB, which are streamed in real-time through Kinesis Data Streams and Firehose, processed by Lambda if needed, and finally stored in S3.

Setting Up a High Availability Kubernetes Cluster on AWS with Kubeadm, Terraform, and Ansible

Manasseh M — Sat, 12 Oct 2024 13:16:55 +0000

Tech Stack

Terraform, Ansible, Docker, cri-dockerd, kubeadm, Kubernetes, Ubuntu, AWS {VPC, EC2, NLB}

This project contain the all required automation code for setting up Kubernetes cluster using kubeadm in AWS cloud environment.

Infrastructure Provisioning

Terraform for all the infrastructure provisioning automation.

Kubernetes Cluster Setup

Ansible for all Server & Cluster configurations.

Architecture Diagram

Writeup

Github Repo
Step 1: Clone the GitHub Repository and Initialize Terraform
terraform init
This command initializes the working directory, downloading provider plugins and preparing the environment.
terraform validate
This command checks if your Terraform configuration is syntactically valid and consistent
terraform apply
To apply the execution plan and deploy your resources

Step 2: Provisioning of resources on AWS and Ansible Playbook Configuation

After running terraform apply and confirming the action with "yes," Terraform is creating the specified resources such as VPC, subnets, NAT gateway, security groups, load balancer, and EC2 instances (workers, masters, bastion).

The Ansible playbook successfully gathers system information (facts) from all the EC2 instances (ip-10-0-*) without issues. Fact gathering is a default step that collects information about the managed nodes.

The following tasks are being executed across multiple EC2 instances:

Docker Installation: This task changes the system state, indicating that Docker was successfully installed on the target instances.
Install APT Transport HTTPS: Another package installation task required for using APT with HTTPS-based repositories.
Install curl: Curl was already installed, so no changes were made, but the task ran successfully on all instances.
Get Kubernetes package key: The task retrieves the Kubernetes package signing key, which is necessary for adding the Kubernetes repository.
Install Kubernetes repository: The repository is successfully installed, allowing Kubernetes tools to be installed.
Kubelet & Kubeadm Installation: These are essential Kubernetes components, and their installation completed successfully.
CRI-Dockerd Version: The playbook retrieves the latest version of cri-dockerd, which is a Docker runtime shim for Kubernetes. The output shows the version as v0.3.15.

The output shows:

The playbook generates a Master Join command using kubeadm token create.
The same process happens for worker nodes, where a join command is generated and copied locally for use.
kubectl, the Kubernetes command-line tool, is installed on the control plane nodes.
The additional control plane nodes are joined to the cluster using the join command with the token and certificate key.
The worker nodes are similarly joined to the cluster, completing the setup.

Play Recap: Shows the status for each EC2 instance involved (both control plane and worker nodes). Kubernetes cluster should be fully set up.

ok: Number of tasks successfully executed.
changed: Number of tasks that resulted in changes.
unreachable: 0 indicates all nodes were reachable.
failed: 0 means no tasks failed.
Total Resources Added: 49 added shows the creation of 49 resources (e.g., EC2 instances, network resources, etc.).

Output: Bastion Host IP: bastion_host_public_ip = "3.235.87.56" indicates the public IP of the bastion host, which you can use to access the cluster.

PART TWO:

resource "tls_private_key" "ssh" {
  algorithm = "RSA"
  rsa_bits  = "4096"
}

resource "local_file" "k8_ssh_key" {
    filename = "k8_ssh_key.pem"
    file_permission = "600"
    content  = tls_private_key.ssh.private_key_pem
}

resource "aws_key_pair" "k8_ssh" {
  key_name   = "k8_ssh"
  public_key = tls_private_key.ssh.public_key_openssh
}

File keys.tf generates SSH key pair using Terraform's tls_private_key resource and stores the private key locally in a PEM file(k8_ssh_key.pem). Then Uploads the public key to AWS as a key pair (k8_ssh) for use with EC2 instances.

Therefore, a PEM file (k8_ssh_key.pem) will be generated in your current working directory with the correct permissions.

Step 1: Connect to the Bastion host

Step 2: Get private ipv4 address for one of the Control Plane and ssh into it

kubectl get nodes
This command shows the nodes in your Kubernetes cluster. Each node represents a machine (physical or virtual) on which Kubernetes runs your applications.

Troubleshooting

kubectl get pods -A
This shows all the pods running across all namespaces (-A means all namespaces).

kube-flannel pods are crashing and coredns pods are stuck in ContainerCreating state.

Since Flannel is a networking tool, it may fail due to issues with the node network setup.
step1: Check the kube-flannel ConfigMap to ensure that the Network CIDR matches the PodCIDR assigned to your nodes.
kubectl get configmap -n kube-flannel kube-flannel-cfg -o yaml
step2: Check Node Configuration
`kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

Based on the output from the kubectl get configmap and kubectl get nodes commands, we can see that there is a mismatch between the Flannel network configuration and the PodCIDRs assigned to the nodes.
step3: Update the Flannel configuratioN and edit the ConfigMap.
kubectl edit configmap -n kube-flannel kube-flannel-cfg

All pods are up and Running

These pods are essential Kubernetes components:

coredns: Responsible for DNS resolution within the cluster.
kube-apiserver, kube-controller-manager, kube-scheduler: These components are essential to the control plane and are all running without issues, meaning the control plane is functional.
kube-proxy: This is a network proxy that maintains network rules on each node. All kube-proxy pods are running successfully.

PART THREE:

High availability is critical in Kubernetes deployments, especially for production environments. By leveraging a load balancer to manage traffic and implementing a multi-AZ architecture, we ensure that the Kubernetes cluster remains resilient, scalable, and secure. This approach minimizes the risk of downtime and ensures that your applications are always available, even in the event of failures.

Enhancing Security using AWS Secret Manager

Manasseh M — Fri, 09 Feb 2024 07:59:33 +0000

Managing secrets such as database credentials, API keys, and other sensitive information is a critical aspect of cloud security. Exposing these secrets can lead to unauthorized access and potentially compromise your cloud environment.

Secrets Manager

This service is designed specifically for handling secrets, it enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Secrets Manager integrates with AWS services such as such as Amazon Redshift, Amazon DocumentDB (MongoDb), and Amazon Relational Database Service (Amazon RDS) and supports the automatic rotation of secrets.

Lab
In this hands-on task, we will illustrate the process of safeguarding an application through the secure storage of a Google MapsApi key and a DockerHubPassword where we will employ secret rotation techniques. We will utilize an AWS Lambda function to access the stored secrets.

Step 1: Setting Up AWS Secrets Manager

• Navigate to the AWS Secrets Manager console and create a new secret

• Choose "Other type of secrets" if your application uses a custom type of secret Input your Google mapsAPI credentials or secret information and move on to the next step.

• Configure a secret name eg. test-key and a description, and move on to the next step. In this part, we will create a secret that does not require automatic rotation.

Step 2: Accessing Secrets from Your AWS Lambda Function

• Create a lambda function eg.SecretsMangerKeyRetrival. Adapt the AWS Lambda function to fetch the secret eg. test-key.

• Utilize the AWS SDK (e.g., Boto3 for Python) in Lambda to invoke the get_secret_value API.

import json
import boto3
import base64

# Your secret's name and region
secret_name = "test-key"
region_name = "us-east-1"

#Set up our Session and Client
session = boto3.session.Session()
client = session.client(
    service_name='secretsmanager',
    region_name=region_name
)

def lambda_handler(event, context):

    # Calling SecretsManager
    get_secret_value_response = client.get_secret_value(
        SecretId=secret_name
    )

    #Raw Response
    print(get_secret_value_response)

    #Extracting the key/value from the secret
    secret = get_secret_value_response['SecretString']
    print(secret)

• Click on the role on the configuration which will direct you to the IAM console. Add the custom policy that will enable the Lambda function Assign with the necessary permissions for accessing secrets in Secrets Manager.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:us-east-1:799151699415:secret:DockerHubSecret-3Y8fsD"
        }
    ]
}

• When one tests the function the credentials that were stored in the AWS Secrets Manager will be displayed. In this case, the secret name test-key displays the Google Maps API credentials.

Using Secret Rotation configuration

Step 1: Setting Up AWS Secrets Manager

• Navigate to the AWS Secrets Manager console and create a new secret e.g DockerHubPaasword

• Configure the Rotation Schedule minimum time unit is 4 hours. One can specify the secret to change after hours, days, weeks, or months.

Step 2: Accessing Secrets from Your AWS Lambda Function

• Create a lambda function eg. SecretsRotationalFunction. Adapt the AWS Lambda function to fetch the secret eg.DockerHubSecret.

• Utilize the AWS SDK (e.g., Boto3 for Python) in Lambda to invoke the get_secret_value API.

import json
import boto3
import base64

def lambda_handler(event, context):
    arn = event['SecretId']
    token = event['ClientRequestToken']
    step = event['Step']

    # Setup the client
    service_client = boto3.client('secretsmanager')

    # Make sure the version is staged correctly
    metadata = service_client.describe_secret(SecretId=arn)
    if not metadata['RotationEnabled']:
        raise ValueError("Secret %s is not enabled for rotation" % arn)

    if step == "createSecret":
        create_secret(service_client, arn, token)

    elif step == "setSecret":
        set_secret(service_client, arn, token)

    elif step == "testSecret":
        test_secret(service_client, arn, token)

    elif step == "finishSecret":
        finish_secret(service_client, arn, token)

    else:
        raise ValueError("Invalid step parameter")


def create_secret(service_client, arn, token):

    service_client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")

    # Now try to get the secret version, if that fails, put a new secret
    try:
        service_client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")
    except service_client.exceptions.ResourceNotFoundException:
        # Generate a random password
        passwd = service_client.get_random_password(ExcludeCharacters='/@"\'\\')
        # Put the secret
        service_client.put_secret_value(SecretId=arn, ClientRequestToken=token, SecretString=passwd['RandomPassword'], VersionStages=['AWSPENDING'])


def set_secret(service_client, arn, token):
    print("No database user credentials to update...")


def test_secret(service_client, arn, token):
    print("No need to testing against any service...")


def finish_secret(service_client, arn, token):

    # First describe the secret to get the current version
    metadata = service_client.describe_secret(SecretId=arn)

    for version in metadata["VersionIdsToStages"]:
        if "AWSCURRENT" in metadata["VersionIdsToStages"][version]:
            if version == token:
                # The correct version is already marked as current, return
                return

            # Finalize by staging the secret version current
            service_client.update_secret_version_stage(SecretId=arn, VersionStage="AWSCURRENT", MoveToVersionId=token, RemoveFromVersionId=version)
            break

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetRandomPassword",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:PutSecretValue",
                "secretsmanager:UpdateSecretVersionStage"
            ],
            "Resource": "*"
        }
    ]
}

• On the configuration tab under the Resource-based policy statements edit the policy statement to enable the Secret Manager to invoke the lambda function.

• After this, when one goes back to the Secrets manager, then to the secret name under the rotation tab select the function to enable it.

• When one views the secret, one identifies that the previous password has been automatically changed therefore proving the lambda function is being invoked and there is rotation of secrets.

After 4 hours.

AWS Secrets Manager pricing is generally structured around two main cost factors: The storage of secrets AWS Secrets Manager charges a monthly fee for each secret stored. The usage of API calls typically a certain number of free API calls per month, after which you are charged a rate for additional calls.

Access to AWS Secrets Manager is managed through AWS Identity and Access Management (IAM), and activity can be monitored using AWS CloudTrail and Amazon CloudWatch.

*References *
What is AWS Secrets Manager AWS Secrets Manager

Endre Synnes AWS Secrets Manager - Rotate Secrets

Be A Better Dev - AWS Lambda and Secrets Manager Tutorial in Python