DEV Community: Esther Ninyo

Implementation of a notification system using AWS part 2 - Lambda function and AWS SES setup

Esther Ninyo — Mon, 31 Mar 2025 21:02:19 +0000

INTRODUCTION

In this post, we'll continue from where we left off and set up a Lambda function to process messages from the SQS queue and send emails using Amazon SES.

Resources to be created

Lambda
AWS SES

Step 1
Create a lambda function that consumes the messages from the sqs queue, processes it and send it to the receiver's email address or send it to a dead letter queue when faced with a problem.

Choose Author from scratch
Enter the function name emailNotificationFunction
Under runtime, choose python from the dropdown list
Then click on create function. This will automatically create an IAM role. Follow the steps below to see the role created and to also update the policy to what is required.
Click on the function you just created to see the overview. Scroll down a bit, click on the configuration tab.
On the left pane, click on permissions. Locate the execution role and click on the link under the role name to take you to the role created.
You will see a policy name that starts with AWSLambdaBasicExecutionRole-. Edit it with the permission below and save it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:REGION:ACCOUNT_ID:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/lambda/emailNotificationFunction:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "arn:aws:sqs:REGION:ACCOUNT_ID:emailNotificationQueue"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ses:SendEmail",
            "Resource": "*"
        }
    ]
}

Leave everything else as it is. We will update it later.
Check the images below for pictorial reference

EMAIL SETUP
The next thing we will doing is to setup the email address that our notification will go to. Once you have your email address handy, head over to the step below to verify your email address.

SUBSCRIPTION
Subscribe lambda to SQS

Click on the sqs queue you created, scroll down a bit and click on the lambda triggers tab.
Click on configure lambda function trigger, choose your lambda function from the drop down or type in the lambda arn and click on save. See images below for reference

LAMBDA FUNCTION LOGIC
Step One: Write the code to process messages from the SQS queue and send emails using SES.

Step Two: Deploy the function and test it to ensure it works correctly.

Example lambda logic

import json
import logging
import boto3
from botocore.exceptions import ClientError

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Initialize SES client
ses_client = boto3.client('ses', region_name='REGION-NAME')  # Replace 'REGION-NAME' with the AWS region where SES is available

SENDER = "Your Name <EMAIL-ADDRESS>"  # Replace with your sender email
CHARSET = "UTF-8"

def send_email(to_email, subject, body):
    try:
        response = ses_client.send_email(
            Destination={
                'ToAddresses': [
                    to_email,
                ],
            },
            Message={
                'Body': {
                    'Text': {
                        'Charset': CHARSET,
                        'Data': body,
                    },
                },
                'Subject': {
                    'Charset': CHARSET,
                    'Data': subject,
                },
            },
            Source=SENDER,
        )
    except ClientError as e:
        logger.error(f"Failed to send email: {e.response['Error']['Message']}")
        return False
    else:
        logger.info(f"Email sent! Message ID: {response['MessageId']}")
        return True

def process_message(message_body):
    try:
        msg = json.loads(message_body)
    except json.JSONDecodeError:
        logger.error(f"Failed to decode message: {message_body}")
        return

    try:
        message = json.loads(msg.get('Message'))
    except json.JSONDecodeError:
        logger.error(f"Failed to decode message: {msg.get('Message')}")
        return

    print(message)

    event_type = message.get('eventType')
    if event_type == 'UserRegistration':
        user_name = message.get('userName')
        email = message.get('email')

        if user_name and email:
            subject = "Welcome to Our Service!"
            body = f"Hello {user_name},\n\nThank you for registering with us.\n\nBest regards,\nYour Company"
            send_email(email, subject, body)
        else:
            logger.error(f"Missing userName or email in message: {message}")
    else:
        logger.warning(f"Unknown event type: {event_type}")
        raise ValueError("Unknown EventType")

def lambda_handler(event, context):
    for record in event['Records']:
        message_body = record['body']
        logger.info(f"Received message: {message_body}")
        process_message(message_body)

    return {
        'statusCode': 200,
        'body': json.dumps('Messages processed successfully!')
    }

CONCLUSION

By following these steps, you've successfully created a Lambda function and set up Amazon SES to send email notifications. Your system is now ready to process messages and send emails based on the notifications received.

Implementation of a notification system using AWS - SNS and SQS setup

Esther Ninyo — Mon, 31 Mar 2025 21:01:28 +0000

INTRODUCTION

In this post, we'll walk through the steps to set up Amazon SNS and SQS to handle email notifications. This setup will allow you to send messages to an SQS queue via SNS, which can then be processed by a Lambda function.

Resources to be created

SNS (Simple Notification System)
SQS (Simple Queue Service)

Step 1
From the AWS console, navigate to the SNS dashboard and create a topic called emailNotificationTopic.

Choose sns-type standard
Enter the sns topic name emailNotificationTopic and display name (optional).
Then click on create topic.
On the emailNotificationTopic page, scroll down a bit and locate the access policy tab. Edit the access policy by adding the policy below

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish",
        "SNS:RemovePermission",
        "SNS:SetTopicAttributes",
        "SNS:DeleteTopic",
        "SNS:ListSubscriptionsByTopic",
        "SNS:GetTopicAttributes",
        "SNS:AddPermission",
        "SNS:Subscribe"
      ],
      "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC_NAME",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "ACCOUNT_ID"
        }
      }
    }
  ]
}

Check the images below for pictorial reference

Step 2
Create an SQS queue that receives and stores the messages sent from the SNS topic until it is consumed.

Choose queue-type standard
Enter the sqs name emailNotificationQueue.
Then click on create queue.
On the emailNotificationQueue page, scroll down a bit and locate the access policy tab. Edit the access policy by adding the policy below

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC_NAME"
        }
      }
    }
  ]
}

Leave everything else as it is. We will update it later.
Check the images below for pictorial reference

SUBSCRIPTION
In this section, we will subscribe the sqs queue to the sns topic so that the queue can consume any message from the topic.
Subscribe queue to SNS

Click on the sns topic you created, scroll down a bit and click on the subscription tab.
Click on create subscription, fill in the details and click on create. See images below for reference

CONCLUSION

By following these steps, you've successfully set up an SNS topic and an SQS queue, and subscribed the queue to the topic. In the next post, we'll cover how to create a Lambda function to process these messages and send emails using Amazon SES.

Provisioning AWS CloudTrail using Terraform (Step-by-Step)

Esther Ninyo — Sun, 31 Mar 2024 05:09:00 +0000

CloudTrail is an AWS service that enables governance, compliance, operational and risk auditing of your AWS account. It captures all the events that happens within your AWS account and it is recorded as events in CloudTrail. The trail will be stored in an s3 bucket of your choice.
To read more on CloudTrail, please visit the documentation page.

In this technical post, we'll walk through the steps to provision CloudTrail using Terraform.

Resources to be created:

S3 bucket
KMS (Key Management System) for S3 objects encryption
CloudTrail

Prerequisite:

An AWS account with permissions to create CloudTrail resources
AWS cli configured
Terraform installed

Folder Structure

cloudtrail  #this is a folder
---> providers.tf
---> s3.tf
---> main.tf
---> kms.tf

Step 1:

Create a new directory (cloudtrail) and navigate into it

mkdir cloudtrail
cd cloudtrail

providers.tf

terraform {
  required_version = "~> 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "eu-west-1"

  default_tags {
    tags = {
      Environment = terraform.workspace,
      ManagedBy   = "Terraform"
    }
  }
}

Step 2:

Initialize your project

terraform init

A successful initialization should look like the image below:

Copy the code below to your main.tf file

main.tf

resource "aws_cloudtrail" "cloudtrail" {
  name                       = "cloudtrail-tutorial"
  s3_bucket_name             = aws_s3_bucket.cloudtrail_s3.id
  kms_key_id                 = aws_kms_key.cloudtrail_kms_key.arn
  enable_log_file_validation = true
  is_multi_region_trail      = true
  enable_logging             = true

  depends_on = [
    aws_s3_bucket.cloudtrail_s3,
    data.aws_iam_policy_document.cloudtrail_s3_policy,
    aws_kms_key.cloudtrail_kms_key
  ]
}

Copy the code below to your s3.tf file
s3.tf

resource "aws_s3_bucket" "cloudtrail_s3" {
  bucket        = "cloudtrail-bucket"
  force_destroy = true
}

resource "aws_s3_bucket_acl" "s3_bucket" {
  bucket = aws_s3_bucket.cloudtrail_s3.id

  acl = "private"
}

resource "aws_s3_bucket_public_access_block" "pub_access" {
  bucket                  = aws_s3_bucket.cloudtrail_s3.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}


resource "aws_s3_bucket_policy" "cloudtrail_s3_policy" {
  bucket = aws_s3_bucket.cloudtrail_s3.id
  policy = data.aws_iam_policy_document.cloudtrail_s3_policy.json
}

data "aws_iam_policy_document" "cloudtrail_s3_policy" {
  statement {
    sid       = "AWSCloudTrailAclCheck"
    effect    = "Allow"
    resources = ["${aws_s3_bucket.cloudtrail_s3.arn}"]
    actions   = ["s3:GetBucketAcl"]

    condition {
      test     = "StringEquals"
      variable = "AWS:SourceArn"
      values   = ["arn:aws:cloudtrail:eu-west-1:${data.aws_caller_identity.current.account_id}:trail/cloudtrail-${terraform.workspace}"]
    }

    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
  }

  statement {
    sid       = "AWSCloudTrailWrite"
    effect    = "Allow"
    resources = ["${aws_s3_bucket.cloudtrail_s3.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"]
    actions   = ["s3:PutObject"]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }

    condition {
      test     = "StringEquals"
      variable = "AWS:SourceArn"
      values   = ["arn:aws:cloudtrail:eu-west-1:${data.aws_caller_identity.current.account_id}:trail/cloudtrail-${terraform.workspace}"]
    }

    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
  }
}

Copy the code below to your kms.tf file
kms.tf

data "aws_caller_identity" "current" {}

resource "aws_kms_key" "cloudtrail_kms_key" {
  description         = "KMS key for cloudtrail"
  enable_key_rotation = true
  policy              = data.aws_iam_policy_document.kms_policy.json

  depends_on = [
    data.aws_iam_policy_document.kms_policy
  ]
}

resource "aws_kms_alias" "kms_alias" {
  name          = "alias/cloudtrail_kms"
  target_key_id = aws_kms_key.cloudtrail_kms_key.key_id
}


# KMS KEY POLICY
data "aws_iam_policy_document" "kms_policy" {
  # Allow root users full management access to key
  statement {
    sid       = "Enable IAM User Permissions"
    effect    = "Allow"
    actions   = ["kms:*"]
    resources = ["*"]
    principals {
      type = "AWS"
      identifiers = [
      "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
  }
  statement {
    sid       = "Allow CloudTrail to encrypt logs"
    effect    = "Allow"
    actions   = ["kms:GenerateDataKey*"]
    resources = ["*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "AWS:SourceArn"
      values   = ["arn:aws:cloudtrail:eu-west-1:${data.aws_caller_identity.current.account_id}:trail/cloudtrail-${terraform.workspace}"]
    }
    condition {
      test     = "StringLike"
      variable = "kms:EncryptionContext:aws:cloudtrail:arn"
      values   = ["arn:aws:cloudtrail:*:${data.aws_caller_identity.current.account_id}:trail/*"]
    }
  }
  statement {
    sid       = "Allow CloudTrail to describe key"
    effect    = "Allow"
    actions   = ["kms:DescribeKey"]
    resources = ["*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
  }
  statement {
    sid    = "Allow principals in the account to decrypt log files"
    effect = "Allow"
    actions = [
      "kms:Decrypt",
      "kms:ReEncryptFrom"
    ]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    condition {
      test     = "StringEquals"
      variable = "kms:CallerAccount"
      values   = ["${data.aws_caller_identity.current.account_id}"]
    }
    condition {
      test     = "StringLike"
      variable = "kms:EncryptionContext:aws:cloudtrail:arn"
      values   = ["arn:aws:cloudtrail:*:${data.aws_caller_identity.current.account_id}:trail/*"]
    }
  }
  statement {
    sid       = "Allow alias creation during setup"
    effect    = "Allow"
    actions   = ["kms:CreateAlias"]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    condition {
      test     = "StringEquals"
      variable = "kms:CallerAccount"
      values   = ["${data.aws_caller_identity.current.account_id}"]
    }
    condition {
      test     = "StringEquals"
      variable = "kms:ViaService"
      values   = ["ec2.eu-west-1.amazonaws.com"]
    }
  }
  statement {
    sid    = "Enable cross account log decryption"
    effect = "Allow"
    actions = [
      "kms:Decrypt",
      "kms:ReEncryptFrom"
    ]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    condition {
      test     = "StringEquals"
      variable = "kms:CallerAccount"
      values   = ["${data.aws_caller_identity.current.account_id}"]
    }
  }
}

Step 3:

terraform plan

A successful plan should look like the image below:

Step 4:

terraform apply

A successful plan should look like the image below:

Conclusion

You have successfully created an AWS CloudTrail service.
Do you have any question? Please send it my way. Kindly follow me on LinkedIn.

Monitor EC2 instance metrics with Datadog (step-by-step)

Esther Ninyo — Wed, 27 Mar 2024 15:27:44 +0000

Hi there,
Ever thought of a straight forward way to monitor your EC2 instance metrics with Datadog but couldn't get a simplified solution? Look no further!

The three phases to get this up and running are:
Phase one: Enable the AWS integration in Datadog
Phase two: Deploy the Datadog agent on your EC2 instance
Phase three: Start creating your monitors

Datadog agent can be installed directly on your EC2 instances which gives you the ability to collect metrics such as memory, CPU, disk etc within a short period of time.
To have a robust understanding of how this works, please visit the Datadog blog post for more detail.

Pre-requisite:

To continue with this hands-on, make sure you have the following:

EC2 instance
Datadog account

Project deep dive

For the scope of this project, we will be monitoring the following system-level EC2 metrics such as:

High CPU Utilization
High Memory Utilization
High Disk Utilization

PHASE ONE

This phase consist of enabling the AWS integration in Datadog to allow monitoring of the EC2 instance.

We will setup the Datadog integration using terraform. You can get it here.

Folder structure

--> EC2 monitoring
------> provider.tf
------> main.tf
------> variables.tf

main.tf

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "datadog_aws_integration_assume_role" {
   statement {
   actions = ["sts:AssumeRole"]

   principals {
      type = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
   }
   condition {
      test = "StringEquals"
      variable = "sts:ExternalId"

      values = [
         "${datadog_integration_aws.sandbox.external_id}"
      ]
   }
   }
}

data "aws_iam_policy_document" "datadog_aws_integration" {
   statement {
        actions = [
            "ec2:Describe*",
            "ec2:GetTransitGatewayPrefixListReferences",
            "ec2:SearchTransitGatewayRoutes"
        ]

   resources = ["arn:aws:ec2:${var.region}:${data.aws_caller_identity.current.account_id}:instance/${var.instance_id}"]
   }
}

resource "aws_iam_policy" "datadog_aws_integration" {
   name = "TutorialDatadogAWSIntegrationPolicy"
   policy = "${data.aws_iam_policy_document.datadog_aws_integration.json}"
}

resource "aws_iam_role" "datadog_aws_integration" {
   name = "TutorialDatadogAWSIntegrationRole"
   description = "Role for Datadog AWS Integration"
   assume_role_policy = "${data.aws_iam_policy_document.datadog_aws_integration_assume_role.json}"
}

resource "aws_iam_role_policy_attachment" "datadog_aws_integration" {
   role = "${aws_iam_role.datadog_aws_integration.name}"
   policy_arn = "${aws_iam_policy.datadog_aws_integration.arn}"
}

resource "datadog_integration_aws" "sandbox" {
   account_id  = "${data.aws_caller_identity.current.account_id}"
   role_name   = "TutorialDatadogAWSIntegrationRole"
}

variable.tf

variable "region" {
  type        = string
  description = "The AWS region to use."
  default     = "eu-west-1"
}

variable "datadog_api_key" {
  type        = string
  description = "The Datadog API key."
  default     = "<REDACTED>"
}

variable "datadog_app_key" {
  type        = string
  description = "The Datadog application key."  
  default     = "<REDACTED>"
}

variable "instance_id" {
  type        = string
  description = "EC2 instance ID."  
  default     = "<REDACTED"
}

provider.tf

terraform {
  required_version = "~> 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
    datadog = {
      source  = "DataDog/datadog"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = var.region

  default_tags {
    tags = {
      Environment = terraform.workspace,
      ManagedBy   = "Terraform"
    }
  }
}

# Configure the Datadog provider
provider "datadog" {
    api_key = var.datadog_api_key
    app_key = var.datadog_app_key
    api_url = "https://api.datadoghq.eu"
}

Get datadog app key, api key and api url

Go to your datadog profile at the bottom left and click on organisation settings.

Locate the navigation pane at the left (1), under access (2), click on application key (3) to create a new key. Also click on the api key (4) to create a new key to be used.

Click on this link to access the api url depending the Datadog site you use. Replace app with api.

WHAT NEXT?
The next line of action is to initialise, plan and apply your terraform changes. To do this, use the command below in your folder home directory:
terraform init

terraform plan

terraform apply

If the terraform plan is successful, you should see the resources that will be created after running terraform apply like the result below:

terraform plan
data.aws_iam_policy_document.datadog_aws_integration: Reading...
data.aws_caller_identity.current: Reading...
data.aws_iam_policy_document.datadog_aws_integration: Read complete after 0s [id=1400131043]
data.aws_caller_identity.current: Read complete after 0s [id=134130342652]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.datadog_aws_integration_assume_role will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "datadog_aws_integration_assume_role" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions = [
              + "sts:AssumeRole",
            ]

          + condition {
              + test     = "StringEquals"
              + values   = [
                  + (known after apply),
                ]
              + variable = "sts:ExternalId"
            }

          + principals {
              + identifiers = [
                  + "arn:aws:iam::<REDACTED>:root",
                ]
              + type        = "AWS"
            }
        }
    }

  # aws_iam_policy.datadog_aws_integration will be created
  + resource "aws_iam_policy" "datadog_aws_integration" {
      + arn         = (known after apply)
      + id          = (known after apply)
      + name        = "TutorialDatadogAWSIntegrationPolicy"
      + name_prefix = (known after apply)
      + path        = "/"
      + policy      = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "ec2:SearchTransitGatewayRoutes",
                          + "ec2:GetTransitGatewayPrefixListReferences",
                          + "ec2:Describe*",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:ec2:<REDACTED>:instance/<REDACTED>"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + policy_id   = (known after apply)
      + tags_all    = {
          + "Environment" = "default"
          + "ManagedBy"   = "Terraform"
        }
    }

  # aws_iam_role.datadog_aws_integration will be created
  + resource "aws_iam_role" "datadog_aws_integration" {
      + arn                   = (known after apply)
      + assume_role_policy    = (known after apply)
      + create_date           = (known after apply)
      + description           = "Role for Datadog AWS Integration"
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "TutorialDatadogAWSIntegrationRole"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = {
          + "Environment" = "default"
          + "ManagedBy"   = "Terraform"
        }
      + unique_id             = (known after apply)
    }

  # aws_iam_role_policy_attachment.datadog_aws_integration will be created
  + resource "aws_iam_role_policy_attachment" "datadog_aws_integration" {
      + id         = (known after apply)
      + policy_arn = (known after apply)
      + role       = "TutorialDatadogAWSIntegrationRole"
    }

  # datadog_integration_aws.sandbox will be created
  + resource "datadog_integration_aws" "sandbox" {
      + account_id                       = "<REDACTED>"
      + cspm_resource_collection_enabled = (known after apply)
      + external_id                      = (known after apply)
      + id                               = (known after apply)
      + metrics_collection_enabled       = (known after apply)
      + resource_collection_enabled      = (known after apply)
      + role_name                        = "TutorialDatadogAWSIntegrationRole"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

PHASE TWO

The second phase is to deploy the agent.
Use the command below to install the agent on ubuntu server:

DD_API_KEY=<API_KEY DD_SITE=<DATADOG_SITE> DD_APM_INSTRUMENTATION_ENABLED=host bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

where:
API_KEY = your Datadog api key
DATADOG_SITE = Datadog site. For this exercise, we use "datadog.eu".

Depending on the operating system you use, navigate to this site to get the command for installing Datadog agent.

After the Datadog agent agent has been installed, go to your Datadog account, navigate to metrics and you will start to see the reports of your EC2 metrics in Datadog like the image below:

PHASE THREE

In this phase, we will create monitors for our EC2 instance for the metrics listed at the beginning of this tutorial.

A. HIGH CPU UTILISATION

On the monitors page in Datadog, on the top right, click on new monitor

Click on metrics and configure your monitor

The image below shows the configuration needed to monitor your EC2 cpu utilisation

Your monitor should look like this after creation:

To understand each options used in creating the monitor, click here

B. HIGH MEMORY UTILISATION

Your monitor should look like this after creation:

C. HIGH DISK UTILISATION

Conclusion

I hope you are able to follow through and also are able to create the Datadog monitors for your metrics. Do you have any question? Please send it my way. Kindly follow me on LinkedIn.

Create AWS ECR Repository using Cloudformation

Esther Ninyo — Fri, 12 Aug 2022 18:42:00 +0000

Cloudformation is an infrastructure as code (iac) tool that gives us the super powers to easily replicate infrastructures on AWS platform and also gives the users the ability to easily control and track changes to the infrastructure.

Cloudformation helps set up AWS resources using templates written in json or yaml format which reduces the time spent on managing the resources and gives us more time to focus on the application running in AWS. To read more on the super powers of cloudformation, please find the documentation here.

This article comprises on how to use cloudformation to spin up ecr repo on AWS. Should in case you're interested on how to use terraform to spin up ecr repo, please find my article on how to do that here.

Prerequisite:

Download and install AWS CLI so you can interact with AWS services from your command line interface
Configure your AWS credentials via your cli using AWS configure command. To do that, you need to create a user using IAM on AWS.

Project Structure

For this project, we need three files as seen and explained below:
ecr.yml: This is the cloudformation template used in creating the ecr repo resource on AWS.

ecr.json: This is the cloudformation parameters that allows us to dynamically input values whenever a stack is created or updated. Cloudformation parameters are written in json format.

create.sh: This is mainly a file that contains the command that creates the cloudformation stack instead of typing the command repeatedly.

Ecr_cloudformation

ecr.yml
ecr.json
create.sh

Project configuration

Step 1: Open up the ecr.json file, paste the following code in it and save. This is our parameter template.



[
   {
       "ParameterKey": "repoName",
       "ParameterValue": "ecr"
   }
]

Step 2: Open up the ecr.yml file, paste the following lines of code in it and save. This is our resource template written in yaml format and the indentation is very important.



AWSTemplateFormatVersion: 2010-09-09
Description: Elastic Container Registory Repository using Cloudformation

#------------------------
#   PARAMETERS
#------------------------
Parameters:
  repoName: 
    Description: Name for ecr repo 
    Type: String

#------------------------
#   RESOURCES
#------------------------ 
Resources:
  ecrRepo:
    Type: AWS::ECR::Repository
    Properties: 
      RepositoryName: !Sub ${repoName}-repo
      ImageScanningConfiguration: 
        ScanOnPush: true

line 1: This identifies the capabilities of the template and the only valid version value is 2010-09-09. The value must be a literal string and this section is optional.

line 2: This section enables you to add comment about the template and it is optional.

line 7 - 10: This is the parameter section and the parameter declared here is repoName with its default value declared in the ecr.json file.

line 15 - 21: This is the resource section that declares the AWS resources you want to include in the stack. The resource declared here is ecrRepo. Under the properties, the repositoryName made use of the parameter by substituting it with additional extension -repo.

Step 3: To spin up the resources, we will be using cloudformation create-stack command but we will be including it in the create.sh file we created so it can be easily accessed and used.
Open up the create.sh file and add the following command to it:



aws cloudformation create-stack --stack-name $1 --template-body file://$2 --parameter file://$3 --region=us-east-1

Before you can execute that command, you need to move into the project directory and give the create.sh file execute access by using the command below:



chmod +x create.sh

After the file has been given execute access, you can then safely create the cloudformation stack that will spin up the ecr repo using the command below:



./create.sh ecr-repo ecr.yml ecr.json

The response above shows that the cloudformation stack has been created successfully.

Note the following in the create.sh file:
$1 = ecr-repo
$2 = ecr.yml
$3 = ecr.json

Completely created cloudformation stack

Successfully created ecr repo

You have successfully created an elastic container registry repository using cloudformation and you can safely publish your container images to it.

Thank you for reading to the end. Kindly reach out to me in the comment section if you have any questions or on LinkedIn and Twitter on ways to improve or to say hi.

Till next time, cheers.

Provision AWS Elastic Container Registry Repository using Terraform

Esther Ninyo — Thu, 21 Jul 2022 13:48:00 +0000

Automation is very crucial in development which eliminates/reduces human error during manual provisioning. Automation enables teams to focus on work that adds value to the business which in turn increases revenue.
In this article, I will walk you through how to set up AWS elastic container registry repository using terraform.

Terms and Technologies
Terraform : Terraform is an open-source iac (infrastructure as code) tool that is used to provision, change and improve infrastructure on any environment. To read more on terraform, visit here

Terraform backend config: The terraform backend defines where terraform stores its state data files.

Terraform provider: In terraform, infrastructures are provisioned across public cloud providers such as amazon web services, google cloud, azure etc. The provider block is used to state the cloud provider you want to use and in this tutorial, we will be making use of amazon web service.

Prerequisite:

Download and install AWS CLI so you can interact with aws services from your command line interface
Configure your AWS credentials via your cli using aws configure command. To do that, you need to create a user using IAM on AWS.
Download and install terraform

Project Structure
Terraform-ecr

Backend.tf
Main.tf
Provider.tf
Variable.auto.tfvars
Variables.tf

Project Configuration
The first thing we will be doing is to create an s3 bucket that will hold the state files. Then, add the terraform configuration block that stores state data files for this project inside the s3 bucket created.

Open up your backend.tf file and add the following line of code

terraform {
 backend "s3" {
   bucket = "myawspracticebucket12"
   key = "~/.aws/config"
   region = "us-east-1"
 }
}

Line 1: This is the terraform block needed to create our backend config
Line 2: describes the backend block which contains the information of where the backend will stored
Line 3: The name of the bucket we created
Line 4: The path to your credential file. This is available when you do aws configure. You can also decide to use a different credential. All you need to do is to specify the path.
Line 5: Region of the s3 bucket
Before we move on, let’s initialize the backend using the terminal to see if we’ve not made any mistake by using this command:
Terraform init
This will be the output if everything is done correctly

If you do not have that, crosscheck the path to your credential file and also your bucket name if they are correct.

Step 2: In this step, i add the terraform provider which is “hashicorp/aws” with its version, i also added the aws region declared in the variable.tf Open up your provider.tf file and add the following lines of code

terraform {
 required_providers {
     aws = {
         source = "hashicorp/aws"
         version = "~>4.19.0"
     }
 }
}

provider "aws" {
 region = var.region
}

Step 3: In this step, I added the resource block that creates the elastic container registry (ecr) on aws. Open up your main.tf file and add the following lines of code

resource "aws_ecr_repository" "ecr_repo" {
 name                 = var.ecr
 image_tag_mutability = "IMMUTABLE"

 image_scanning_configuration {
   scan_on_push = true
 }
}

Line 1: This is the resource block that holds information used in creating the elastic container registry
Line 2: The name of the repository stored in the variable file
Line 3: This prevents image tags from being overwritten
Line 6: This scans images for vulnerabilities

Step 3: In this step, I will declare the variables used in the previous files such as the region and repository name. Open up your variables.tf file and add the following code

variable "region" {
 description = "AWS region"
 type = string
}

variable "ecr" {
 description = "Repository name"
 type = string
}

You will notice that we do not have any default value in the variables.tf file, the values will be stored in the variables.auto.tfvars file.
Open up your variables.auto.tfvars file and add the following code

region = "us-east-1"
ecr = "web"

Step 4: In this step, i will use terraform command to spin up our infrastructure
Terraform init

Terraform plan to see the resources we’re creating

Terraform apply to spin up the resource

Let’s head over to the console to see what we created

Repository created

You have successfully created an elastic container registry repository and you can safely publish your container images to it.

Thank you for reading to the end. Kindly reach out to me in the comment section if you have any question or on linkedin and twitter on ways to improve.

Till next time, cheers.