DEV Community: Ahmed Salem

Hey folks! I’m building my Medium profile where I share real-world AWS, GCP, and DevOps tutorials. I’m aiming to reach 100 followers — would love your support here ❤️: https://medium.com/@ahmedSalem2020

Ahmed Salem — Wed, 09 Apr 2025 05:13:34 +0000

Hey folks! I’m building my Medium profile where I share real-world AWS, GCP, and DevOps tutorials. I’m aiming to reach 100 followers. I would love your support here ❤️: https://medium.com/@ahmedSalem2020

Ahmed Salem — Mon, 07 Apr 2025 07:01:19 +0000

Automating EC2 Instance Resizing with AWS Systems Manager (SSM) Document

Ahmed Salem — Wed, 29 Jan 2025 06:13:37 +0000

Managing EC2 instance sizes manually can be time-consuming, especially when scaling resources across multiple instances. In this guide, we’ll deploy an AWS Systems Manager (SSM) document using AWS CloudFormation (CFN) to automate the resizing of EC2 instances in bulk. This setup allows users to resize instances across different Availability Zones with a single click.

Overview

Our goal is to create a CloudFormation stack that:

Deploys multiple EC2 instances in different Availability Zones
Creates an SSM document to automate the resizing of EC2 instances
Allows resizing based on Availability Zone, Tags, and Instance Type
Enables automation through AWS Systems Manager

Instead of manually stopping, resizing, and restarting instances, this solution streamlines the entire process into an automated workflow.

Architecture

The architecture consists of:

EC2 Instances - Launched with CloudFormation, spread across different Availability Zones
AWS SSM Document - Defines the automation process for resizing instances
IAM Role for SSM - Grants permissions to modify EC2 instance attributes

Step-by-Step Guide

1. Create a CloudFormation Template for EC2 Instances

We first create an EC2 CloudFormation (CFN) template (ec2.yaml) to launch instances in different Availability Zones.

ec2.yaml

AWSTemplateFormatVersion: 2010-09-09
#Description: EC2 with user data from CloudFormation.
Parameters:
  InstanceType:
    Description: WebServer EC2 instance type.
    Type: String
    Default: t2.micro
    AllowedValues:
      - t1.micro
      - t2.nano
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
      - m1.small
      - m1.medium
      - m1.large
      - m1.xlarge
      - m2.xlarge
      - m2.2xlarge
      - m2.4xlarge
      - m3.medium
      - m3.large
      - m3.xlarge
      - m3.2xlarge
      - m4.large
      - m4.xlarge
      - m4.2xlarge
      - m4.4xlarge
      - m4.10xlarge
      - c1.medium
      - c1.xlarge
      - c3.large
      - c3.xlarge
      - c3.2xlarge
      - c3.4xlarge
      - c3.8xlarge
      - c4.large
      - c4.xlarge
      - c4.2xlarge
      - c4.4xlarge
      - c4.8xlarge
      - g2.2xlarge
      - g2.8xlarge
      - r3.large
      - r3.xlarge
      - r3.2xlarge
      - r3.4xlarge
      - r3.8xlarge
      - i2.xlarge
      - i2.2xlarge
      - i2.4xlarge
      - i2.8xlarge
      - d2.xlarge
      - d2.2xlarge
      - d2.4xlarge
      - d2.8xlarge
      - hi1.4xlarge
      - hs1.8xlarge
      - cr1.8xlarge
      - cc2.8xlarge
      - cg1.4xlarge

  # Ec2Name:
  #   Description: Ec2 Resource name.
  #   Type: String

  # BucketName:
  #   Description: BucketName.
  #   Default: ahmedsalem-testbuckettt
  #   Type: String

  # ObjectPrefix:
  #   Description: idex.html prefix,for ex / 
  #   Type: String
  #   Default: /

  # KeyName:
  #   Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
  #   Type: 'AWS::EC2::KeyPair::KeyName'
  #   ConstraintDescription: must be the name of an existing EC2 KeyPair.

  # SubnetId:
  #  Description: Subnet ID which will run the web server instanc into.
  #  Type: 'AWS::EC2::Subnet::Id'

  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'

  SSHLocation:
    Description: The IP address range that can be used to SSH to the EC2 instance.
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})'
    ConstraintDescription: Must be a valid IP CIDR range of the form x.x.x.x/x

Resources:
  WebServerInstance1:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-ed2f8486
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_1"
       - Key: "ssm-tag"
         Value: "test-ssm"
      # SecurityGroupIds:
      #       - !GetAtt "WebServerSecurityGroup.GroupId"
      #IamInstanceProfile: !Ref IAMInstanceProfile

  WebServerInstance2:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-0556b778
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_2"
       - Key: "ssm-tag"
         Value: "test-ssm"
      # SecurityGroupIds:
      #       - !GetAtt "WebServerSecurityGroup.GroupId"
      #IamInstanceProfile: !Ref IAMInstanceProfile

  WebServerInstance3:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-34531b78
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_3"
      #  - Key: "ssm-tag"
      #    Value: "test-ssm"
      # SecurityGroupIds:
      #       - !GetAtt "WebServerSecurityGroup.GroupId"
      #IamInstanceProfile: !Ref IAMInstanceProfile

  WebServerInstance4:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-ed2f8486
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_4"

  WebServerInstance5:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-ed2f8486
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_5"

  WebServerInstance6:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-ed2f8486
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_6"

  WebServerInstance7:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref LatestAmiId
      NetworkInterfaces: 
      - AssociatePublicIpAddress: "true"
        DeviceIndex: "0"
        SubnetId: subnet-ed2f8486
        GroupSet:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      Tags:
       - Key: "Name"
         Value: "server_7"

  WebServerSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Enable HTTP access via port 80 and SSH Access port 22.
      # VpcId: vpc-0078f41e6b5568b6e
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: !Ref SSHLocation

  # IAMInstanceProfile:
  #  Type: AWS::IAM::InstanceProfile
  #  Properties:
  #    Roles:
  #      - !Ref IAMRole
  #    InstanceProfileName: ec2-access-s3

  # IAMRole:
  #  Type: AWS::IAM::Role
  #  Properties:
  #    RoleName: ec2-s3-access
  #    AssumeRolePolicyDocument:
  #      Version: '2012-10-17'
  #      Statement:
  #        - Effect: Allow
  #          Principal:
  #            Service: ec2.amazonaws.com
  #          Action: sts:AssumeRole
  #    Path: '/'
  #    Policies:
  #      - PolicyName: 'EC2Access'
  #        PolicyDocument:
  #          Version: '2012-10-17'
  #          Statement:
  #            - Effect: 'Allow'
  #              Action:
  #                - 's3:GetObject'
  #              Resource: !Sub 'arn:aws:s3:::${BucketName}/index.html'

What This Template Does:

• Launches multiple EC2 instances with different tags
• Ensures instances are deployed in multiple Availability Zones
• Assigns instances a security group for controlled access

2. Create a CloudFormation Template for the SSM Document

We now create another CloudFormation template (ssm.yaml) that defines the SSM document for resizing EC2 instances.

ssm.yaml

AWSTemplateFormatVersion: 2010-09-09

# Description: SSM Document for resizing list of EC2 instances

Resources:
  SSMDocumentRole:
   Type: AWS::IAM::Role
   Properties:
     AssumeRolePolicyDocument:
       Version: '2012-10-17'
       Statement:
         - Effect: Allow
           Principal:
             Service: ssm.amazonaws.com
           Action: sts:AssumeRole
     ManagedPolicyArns:
     - arn:aws:iam::aws:policy/AmazonEC2FullAccess
     Path: "/"

  document: 
    Type: AWS::SSM::Document
    Properties:
      DocumentType: Automation
      Content:
        schemaVersion: '0.3'
        description: 'Resize EC2 instances.'
        assumeRole: '{{ AutomationAssumeRole }}'
        parameters:
          AvailabilityZone:
            type: String
            description: "(Required) The impacted Availability."
            default: 'us-east-2a'

          TagName:
            type: String
            description: "(Required) The Tag Name."
            default: 'ssm-tag'  

          TagValue:
            type: String
            description: "(Required) The Tag Value."
            default: 'test-ssm' 

          InstanceType:
            type: String
            description: "(Required) The Desired Instance Type."
            default: 't2.small'  

          # If I need to specify another IAMRole
          AutomationAssumeRole:
            type: String
            description: "(Required) The Arn of the role that allows the automation."
            default: !GetAtt SSMDocumentRole.Arn

        mainSteps:
        - name: listInstances
          action: aws:executeAwsApi
          timeoutSeconds: '60'
          inputs:
            Service: ec2
            Api: DescribeInstances
            Filters:
              - Name: availability-zone
                Values: ["{{ AvailabilityZone }}"]

              - Name: instance-state-name
                Values: ["running"]

              - Name: tag:{{ TagName }}
                Values: ["{{ TagValue }}"]
          outputs:
          - Name: InstanceIds
            Selector: "$.Reservations..Instances..InstanceId"
            Type: StringList

        - name: stopInstances
          action: aws:changeInstanceState
          onFailure: Continue
          inputs:
            InstanceIds:
              - "{{ listInstances.InstanceIds }}"
            DesiredState: stopped

        - name: ResizeInstances
          action: "aws:executeScript"
          inputs: 
            Runtime: "python3.8"
            Handler: resizeInstance
            InputPayload: 
              InstanceIds:
                - "{{ listInstances.InstanceIds }}"
            Script: |-
              def resizeInstance(events, context):
                import boto3
                client = boto3.client('ec2')
                for instance_id in events['InstanceIds']:
                  #client.modify_instance_attribute(InstanceId=instanceId, InstanceType={'Value': 't2.small',},)
                  print(instance_id)
                  print(type(instance_id))
                  client.modify_instance_attribute(InstanceId=instance_id, Attribute='instanceType', Value='t2.small')
                return True
          onFailure: Continue

        - name: startInstances
          action: aws:changeInstanceState
          inputs: 
            InstanceIds:
                - "{{ listInstances.InstanceIds }}"
            DesiredState: running
          isEnd: true

What This Template Does:

• Creates an SSM document that:
• Identifies EC2 instances based on Availability Zone and Tags
• Stops the instances before resizing
• Modifies the instance type using boto3
• Restarts the instances after resizing
• Defines an IAM Role for SSM to interact with EC2

3. Deploy the CloudFormation Templates

Launch the EC2 Stack
- Deploy the ec2.yaml template to provision instances.
- Ensure the Tag Name and Tag Value are correctly set.
Launch the SSM Stack
- Deploy the ssm.yaml template to create the SSM document.
- Verify that the IAM role is correctly attached.

4. Execute the SSM Document to Resize Instances

Once both CloudFormation stacks are deployed:

Go to AWS Systems Manager → Shared Documents
Click on “Shared with me”
Select your custom SSM document and click “Execute document”
Enter the required parameters (Availability Zone, Tag Name, New Instance Type)
Click Execute

The SSM automation will now:

• Identify instances in the specified Availability Zone & Tag
• Stop instances before resizing
• Modify instance types dynamically
• Restart instances

You should see all EC2 instances updated successfully!

5. Clean Up Resources

To remove the deployed resources:

Delete the SSM CloudFormation Stack
Delete the EC2 CloudFormation Stack This ensures that no unintended costs are incurred.

Conclusion

By leveraging AWS Systems Manager (SSM) Documents, we can automate EC2 instance resizing across multiple environments with minimal effort. This solution eliminates manual resizing, ensuring faster and more efficient instance modifications.

Start implementing this in your AWS environment to save time and improve scalability!

Assigning a FQDN (Fully Qualified Domain Name) to an EC2 Instance Using Route 53

Ahmed Salem — Mon, 20 Jan 2025 11:16:22 +0000

In this guide, we’ll explore how to assign a Fully Qualified Domain Name (FQDN) to an EC2 instance running a web server using AWS Route 53. This process includes deploying CloudFormation templates, configuring Route 53, and ensuring the FQDN resolves to the EC2 instance’s public IP.

Overview

We aim to assign a domain name to a web server hosted on an EC2 instance. Instead of accessing the server using a public IP, we’ll use a friendly FQDN (e.g., www.cmcloudlab1589.info).

Key Benefits

User-Friendly Access
• The FQDN makes it easier for users to access the web server instead of remembering an IP address.
Scalability
• The setup can easily accommodate additional domains and services under the same hosted zone.
AWS Automation
• Using CloudFormation ensures repeatability and simplifies deployment.

Key Objectives:

Deploy an EC2 instance running a web application.
Create a Route 53 RecordSet in the existing public hosted zone.
Ensure the FQDN resolves to the EC2 instance’s public IP.

Architecture
The architecture consists of:

An EC2 instance running a web server.
A Route 53 public-hosted zone with a Type A RecordSet pointing to the instance’s public IP.

Diagram:

Step-by-Step Guide

1. Create a CloudFormation Template for the EC2 Instance

The following ec2.yaml file sets up an EC2 instance, including:

• A web server with httpd installed and configured.

• Security groups to allow HTTP (port 80) and SSH (port 22) traffic.

• An IAM role with S3 access to retrieve the index.html file for the web server.

AWSTemplateFormatVersion: 2010-09-09
#Description: [CET-004] EC2 with user data from CloudFormation.
Parameters:
  InstanceType:
    Description: WebServer EC2 instance type.
    Type: String
    Default: t2.micro
    AllowedValues:
      - t1.micro
      - t2.nano
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
      - m1.small
      - m1.medium
      - m1.large
      - m1.xlarge
      - m2.xlarge
      - m2.2xlarge
      - m2.4xlarge
      - m3.medium
      - m3.large
      - m3.xlarge
      - m3.2xlarge
      - m4.large
      - m4.xlarge
      - m4.2xlarge
      - m4.4xlarge
      - m4.10xlarge
      - c1.medium
      - c1.xlarge
      - c3.large
      - c3.xlarge
      - c3.2xlarge
      - c3.4xlarge
      - c3.8xlarge
      - c4.large
      - c4.xlarge
      - c4.2xlarge
      - c4.4xlarge
      - c4.8xlarge
      - g2.2xlarge
      - g2.8xlarge
      - r3.large
      - r3.xlarge
      - r3.2xlarge
      - r3.4xlarge
      - r3.8xlarge
      - i2.xlarge
      - i2.2xlarge
      - i2.4xlarge
      - i2.8xlarge
      - d2.xlarge
      - d2.2xlarge
      - d2.4xlarge
      - d2.8xlarge
      - hi1.4xlarge
      - hs1.8xlarge
      - cr1.8xlarge
      - cc2.8xlarge
      - cg1.4xlarge
  Ec2Name:
    Description: Ec2 Resource name.
    Type: String
  BucketName:
    Description: BucketName.
    Default: ahmedsalem-testbuckettt
    Type: String
  ObjectPrefix:
    Description: idex.html prefix,for ex / 
    Type: String
    Default: /
  # KeyName:
  #   Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
  #   Type: 'AWS::EC2::KeyPair::KeyName'
  #   ConstraintDescription: must be the name of an existing EC2 KeyPair.
  # SubnetId:
  #  Description: Subnet ID which will run the web server instanc into.
  #  Type: 'AWS::EC2::Subnet::Id'
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
  SSHLocation:
    Description: The IP address range that can be used to SSH to the EC2 instance.
    Type: String
    MinLength: '9'
    MaxLength: '18'
    Default: 0.0.0.0/0
    AllowedPattern: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})'
    ConstraintDescription: Must be a valid IP CIDR range of the form x.x.x.x/x

Resources:
  WebServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      Tags:
       - Key: "Name"
         Value: !Ref Ec2Name
       - Key: "value"
         Value: "to be deleted"
      # KeyName: !Ref KeyName
      # NetworkInterfaces: 
      #   - AssociatePublicIpAddress: "true"
      #     DeviceIndex: "0"
      # SubnetId: !Ref SubnetId
      ImageId: !Ref LatestAmiId
      SecurityGroupIds:
            - !GetAtt "WebServerSecurityGroup.GroupId"
      IamInstanceProfile: !Ref IAMInstanceProfile
      UserData: 
        Fn::Base64:
          !Sub |
            #!/bin/bash -xe
            yum update -y aws-cfn-bootstrap
            yum install -y httpd
            systemctl start httpd
            systemctl enable httpd
            aws s3 cp s3://${BucketName}${ObjectPrefix}index.html /var/www/html/

  WebServerSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Enable HTTP access via port 80 and SSH Access port 22.
      VpcId: vpc-0078f41e6b5568b6e
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: !Ref SSHLocation

  IAMInstanceProfile:
   Type: AWS::IAM::InstanceProfile
   Properties:
     Roles:
       - !Ref IAMRole
     InstanceProfileName: ec2-access-s3

  IAMRole:
   Type: AWS::IAM::Role
   Properties:
     RoleName: ec2-s3-access
     AssumeRolePolicyDocument:
       Version: '2012-10-17'
       Statement:
         - Effect: Allow
           Principal:
             Service: ec2.amazonaws.com
           Action: sts:AssumeRole
     Path: '/'
     Policies:
       - PolicyName: 'EC2Access'
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Effect: 'Allow'
               Action:
                 - 's3:GetObject'
               Resource: !Sub 'arn:aws:s3:::${BucketName}/index.html'

Outputs:
  WebsiteURL:
    Description: URL for newly apache webserver created.
    Value: !Join 
      - ''
      - - 'http://'
        - !GetAtt 
          - WebServerInstance
          - PublicDnsName

  PublicIp:
    Description: The publicIP of the webserver
    Value: !GetAtt WebServerInstance.PublicIp
    Export:
      Name: !Sub "${AWS::StackName}-PublicIp"

2. Create the Route 53 CloudFormation Template

The following r53.yaml file creates a Route 53 RecordSet to associate the FQDN with the EC2 instance’s public IP:

Parameters:
  WebserverStackParameter:
    Type: String
    Default: Webserver

  HostedZoneId:
    #Type: AWS::Route53::HostedZone::Id
    Type: String
    Description: HostedZone ID
    ConstraintDescription: must be a valid HostedZone ID
    Default: Z0775680V6B0O09IGAKU

  WebserverFQDN:
    Type: String
    # ConstraintDescription: must be a valid HostedZone ID

Resources:
  PublicDNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref HostedZoneId
      Name: !Sub "${WebserverFQDN}.cmcloudlab1832.info"
      Type: A
      TTL: 900
      ResourceRecords:
        -
          Fn::ImportValue: 
             !Sub "${WebserverStackParameter}-PublicIp"

Outputs:
  Hostname:
    Description: The Hostname attached to our Webserver
    Value: !Ref PublicDNSRecord

3. Deploy the Infrastructure

Launch the EC2 Stack
• Deploy the ec2.yaml CloudFormation template to set up the EC2 instance.
• Note the exported PublicIp value for use in the Route 53 stack.
Launch the Route 53 Stack
• Deploy the r53.yaml CloudFormation template to create the RecordSet in the Route 53 hosted zone.

4. Test the Configuration

After deploying both stacks, you can access your web server using the FQDN assigned in Route 53.

Example FQDN:
www.cmcloudlab1589.info

Conclusion
Assigning a Fully Qualified Domain Name (FQDN) to an EC2 instance using Route 53 is a straightforward yet powerful solution for enhancing accessibility and scalability of your applications. By leveraging AWS CloudFormation, we automated the deployment process, making it efficient, repeatable, and easy to manage.

Leverage On-Premises Infrastructure in Amazon EKS Clusters with Amazon EKS Hybrid Nodes

Ahmed Salem — Fri, 13 Dec 2024 12:13:29 +0000

Amazon Web Services (AWS) continues to bridge the gap between on-premises infrastructure and the cloud. The new Amazon EKS Hybrid Nodes feature enables organizations to extend their Amazon Elastic Kubernetes Service (EKS) clusters to include on-premises resources. This hybrid approach provides enhanced flexibility and scalability, empowering businesses to leverage their existing infrastructure while enjoying the benefits of Kubernetes orchestration in the cloud.

In this article, we’ll explore what Amazon EKS Hybrid Nodes are, how they work, and how they can help you integrate on-premises resources seamlessly into your EKS clusters.

Overview: What Are Amazon EKS Hybrid Nodes?

Amazon EKS Hybrid Nodes allow organizations to use their on-premises servers, virtual machines (VMs), or edge devices as worker nodes in an EKS cluster. This feature makes it easier to manage workloads that require low latency, data residency compliance, or closer proximity to on-premises systems.

With Amazon EKS Hybrid Nodes, you can deploy containerized applications on your existing infrastructure while managing everything from a single EKS control plane.

Key Benefits

1. Extend Kubernetes Clusters On-Premises

EKS Hybrid Nodes enable you to extend your Kubernetes clusters beyond AWS, incorporating on-premises resources into a unified infrastructure.

2. Seamless Integration

EKS Hybrid Nodes operate independently and do not require services like AWS Outposts or Amazon ECS Anywhere. Instead, they seamlessly integrate on-premises infrastructure into EKS clusters, providing flexibility without additional dependencies.

3. Centralized Management

Manage your on-premises and cloud-based workloads using the same EKS control plane, simplifying operations and providing a consistent Kubernetes experience.

4. Support for Edge Use Cases

Deploy workloads on edge devices or in locations with limited cloud connectivity while maintaining centralized control through EKS.

How Amazon EKS Hybrid Nodes Work

The following diagram illustrates the hybrid network connectivity, node setup, and integration between on-premises environments and AWS:

Amazon EKS Hybrid Nodes enable seamless integration between your on-premises infrastructure and the Amazon EKS control plane. Here’s how it works:

• Hybrid network connectivity using:

• AWS Site-to-Site VPN
• AWS Direct Connect
• Another VPN solution
• A VPC with routing configurations for remote nodes and pods, using a Virtual Private Gateway (VGW) or Transit Gateway (TGW).

• Physical or virtual machines as infrastructure.

• Compatible operating systems, such as:

• Amazon Linux 2023
• Ubuntu (20.04, 22.04, 24.04)
• Red Hat Enterprise Linux (RHEL 8 or 9)
• IAM roles:

• EKS Cluster IAM Role
• EKS Hybrid Nodes IAM Role
• AWS Systems Manager or IAM Roles Anywhere for node authentication.

Setting Up Amazon EKS Hybrid Nodes

1. Prerequisites

Ensure the prerequisites we mentioned above are in place before your on-premises infrastructure can join your EKS cluster as hybrid nodes.

2. Create an EKS Cluster and Enable Hybrid Nodes

1. Navigate to the Amazon EKS Console

Open the Amazon EKS Console and begin creating a new cluster.

2. Specify Networking for Hybrid Nodes

On the Step 2: Specify Networking screen, enable the option Configure remote networks to enable hybrid nodes and specify the Classless Inter-Domain Routing (CIDR) blocks for your on-premises environments.

3. CIDR Configuration Guidelines

• The CIDRs for remote nodes and remote pods must:

Use RFC-1918 IPv4 addresses (private IP ranges).
Not overlap with the VPC CIDR or the EKS cluster Kubernetes service CIDR.

• The remote node CIDR and remote pod CIDR must not overlap.

• Specifying a pod CIDR block is mandatory if:

You run webhooks on your hybrid nodes.
Your CNI plugin doesn’t use NAT for pod addresses as traffic leaves your nodes.

3. Connect Your Hybrid Nodes to the EKS Cluster

With the EKS cluster configured, the next step is to connect your on-premises infrastructure to the cluster as hybrid nodes. Use the Amazon EKS Hybrid Nodes CLI (nodeadm) to automate the installation, configuration, and registration of your hybrid nodes.

Install the required components:

$ nodeadm install 1.31 --credential-provider <ssm, iam-ra>

2. Create a nodeConfig.yaml file:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
metadata:
  name: hybrid-node
spec:
  cluster:
    name: my-cluster
    region: us-east-1
  hybrid:
    ssm:
      activationCode: <activation-code>
      activationId: <activation-id>

3. Register the hybrid node with the cluster:

$ nodeadm init -c file://nodeConfig.yaml

4. Verify node readiness:

Use the EKS console or the following command to confirm that the node has joined:

kubectl get nodes

Ensure the nodes are in the Ready state after installing the required CNI plugin.
Note: Before the hybrid nodes are marked as Ready, you need to install a compatible CNI plugin. For more information, refer to Install CNI for EKS Hybrid Nodes.

4. Deploy Workloads

Use Kubernetes-native tools such as kubectl and Helm charts to deploy workloads to both cloud-based and hybrid nodes.

5. Monitor and Optimize

Monitor your cluster’s performance using Amazon CloudWatch and other observability tools. Ensure that hybrid nodes are scaling effectively and meeting workload demands.

Conclusion

Amazon EKS Hybrid Nodes open new possibilities for organizations looking to unify their cloud and on-premises resources under a single Kubernetes control plane. By enabling seamless integration of on-premises infrastructure with EKS, this feature offers unparalleled flexibility, scalability, and consistency for modern application deployment.

Whether you’re running edge workloads, ensuring data compliance, or modernizing legacy systems, EKS Hybrid Nodes provide a powerful solution to meet your hybrid cloud needs. Explore this feature today and unlock the full potential of Kubernetes on your terms.

References

AWS EKS Auto Mode: Automating Kubernetes Cluster Management

Ahmed Salem — Thu, 12 Dec 2024 10:58:27 +0000

AWS has taken a significant step in simplifying Kubernetes cluster management with the introduction of EKS Auto Mode. This new feature is designed to reduce the operational complexity of managing Kubernetes clusters while ensuring cost efficiency, scalability, and security. This article will provide a detailed overview of EKS Auto Mode, explore its benefits, and guide you through its implementation.

Overview: What Is EKS Auto Mode?

Amazon Elastic Kubernetes Service (EKS) Auto Mode automates the management of Kubernetes cluster infrastructure, including compute, storage, and networking. By handling tasks such as scaling, upgrades, and security compliance, EKS Auto Mode enables teams to focus on building and deploying applications rather than managing infrastructure.

Key Features

1. Simplified Cluster Management

EKS Auto Mode provides production-ready clusters with minimal operational overhead, enabling organizations to deploy workloads without deep Kubernetes expertise.

2. Dynamic Scaling

The service dynamically scales the number of nodes in your cluster based on workload demand, ensuring efficient resource utilization.

3. Cost Optimization

EKS Auto Mode minimizes costs by automatically terminating unused nodes and consolidating workloads.

4. Enhanced Security

The service uses immutable Amazon Machine Images (AMIs) with enforced security measures, such as SELinux controls and read-only root file systems. Nodes have a maximum lifetime of 21 days, after which they are replaced to maintain security compliance.

5. Automated Upgrades

EKS Auto Mode automates updates to Kubernetes clusters, nodes, and associated components, ensuring they are always running the latest versions.

6. Integration with AWS Services

The feature integrates seamlessly with AWS services like Amazon EC2, Elastic Load Balancing, and Amazon EBS, leveraging AWS’s cloud-native capabilities.

Getting Started with EKS Auto Mode

1. Creating an EKS Auto Mode Cluster

To create a new EKS Auto Mode cluster:
1. AWS Management Console: Navigate to the EKS Console and select the Auto Mode option when creating a cluster.
2. CLI or eksctl: Use commands or YAML configurations to define the cluster with Auto Mode enabled.
3. Key Settings:
• Choose regions and Availability Zones.
• Define default NodePools, which are automatically created and managed.

2. Migrating Existing Clusters to Auto Mode

AWS allows you to transition existing EKS clusters to Auto Mode.
• Steps:
• Evaluate workloads for compatibility with EKS Auto Mode.
• Modify the cluster’s configuration to enable Auto Mode.
• Considerations:
• Existing configurations may need adjustments, such as migrating
manually managed NodeGroups to Auto-managed NodePools.

3. Deploying Workloads on Auto Mode Clusters

EKS Auto Mode is fully compatible with Kubernetes-native tools and workflows. You can deploy workloads using:
• kubectl: Standard Kubernetes CLI for managing workloads.
• Helm Charts: Package manager for Kubernetes applications.
• GitOps Tools: Integrate with tools like ArgoCD for declarative application management.

4. Configuring Auto Mode Clusters
• NodePools:
• Default NodePools are created automatically, but you can define custom NodePools to meet specific requirements.
• NodePools support a variety of instance types, including GPU-enabled instances for AI/ML workloads.
• Scaling Policies:
• Configure scaling policies to handle workload spikes.
• Use Pod Disruption Budgets (PDBs) to ensure availability during scaling events.

5. Understanding Instance Management in Auto Mode

EKS Auto Mode manages instances as immutable appliances, ensuring:
• Security: No SSH or direct access to nodes is allowed.
• Automated Replacement: Instances are replaced regularly to maintain compliance with security standards.
• Compatibility: Supports both EC2 On-Demand Instances and Spot Instances for cost optimization.

Key Benefits
1. Reduced Operational Overhead: EKS Auto Mode automates infrastructure management tasks, freeing teams to focus on application development.
2. Cost Efficiency: Dynamic scaling and Spot Instance support optimize costs without compromising performance.
3. Scalability: The service scales resources up or down based on workload demands, ensuring high availability.
4. Enhanced Security: Regular instance replacement and immutable configurations ensure compliance with the latest security standards.
5. Seamless Integration: Works with Kubernetes-native tools and AWS services for a consistent and familiar user experience.

Common Use Cases
1. Web Applications: Automatically scale resources to handle traffic spikes.
2. AI/ML Workloads: Use GPU-enabled NodePools to run training and inference jobs efficiently.
3. Microservices: Deploy containerized workloads with Kubernetes-native tools.
4. CI/CD Pipelines: Integrate with GitOps tools to streamline deployment workflows.

Considerations
• Compatibility: Not all workloads are immediately compatible with Auto Mode. Evaluate your existing configurations before migrating.
• Custom Configurations: While default settings are sufficient for many use cases, some applications may require custom NodePools or scaling policies.
• Access Restrictions: Auto Mode enforces strict security policies, limiting direct access to managed nodes.

Conclusion

EKS Auto Mode is a significant advancement for organizations looking to simplify Kubernetes cluster management. By automating tasks like scaling, upgrades, and security compliance, it allows teams to focus on delivering value through applications rather than managing infrastructure. Whether you’re starting fresh or migrating existing clusters, EKS Auto Mode offers a robust, scalable, and cost-effective solution for modern cloud-native applications.

Additionally, with the release of Terraform AWS Provider v5.79.0, managing EKS Auto Mode clusters has become even more accessible. This version introduces support for EKS Auto Mode, enabling Infrastructure as Code (IaC) configurations for compute, storage, and networking components. The integration further streamlines cluster deployments and simplifies automation workflows, making it easier for teams to adopt EKS Auto Mode within their existing Terraform-based setups.

Start exploring EKS Auto Mode today and unlock new levels of efficiency, scalability, and simplicity in Kubernetes management!

References

Subnet Settings in AWS: A Subtle Configuration That Can Cause Big Headaches

Ahmed Salem — Fri, 06 Dec 2024 05:54:59 +0000

I’ve spent countless hours configuring VPCs, subnets, and routing tables, but every now and then, something as simple as a checkbox reminds me how much there is to learn in AWS networking. Let me share a recent discovery that saved me hours of troubleshooting—and might save you even more.

The Problem: Internet Access Issues in a New Subnet

I was manually creating a new subnet in AWS. Everything seemed perfectly configured:
• Internet Gateway (IGW).
• Route Table pointing to the IGW.
• Security Groups allowing outbound traffic.

Yet when I launched an EC2 instance in the subnet, it had no internet access. At first, I assumed I had missed something minor, but after reviewing my setup repeatedly, I was stumped.

The Missing Link: Auto-Assign Public IPv4 Address

After hours of digging, I found the issue buried in the subnet settings. By default, when you manually create a subnet, the Auto-assign public IPv4 address option is disabled. This means that even though the route table points to an IGW and the security group allows traffic, instances launched in the subnet won’t be assigned a public IP address—making internet connectivity impossible.

Why This Happens

Public IP assignment is managed per subnet, not at the instance level or in the Launch Template. Even if your Launch Template includes settings for a public IP, it won’t override the subnet’s configuration. Without enabling this option, your instances are essentially isolated from the internet.

How to Fix It
Enable Public IP Assignment in Subnet Settings

Go to the VPC Console > Subnets.
Select the subnet in question and click Edit Subnet Settings.
Under Auto-assign IP settings, check the box for Enable auto-assign public IPv4 address.
Click Save.

Alternative: Assign Elastic IPs Manually

If you prefer not to enable auto-assignment, you can manually assign an Elastic IP to your instance. This approach gives you more control but adds extra steps during setup.

Lessons Learned

This experience reinforced an important AWS networking concept:

Even with the correct routing table and security group settings, an instance cannot access the internet without a public IP.
When creating subnets manually, always remember:
• If the instances in the subnet require internet access, enable the Auto-assign public IPv4 address option.
• Alternatively, assign Elastic IPs or use a NAT Gateway for outbound internet access.

Deploying Multiple PHP Applications Using AWS Elastic Beanstalk with a Standalone ALB

Ahmed Salem — Sat, 30 Nov 2024 19:08:49 +0000

In this guide, we’ll deploy multiple PHP applications using AWS Elastic Beanstalk (EB) environments, and configure a single standalone Application Load Balancer (ALB) for all environments. Based on the actual implementation, this article clarifies how to manage multiple Elastic Beanstalk environments with dedicated target groups under one centralized ALB.

Overview

We’ll set up multiple PHP applications as separate EB environments. Instead of configuring a load balancer for each environment, we’ll use one ALB with dedicated target groups for each environment. This approach is cost-efficient, simplifies management, and ensures centralized control over routing and scaling.

Architecture

Key Components:

• Elastic Beanstalk Environments: Each PHP application runs in its environment.
• Standalone ALB: A single ALB handles all incoming traffic and routes it to the appropriate target group.
• Target Groups: Each Elastic Beanstalk environment has its target group for routing.

Architecture Workflow:

ALB receives traffic for all applications.
Listener rules on the ALB route traffic to the correct target group based on host headers or path patterns.
Target groups forward traffic to the registered instances of the respective Elastic Beanstalk environments.

Step-by-Step Guide

Step 1: Set Up Elastic Beanstalk Environments
Create Separate Environments for PHP Applications:

Open the Elastic Beanstalk Console.
Click Create Application and Configure: • Application Name: PHP-App-1. • Platform: Select PHP. • Environment: Choose Web Server Environment.
Upload your .zip package containing the PHP application (e.g., index.php, composer.json).
Deploy the application.
Repeat these steps for additional applications (e.g., PHP-App-2, PHP-App-3).

Step 2: Create a Standalone ALB

Create the ALB:

Go to the EC2 Console > Load Balancers.
Click Create Load Balancer and select Application Load Balancer.
Configure: • Name: standalone-alb. • Scheme: Internet-facing. • Listeners: Add an HTTPS listener (port 443). • Availability Zones: Choose the same zones as your Elastic Beanstalk environments.
Click Create.

Register ALB with Elastic Beanstalk:

Navigate to each Elastic Beanstalk environment.
Under Configuration, link the environment to the newly created ALB.
Ensure health checks are consistent with the ALB configuration.

Step 3: Configure Target Groups for Each Environment

Create Target Groups:

Go to EC2 Console > Target Groups.
Click Create Target Group for each Elastic Beanstalk environment. • Name: Example: php-app-1-tg. • Target Type: Instance. • Protocol: HTTP. • Port: 80. • Health Check Path: / (or a custom endpoint defined in your application).
Add instances of the respective Elastic Beanstalk environment to the target group.
Navigate to the Targets tab in each target group and confirm the registered instances are healthy.

Step 4: Add Listener Rules to the ALB

Go to the ALB Console > Listeners > HTTP:80 > Edit Rules.
Add a rule for each target group: • Condition: Use Host Header to match the Elastic Beanstalk environment domain (e.g., php-app-1.elasticbeanstalk.com). • Action: Forward traffic to the corresponding target group (e.g., php-app-1-tg).
Repeat this process for all environments.

Testing the Setup
• Simulate traffic to verify that the ALB forwards requests correctly to the appropriate target groups based on listener rules.
• Check the health of each target group to ensure all instances are healthy and receiving traffic as expected.
• Use tools like curl or Postman to send requests directly to the ALB DNS endpoint. Confirm that the traffic is routed to the correct Elastic Beanstalk environment and returns the expected responses.

Key Benefits:

1. Cost Efficiency: Reduces infrastructure costs by using one ALB for all environments.
2. Simplified Management: Centralizes traffic routing and listener rule configuration in one place.
3. Scalability: Supports independent scaling of target groups for each environment.
4. Enhanced Traffic Control: Provides granular routing with ALB listener rules.
5. Centralized Health Monitoring: Consolidates health checks for all environments.

Conclusion

By using a single ALB with target groups for multiple Elastic Beanstalk environments, you achieve a cost-effective, scalable, and centralized solution for hosting PHP applications.

Building and Integrating Lambda Layers with Serverless API

Ahmed Salem — Fri, 12 Jul 2024 12:05:44 +0000

In this tutorial, we’ll explore how to use AWS Lambda Layers to organize and reuse code across multiple AWS Lambda functions. Specifically, we’ll move DynamoDB queries to a separate Lambda Layer and integrate it with a Serverless API. This approach enhances maintainability and promotes code reuse, making your serverless applications more scalable and efficient.

Overview

Our solution involves creating a Lambda Layer containing DynamoDB operations (list, add, and get countries) and integrating this layer with our existing Serverless API. We will define the required resources using the Serverless Framework and deploy them to AWS.

Architecture

The architecture includes:

A DynamoDB table to store country information.
A Lambda Layer to encapsulate DynamoDB operations.
Multiple Lambda functions to handle API requests (add, list, get countries).
API Gateway to expose these Lambda functions as RESTful endpoints.

Directory Structure

Here is the directory structure for our project:

Step-by-Step Guide
1. Create a Separate Folder for Lambda Layers
Create a folder named Dynamo_layer and add the following:

A folder named modules-layer.
A folder named dynamo-sdk-layer.
A serverless.yml file.

2. Add Layers Resources in serverless.yml
In the Dynamo_layer directory, add the following content to serverless.yml:

# Service name has to be unique for your account.
service: my-dependencies

# framework version range supported by this service.
frameworkVersion: ">=1.1.0 <8.0.0"

# configuration of the cloud provider. As we are using AWS so we defined AWS corresponding configuration.
provider:
  name: aws
  lambdaHashingVersion: 20201221
  runtime: nodejs14.x
  #stage: dev
  region: us-east-2
  # Create an ENV variable to be able to use it in my JS code. *** Check line 4 in get-country-by-name JS file ***
  environment:
    tableName: ${self:custom.tableName}
  # To Give a permission to each lambda function to access DynamoDB table
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:*
        # - dynamodb:Query
        # - dynamodb:Scan
        # - dynamodb:GetItem
        # - dynamodb:PutItem
      Resource: "*"

layers:
  ModulesLayer:
    path: ./modules-layer
    description: "my dependencies"

  DynamoSdkLayer:
    path: ./dynamo-sdk-layer
    description: "my dynamo dependencies"

custom:
  tableName: countries

3. Deploy Lambda Layers

Deploy the Dynamo_layer to create a CloudFormation stack for Lambda Layers by running the following command:

sls deploy

4. Navigate to the my-service folder, which contains the serverless.yml file for your API Gateway endpoints and Lambda functions.
Ensure the configuration includes:

A DynamoDB table.
Lambda functions for countries_loadData, add-country, list-countries, and get-country-by-name.

serverless.yml

# Service name has to be unique for your account.
service: my-service

# framework version range supported by this service.
frameworkVersion: ">=1.1.0 <8.0.0"

# configuration of the cloud provider. As we are using AWS so we defined AWS corresponding configuration.
provider:
  name: aws
  lambdaHashingVersion: 20201221
  runtime: nodejs14.x
  #stage: dev
  region: us-east-2
  # Create an ENV variable to be able to use it in my JS code. *** Check line 4 in get-country-by-name JS file ***
  environment:
    tableName: ${self:custom.tableName}
    NODE_PATH: "./:/opt/node_modules"
  # To Give a permission to each lambda function to access DynamoDB table
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:*
        # - dynamodb:Query
        # - dynamodb:Scan
        # - dynamodb:GetItem
        # - dynamodb:PutItem
      Resource: "*"

# package:
#   exclude:
#     - Dynamo-SDK/**

custom:
  tableName: countriees         

# As shown below, when the HTTP POST or GET request is made, the handler should be invoked.
functions:

#(1) Lambda function to initially fill DynamoDB
  FillDynamoDB:
    handler: lambdas/countries_loadData.fill
    description: fill DynamoDB table with set of countries.
    events:
      - http: 
          path: fill-dynamoDB
          method: POST
          cors: true
    layers:
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:ModulesLayer:3
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:DynamoSdkLayer:3

#(2) Lambda function to list all the countries 
  GetAllCountries:
    handler: lambdas/list-countries.list
    description: get all the countries information.
    events:
      - http: 
          path: list-countries
          method: GET
          cors: true
    layers:
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:ModulesLayer:3
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:DynamoSdkLayer:3

#(3) Lambda function to get a country by name
  GetCountryByName:
    handler: lambdas/get-country-by-name.get
    description: get country By Name.
    events:
      - http: 
          path: get-country/{NAME}
          method: GET
          cors: true
    layers:
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:ModulesLayer:3
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:DynamoSdkLayer:3

#(4) Lambda function to add a new country 
  AddNewCountry:
    handler: lambdas/add-country.add
    description: add a new country.
    events:
      - http: 
          path: add-country
          method: POST
          cors: true
    layers:
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:ModulesLayer:3
      - arn:aws:lambda:us-east-2:<ACCOUNT-ID>:layer:DynamoSdkLayer:3



#Resources are AWS infrastructure components which your Functions use. 
#The Serverless Framework deploys an AWS components your Functions depend upon.
resources:
  Resources:
    myDynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      #DeletionPolicy: Retain
      Properties:       
        TableName: ${self:custom.tableName}

        AttributeDefinitions:
          -
            AttributeName: "NAME"
            AttributeType: "S" 

        KeySchema:
          -
            AttributeName: "NAME"
            KeyType: "HASH"

        BillingMode: PAY_PER_REQUEST

    # Custom resource to invoke lambda function to fill the countries DynamoDB table
    TriggerFillDynamoDBFunction:
     Type: AWS::CloudFormation::CustomResource
     Properties:
       ServiceToken: !GetAtt 'FillDynamoDBLambdaFunction.Arn'

5. Deploy Serverless API

Deploy the serverless configuration by running:

sls deploy

6. Access API Endpoints

You can access the endpoints using the following URLs:

List countries: GET

https://nymaeazapc.execute-api.us-east-2.amazonaws.com/dev/list-countries

Get country by name: GET

https://nymaeazapc.execute-api.us-east-2.amazonaws.com/dev/get-country/{NAME}

Add a country: POST

https://nymaeazapc.execute-api.us-east-2.amazonaws.com/dev/add-country

7. Remove All Functions, Events, and Resources

To clean up, remove all AWS resources created by running:

sls remove

Conclusion

By following these steps, you have successfully utilized AWS Lambda Layers to modularize and reuse DynamoDB queries across multiple Lambda functions. This approach demonstrates the power of Lambda Layers in promoting code reuse and maintainability, making your serverless applications more efficient and scalable.

Building Scalable Serverless Applications with AWS SQS and Lambda using SAM

Ahmed Salem — Sat, 17 Feb 2024 21:59:41 +0000

In this tutorial, we will explore how to leverage the AWS Serverless Application Model (SAM) to build a scalable serverless application that utilizes Amazon Simple Queue Service (SQS) and AWS Lambda. Our goal is to create a system where a scheduled Lambda function sends random messages to an SQS queue, triggering another Lambda function to process and log these messages.

Overview
Our solution involves creating two Lambda functions - one to produce random messages and push them into an SQS queue, and another to consume these messages from the queue and log them to CloudWatch. We'll use AWS SAM to define and deploy our serverless application.

Architecture

Step-by-Step Guide
Let's break down the implementation into the following steps:

1. Setting Up AWS SAM CLI
Ensure the AWS SAM CLI is installed on your local machine. Follow the installation instructions provided here for Windows or refer to the official documentation for other operating systems.

2. Creating an AWS SAM Project using PowerShell Window

1) Use SAM init command to generate a sample module. This will
give us a preformatted module that is ready to be deployed
after we change a few things.

 # sam init

          sam-app/
            ├── README.md
            ├── events/
            │   └── event.json
            ├── hello_world/
            │   ├── __init__.py
            │   ├── app.py            #Contains your AWS Lambda handler logic.
            │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
            ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
            └── tests/
                  └── unit/
                  ├── __init__.py
                  └── test_handler.py

2) the project will be created under C:\Windows\System32. Copy the project into your target directory and navigate to it in
PowerShell using the below command:

# cd "C:\Users\ahmedsalem"

3. Defining SQS Resource
Create SQS resource in template.yaml file

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  ahmedsalem

  Sample SAM Template for ahmedsalem

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  MyQueue: 
    Type: AWS::SQS::Queue
    Properties: 
      QueueName: "ahmedsalem-sqs"
      VisibilityTimeout: 120

  SendDataToSQS:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: Sender/
      Handler: sender.send_data
      Runtime: nodejs12.x
      Timeout: 60

  LambdaScheduledRule: 
      Type: AWS::Events::Rule
      Properties: 
        Description: "ScheduledRule"
        ScheduleExpression: cron(*/2 * * * ? *)
        State: "ENABLED"
        Targets: 
          - 
            Arn: !GetAtt 'SendDataToSQS.Arn'
            Id: "TargetFunctionV1"

  PermissionForEventsToInvokeLambda:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !Ref SendDataToSQS
        Action: "lambda:InvokeFunction"
        Principal: "events.amazonaws.com"
        SourceArn: !GetAtt LambdaScheduledRule.Arn

  LambdaRoleForSQS:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
              - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        - "arn:aws:iam::aws:policy/AmazonSQSFullAccess"
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
      # Policies:
      #   - PolicyName: SQSpermissions
      #     PolicyDocument:
      #       Version: 2012-10-17
      #       Statement:
      #         - Effect: Allow
      #           Action:
      #           - "sqs:*"
      #           Resource: '*'

  SampleSQSPolicy: 
    Type: AWS::SQS::QueuePolicy
    Properties: 
      Queues: 
        - !Ref MyQueue
      PolicyDocument: 
        Statement: 
          - 
            Action: 
            - "SQS:*"
            # - "SQS:SendMessage" 
            # - "SQS:ReceiveMessage"
            Effect: "Allow"
            Resource: "*"
            Principal:  
              Service:
              - lambda.amazonaws.com     

  LambdaConsumer:
    Type: AWS::Lambda::Function
    Properties:
        #CodeUri: Receiver/
        Handler: receiver.receive_data
        Runtime: nodejs12.x
        Timeout: 60
        Role: !GetAtt LambdaRoleForSQS.Arn

  LambdaFunctionEventSourceMapping:
   Type: AWS::Lambda::EventSourceMapping
   Properties:
     BatchSize: 1
     Enabled: true
     EventSourceArn: !GetAtt MyQueue.Arn
     FunctionName: !GetAtt LambdaConsumer.Arn

4. Configuring CloudWatch Event Rule
Define a CloudWatch Event Rule in the template.yaml file to trigger the Lambda function every two minutes.

5. Implementing Lambda Functions
Create lambda function resource that sends random data to SQS with its nodeJS backend code.

sender.js

// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');

// Set the region 
//AWS.config.update({region: 'REGION'});

// Create an SQS service object
var sqs = new AWS.SQS({apiVersion: '2012-11-05'});

module.exports.send_data = (event, context) => {
    var sqs_url = "https://sqs.us-east-2.amazonaws.com/944163165741/ahmedsalem-sqs"

    var params = {
        // Remove DelaySeconds parameter and value for FIFO queues
       DelaySeconds: 10,
       MessageAttributes: {
         "Title": {
           DataType: "String",
           StringValue: "The Whistler"
         },
         "Author": {
           DataType: "String",
           StringValue: "John Grisham"
         },
         "WeeksOn": {
           DataType: "Number",
           StringValue: "6"
         }
       },
       MessageBody: "Information about current NY Times fiction bestseller for week of 12/11/2016.",
       // MessageDeduplicationId: "TheWhistler",  // Required for FIFO queues
       // MessageGroupId: "Group1",  // Required for FIFO queues
       QueueUrl: sqs_url
    };

     sqs.sendMessage(params, function(err, data) {
       if (err) {
         console.log("Error", err);
       } else {
         console.log("Success", params.MessageBody);
       }
     });

     return;



}

6. Create a lambda permission resource for events to invoke sender.js Lambda Function.

7. Create an IAM Role to allow lambda access to SQS.

8. Create SQS QueuePolicy resource to grant lambda access to SQS.

9. Create lambda function resource that receives that random data from SQS and send it to CloudWatch logs with its nodeJS backend code.

recevier.js

exports.receive_data = async function(event, context) {
    event.Records.forEach(record => {
      const { body } = record;
      console.log(body);
    });
    return {};
  }

10. Build your application
change into the project directory, where the template.yaml file for the sample application is located. Then run the below command:

# sam build

11. Deploy your application to the AWS Cloud

# sam deploy --guided

12.Clean up
If you no longer need the AWS resources that you created, you can remove them by deleting the AWS CloudFormation stack that you deployed.

You can delete the AWS CloudFormation stack using one of the below options:

1- From the AWS Management Console.

2- By running the following AWS CLI command:

# aws cloudformation delete-stack --stack-name ahmedsalem --region us-east-2

Conclusion
By following these steps, you can build a scalable serverless application using AWS SAM, SQS, and Lambda. This architecture enables efficient processing of messages and facilitates the development of event-driven applications.

Leveraging Kinesis Stream and ElasticSearch with Serverless Framework

Ahmed Salem — Sun, 14 Jan 2024 08:17:09 +0000

Overview
This guide outlines a dynamic architecture that seamlessly integrates Amazon Kinesis Stream, Elasticsearch, and Serverless Framework for efficient data collection, processing, and analysis.

Introduction
In this guide, we'll explore the integration of Amazon Kinesis Stream and ElasticSearch using the Serverless Framework. This implementation involves creating a scheduled Node.js Lambda function to generate random data, and pushing it to a Kinesis stream. Subsequently, another Lambda function consumes this stream and stores the data in ElasticSearch.The ultimate goal is to enable querying of inserted data via Kibana over ElasticSearch.

Key Components:

- AWS Kinesis Stream: A scalable stream for real-time data ingestion and buffering.

- Elasticsearch: A powerful search and analytics engine for storing, indexing, and querying data.

- Serverless Framework: A toolkit for building and deploying serverless applications, simplifying infrastructure management.

- Lambda Functions: Serverless functions triggered by events to execute specific tasks.

Architecture

Data Flow:

Data Generation:

A time-based event (CloudWatch Event Rule) triggers a Lambda function (sender.js) every 2 minutes. ** "sender.js" generates random data and pushes it into the Kinesis Stream.**

Data Consumption and Storage:

A second Lambda function (receiver.js) continuously consumes data from the Kinesis Stream.
"receiver.js" extracts and processes the data, preparing it for storage.
"receiver.js" sends the processed data to Elasticsearch for indexing and storage.

Data Analysis:

Kibana, a visualization tool integrated with Elasticsearch, enables interactive exploration and analysis of the stored data.

Prerequisites

Before we delve into the implementation, ensure you have Serverless Framework installed on your machine:

npm install serverless -g

Step-by-Step Implementation

1. Create a Serverless Project

serverless create --template aws-nodejs --path my-service

The above command will create "my-service" directory with the following structure:
.
├── .npmignore
├── handler.js
└── serverless.yml

2. Create a Kinesis Stream
Follow the AWS documentation to create a Kinesis stream as a CloudFormation resource.

3. Schedule Data Generation
Create a CloudWatch Event Rule resource to trigger the "sender.js" Lambda function every 2 minutes.

4. Lambda Permission for Event
Create a Lambda Permission resource for events to invoke the "sender.js" Lambda function.

Check the "handler.js" code:

const { KinesisClient, PutRecordCommand } = require("@aws-sdk/client-kinesis"); // CommonJS import
const {v4: uuidv4} = require("uuid")
const faker = require("faker")

const kinesisInstance = new KinesisClient();

const kinesisName = process.env.kinesisName

module.exports.handler = async function(event) {
  const response = await kinesisInstance.send(new PutRecordCommand({
    Data: Buffer.from(JSON.stringify({
      name: faker.name.firstName(),
      jobTitle: faker.name.jobTitle()  
    })),
    PartitionKey: uuidv4(),
    StreamName: kinesisName
  }));
  console.log(response)
}

Check the below "serverless.yaml" file:

# The service name has to be unique to your account.
service: my-service

# framework version range supported by this service.
frameworkVersion: '2'

# Configuration of the cloud provider. As we are using AWS we defined AWS corresponding configuration.
provider:
  name: aws
  #runtime: nodejs14.x
  #stage: dev
  runtime: nodejs12.x
  lambdaHashingVersion: 20201221
  region: us-east-2

  # Create an ENV variable to be able to use it in my JS code. 
  environment:
    kinesisName: ${self:resources.Resources.MyKinesisStream.Properties.Name}

  # To Permit each lambda function to access the DynamoDB table
  iamRoleStatements:
    - Effect: Allow
      Action:
        - kinesis:*
        - es:*
      Resource: "*"


custom:
  CronExpression: cron(*/2 * * * ? *)  


functions:
  #(1) Lambda function that sends random data to kinesis stream.
  SendDataTokinesis:
    handler: sender.handler

  #(2) Lambda function that receives random data from kinesis stream.
  ReceiveDataFromkinesis:
    handler: receiver.handler
    events:
      - stream:
          type: kinesis
          arn:
            Fn::GetAtt:
              - MyKinesisStream
              - Arn

#Resources are AWS infrastructure components which your Functions use. 
#The Serverless Framework deploys an AWS components your Functions depend upon.
resources:
  Resources:
    MyKinesisStream: 
      Type: AWS::Kinesis::Stream 
      Properties: 
          Name: ahmedsalem-KinesisStream
          RetentionPeriodHours: 168 
          ShardCount: 3

    LambdaScheduledRule: 
      Type: AWS::Events::Rule
      Properties: 
        Description: "ScheduledRule"
        ScheduleExpression: ${self:custom.CronExpression}
        State: "ENABLED"
        Targets: 
          - 
            Arn: !GetAtt 'SendDataTokinesisLambdaFunction.Arn'
            Id: "TargetFunctionV1"

    PermissionForEventsToInvokeLambda:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !Ref SendDataTokinesisLambdaFunction
        Action: "lambda:InvokeFunction"
        Principal: "events.amazonaws.com"
        SourceArn: !GetAtt LambdaScheduledRule.Arn

    ElasticSearchInstance:
      Type: AWS::Elasticsearch::Domain
      Properties:
        DomainName: 'test'
        EBSOptions:
          EBSEnabled: true
          VolumeType: gp2
          VolumeSize: 10
        AccessPolicies:
          Version: '2012-10-17'
          Statement:
            -
              Effect: 'Allow'
              Principal:
                Service: 'lambda.amazonaws.com'
                #AWS: 'arn:aws:iam::<ACCOUNT-ID>:user/<USER-ID>'
              Action: 'es:*'
              #Resource: 'arn:aws:es:us-east-2:944163165741:domain/test/*'
              Resource: '*'
        ElasticsearchClusterConfig:
          InstanceType: t2.small.elasticsearch
          InstanceCount: 1
          DedicatedMasterEnabled: false
          ZoneAwarenessEnabled: false
        ElasticsearchVersion: 5.3

5. Implement sender.js
Create a Lambda function sender.js that sends random data to the Kinesis stream.

const { KinesisClient, PutRecordCommand } = require("@aws-sdk/client-kinesis"); // CommonJS import
const {v4: uuidv4} = require("uuid")
const faker = require("faker")

const kinesisInstance = new KinesisClient();

const kinesisName = process.env.kinesisName

module.exports.handler = async function(event) {
  const response = await kinesisInstance.send(new PutRecordCommand({
    Data: Buffer.from(JSON.stringify({
      name: faker.name.firstName(),
      jobTitle: faker.name.jobTitle()  
    })),
    PartitionKey: uuidv4(),
    StreamName: kinesisName
  }));
  console.log(response)
}

Then, install the below-required packages:

npm init
npm install @aws-sdk/client-kinesis
npm install uuid
npm install faker

6. Implement receiver.js
Create a Lambda function receiver.js that consumes data from the Kinesis stream.

const { HttpRequest} = require("@aws-sdk/protocol-http");
const { defaultProvider } = require("@aws-sdk/credential-provider-node");
const { SignatureV4 } = require("@aws-sdk/signature-v4");
const { NodeHttpHandler } = require("@aws-sdk/node-http-handler");
const { Sha256 } = require("@aws-crypto/sha256-browser");
const {v4: uuidv4} = require("uuid")


var region = 'us-east-2';
var domain = 'search-test-mz4py3tpigmamsdhngu5gjdvaq.us-east-2.es.amazonaws.com'; // e.g. search-domain.region.es.amazonaws.com
var index = 'node-test';
var type = 'doc';
var id = '8';


module.exports.handler = async function(event) {
    event.Records.forEach(async function(record) {
        console.log(record.kinesis.data);
        // Kinesis data is base64 encoded so decode here
        var payload = Buffer.from(record.kinesis.data, 'base64').toString();
        console.log('Decoded payload:', payload); 
        await indexDocument(payload)

    });
}


//indexDocument(payload).then(() => process.exit())

async function indexDocument(document) {

    // Create the HTTP request
    var request = new HttpRequest({
        body: document,
        headers: {
            'Content-Type': 'application/json',
            'host': domain
        },
        hostname: domain,
        method: 'PUT',
        path: index + '/' + type + '/' + id
    });

    // Sign the request
    var signer = new SignatureV4({
        credentials: defaultProvider(),
        region: region,
        service: 'es',
        sha256: Sha256
    });

    var signedRequest = await signer.sign(request);

    // Send the request
    var client = new NodeHttpHandler();
    var { response } =  await client.handle(signedRequest)
    //console.log(response.statusCode + ' ' + response.body.statusMessage);
    console.log(response);

    var responseBody = '';
    await new Promise(() => {
      response.body.on('data', (chunk) => {
        responseBody += chunk;
      });
      response.body.on('end', () => {
        console.log('Response body: ' + responseBody);
      });
    }, (error) => {
        console.log('Error: ' + error);
    });
};

7. Set Up ElasticSearch
Create an ElasticSearch resource with the necessary permissions for Lambda to access it.

8. Update receiver.js
Update the receiver.js Lambda function to consume the Kinesis stream and send data to ElasticSearch. Install required package:

npm install http-aws-es

Conclusion
Following these detailed steps, you've successfully integrated Amazon Kinesis Stream and ElasticSearch using the Serverless Framework. This architecture allows you to generate and process data in real-time, storing it in ElasticSearch for easy querying and analysis using tools like Kibana.

References
Serverless Framework - DynamoDB / Kinesis Streams

Tutorial: Using AWS Lambda with Amazon Kinesis

What is Amazon OpenSearch Service?

KinesisClient

AWS::Elasticsearch::Domain

Leveraging AWS Cognito with Serverless for User Authentication and Data Management

Ahmed Salem — Fri, 13 Oct 2023 06:06:30 +0000

In this comprehensive guide, we will demonstrate how to integrate AWS Cognito with a Serverless application to handle user registration, authentication, and data management. This powerful combination allows you to build secure and scalable serverless APIs with user authentication and data storage features. We'll break down the process step by step and provide code snippets for each task.

Overview:
Our objective is to create a Serverless API using Node.js Lambda functions that enable user registration and list countries from DynamoDB for authenticated users. We'll leverage the Serverless Framework for this purpose.

Acceptance Criteria:
we aim to achieve the following :
1- Register users through the Serverless API and verify their email addresses.
2- Enable users to list countries using a REST client by providing a token obtained during registration.
4- Store user data securely in DynamoDB.

Architecture:

Step-by-Step Implementation:

Step 1: Install Serverless Framework

The Serverless Framework simplifies the deployment and management of serverless applications. To get started, you can install it globally using npm (Node Package Manager).

npm install serverless -g

This command installs the Serverless Framework globally on your machine, allowing you to create and manage serverless projects with ease.

Step 2: Create a Serverless Project

The Serverless Framework provides a convenient way to create a new serverless project. You can use predefined templates to scaffold your project.

serverless create --template aws-nodejs --path my-service

This command creates a new serverless project in a directory called my-service. Inside this directory, you'll find essential project files such as serverless.yml and handler.js.

Step 3: Create a DynamoDB Table for Countries

In your serverless.yml file shown below, you need to define the DynamoDB table that will store country data. DynamoDB is a fully managed NoSQL database service provided by AWS.

serverless.yml

# Service name has to be unique for your account.
service: my-service

# framework version range supported by this service.
frameworkVersion: '2'

# Configuration of the cloud provider. As we are using AWS so we defined AWS corresponding configuration.
provider:
  name: aws
  runtime: nodejs14.x
  #lambdaHashingVersion: 20201221
  stage: dev
  region: us-east-2

  # Create an ENV variable to be able to use it in my JS code. *** Check line 4 in get-country-by-name JS file ***
  environment:
    countriestableName: ${self:custom.countriestableName}
    userstableName: ${self:custom.userstableName}

  # To Give a permission to each lambda function to access DynamoDB table
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:*
        # - dynamodb:Query
        # - dynamodb:Scan
        # - dynamodb:GetItem
        # - dynamodb:PutItem
      Resource: "*"


custom:
  countriestableName: countriees  
  userstableName: useres
  userPoolCallback: http://localhost

# As shown below, when the HTTP POST or GET request is made, the handler should be invoked.
functions:
  #(1) Lambda function to initially fill DynamoDB
  FillDynamoDB:
    handler: lambdas/common/countries_loadData.fill
    description: fill DynamoDB table with set of countries.
    events:
      - http: 
          path: fill-dynamoDB
          method: POST
          cors: true

  #(2) Lambda function to list all the countries 
  GetAllCountries:
    handler: lambdas/lambda-endpoints/list-countries.list
    description: get all the countries information.
    events:
      - http: 
          path: list-countries
          method: GET
          cors: true 
          integration: lambda
          authorizer:
            #name: ${self:resources.Resources.ApiGatewayAuthorizer.Properties.Name}
            name: CognitoUserPoolAuthorizer
            # The type of authorizer. COGNITO_USER_POOLS: An authorizer that uses Amazon Cognito user pools.
            type: COGNITO_USER_POOLS
            scopes: email
            # The source of the identity in an incoming request.
            identitySource: method.request.header.Authorization
            # The ID of the RestApi resource that API Gateway creates the authorizer in.
            RestApiId: ApiGatewayRestApi
            arn:
              Fn::GetAtt:
                - UserPool
                - Arn

  #(3) Lambda function to add the user data in users dynamoDB table 
  RegisterNewUser:
    handler: lambdas/lambda-endpoints/addusertoDB.putNewUser
    description: get all the countries information.
    events:
      - http: 
          path: add-new-user
          method: POST
          cors: true

#Resources are AWS infrastructure components which your Functions use. 
#The Serverless Framework deploys an AWS components your Functions depend upon.
resources:
  ${file(./CF.yaml)}

CF.yaml

 Resources:
    countriesDynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      #DeletionPolicy: Retain
      Properties:       
        TableName: ${self:custom.countriestableName}

        AttributeDefinitions:
          -
            AttributeName: "NAME"
            AttributeType: "S" 


        KeySchema:
          -
            AttributeName: "NAME"
            KeyType: "HASH"

        BillingMode: PAY_PER_REQUEST

    usersDynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      #DeletionPolicy: Retain
      Properties:       
        TableName: ${self:custom.userstableName}

        AttributeDefinitions:
          -
            AttributeName: "NAME"
            AttributeType: "S" 


        KeySchema:
          -
            AttributeName: "NAME"
            KeyType: "HASH"

        BillingMode: PAY_PER_REQUEST

    # Custom resource to invoke lambda function to fill the countries DynamoDB table
    TriggerFillDynamoDBFunction:
     Type: AWS::CloudFormation::CustomResource
     #DependsOn: !Ref FillDynamoDB
     Properties:
       #ServiceToken: arn:aws:lambda:us-east-2:944163165741:function:my-service-dev-FillDynamoDB
       ServiceToken: !GetAtt 'FillDynamoDBLambdaFunction.Arn'

      # User Pool Resources

    # Cognito User Pool Resource
    UserPool:
      Type: AWS::Cognito::UserPool
      Properties:
        AdminCreateUserConfig:
        UserPoolName: ahmedsalem-UserPool
        Schema:
          - Name: email
            AttributeDataType: String
            Mutable: false
            Required: true
        AutoVerifiedAttributes:
          - email
        UsernameConfiguration:
          CaseSensitive: false
        AccountRecoverySetting:
          RecoveryMechanisms:
            - Priority: 1
              Name: "verified_email"
        LambdaConfig:
          PostConfirmation: !GetAtt RegisterNewUserLambdaFunction.Arn

    UserPoolToRegisterNewUserLambdaPermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt RegisterNewUserLambdaFunction.Arn
        Principal: cognito-idp.amazonaws.com
        Action: lambda:InvokeFunction
        SourceArn: !GetAtt UserPool.Arn

    UserPoolClient:
      Type: AWS::Cognito::UserPoolClient
      Properties:
        ClientName: ahmedsalem-UserPoolClient
        UserPoolId: !Ref UserPool
        GenerateSecret: false
        AllowedOAuthFlows:
          - implicit
        AllowedOAuthFlowsUserPoolClient: true
        AllowedOAuthScopes:
          - phone
          - email
          - openid
          - profile
          - aws.cognito.signin.user.admin
        CallbackURLs:
          - ${self:custom.userPoolCallback}
        SupportedIdentityProviders:
          - COGNITO

    UserPoolDomain:
      Type: AWS::Cognito::UserPoolDomain
      Properties:
        Domain: ahmedsalem-app
        UserPoolId: !Ref UserPool
 Outputs:
  TokenURL:
    Description: Url for users signing in/up
    Value: !Sub "https://${UserPoolDomain}.auth.us-east-2.amazoncognito.com/oauth2/authorize?response_type=token&client_id=${UserPoolClient}&redirect_uri=${self:custom.userPoolCallback}"

Step 4: Create Lambda function to initially fill countries DynamoDB table in serverless.yaml file and its backend nodejs code.

API_Responses.js

const Responses = {
    _200(data = {}) {
        return {
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Methods': '*',
                'Access-Control-Allow-Origin': '*',
            },
            statusCode: 200,
            body: JSON.stringify(data),
        };
    },

    _400(data = {}) {
        return {
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Methods': '*',
                'Access-Control-Allow-Origin': '*',
            },
            statusCode: 400,
            body: JSON.stringify(data),
        };
    },
};

module.exports = Responses;

countries.json

[

    {
        "name": "Albania",
        "code": "AL"
    },

    {
        "name": "Australia",
        "code": "AU"

    },

    {
        "name": "Belgium",
        "code": "BE"

    },

    {
        "name": "Brazil",
        "code": "BR"

    },

    {
        "name": "China",
        "code": "CN"

    },

    {
        "name": "Greece",
        "code": "GR"

    }
]

Dynamo.js

//  To access AWS resources  
const AWS = require('aws-sdk');

// when you're working with dynamoDB you have to use the document client to acces the files in DynamoDB
const documentClient = new AWS.DynamoDB.DocumentClient();

// Create an obj named Dynamo that has a get method that takes an NAME and tablename
// this a Async process looking into DynamoDB, So we need to await this 
const Dynamo = {
    async get(NAME, TableName) {
        const params = {
            TableName,
            Key: {
                NAME,
            },
        };

        const data = await documentClient.get(params).promise();

        if (!data || !data.Item) {
            throw Error(`There was an error fetching the data for NAME of ${NAME} from ${TableName}`);
        }
        console.log(data);

        return data.Item;
    },

    async write(data, TableName) {
        if (!data.NAME) {
            throw Error('no NAME on the data');
        }

        const params = {
            TableName,
            Item: data,
        };

        const res = await documentClient.put(params).promise();

        if (!res) {
            throw Error(`There was an error inserting NAME of ${data.NAME} in table ${TableName}`);
        }

        return data;
    },
    async list(TableName) {
        if (!data) {
            throw Error('error in listing all countries');
        }

        const params = {
            TableName
        };

        const res = await documentClient.scan(params).promise();

        if (!res) {
            throw Error(`error in listing all countires ${TableName}`);
        }

        return data;
    },
};
module.exports = Dynamo;

countries_loadData.js

var AWS = require("aws-sdk");
var fs = require('fs');
const Responses = require('../common/API_Responses');
const Dynamo = require('../common/Dynamo');
const countriestableName = process.env.countriestableName;
var response = require('cfn-response');

var documentClient = new AWS.DynamoDB.DocumentClient();

console.log("Importing countries into DynamoDB. Please wait...");

var allcountries = JSON.parse(fs.readFileSync('lambdas/common/countries.json', 'utf8'));

console.log(allcountries);
exports.fill = (event, context) => {

    if (event.RequestType != "Create") {
        return response.send(event, context, response.SUCCESS, {})
    }

allcountries.forEach( function(country) {
    var params = {
        TableName: countriestableName,
        Item: {
            "NAME":  country.name,
            "Code": country.code
        }
    };
    documentClient.put(params, function(err, data) {
       if (err) {
           console.error("Unable to add country", country.name, ". Error JSON:", JSON.stringify(err, null, 2));
           return response.send(event, context, response.FAILED, {})

       } else {
           console.log("PutItem succeeded:", country.name);
           return response.send(event, context, response.SUCCESS, {})
       }
    });
});
};

Step 5: Create a Custom Resource to fill the DynamoDB Table
To automate the population of the DynamoDB table during deployment, you can define a custom resource in your serverless.yml file above. This custom resource triggers the Lambda function responsible for filling the countries' DynamoDB table.

Step 6: Create Lambda Function for Listing Countries

Create a Lambda function to list countries from the DynamoDB table. This function should be defined in Node.js and include code for querying DynamoDB.

// const Responses = require('../common/API_Responses');
const AWS = require('aws-sdk');
const Dynamo = require('../common/Dynamo');
const documentClient = new AWS.DynamoDB.DocumentClient();
const countriestableName = process.env.countriestableName;

const params = {
 TableName : countriestableName
}

async function listItems(){
 try {
   const data = await documentClient.scan(params).promise()
   return data
 } catch (err) {
   return err
 }
}

exports.list = async (event, context) => {
 try {
   const data = await listItems()
   return { body: JSON.stringify(data) }
 } catch (err) {
   return { error: err }
 }
}

Step 7: Access the API
To access your Serverless API, you can use the provided URLs. Utilize a REST client to test the API, specifically the /list-countries endpoint.

Step 8: Create a DynamoDB Table for Users
Similar to step 3, you need to define another DynamoDB table to store user data. This table will hold user information after registration.

Step 9: Create a Cognito User Pool
Amazon Cognito is an identity management service that simplifies user authentication and authorization. Here, you create a Cognito user pool, which is essentially a user directory where your users can sign up and sign in.

a) Create a User Pool

b) Create a Lambda Permission to Register New Users
Create a Lambda permission to allow Cognito to invoke your Lambda function for registering new users. This step establishes the connection between Cognito and your Lambda function.

c) Create a User Pool Client
A user pool client represents a web or mobile application that interacts with your Cognito User Pool. Define the client properties to configure how users interact with your app.

d) Create a User Pool Domain
A user pool domain provides a custom domain name for your Cognito user pool's hosted UI. It's used for user sign-up and sign-in.

Step 10: Implement Lambda Function for User Registration
Create a Lambda function that handles user registration. This function should be triggered when a user signs up through the Cognito user pool. Ensure that you have installed any required dependencies for your Lambda function.

const {
    DynamoDBClient,
    PutItemCommand,
} = require("@aws-sdk/client-dynamodb");

console.log("hello");

const dynamoDbClient = new DynamoDBClient();
const USER_TABLE = process.env.userstableName;
module.exports.putNewUser = async (event, context, callback) => {
    console.log(USER_TABLE);
    console.log(event.request.userAttributes.email)
    console.log(event.userName)

    await dynamoDbClient.send(new PutItemCommand({

        TableName: USER_TABLE,
        Item: {
            NAME: {S: event.userName},
            Email: {S: event.request.userAttributes.email}
        }
    }))
    callback(null, event)
};

Step 11: Test User Registration
To Confirm that lambda function will to be fired when new user sign up using AWS Cognito and its data will be saved in users table.

(a) Go to the created User Pool and select "App Client Settings"
(b) Click on "Launch Hosted UI"
(c) Click on sign up, add new user, and paste the confirmation code that will be sent to you through the mail
(d) Check the users DynamoDB table, it should have the new registered user data.

Step 12: Create an API Authorizer
API authorizers are used to control access to your API endpoints. In this step, you create an API authorizer that checks the access_token header in the request to ensure that only authorized users can access your API.

Step 13: Test API Authorization
To confirm that the API endpoint is restricted to authorized users, follow these steps:

1- Go to your Cognito User Pool settings.
2- Select "App Client Settings."
3- Launch the hosted UI.
4- Sign in with a user account.
5- After signing in, you'll be directed to a rollback URL;
6- copy the URL to obtain the access_token.
7- Open a REST client and paste the API endpoint URL.
8- Set the action to "GET."
9- If you attempt to send the request without the access_token, you should receive an "unauthorized request" response.
10- Add an "Authorization" header with the access_token, and you should receive the expected response from the API endpoint.

Step 14: Deploy Your Serverless Application
Deploying your Serverless application is straightforward using the Serverless Framework. The sls deploy command packages and deploys your entire application stack to AWS.

sls deploy

This command uploads your Lambda functions, API Gateway, and other resources to AWS.

Step 15: Remove Resources (Optional)
If you ever need to remove all the functions, events, and resources created by your Serverless application from your AWS account, you can use the sls remove command.

sls remove

Conclusion:
In this article, we've covered the complete process of integrating AWS Cognito with a Serverless application for user registration, authentication, and data management. Leveraging the power of Serverless and AWS Cognito, you can build secure and scalable serverless APIs with user authentication.