DEV Community: Dhaval Mehta

Unleashing the Power of Observability: Serverless Monitoring and Logging with AWS CloudWatch - Part 2

Dhaval Mehta — Fri, 21 Jul 2023 14:48:50 +0000

Part 2: Collecting and Monitoring Serverless Metrics

Welcome to the second part of our blog series on "Unleashing the Power of Observability: Serverless Monitoring and Logging with AWS CloudWatch." In this post, we'll focus on metrics, a critical component for understanding the performance of serverless functions. We'll cover how to enable CloudWatch metrics for AWS Lambda functions and explore the available metrics to gain valuable insights into your serverless application's behavior.

Enabling CloudWatch Metrics for AWS Lambda Functions

Enabling CloudWatch metrics for AWS Lambda functions is a straightforward process that allows you to start collecting valuable data about the performance of your serverless application.

1. Using the AWS Management Console:

Step 1: Go to the AWS Management Console and navigate to the Lambda service.

Step 2: Select your Lambda function from the list and click on "Monitoring" in the navigation menu.

Step 3: In the Monitoring tab, you'll find the option to enable CloudWatch metrics. Simply click on the "Edit" button and choose "Enable" to start collecting metrics.

2. Using the AWS Command Line Interface (CLI):

You can also enable CloudWatch metrics for your Lambda function programmatically using the AWS CLI.

aws lambda put-function-concurrency --function-name YourLambdaFunctionName --reserved-concurrent-executions 100

By enabling CloudWatch metrics, you'll gain insights into essential performance metrics like the number of invocations, duration, errors, and throttles, which will help you optimize your serverless application.

**Exploring Available Metrics for AWS Lambda Functions

Once you've enabled CloudWatch metrics for your Lambda function, you can explore and analyze a wealth of valuable data.**

1. Using the AWS Management Console:

Step 1: Go to the AWS Management Console and navigate to the CloudWatch service.

Step 2: In the navigation pane, click on "Metrics."

Step 3: Under "All metrics," select "AWS/Lambda" as the Namespace.

Step 4: You'll see a list of available metrics for your Lambda functions, such as "Invocations," "Duration," "Errors," "Throttles," and more.

2. Using the AWS CLI:

You can programmatically retrieve the available CloudWatch metrics for your Lambda functions using the AWS CLI.

aws cloudwatch list-metrics --namespace "AWS/Lambda" --metric-name "Invocations" --dimensions Name=FunctionName,Value=YourLambdaFunctionName

The metrics provided by CloudWatch will give you a comprehensive view of how your Lambda function is performing and help you identify any areas that may need improvement or optimization.

Creating Custom Metrics for AWS Lambda Functions

In addition to the default metrics provided by CloudWatch, you can create and publish custom metrics to monitor specific aspects of your serverless application.

1. Using the AWS SDK:

You can use the AWS SDK for your preferred programming language to publish custom metrics to CloudWatch.

Python Example:


import boto3

cloudwatch = boto3.client('cloudwatch')
response = cloudwatch.put_metric_data(
    Namespace='CustomMetrics',
    MetricData=[
        {
            'MetricName': 'CustomMetric',
            'Value': 1,
            'Unit': 'Count'
        },
    ]
)

Custom metrics allow you to track application-specific data and business metrics, enabling you to make informed decisions based on the behavior of your serverless functions.

Conclusion

In this post, we focused on the importance of metrics in understanding the performance of serverless functions. We explored how to enable CloudWatch metrics for AWS Lambda functions using both the AWS Management Console and the AWS CLI. Additionally, we learned how to explore the available metrics and how to create and publish custom metrics for more tailored monitoring.

In the next part of our blog series, "Centralized Logging with CloudWatch Logs," we'll delve into how to enable CloudWatch Logs for Lambda functions and effectively manage and analyze logs to gain deep insights into your serverless applications. Stay tuned for more exciting insights and practical examples on leveraging AWS CloudWatch for effective observability in your serverless architecture.

Automating AWS Batch Jobs with Amazon EventBridge Scheduler

Dhaval Mehta — Fri, 21 Jul 2023 12:41:23 +0000

Introduction

Amazon EventBridge Scheduler, introduced by AWS in November 2022, is a powerful serverless scheduler that allows you to create, run, and manage tasks across various AWS services from one central, managed platform. This highly scalable service enables you to schedule millions of tasks, invoking any AWS service as a target, without the need for provisioning or managing underlying infrastructure. In this blog post, we will explore how to use EventBridge Scheduler to automate AWS Batch jobs, ensuring smooth and efficient task execution.

What is Amazon EventBridge Scheduler?

Amazon EventBridge Scheduler is a serverless scheduler introduced by AWS to facilitate the easy creation, execution, and management of tasks across multiple AWS services. With support for over 200 services and more than 6,000 APIs, the scheduler allows you to configure schedules with a minimum granularity of one minute, ensuring precise control over task execution. It also offers at-least-once event delivery to targets, along with the ability to set delivery windows, retries, retention times, and dead letter queues.

How does EventBridge Scheduler work?

EventBridge Scheduler enables you to create one-time or recurring schedules, making it flexible enough to handle various use cases. Whether you need to trigger a task just once or run it periodically, EventBridge Scheduler has you covered. The scheduler requires no event bus, streamlining the process of setting up schedules and reducing complexity. Here are some key features of EventBridge Scheduler:

Quota on schedules: Up to 1 million schedules per AWS account.
Event invocation throughput: Capable of supporting throughput in thousands of transactions per second (TPS).
Targets: Over 270 services and more than 6,000 API actions with AWS SDK targets.
Time expressions and time-zones: Supports at(), cron(), and rate() expressions in all time-zones and with Daylight Saving Time (DST) adjustments.
One-time schedules: Allows the creation of single, one-off schedules with precise execution times.
Time window schedules: Enables the configuration of schedules with a time window for more flexibility.
Rule quota consumption: No rule quota consumption; soft limit of 1 million schedules.

Scheduling AWS Batch Jobs with EventBridge Scheduler

Now, let's dive into the process of automating AWS Batch jobs using Amazon EventBridge Scheduler. Below are the steps to achieve this:

Creating the Required Policies

To get started, we need to create a policy that grants the necessary permissions for the Scheduler to execute our AWS Batch job. The policy should include permissions for "batch:SubmitJob", "batch:DescribeJobDefinitions", and "batch:DescribeJobQueues".

# CloudFormation code snippet for creating the BatchSubmitPolicy
BatchSubmitPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    Description: Policy to allow Scheduler to execute AWS Batch jobs
    ManagedPolicyName: YourBatchSubmitPolicy
    PolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: 'Allow'
          Action: 
            - "batch:SubmitJob"
            - "batch:DescribeJobDefinitions"
            - "batch:DescribeJobQueues"
          Resource: 
            - YourBatchJobDefinitionARN
            - YourBatchJobQueueARN

Creating the required role

Next, we create an IAM role that the Scheduler will assume to execute the tasks. This role will have the previously created BatchSubmitPolicy attached.

# CloudFormation code snippet for creating the EventBridgeRole
EventBridgeRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: 'Allow'
          Principal:
            Service:
              - 'scheduler.amazonaws.com'
          Action: 'sts:AssumeRole'
    Path: "/"
    ManagedPolicyArns:
      - !Ref BatchSubmitPolicy
    Description: EventBridge role for AWS Batch job execution
    RoleName: YourEventBridgeRole

Configuring the Schedule

Now we create the EventBridge schedule, specifying the name, description, and the desired ScheduleExpression. In this example, we use a rate expression to trigger the job every 8 hours.

# CloudFormation code snippet for creating the EventSchedule
EventSchedule:
  Type: AWS::Scheduler::Schedule
  Properties: 
    Description: 'EventBridge schedule to trigger the AWS Batch job'
    FlexibleTimeWindow: 
      Mode: "OFF"
    Name: YourBatchExecSchedule
    ScheduleExpression: 'rate(8 hours)'
    State: 'ENABLED'

Defining the Target

The TargetArn needs to be defined in a specific format, and in this case, we'll set it to invoke the "batch:SubmitJob" API action of AWS Batch.

# CloudFormation code snippet for defining the Target
Target: 
  Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
  RoleArn: !GetAtt EventBridgeRole.Arn     
  Input: !Sub |
    {
      "JobName": "scheduled-batch-job",
      "JobDefinition": "YourBatchJobDefinitionARN",
      "JobQueue": "YourBatchJobQueueARN",
      "ContainerOverrides": {
          "Environment": [ 
            { 
                "Name": "LoggingLevel",
                "Value": "${LoggingLevel}"
            },
            {
              "Name": "Prefix",
              "Value": "${Prefix}"
            }
          ]
        }            
    }

Complete CloudFormation Code

Here's the complete CloudFormation template to create the EventBridge schedule for your AWS Batch job.

AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation template to create an EventBridge schedule for AWS Batch jobs

Parameters:
  LoggingLevel:
    Type: String
    AllowedValues: [NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL]    
    Default: "INFO"     

  Prefix:
    Type: String
    Default: "prefix-value"   

Resources:
  BatchSubmitPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Policy to allow Scheduler to execute AWS Batch jobs
      ManagedPolicyName: YourBatchSubmitPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: 'Allow'
            Action: 
              - "batch:SubmitJob"
              - "batch:DescribeJobDefinitions"
              - "batch:DescribeJobQueues"
            Resource:
              - YourBatchJobDefinitionARN
              - YourBatchJobQueueARN

  EventBridgeRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: 'Allow'
            Principal:
              Service:
                - 'scheduler.amazonaws.com'
            Action: 'sts:AssumeRole'
      Path: "/"
      ManagedPolicyArns:
        - !Ref BatchSubmitPolicy
      Description: EventBridge role for AWS Batch job execution
      RoleName: YourEventBridgeRole

  EventSchedule:
    Type: AWS::Scheduler::Schedule
    Properties: 
      Description: 'EventBridge schedule to trigger the AWS Batch job'
      FlexibleTimeWindow: 
        Mode: "OFF"
      Name: YourBatchExecSchedule
      ScheduleExpression: 'rate(8 hours)'
      State: 'ENABLED'
      Target: 
        Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
        RoleArn: !GetAtt EventBridgeRole.Arn     
        Input: !Sub |
          {
            "JobName": "scheduled-batch-job",
            "JobDefinition": "YourBatchJobDefinitionARN",
            "JobQueue": "YourBatchJobQueueARN",
            "ContainerOverrides": {
                "Environment": [ 
                  { 
                      "Name": "LoggingLevel",
                      "Value": "${LoggingLevel}"
                  },
                  {
                    "Name": "Prefix",
                    "Value": "${Prefix}"
                  }
                ]
              }            
          }

Conclusion

Amazon EventBridge Scheduler offers a powerful and flexible solution for automating tasks across various AWS services. In this blog post, we explored how to use EventBridge Scheduler to schedule and trigger AWS Batch jobs. By following the steps outlined in this guide, you can efficiently manage your tasks and improve operational efficiency.

References

Unleashing the Power of Observability: Serverless Monitoring and Logging with AWS CloudWatch - Part 1

Dhaval Mehta — Sun, 09 Jul 2023 17:08:52 +0000

Part 1: Introduction to AWS CloudWatch
Welcome to the first part of our blog series on "Unleashing the Power of Observability: Serverless Monitoring and Logging with AWS CloudWatch." In this introductory post, we'll explore the fundamental concepts and capabilities of AWS CloudWatch, a powerful monitoring and observability service offered by Amazon Web Services. By the end of this post, you'll have a clear understanding of what CloudWatch is and how it can benefit your serverless applications.

What is AWS CloudWatch?
AWS CloudWatch is a fully managed monitoring and observability service that provides real-time insights into the performance, health, and behavior of your AWS resources and applications. It enables you to collect and monitor metrics, collect and analyze logs, set alarms, and create dashboards to gain valuable operational insights.

CloudWatch offers a centralized platform for monitoring all your AWS services, making it easier to identify and resolve issues, optimize resource utilization, and ensure the overall efficiency of your cloud-based applications.

Key Features of AWS CloudWatch
1. Metrics
Metrics are fundamental to monitoring the performance of your AWS resources and services. CloudWatch allows you to collect, store, and visualize various metrics for different AWS services. These metrics provide crucial data points such as CPU utilization, network traffic, request counts, latency, and more. You can use these metrics to gain insights into the behavior of your resources and make informed decisions to improve performance and efficiency.

Example Code Snippet: Enabling CloudWatch Metrics for an AWS Lambda Function

import boto3

# Enable CloudWatch metrics for an AWS Lambda function
client = boto3.client('lambda')
response = client.put_function_concurrency(
    FunctionName='YourLambdaFunctionName',
    ReservedConcurrentExecutions=100
)
print(response)

2. Logs
CloudWatch Logs allows you to collect and store logs from various AWS services and applications. You can use CloudWatch Logs to troubleshoot issues, monitor application events, and analyze log data efficiently. With support for real-time log streaming, you can stay on top of critical events and identify potential problems before they escalate.

Example Code Snippet: Configuring CloudWatch Logs for an AWS Lambda Function

import logging

# Configure CloudWatch logging for an AWS Lambda function
logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)

3. Alarms
CloudWatch Alarms enable you to set up automated actions based on predefined thresholds for specific metrics. When a metric breaches the defined threshold, CloudWatch can trigger notifications or execute actions such as sending an SNS notification or performing an Auto Scaling action.

Example Code Snippet: Setting Up a CloudWatch Alarm for High CPU Utilization

import boto3
# Create a CloudWatch Alarm
cloudwatch = boto3.client('cloudwatch')
cloudwatch.put_metric_alarm(
    AlarmName='YourAlarmName',
    ComparisonOperator='GreaterThanThreshold',
    EvaluationPeriods=1,
    MetricName='CPUUtilization',
    Namespace='AWS/EC2',
    Period=60,
    Statistic='Average',
    Threshold=80.0,
    ActionsEnabled=False,
    AlarmDescription='Alarm when CPU exceeds 80%',
    AlarmActions=['arn:aws:sns:us-east-1:123456789012:YourSNSTopic'],
)

4. Dashboards
CloudWatch Dashboards allow you to create custom visualizations of your metrics and logs in a centralized location. You can design dashboards with various widgets like line charts, bar charts, and text widgets, providing you with a consolidated view of the health and performance of your AWS resources.

5. Event Rules
CloudWatch Event Rules help you automate responses to operational changes within your AWS environment. By defining rules based on events or scheduled tasks, you can trigger actions like invoking a Lambda function, publishing to an SNS topic, or sending data to Kinesis Data Streams.

Conclusion
In this introductory post, we explored AWS CloudWatch and its essential features, which include metrics, logs, alarms, dashboards, and event rules. As a monitoring and observability service, CloudWatch empowers you to gain real-time insights into your AWS resources and applications, ensuring their optimal performance and reliability.

In the next part of this blog series, we'll dive deeper into Monitoring Serverless Applications with CloudWatch Metrics, where we'll explore how to enable CloudWatch metrics for AWS Lambda functions and leverage these metrics to monitor and optimize your serverless applications. Stay tuned for more exciting insights and practical examples on leveraging AWS CloudWatch for effective observability in your serverless architecture.

Streamlining Software Delivery: Mastering Serverless CI/CD Pipelines with AWS

Dhaval Mehta — Sun, 09 Jul 2023 17:04:16 +0000

Introduction:
Continuous Integration and Continuous Deployment (CI/CD) pipelines are essential for automating the software delivery process and ensuring fast and reliable releases. In this blog post, we'll explore how to build serverless CI/CD pipelines using AWS services. We'll leverage AWS CodePipeline, AWS CodeBuild, and AWS Lambda to create a fully automated pipeline that builds, tests, and deploys our applications.

Prerequisites:
To follow along with this tutorial, you'll need:

An AWS account
Basic knowledge of AWS services, specifically AWS CodePipeline, AWS CodeBuild, and AWS Lambda
A sample application (e.g., a Node.js application hosted on GitHub)

Step 1: Set up the CodePipeline

First, let's create the CodePipeline that will orchestrate our CI/CD process:

Go to the AWS Management Console and open the AWS CodePipeline service.
Click on "Create pipeline" to start creating a new pipeline. Provide a pipeline name and select your source provider (e.g., GitHub).
Connect to your GitHub repository and choose the branch to monitor for changes.
Configure the build stage by selecting AWS CodeBuild as the build provider and specifying the build configuration.
Set up the deployment stage to deploy your application to the desired environment (e.g., AWS Elastic Beanstalk or AWS Lambda).
Review the pipeline settings and click on "Create pipeline" to create the pipeline.

Step 2: Create the CodeBuild Project
Next, let's create a CodeBuild project that will compile and test our application:

Open the AWS CodeBuild service in the AWS Management Console.
Click on "Create build project" to create a new CodeBuild project.
Provide a project name and select the source provider (e.g., GitHub).
Connect to your GitHub repository and choose the branch to build.
Configure the build environment with the desired runtime and build specifications.
Define the build specifications in the buildspec.yml file, which includes build steps, tests, and artifacts.
Review the project settings and click on "Create build project" to create the project.

Step 3: Set up Deployment with AWS Lambda
In this step, we'll configure AWS Lambda as our deployment provider to deploy the application:

Open the AWS Lambda service in the AWS Management Console.
Click on "Create function" to create a new Lambda function.
Select the desired runtime (e.g., Node.js) and provide a function name.
Choose an appropriate IAM role for the Lambda function with the necessary permissions.
Write the deployment logic in the Lambda function using the desired deployment approach (e.g., AWS SDK, AWS CLI, or a deployment framework like Serverless Framework).
Review the function settings and click on "Create function" to create the Lambda function.

Step 4: Configure CodePipeline to Use AWS Lambda for Deployment
Now, let's configure CodePipeline to use the Lambda function for deployment:

Open the AWS CodePipeline service in the AWS Management Console.
Select your pipeline and click on "Edit" to modify the pipeline configuration.
Navigate to the deployment stage and choose "AWS Lambda" as the deployment provider.
Specify the Lambda function name and any additional configuration settings.
Save the pipeline changes.

Conclusion:
In this blog post, we explored how to build serverless CI/CD pipelines using AWS services. We set up an end-to-end pipeline using AWS CodePipeline, AWS CodeBuild, and AWS Lambda. This pipeline automatically builds, tests, and deploys our application whenever changes are pushed to the source repository. With serverless CI/CD pipelines, you can ensure fast and reliable software delivery in a scalable and cost-efficient manner.

Extending CloudFormation's Power: Creating Custom Resources for Enhanced AWS Resource Management

Dhaval Mehta — Tue, 04 Jul 2023 17:38:24 +0000

Introduction:
CloudFormation templates are powerful tools for programmatically provisioning AWS resources. While they provide a wide range of predefined resources and templates, there are situations that call for additional custom actions. In this blog post, we will explore the process of creating and utilizing custom resources in CloudFormation to enhance your AWS resource management.

Why Do We Need Custom Resources?
CloudFormation templates excel at creating AWS resources through code. However, there are instances where you may require specific actions to be performed on the created resources. These actions might involve tasks such as creating directories within newly generated buckets, configuring OpenSearch indexes, inserting essential data into databases, or cleaning up resources when shutting down infrastructure. Custom resources offer the means to extend CloudFormation's capabilities, empowering you to address these specialized requirements seamlessly.

Creating a Custom Resource: Step-by-Step Guide
To create a custom resource using CloudFormation, follow these three steps:

Step 1: Create a Role for the Lambda Function
Start by creating a role that the Lambda function we will use. This role should have the necessary permissions to perform the required actions. In the example below, we grant the Lambda function permissions for executing code and accessing S3:

CustomLambdaRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - sts:AssumeRole
    Path: "/"
    ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/AWSLambdaExecute"
      - "arn:aws:iam::aws:policy/AmazonS3FullAccess"
    Description: "Custom Lambda Role"

Step 2: Create the Lambda Function
Next, create the Lambda function itself using CloudFormation. The example below demonstrates how to define the function and its properties:

CustomFunction:
  Type: AWS::Lambda::Function
  Properties: 
    Architectures: 
      - "x86_64"
    Code:
      ZipFile: |
        import boto3
        import json
        import os
        import urllib3

        # Retrieve Environment Variables
        S3Bucket=os.environ["S3Bucket"]

        http = urllib3.PoolManager()
        SUCCESS = "SUCCESS"
        FAILED = "FAILED"

        def lambda_handler(event, context):
            # Retrieve parameters
            dirs = event['ResourceProperties']['dirs']
            response_data = {}
            try:
                if event['RequestType'] in ('Create', 'Update'):
                    createUpdateEvent(dirs)
                elif event['RequestType'] == 'Delete':
                    deleteEvent()

                # Everything OK... send the signal back                
                send(event,
                    context,
                    SUCCESS,
                    response_data)

            except Exception as e:
                response_data['Data'] = str(e)
                send(event,
                    context,
                    FAILED,
                    response_data)

        def createUpdateEvent(dirs):
            s_3 = boto3.client('s3')
            for dir_name in dirs:
                print("Creating: ", str(dir_name))
                s_3.put_object(Bucket=S3Bucket, Key=(dir_name + '/'))

        def deleteEvent():
            print("Deleting S3 content...")
            b_operator = boto3.resource('s3')
            b_operator.Bucket(str(S3Bucket)).objects.all().delete()

        def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False):
            responseUrl = event['ResponseURL']
            print(responseUrl)

            responseBody = {
                'Status': responseStatus,
                'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,
                'PhysicalResourceId': physicalResourceId or context.log_stream_name,
                'StackId': event['StackId'],
                'RequestId': event['RequestId'],
                'LogicalResourceId': event['LogicalResourceId'],
                'NoEcho': noEcho,
                'Data': responseData
            }

            json_responseBody = json.dumps(responseBody)

            print("Response body:\n" + json_responseBody)

            headers = {
                'content-type': '',
                'content-length': str(len(json_responseBody))
            }

            try:
                response = http.request('PUT', responseUrl, body=json_responseBody.encode('utf-8'), headers=headers)
                print("Status code: " + response.reason)

            except Exception as e:
                print("send(..) failed executing requests.put(..): " + str(e))

Step 3: Define the Custom Resource
Finally, define the custom resource itself in the CloudFormation template. Use the following code snippet as a reference:

CustomResource:
  Type: Custom::CustomResource
  Properties:
    ServiceToken: !GetAtt CustomFunction.Arn
    dirs: !Ref Dirs

Creating Parameters for the Custom Resource
To make the custom resource more flexible, you can add parameters to the CloudFormation template. In this example, we create a parameter named "Dirs" to specify a comma-delimited list of directories to be created:

Parameters:
  Dirs:
    Description: "Comma-delimited list of directories to create."
    Type: CommaDelimitedList
    Default: test-dir

Executing this updated CloudFormation template will create an S3 bucket and automatically create a folder within that bucket as part of your infrastructure setup. You can customize the directory names by providing a comma-delimited list of values for the "Dirs" parameter.

Conclusion:
Custom resources in CloudFormation offer a flexible way to extend the provisioning capabilities of AWS resources. By following the steps outlined in this blog post and using the updated Lambda code, you can create custom resources and seamlessly integrate them into your CodePipeline workflows. Whether it's setting up additional resources, performing specific actions, or cleaning up resources, custom resources provide a powerful solution to meet your specific requirements.