DEV Community: Omar Omar

AWS Cognito for Bedrock Storyteller running on Containerized Lambda Function with Web Adapter

Omar Omar — Sun, 31 Mar 2024 21:10:03 +0000

This is a sample application that demonstrates how to use AWS Cognito to add login functionality to a Bedrock Storyteller Flask application running on AWS Lambda with Web Adapter. The main purpose of this tutorial is to exhibit how to use AWS Cognito to authenticate users and authorize them to access the application.

Adding logging in and logging out functionality to your application makes it more secure. AWS Cognito is a great service that provides authentication, authorization, and user management for your applications. It is easy to use and provides a lot of features out of the box.

Throughout this tutorial, you will learn how to:

Create a new AWS Cognito User Pool
Add a new user to the User Pool
Confirm the user using pre-signup Lambda trigger
Update the Bedrock Storyteller Flask application to use AWS Cognito for authentication
Deploy the application to a containerized AWS Lambda with Web Adapter
Use AWS SAM to deploy the application

The ultimate goal is to learn how to use AWS Cognito to authenticate users and authorize them to access the application. By the end of this tutorial, you will have a Bedrock Storyteller Flask application running on AWS Lambda with Web Adapter that uses AWS Cognito for authentication and authorization.

Prerequisites

Before you begin, ensure you have the following:

AWS CLI installed and configured
Docker installed
Python installed
AWS SAM CLI installed
Clone the Bedrock Storyteller Flask application repository.

Step 1: Create a new AWS Cognito User Pool

1- Open the AWS Management Console and navigate to the AWS Cognito service.
2- Click on Create user pool.
3- Check the Email and then click on Next.
4- Leave the Password policy as default and select No MFA.
5- Uncheck the Enable self-service account recovery and then click on Next.

6- Check the Enable self-registration, select Don't automatically send messages, leave the rest as default, and then click on Next.
7- Select Send email with Cognito and then click on Next.

8- Enter a name for the user pool and App client name and select Generate a client secret.

9- Under Advanced app client settings, make sure you select (ALLOW_REFRESH_TOKEN_AUTH), (ALLOW_ADMIN_USER_PASSWORD_AUTH), (ALLOW_USER_PASSWORD_AUTH) and then click on Next.

10- Finally, review the Review and create page and then click on Create user pool.

Note:

Capture the following information as you will need it later:

User Pool ID
App client ID
App client secret (navigate to App integration -> App clients and analytics -> click on App client name -> Show client secret)

Step 2: Add a new user to the User Pool

Creating a Lambda function to confirm new users

1- Navigate to the AWS Lambda service.
2- Click on Create function.
3- Enter a name for the function, select Python 3.12 as the runtime, and then click on Create function.
4- Copy the code from the lambda_function.py file in the assets directory and paste it into the Lambda function code editor. Then click on Deploy.

Adding the Lambda trigger to the User Pool

1- Navigate back to the AWS Cognito service.
2- Click on the newly created user pool.
3- Select User pool properties and click on Add Lambda trigger.
4- Select Sign-up as a Trigger type and Pre sign-up trigger as a lambda trigger type.
5- Under Lambda function, assign the lambda function that you created earlier and then click on Add Lambda trigger.

Note:

The purpose of having a pre-signup Lambda trigger is to confirm the user before adding them to the user pool. Usually, the users confirm their email address by clicking on the link sent to their email address. In this tutorial, we are confirming the user automatically using the Lambda trigger because we are not sending any email to the users and we are limiting access to the application to only the users that we add manually. Moreover, we can use dummy email addresses for the users.

Adding a new user to the User Pool

We will add a new user to the user pool using the AWS CLI and the Lambda trigger will confirm the user automatically.

1- Compute a secret stash value for the user's password using the following command:

echo -n "[username][app client ID]" | openssl dgst -sha256 -hmac [app client secret] -binary | openssl enc -base64

Actual example:

echo -n "omar@omar.com4njih1dqgurs5r529l0mgl4j21" | openssl dgst -sha256 -hmac r6v7qk0ui0aunhkc4jk8eh1pb4g8nl4715fg4l0aejfithdmg5r -binary | openssl enc -base64

Replace [username] with the email of the user you want to add.
Replace [app client ID] with the app client ID of the user pool.
Replace [app client secret] with the app client secret of the user pool.
Save the output of the command as the secret stash value.

Note:

To learn more about how to compute the secret stash value, refer to the AWS documentation.
My username is omar@omar.com.
My app client ID is 4njih1dqgurs5r529l0mgl4j21.
My app client secret is r6v7qk0ui0aunhkc4jk8eh1pb4g8nl4715fg4l0aejfithdmg5r.

2- Add the user to the user pool using the following command:

aws cognito-idp sign-up \
--client-id 'APP_CLIENT_ID' \
--secret-hash `SECRET_HASH` \
--username 'USERNAME' \
--password 'PASSWORD' \
--user-attributes Name=email,Value=`USERNAME` \
--region 'REGION' \
--profile 'PROFILE'

Actual example:

aws cognito-idp sign-up \
--client-id 4njih1dqgurs5r529l0mgl4j21 \
--secret-hash o1364n3vGN3BBLJNJCb858KjGDMdx+Jyt85KIFVJwXc= \
--username omar@omar.com \
--password Test_Pass123 \
--user-attributes Name=email,Value=omar@omar.com \
--region us-east-1 \
--profile default

Note:

Replace [PROFILE] with the name of your AWS profile, if it's not the default profile.
Once you run the command, you could navigate to the AWS Cognito service and check the Users tab to see the newly added user. Also, if you navigate to the Monitor section of the Lambda function, you will see that the Lambda function was triggered and the user was confirmed.

Step 3: Update the Bedrock Storyteller Flask application to use AWS Cognito for authentication

1- Open the Bedrock Storyteller Flask application in your favorite code editor.
2- Update the values of the following variables in the cognito_config.py file in the app directory:

COGNITO_REGION = 'us-east-1'
COGNITO_USER_POOL_ID = 'us-east-1_Q0l9hNEw1'
COGNITO_CLIENT_ID = '2p87n43ntcdv8f4ml5jh3cd8ji'
COGNITO_CLIENT_SECRET = 'vivq45062h98669dgv3rthnhppi6da0virnrlapn02b7cs4r54l'

3- Notice in line 16 of the app.py file in the app directory that we are using the app.secret_key. This is used to sign the session cookie. You can generate a secret key using the following command:

python -c 'import secrets; print(secrets.token_hex())'

Note:

Refer to the Flask documentation to learn how to compute the secret key value.

Step 4: Deploy the application to a containerized AWS Lambda with Web Adapter using AWS SAM

1- Using your terminal, navigate to the root directory of the Bedrock Storyteller Flask application.

The directory structure

├── README.md
├── assets
├── imgs
├── app
│   ├── Dockerfile
│   ├── __init__.py
│   ├── app.py
│   ├── cognito_config.py
│   ├── requirements.txt
│   ├── static
│   └── templates
│       ├── about.html
│       ├── index.html
│       └── login.html
└── template.yaml

2- Build the Docker image using the following command:

sam build --use-container --profile `NAME_OF_YOUR_AWS_PROFILE`

Note:

Replace [NAME_OF_YOUR_AWS_PROFILE] with the name of your AWS profile, if it's not the default profile.

3- Deploy the application using the following command:

sam deploy --guided --profile `NAME_OF_YOUR_AWS_PROFILE`

Note:

Follow the prompts to deploy the application:

Stack Name [sam-app]: Bedrock Storyteller
AWS Region [us-east-1]: us-east-1
Confirm changes before deploy [y/N]: y
Allow SAM CLI IAM role creation [Y/n]: y
Disable rollback on stack creation failures [y/N]: n
FlaskFunction Function URL has no authentication. Is this okay? [y/N]: y
Save arguments to configuration file [Y/n]: y
SAM configuration file [samconfig.toml]: `click enter`
SAM configuration environment [default]: `click enter`

4- Once the deployment is complete, you will see the output with the FlaskFunction URL. Copy the URL and open it in your browser.

Conclusion

In this tutorial, you learned how to use AWS Cognito to authenticate users and authorize them to access a Bedrock Storyteller Flask application running on AWS Lambda with Web Adapter. You created a new AWS Cognito User Pool, added a new user to the User Pool, confirmed the user using a pre-signup Lambda trigger, updated the Bedrock Storyteller Flask application to use AWS Cognito for authentication, and deployed the application to a containerized AWS Lambda with Web Adapter using AWS SAM.

The next step is to explore adding AWS Cognito sign-in and sign-out functionality to the application. I hope you have enjoyed this tutorial and found it helpful. Feel free to reach out if you have any questions or feedback.

Future Work

[ ] enhance data chunking of the Bedrock Storyteller Flask application.

References

Alexa to Run Systems Manager Documents

Omar Omar — Sun, 25 Dec 2022 19:30:04 +0000

Introduction:

Alexa is Amazon's cloud-based voice service that powers hundreds of millions of devices. It also enables developers to build skills, which are like applications for Alexa. The Alexa skill is a cloud-based solution that provides the logic and functionality to perform certain tasks using voice commands. The skill is hosted on AWS Lambda and is built using Alexa Skills Kit SDK (ASK) framework. The communication between Alexa service and the Lambda function hosting the Alexa skill is encrypted and the access permissions to the Lambda function are protected by AWS Identity and Access Management (IAM) policies. Therefore, we can be confident that Alexa skills are secure.

This step-by-step tutorial walks you through the process of developing an Alexa cloud-based solution. We will build our Alexa skill using Alexa Skills Kit SDK (ASK) for Python that will allow us to run AWS Systems Manager Documents (SSM documents) using Run Command, which is a capability of Systems Manager. The SSM documents run on AWS Systems Manager managed nodes (SSM managed EC2 instances) using AWS-RunDocument. Utilizing the AWS-RunDocument SSM document with Run Command enables us to run any SSM document on our managed nodes without having to modify our backend source code (Python). It is a great way to standardize the process of running SSM documents on our managed nodes. The SSM documents define the actions to perform on the managed instances and can be used to perform a variety of tasks such as patching OS, stopping EC2 instances, installing software, updating software, configuring operating systems, and much more. Basically, sky is the limit when it comes to the functionality of running SSM documents.

It is worth mentioning that this serverless solution is built using AWS services and components with minimal costs associated. The solution is also built using the best practices and the least privilege principle. The access permissions to the AWS services and components are protected by AWS Identity and Access Management (IAM) policies. Therefore, we can be confident that our solution is secure.

The ultimate goal of the tutorial is to simplify the process of building this cloud-based solution, learning how to create an Alexa skill and learning how to create several AWS services and components such as, DynamoDB table, IAM service role, SNS topic, Lambda functions, Secrets Manager secret, and SSM documents. Moreover, it is a great opportunity to learn about the logic behind the solution and the architecture of the solution.

As an AWS Community Builder, this is part of my continuous effort to share knowledge and experience with the community. The value added by this solution is to run SSM commands on our managed instances without having to log into the AWS console or to use the AWS CLI. We can perform tasks remotely and securely from anywhere using Alexa enabled devices such as an Echo device or Alexa mobile app. Run commands and tasks remotely on our managed instances using Alexa is a great way to save time and to automate the process of running SSM documents on our managed instances.
I intend keep the tutorial as simple as possible and will not go into the details unless necessary. I will pass the baton to you to take the solution to the next level. I will also intentionally intermingle between SSM documents and SSM commands. Although, the solution could be built using IaC, we would have lost the learning experience.

Remember, the goal of the tutorial is to learn and to understand the logic and architecture of the solution. It's about the journey, not the destination, so let's buckle up and get started!

Alexa Cloud Solution: Logic and Architecture:

The solution consists of several AWS services and components as shown on the diagram below. The logic flow might be a bit complex at first, but it becomes clearer as we go through the tutorial.

1- The user wakes up the Alexa skill by saying, Alexa, open {name of the skill} on Alexa enabled devices such as an Echo device or Alexa mobile app. Our skill is called Command Control.
2- The user initiates a voice command to run an SSM command for a specific instance tag by saying, {name of the command} {name of the tag}, for example, patch dev. Ofcourse, we can change the utterances to match our needs. But for the purpose of this tutorial, we will keep it simple.
3- The Alexa service sends a JSON body request to invoke the Lambda function hosting the Alexa skill (AlexaSkill function). The AlexaSkill Lambda function validates the request and extracts the command and tag from the request.
4- The AlexaSkill function sends a payload of the command and tag to the Master Lambda function. The Master Lambda function is invoked synchronously.
5- Based on the command and tag received, the Master Lambda function queries a DynamoDB table to obtain the SSM document name and any SSM document parameters corresponds to the command.

Note: not all SSM documents require parameters. It's our responsibility to make sure that the SSM document name and any SSM document parameters are valid. The parameters in the DynamoDB table should be in JSON format.

6- The Master Lambda function validates the response from the DynamoDB table. If the response is not valid, the Master Lambda function returns error message to AlexaSkill function. The handled errors could be such as, the Command, SSM document name, any SSM document parameters or even the DynamoDB table name are not valid or don't exist.
7- The Master Lambda function then queries whether or not the specified tag exists for any running EC2 instances in the region (the instance will be ignored if it's not in the running status). If the condition is met (tag exists and the instance is running), the Master Lambda function then queries whether or not the instances are SSM managed nodes. Again, if this condition is met (the instances are SSM managed nodes), the Master Lambda function sends the AWS-RunDocument attributes to Run Command to run the SSM document on the EC2 instances. If the condition is not met, the Master Lambda function returns error message to AlexaSkill function. The handled errors could be such as, the specified tag does not exist for any running and SSM managed EC2 instances.

8- The Run Command runs the SSM document on the EC2 instances filtering by the specified tag. Once Run Command has started executing the SSM document on the EC2 instances successfully, the Pending status is to be returned to the Master Lambda function.
9- The Master Lambda function sends the Pending status, command, tag and number of instances the SSM document is running on back to AlexaSkill function and AlexaSkill function sends the response to the user.
10- The Run Command also sends notifications to the SNS topic to notify the user once the SSM document has been run on the EC2 instances and also when the status has been updated to InProgress, Failed or Success.

Note: if you prefer to receive notifications via email, SMS, or any other method allowed by SNS, you can subscribe to the SNS topic. It's also worth noting that the Run Command records the status of the SSM document in the SSM document history. Therefore, we can always go to the Run Command history to see the status of the SSM document.

Important Notes:

The Master Lambda function only runs SSM documents on running and SSM managed nodes. Therefore, if the EC2 instance is not running or not SSM managed, the Master Lambda function will ignore it.
To create SSM managed nodes, we need to install the SSM agent on the EC2 instance. The SSM agent is installed by default on Amazon Linux 2 EC2 instances. However, if we are using other Linux distributions, we need to install the SSM agent manually. Then, we need to attach an instance profile IAM role with the AmazonSSMManagedInstanceCore policy to the EC2 instance. Please, refer to AWS documentation for more details on to setup SSM managed nodes.
We also need to tag the EC2 instances with a key-value pair to filter the EC2 instances. I have designated Alexa to be the constant tag key, and we need to specify the tag value in the voice command to match the tag value we have specified for the EC2 instances. Therefore, if the instance does not have Alexa as the tag key, the Master Lambda function ignores the instance. If we need to use a different tag key, we should modify the tag_key = 'Alexa in the Master Lambda function Python source code.
As best practices, I have configured intensive code level logging to cover lots of error scenarios, which should be helpful for troubleshooting and debugging. If an error is raised, the Master Lambda function sends an error message to AlexaSkill and eventually back to the user. The error messages are also logged in CloudWatch Logs for the Master Lambda function log group.

Alexa Cloud-based Solution Architecture Diagram

Tutorial Sections:

The solution is broken down into several sections to make it easier to follow. The sections are as following:

Creating an Alexa skill hosted on AWS Lambda function (AlexaSkill function) using the Alexa Skill Kit (ASK) SDK for Python and Boto3 SDK for Python
Configuring the skill with Alexa service on Alexa Developer Console
Creating an SSM document to stop running EC2 instances. We will also use AWS SSM document AWS-RunPatchBaseline to patch Amazon Linux 2 EC2 instances. This covers the two scenarios; running AWS managed SSM documents and customer created SSM documents
Creating an SNS topic to publish notifications from Systems Manager - Run Command
Creating an IAM service role for Systems Manager to send notifications to the SNS topic
Provisioning an on-demand DynamoDB table to store the commands, SSM document names and any SSM document parameters that are used by the Master Lambda function
Creating a Lambda function (MasterLambda) to send SSM commands to Systems Manager - Run Command
Creating a Lambda function (SlackLambda) to send notifications to Slack or you may use any other method of your choice (optional)
Testing Alexa skill and Run Command
Party time 🎉🎉🎉

Prerequisites:

An AWS account
An AWS Alexa Developer account
Python 3 (>= 3.6)
pip
virtualenv or venv
Alexa enabled device or Alexa app on your mobile device

1: Creating Alexa skill hosted on AWS Lambda function (AlexaSkill)

Step 1: Setting up the ASK SDK in a virtual environment on Linux or macOS

1- Create a virtual environment:

virtualenv command_control

Note: commands is the name of the virtual environment. You can use any name you want. Also for Windows, you may refer to Alexa documentation for more details.

2- Activate the virtual environment:

source command_control/bin/activate

3- Install the ASK SDK:

pip install ask-sdk

Note: The ASK SDK is installed in the virtual environment. You can use the deactivate command to exit the virtual environment.

Step 2: Add skill source code

Create a Python file named lambda_function.py in the command_control root directory and copy the Python code from this Lambda function.

Note: it's important to name the file lambda_function.py because the Lambda function will be looking for this handler when it's invoked.

Step 3: Packaging and Creating Lambda Function (AlexaSkill function)

1- From the command_control root directory, copy the lambda_function.py into lib/python3.8/site-packages/ directory.

cd command_control

cp lambda_function.py lib/python3.8/site-packages/

2- Navigate to the site-packages directory to create a zip file of the dependencies:

cd lib/python3.8/site-packages
zip -r lambda-package.zip .

Note: There are multiple ways to create a zip file for the package. You can use any method you prefer or refer to AWS documentation for more information.

3- Navigate to AWS Lambda console and create a new function with the following settings:

Function name: AlexaSkill or any name you prefer
Runtime: Python 3.9
Architecture: x86_64
Role: Create a new role with basic Lambda permissions
Click Create function

Note: we will come back to add all necessary permissions to the lambda IAM role later.

4- Upload the lambda_package.zip file to the Lambda function.

5- Copy the ARN of the Lambda function. We will need it later.

2: Configuring skill with Alexa service via the Alexa Developer Console

Step 1: Configuring Alexa service

Navigate to the Alexa Developer Console and login to create a skill.
Click on the Create Skill button.

Name the skill Command Control and click Next.

For Experience, Model, Hosting service page, select options as follows:

Other for Choose a type of experience
Custom for Choose a model
Provision your own for Hosting service

Then, click on the Next button.

On Templates page, select the Start from scratch option and click on the Next button.
On Review page, click on the Create Skill button.

Step 2: Configuring skill with Alexa Service

Under CUSTOM section on the left hand side menu, click on Interaction Model. Then, click on JSON Editor tab. Copy the JSON text from this interaction model file and paste it into the JSON editor. Then, click on the Save Model button followed by the Build Model button.

Note: the JSON text is the interaction model for the skill. It's a quicker way to define the intents and utterances for the skill instead of doing it manually. To learn more about the interaction model, refer to Alexa documentation. Also, ensure that you have not received any errors while building the model.

Under CUSTOM section on the left hand side menu, click on Endpoint.
Select the AWS Lambda ARN option, which is selected by default. Then, copy Your Skill ID from the Endpoint page. The Skill ID is the unique identifier for the skill. It allows Alexa to identify the skill and to invoke the Lambda function securely. We will need it later.
We will come back to this page to add our AlexaSkill Lambda function ARN.

Step 3: Configuring Lambda function hosting Alexa skill with Alexa service

Navigate back to the AWS Lambda console and select the AlexaSkill function.
Add an Alexa trigger to the Lambda function as shown below and paste the Your Skill ID from the Endpoint page. Then, click on the Add button.
Head back to the Alexa Developer Console and add the AlexaSkill Lambda function ARN to the Endpoint page. Then, click on the Save Endpoints button. Refer to Step 2, bullet 4.

Under Configuration, select Permissions tab and click on Role name link to open the IAM role on a new tab.
Create an inline policy to allow the lambda to invoke other lambda functions. Define the policy as shown below:

Service: Lambda
Actions: InvokeFunction
Resources: arn:aws:lambda:us-east-1:123456789012:function:* (replace the account number and region with your own)

Or, you can use the following JSON to create the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "InvokeLambda",
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:us-east-1:123456789012:function:*"
        }
    ]
}

Replace the account number and region with your own

Click on the Review policy button, give the policy a name and click on the Create policy button.

Note: you may create a customer managed policy and attach it to the role instead of creating an inline policy. Please, refer to AWS documentation for more information.

3: Creating SSM document to stop EC2 instances

As previously mentioned, we will create our own SSM document to stop EC2 instances, and we will also use an Amazon managed or built-in SSM document to patch Amazon Linux 2 EC2 instances. By following this strategy, we would cover two case scenarios:

Customer created SSM document
Amazon managed SSM document

The reasoning behind this is that we may have our own SSM documents to perform certain tasks. Also, we may need to use Amazon managed or built-in SSM documents to perform other tasks.

Step 1: Customer created SSM document:

The below steps are for creating an SSM document to stop Amazon Linux 2 EC2 instances:

Navigate to AWS Systems Manager console and select Documents from the left hand side menu.
Click on the Create document button and select Command or Session from the drop-down menu.
On the Create document page, enter the following information:

Name: StopEC2Instances
Document type: Command document
Content: YAML
Content: Copy the YAML text from this file and paste it into the Content field and click on the Create document button.

Note: the SSM document works on Amazon Linux 2 instances. If you are using a different operating system, you may need to modify the SSM document to work with your operating system.

Step 2: Amazon managed SSM document:

This step is informational only. The Amazon managed document called AWS-RunPatchBaseline is used to patch EC2 instances. It works on all operating systems (Windows, Linux, and macOS). The document requires parameters to be passed to it. The following parameters are some of the parameters that can be passed to the document, but not all of them are required:

Operation: The operation to perform. Valid values: Scan, Install
RebootOption: The reboot option for the instances. Valid values: RebootIfNeeded, NoReboot
Target selection: The instances to patch. Valid values: InstanceIds, Tags
Timeout(seconds): The maximum time (in seconds) that the command can run before it is considered to have failed. The default value is 3600 seconds.
Rate control: The maximum number of instances that are allowed to run the command at the same time. You can specify a number of instances, such as 10, or a percentage of instances, such as 10%. The default value is 50.
Error threshold: The maximum number of errors allowed before the system stops sending the command to additional targets.
Output S3 bucket: The S3 bucket where the command execution details are stored.
SNS topic: The SNS topic where notifications are sent when the command status changes.
IAM role: The IAM role that allows Systems Manager to send notifications.
Event notifications: The event notifications that trigger notifications about command status changes.

The following are the parameters that we will pass to the AWS-RunPatchBaseline document via RunDocument:

Document name: AWS-RunPatchBaseline
Document version: $LATEST
Operation: Install
RebootOption: RebootIfNeeded
Target selection: Tags
Timeout(seconds): 3600
Rate control: 50
Error threshold: 0
SNS topic ARN: SSMCommandNotifications
IAM role ARN: Systems Manager IAM role
Event notifications: All
Event notification Type: Command
Comment: Alexa - AWS-RunPatchBaseline

Note: we will configure the minimum required parameters to run AWS-RunPatchBaseline document. For more information about these parameters, refer to AWS documentation.

4: Creating SNS topic to receive notifications from Systems Manager - Run Command

Navigate to AWS SNS console and click on the Create topic button.
Enter the following information:

Type: Standard
Topic name: SSMCommandNotifications
Display name: SSMCommandNotifications
Click on the Create topic button.

Capture the Topic ARN from the Topic details page. We will use this ARN later to add it as environment variable to the lambda function.

Note: this SNS topic is to receive notifications from Systems Manager - Run Command.

5: Creating IAM role to allow Systems Manager to send notifications to SNS

This service role is assumed by Systems Manager to publish notifications to the SNS topic when the SSM command status changes.

Step 1: Creating the IAM role:

Navigate to AWS IAM console and click on the Create role button.
Select AWS service from the Select type of trusted entity drop-down menu. Then, select Systems Manager from the Choose a use case for other AWS services drop-down menu and select Systems Manager again. Click on the Next: Permissions button.

Click on the Next button on the Attach permissions page. We will attach an inline policy to the role in the next step.
On the Name, review and create page, give the role a name and click on the Create role button.

Step 2: Attaching inline policy to the IAM role:

Navigate to AWS IAM console and select Roles from the left hand side menu.
Search for the role you created in the previous step and click on the role name.
Click on the Add permissions button and select Create inline policy from the drop-down menu.
Click on the JSON tab and paste the following JSON body into the Policy document field:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "arn:aws:sns:us-east-1:123456789012:SSMCommandNotifications"
        }
    ]
}

Note: replace the region, account ID, and SNS topic name with your own values. Or, you can use "arn:aws:sns:*:*:*" to allow the IAM role to send notifications to all SNS topics.

Click on the Review policy button and give the policy a name. Then, click on the Create policy button.
Capture the IAM role ARN. We will use this ARN later to add it as environment variable to the lambda function.

6: Provisioning on-demand DynamoDB table

The DynamoDB table will be used to store the commands, SSM document names and any SSM document parameters that will be used by the Master Lambda function.

Step 1: Creating DynamoDB table:

Navigate to AWS DynamoDB console and click on the Create table button.
Enter the following information:

Table name: SSMCommands
Primary key: Command (String)
Table settings: Customize settings
Table class: DynamoDB Standard
Read/write capacity settings: On-demand
Click on the Create button.

Capture the table name. We will add the name as as environment variable to the lambda function.

Step 2: Creating DynamoDB table items:

Navigate to AWS DynamoDB console and select the SSMCommands table.
Click on the Actions button and select Create item.
For the Command value field, enter shutdown.

Note: Alexa service does not support the stop command. It is a reserved word. Therefore, we will use the shutdown command to stop the EC2 instances.

Click on the Add new attribute button and select String from the dropdown menu.
For the Attribute name field, enter DocumentName and for the Value field, enter StopEC2Instances. Click on the Create item button.
Repeat steps 2-5 and create a new item with the following information:

Command: patch
DocumentName: AWS-RunPatchBaseline

Click on the Add new attribute button and select String from the dropdown menu. For the Attribute name field, enter Parameters and for the Value field, enter the following JSON:

{
   "Operation":[
      "Install"
   ],
   "RebootOption":[
      "RebootIfNeeded"
   ]
}

Click on the Create item button.

7: Creating Master Lambda function (MasterLambda) to send SSM commands to Run Command

Step 1: Creating Lambda function:

Navigate to AWS Lambda console and create a new function with the following settings:

Function name: MasterLambda or any name you prefer
Runtime: Python 3.9
Architecture: x86_64
Role: Create a new role with basic Lambda permissions
Click Create function

Note: we will revisit the lambda function IAM role to add all necessary permissions.

Copy the Python source code for the MasterLambda function and paste it into the Code source on the Lambda console. Then, click on the Deploy button.
Under Configuration tab, select Permissions and click on Role name link to open the IAM role on a new tab.
Create inline policy and copy the inline policy from this JSON file and paste it into the JSON editor. Replace the account number with your own. Then, click on the Review policy button. Give the policy a name and click on the Create policy button. The added policy will allow the lambda function to perform the following actions:

Service: SSM - Actions: SendCommand, ListCommands, DescribeInstanceInformation
Service: ec2 - Actions: DescribeInstances
Service: SNS - Actions: Publish
Service: DynamoDB - Actions: Query
Service: IAM - Actions: PassRole

Note: following AWS best practices and security principles, we are using the least privilege principle to grant the lambda function only the permissions it needs to run and to communicate other AWS services successfully. For more information, refer to AWS documentation.

Under Configuration tab, select General configuration and click on Edit button. Change the timeout to 30 seconds and click on the Save button.

Note: I have tested 30 seconds timeout and 128 MB memory and they are sufficient for this solution.

Under Configuration tab, select Environment variables and click on Edit button. Add the following environment variables:

DynamoDB_Table_Name: SSMCommands (replace the DynamoDB table name from the previous step)
SNS_Topic_ARN: arn:aws:sns:us-east-1:123456789012:SSMCommandNotifications (replace the SNS topic ARN from the previous step)
SSM_Role_ARN: arn:aws:iam::123456789012:role/SSMCommandRole (replace the IAM role ARN from the previous step)
The environment variables are used by the Master Lambda function to access the DynamoDB table and to pass on the SNS topic and the IAM role to Systems Manager.

Click on the Save button.

Step 2: Configuring communication between MasterLambda function and AlexaSkill Lambda function:

Navigate to AWS Lambda console and select the AlexaSkill function.
Under Configuration tab, select Environment variables and click on Edit button. Add the following environment variables:

MasterLambdaARN: arn:aws:lambda:us-east-1:123456789012:function:MasterLambda (replace the ARN with your own value or copy the Master Lambda function ARN from the previous step)

8: Creating Lambda function (SlackLambda) to send notifications to Slack (Optional)

Step 1: Creating Secrets Manager secret to store Slack webhook URL:

The AWS Secrets Manager is used to store the Slack webhook URL. The webhook URL is a unique URL that is used to send messages to a specific Slack channel. For more information about Slack webhooks, refer to Slack documentation.

Navigate to AWS Secrets Manager console and click on the Store a new secret button.
Enter the following information:

Secret type: Other type of secret
Key/Value pairs: select Plaintext and remove everything from the block.
Paste the full Slack webhook URL and click on the Next button.
Secret name: SlackWebhookURL
Description: Slack webhook URL
Click on the Next button and Next again. Then, click on the Store button.

Note: we are using AWS Secrets Manager to store the Slack webhook URL. For more information, refer to AWS documentation.

Step 2: Creating Slack lambda function:

Navigate to AWS Lambda console and create a new function with the following settings:

Function name: SlackLambda or any name you prefer
Runtime: Python 3.7
Architecture: x86_64
Role: Create a new role with basic Lambda permissions
Click Create function

Note: I have Python 3.7 as a runtime for this function due to the fact that the requests library is supported as part of the AWS Lambda execution environment in Python 3.7 and below. It means that we don't have to create a deployment package. The request library is not supported in Python 3.8 and above. For more information, refer to AWS blog

Copy the Python code for the SlackLambda function and paste it into the Code source on the Lambda console. Then, click on the Deploy button.
Under Configuration tab, select Permissions and click on Role name link to open the IAM role on a new tab.
Create inline policy and copy the inline policy from this JSON and paste it into the JSON editor. Then, click on the Review policy button. Give the policy a name and click on the Create policy button. The added policy is to allow the Lambda function to get the Slack webhook URL from Secrets Manager by performing the following action:

Service: secretsmanager - Actions: GetSecretValue

Under Configuration tab, select General configuration and click on Edit button. Change the timeout to 30 seconds and click on the Save button.
On the SlackLambda.py file, make sure to update the SLACK_CHANNEL variable with the Slack channel name. Then, click on the Deploy button.

SLACK_CHANNEL: the Slack channel name

If you are using a different name for the Secrets Manager secret, make sure to update the SLACK_HOOK_URL variable with the name of the secret for the Slack URL. Then, click on the Deploy button.

SLACK_HOOK_URL = boto3.client('secretsmanager').get_secret_value(SecretId='SlackWebhookURL')['SecretString']
Replace SecretId with your secret name.

Step 3: Subscribing SlackLambda function to the SNS topic:

From the AWS Lambda console, select the SlackLambda function.
Under Function overview, click on the Add trigger button.
Select SNS and again select SSMCommandNotifications topic. Then, click on the Add button.

Note: the SSMCommandNotifications topic is the SNS topic that we created in the previous section. If navigate to the SNS console, you will see that the SlackLambda function is subscribed to the SSMCommandNotifications topic.

9: Testing Alexa skill solution

Now, we have configured the Alexa skill solution, we are ready to test it. We will spin up two Amazon Linux 2 EC2 instances and then run the shutdown command to stop the first instance on Alexa Developer Console simulator. Then, we will run the patch command to patch the second instance. However, we need to tag the instances with the key-value pair first.

Instance	Tag Key	Tag Value
Instance 1	`Alexa`	`testing`
Instance 2	`Alexa`	`patching`

Note: the tags are case sensitive. The A in Alexa has to be capitalized and testing and patching have to be lower case. These are the commands and tags that we have defined in the Alexa Service. If you want to use different commands and tags, you will need to update the Alexa Service manually or update the Interaction Model JSON file.

Step 1: Tagging EC2 instances:

During the provisioning of the two EC2 instances, add the above tags to the instances accordingly. Alternatively, you can add the tags to the instances after provisioning them. However, it might take a few minutes for Systems Manager to detect the tags.

How to add tags to the EC2 instances after provisioning them:

Navigate to the EC2 console and select the first instance.
Click on the Tags tab and click on the Add/Edit tags button.
Add the following tags:

Key	Value
Alexa	testing

Repeat the same steps for the second instance and add the following tags:

Key	Value
Alexa	patching

Note: the EC2 instances should be managed by Systems Manager. To confirm that, navigate to the Systems Manager console and under Node Management, select Fleet Manager. You should see the two instances listed there.

Step 2: Testing Alexa skill:

Navigate to the Alexa Developer Console and click on the name of the skill, Command Control.
Click on the Test tab and toggle the Skill testing is enabled in to Development.

Next to the microphone icon, type open command control and click on the Enter button.
Type shutdown and testing as shown below and click on the Enter button.

If you have not tagged the EC2 instance with the key-value pair Alexa and testing, the instance is not running or the instance is not managed by Systems Manager, you will get the following error message:

I couldn't find any running instances tagged with testing. You can run a different command or say cancel to exit.

If you have tagged the EC2 instance with the key-value pair Alexa and testing, the instance is running and the instance is managed by Systems Manager, you will get the following message:

I have sent command shutdown to 1 instance tagged with testing and its current status is Pending. You will receive a Slack notification when the command starts and completes.

Example:

I had not tagged the EC2 instance with the key-value pair Alexa and testing and received a message by Alexa stating that there is no running instance tagged with testing.
Then, I tagged the EC2 instance with the key-value pair Alexa and testing and waited for a few minutes for Systems Manager to detect the tags.
Then, I ran the shutdown command and received a message by Alexa stating that the command has been sent to the instance. It takes the instance one minute to shutdown according to our StopEC2Instances SSM document as per design.
I also received a Slack notification stating that the command has been sent to the instance and the command status is InProgress and then Success when the command completed.

Now, let's run the patch command to patch the second instance. Type patch and patching and click enter. Since we have spun up the instance recently, there are no patches to install and it would not take long to complete the patching. AWS is doing a great job in keeping the Amazon Linux 2 up to date. We can follow the command status or history in the Systems Manager console. We should receive Slack notifications when the command starts and completes, if we have configured SlackLambda function and Slack webhook URL correctly.

Refer to the GitHub repo to see the video.

OmarCloud20 / alexa-runs-ssm-documents

Alexa to Run Systems Manager Documents

Introduction:

View on GitHub

Important Notes

If the simulator sits idle for 5 minutes or so, it will time out and you will need to refresh the page and start over.
To troubleshoot any issues, you can check the CloudWatch logs for the MasterLambda and AlexaSkills Lambda functions.
You can also check the Systems Manager console to see the command status and the command output by going to the Run Command section and selecting the Command history.
You can enable Your Skills in development mode on Alexa app on your mobile phone and test the skill on your mobile phone.

Conclusion

Congratulations on completing this tutorial! We have learned how to build a complete Alexa skill solution that can run SSM documents on SSM managed EC2 instances. The journey started with creating an Alexa skill hosted on an AWS Lambda function. Then we created a Master Lambda function that is triggered by the Alexa skill Lambda function. The Master Lambda function calls Systems Manager API using Boto3 SDK for Python to run SSM documents on SSM managed EC2 instances. We also created a Slack Lambda function that is subscribed to an SNS topic to send neat Slack notifications when the SSM command starts and completes. We have also learned how to use the Alexa Developer Console to test the Alexa skill.

Yes, it's time to celebrate! 🎉🎉🎉

While you are celebrating, you can also think about how you can extend or improve this solution. Maybe you can add more SSM documents and more Alexa commands, or maybe you can learn more about Alexa utterances and intents. How about we take this solution to the next level and build an Observability solution? Maybe we can use the Alexa service with Grafana or CloudWatch to monitor metrics of our environments. That's another tutorial for another day 😉

The possibilities are endless and learning is a lifelong journey. Innovation is the key to success, so keep learning and keep innovating!

Omar OmarFollow

I’m scalable, highly available and reliable engineer. I strongly believe in education and hands-on experience. Learning is a lifelong journey and innovation is the key to success.

References

OmarCloud20 / alexa-runs-ssm-documents

Alexa to Run Systems Manager Documents

Introduction:

View on GitHub

CDK for Terraform (CDKTF) on AWS: How to Configure an S3 Remote Backend and Deploy a Lambda Function URL using Python

Omar Omar — Tue, 27 Sep 2022 13:18:35 +0000

Update (10/12/22):

On the 3rd of Oct, 2022, HashiCorp released CDKTF version 0.13 introducing breaking changes such as namespaces. Due to this latest major upgrade, this tutorial does not work with CDKTF version 0.13. But it still works with CDKTF version 0.12. I will update the tutorial to work with version 0.13 in the near future.

Introduction
Step 1: Required Prerequisites
Step 2: Initializing First CDKTF Project using Python Template
Step 3: Configuring an S3 Remote Backend
- Option 1: Utilize an existing S3 bucket and DynamoDB to configure the S3 Remote Backend
- Option 2: Create an S3 Bucket and DynamoDB Table for the S3 Remote Backend using CDKTF
Step 4: Learn How to Use Construct Hub and AWS Provider Submodules
- Scenario 1: S3 Bucket
- Scenario 2: ECS Cluster
- CDKTF Commands
Step 5: Deploying a Lambda Function URL using CDKTF
Conclusion

Introduction

After two years of development, on the 1st of August 2022, HashiCorp announced the general availability of CDK for Terraform. As the CDKTF framework finally saw the light of day, the news triggered lots excitement among the community. The CDKTF framework is a new open-source project that enables developers to use their favorite programming languages to define and provision cloud infrastructure resources on AWS. Under the hood, it converts the programming language definitions into Terraform configuration files and uses Terraform to provision the resources.

"Learning is no longer a stage of life; it’s a lifelong journey" - Andy Bird

It's often said that the best way to learn something new is to do it, and the first milestone in learning is often the hardest. However, with the right guidance, you can overcome the challenges and achieve your goals. My objective in this article is to guide you through the process of installing, configuring, and deploying your first AWS resource using CDKTF on AWS. In addition, I will also show you how to use the documentation on Construct Hub to deploy your own AWS resources.

The tutorial should be easy to follow and understand for beginners and intermediate users. The tutorial is devoted to developers with adequate AWS, Terraform and Python knowledge but who are unsure of how and where to begin their CDKTF learning.

The main topics covered in this tutorial are:

Proper installation and configuration of all required prerequisites.
Installing and configuring CDKTF.
Initializing first CDKTF project using Python template and local backend.
Deploying an S3 bucket, DynamoDB table and configuring an S3 remote backend.
Learning how to read/use AWS Provider Submodules and Construct Hub documentation
Provisioning an S3 Bucket using CDKTF
Provisioning an IAM role and Lambda Function using CDKTF

This tutorial is also available on my GitHub - CDKTF-Tutorial repo. By the end of this tutorial, you shall be comfortable deploying AWS resources using the CDKTF framework. I will pass you the learning baton and you can take it from there.
Enough said, let's get started.

Step 1: Required Prerequisites

To complete this tutorial successfully, we should install and configure the following prerequisites properly. To set you up for success, I have to ensure you have the following prerequisites installed and configured properly:

AWS CLI version 2:

Follow AWS Prerequisites to use the AWS CLI version 2 documentation for all required prerequisites.
Follow Installing or updating the latest version of the AWS CLI document to install the AWS CLI v2 as per your local device architecture. In my case, it's an Ubuntu 20.04 LTS OS running on a Linux x86 (64-bit) architecture.
Lastly, follow the Configuration basics - Profiles document to configure the AWS CLI v2 and create a named profile (name it CDKFT as shown below). We will use the named profile to configure AWS Provider credentials later on.



aws configure --profile CDKTF

Note: the profile name does not have to be CDKTF, you can name it however you like. However, I will be using CDKTF in this tutorial.

Terraform (version v1.0+ as per Terraform's recommendation).
Node.js and npm v16+:

Follow NodeSource Node.js Binary Distributions to install Node.js v16.x for Ubuntu (the installation includes npm). For other operating systems and architectures, refer to Node.js official page. Once you have Node.js installed, make sure it's version 16.x by running the below command:



node -v

Python 3.7+: Ubuntu 20.04 LTS distribution comes with Python 3.8.2 pre-installed by default. Run the below command to confirm:



python3 --version

Additionally, we will need to make sure we have pip installed and available.



pip3 --version

Note: if pip is unavailable, run sudo apt install python3-pip to install it as per Python documentation.

pipenv v2021.5.29+: as of Sept 22, 2022, the latest version of pipenv is 2022.9.20. We will use pip to install pipenv.



pip3 install pipenv

Note: if you receive a WARNING stating pipenv is not installed on PATH as shown on the below image, run the below command to add it to the path:



export PATH="the-path-mentioned-in-the-warning:$PATH"

Actual example:



export PATH="/home/CDKTF/.local/bin:$PATH"

We have to confirm that pipenv is installed and available before we proceed. Run the below command to confirm:



pipenv --version

Note: it's tempting to install pipenv by using the package manager sudo apt install pipenv, but be aware that the system repository version of pipenv is outdated and will not work with the CDKTF framework.

CDKFT: now, we are ready to install the CDKTF using npm:



npm install --global cdktf-cli@0.12.3

If you receive a permission denied error, use sudo as shown below:



sudo npm install --global cdktf-cli@0.12.3

Let's confirm the version:



cdktf --version

Make sure you have all required prerequisite versions installed and configured successfully as shown on the below image:

Reaching this point means you have all required prerequisites installed and configured properly. We are ready to move on to the next step. This is a milestone, so take a moment to celebrate it. You deserve it.

Step 2: Initializing the First CDKTF Project using Python Template

In this section, we will learn how to use CDKTF CLI commands to create our first AWS CDKTF Python project. We will use the cdktf init command to create a new project in the current directory. The command also creates a cdktf.json file that contains the project configuration. The cdktf.json file contains e.g. the programming language, and the Terraform provider. The cdktf.json file is used by the cdktf command to determine the project configuration. Follow the below steps to initialize the first CDKTF project using Python template:

Create a new directory for the project and cd into the directory. I will create a directory on my Desktop and name it first_project:



mkdir first_project && cd first_project

Run the following command to initialize our first CDKTF project. We will be promoted to enter the following information:

A. Name of the project: leave it as first_project and hit Enter.

B. Project description: leave it as My first CDKTF project and hit Enter.

C. Send crash reports to CDKTF team. I would highly recommend you say yes to send crash reports to the CDKTF team. This will help improve the product and fix bugs.



cdktf init --template="python" --local

Note 1: we are using the --local flag to store our Terraform state file locally. We will reconfigure the backend to S3 Remote backend in the next section.

Note 2: if you receive error [ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'], you would need to remove virtualenv by running sudo apt remove python3-virtualenv. We should still have virtualenv as part of the pip packages. Run pip3 show virtualenv to confirm. If you don't see virtualenv in the list, run pip3 install virtualenv to install it.

Activate the project's virtual environment (optional but recommended):



pipenv shell

Note: the purpose of creating a virtual environment is to isolate the project and and all its packages and dependencies from the host or local device. It's a self contained environment within the host to prevent polluting the system. It's highly recommended to activate it to keep your host healthy and clean.

Install AWS provider. There are multiple ways to install AWS Provider. We will use pipenv to install the AWS provider. Run the below command to install the AWS provider:



pipenv install cdktf-cdktf-provider-aws

Note: as of the 26th of Sept, 2022, if you decide to install the AWS Provider using cdktf provider add "aws@~>4.0", the installation will fail due to no matching distribution found for version v9.0.36. There are other methods of importing a provider but this tutorial won't discuss to focus on simplicity.

Congratulations, you have successfully initialized your first CDKTF Python project. This is another milestone to celebrate. Get a cup of coffee ☕️ and let's move on to the next section.

Step 3: Configuring an S3 Remote Backend

Terraform stores all managed infrastructure and configuration by default in a file named terraform.tfstate. If a local backend is configured for the project, the state file is stored in the current working directly. However, when working in a team environment to collaborate with other team members, it is important to configure a remote backend. There are several remote backend options such as, consul, etcd, gcs, http and s3. For a full list of remote backends, refer to Terraform documentation.

For this tutorial, we will configure an S3 Remote Backend which includes an S3 bucket for storing the state file and a DynamoDB table for state locking and consistency checking.

Select one of the following options to configure the S3 Remote Backend:

Option 1: Utilize an existing S3 bucket and DynamoDB to configure the S3 Remote Backend.

From Step 2, while we still have the virtual environment activated, let's open the project directory using our choice of an Integrated Development Environment (IDE). In my case, I'm using Visual Studio Code which I downloaded from the Ubuntu Software Store. Run code . on the terminal to open the project directory via VS Code.
Navigate to main.py file and add the AWS provider construct to the imports section:



from cdktf_cdktf_provider_aws import AwsProvider

Note: the final main.py code will be provided at the end of this section.

Configure the AWS provider by adding the following code to MyStack class:



AwsProvider(self, "AWS", region="us-east-1", profile="CDKTF")

Note: we are using the profile attribute to specify the AWS profile to use. This is the AWS CLI profile we have previously discussed and created in the Required Prerequisites section. If you don't have a profile created, you can remove the profile attribute and the AWS provider will use the default profile. Or, you can use a different authentication method as per the AWS Provider documentation.

Add S3Backend class to the other imported classes. The S3Backend class is employing the S3 bucket and DynamoDB table as an S3 remote backend.



from cdktf import App, TerraformStack, S3Backend

Add the S3 Backend construct to the main.py file. We will add the following S3 Backend configurations to the MyStack class:

bucket - the name of the S3 bucket to store the state file. The bucket must exist and be in the same region as the stack. If the bucket doesn't exist, the stack will fail to deploy.
key - the name of the state file and its path. The default value is terraform.tfstate.
encrypt - whether to encrypt the state file using server-side encryption with AWS KMS. The default value is true.
region - the region of the S3 bucket and DynamoDB table. The default value is us-east-1.
dynamodb_table - the name of the DynamoDB table to use for state locking and consistency checking. The table must exist and be in the same region as the stack. If the table doesn't exist, the stack will fail to deploy.
profile - the AWS CLI profile to use. The default value is default. But, we have already configured the AWS provider to use the CDKTF profile.

Here is how the S3 Backend construct will look like:



        S3Backend(self,
        bucket="cdktf-remote-backend",
        key="first_project/terraform.tfstate",
        encrypt=True,
        region="us-east-1",
        dynamodb_table="cdktf-remote-backend-lock",
        profile="CDKTF",
        )

The final main.py file should look like this:



from constructs import Construct
from cdktf import App, TerraformStack, S3Backend
from cdktf_cdktf_provider_aws import AwsProvider

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        AwsProvider(self, "AWS", region="us-east-1", profile="CDKTF")

        S3Backend(self,
        bucket="cdktf-remote-backend",
        key="first_project/terraform.tfstate",
        encrypt=True,
        region="us-east-1",
        dynamodb_table="cdktf-remote-backend-lock",
        profile="CDKTF",
        )

        # define resources here


app = App()
MyStack(app, "first_project")

app.synth()

Run cdktf synth to generate the Terraform configuration files. The Terraform configuration files will be generated in the cdktf.out directory. ~~The synth command will fail if the S3 bucket and DynamoDB table don't exist.~~



cdktf synth

You have successfully configured the S3 Remote Backend. Let's move on to the next section.

To test the configuration of the S3 remote backend, follow the below steps to create an S3 bucket resource and deploy the stack.

Import the S3 bucket library to the main.py file:



from cdktf_cdktf_provider_aws import AwsProvider, s3

Add the S3 bucket resource to the MyStack class:



        my_bucket = s3.S3Bucket(self, "my_bucket",
        bucket="Name-of-the-bucket",
        )

Replace Name-of-the-bucket with the name of the bucket you want to create. Note, S3 bucket names must be unique across all existing bucket names in Amazon S3.
Run cdktf deploy to deploy the stack and create the S3 bucket.



cdktf deploy

Note: if you get Incomplete lock file information for providers warning, you can either ignore it or you can run terraform providers lock -platform=linux_amd64 from the project root directory to validate the lock file. For more information, refer to Terraform documentation.

Congratulations, you have successfully configured an S3 remote backend and created an S3 bucket using CDKTF. It's time to take a break and stretch your legs.

Option 2: Create an S3 Bucket and DynamoDB Table for the S3 Remote Backend using CDKTF

For this option, we will take a different approach. We will create the S3 bucket and DynamoDB table using CDKTF and then configure the S3 remote backend to use the newly created resources. Follow the below steps to create the S3 bucket and DynamoDB table:

Open the project directory using your preferred IDE. If you are using VS Code, cd into the project folder and run code . on the terminal to open the project directory using VS Code.
Navigate to the main.py file and replace the default code with the following. This code creates an S3 bucket and DynamoDB table for the S3 remote backend.



from constructs import Construct
from cdktf import App, TerraformStack, S3Backend
from cdktf_cdktf_provider_aws import AwsProvider, s3, dynamodb

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        AwsProvider(self, "AWS", region="us-east-1", profile="CDKTF")

        # define resources here
        s3_backend_bucket = s3.S3Bucket(self,
        "s3_backend_bucket",
        bucket="cdktf-remote-backend-2",
        )
        dynamodb_lock_table = dynamodb.DynamodbTable(self, 
        "dynamodb_lock_table",
        name="cdktf-remote-backend-lock-2",
        billing_mode="PAY_PER_REQUEST",
        attribute=[
            {
                "name": "LockID",
                "type": "S"
            }
        ],
        hash_key="LockID",
        )



app = App()
MyStack(app, "first_project")

app.synth()

Note, if the S3 bucket and DynamoDB table already exist, an error will be thrown. The S3 bucket names must be globally unique across all existing bucket names in Amazon S3 and DynamoDB table names must be unique within an AWS account. If you get an error, you can change the bucket and table names to unique names.

Run cdktf deploy to deploy the stack and create the S3 bucket and DynamoDB table.



cdktf deploy

Note: if you get Incomplete lock file information for providers warning, you can either ignore it or you can run terraform providers lock -platform=linux_amd64 from the project root directory to validate the lock file.

Now, we will configure the S3 remote backend to use the newly created S3 bucket and DynamoDB table. Open the main.py file and replace the code with the following. This code configures the S3 remote backend to use the newly created S3 bucket and DynamoDB table. Make sure to replace the bucket and dynamodb_table values with the names of the S3 bucket and DynamoDB table you created in the previous step.



from constructs import Construct
from cdktf import App, TerraformStack, S3Backend
from cdktf_cdktf_provider_aws import AwsProvider, s3, dynamodb

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        AwsProvider(self, "AWS", region="us-east-1", profile="CDKTF")

        #S3 Remote Backend
        S3Backend(self,
        bucket="cdktf-remote-backend-2",
        key="first_project/terraform.tfstate",
        encrypt=True,
        region="us-east-1",
        dynamodb_table="cdktf-remote-backend-lock-2",
        profile="CDKTF",
        )

        # Resources
        s3_backend_bucket = s3.S3Bucket(self, "s3_backend_bucket",
        bucket="cdktf-remote-backend-2",
        )

        dynamodb_lock_table = dynamodb.DynamodbTable(self, "dynamodb_lock_table",
        name="cdktf-remote-backend-lock-2",
        billing_mode="PAY_PER_REQUEST",
        attribute=[
            {
                "name": "LockID",
                "type": "S"
            }
        ],
        hash_key="LockID",
        )



app = App()
MyStack(app, "first_project")

app.synth()

Run cdktf synth to generate the Terraform configuration files. The Terraform configuration files will be generated in the cdktf.out directory.



cdktf synth

To migrate the local state backend to an S3 remote backend, navigate to the cdktf.out/stacks/first_project directory and run the following command to start the migration process. The first_project is the name of the project. If you have named your project differently, navigate to the cdktf.out/stacks/<project_name> directory and run the command.

Note: before running this command, read Important Notes below



cd cdktf.out/stacks/first_project



terraform init --migrate-state

Important Notes:

When you run terraform init --migrate-state, Terraform prompts you to answer the following question:

Do you want to copy existing state to the new backend?

A. If you enter, yes to migrate the state file to the S3 backend, CDKTF will manage the S3 remote backend (S3 bucket and DynamoDB table) for you. Therefore, if you delete the stack, the S3 bucket and DynamoDB table will be vurnerable to deletion. Note, we can't delete a non-empty S3 unless we add force_destroy=True to the S3 bucket configuration. This option is not recommended if you want to keep the S3 bucket and DynamoDB table, especially if you are using the S3 bucket and DynamoDB table as a remote backend for other Terraform projects. But, if you are just experimenting with CDKTF, this option is fine.

B. If you enter no, to migrate the state file to the S3 backend, CDKTF will not manage the S3 bucket and DynamoDB table. If you delete the stack, the S3 bucket and DynamoDB table will not be deleted and you will have to manually delete the S3 bucket and DynamoDB table. Moreover, you will also need to remove the S3 bucket and DynamoDB table constructs from the main.py file.

To read more about initializing remote backend manually, refer to the Terraform documentation.

The below terminal recording demonstrates the steps above. In the recording, I have shown the error message that you may get if you attempt to run cdktf deploy prior to reconfiguring from local backend to an S3 remote backend.
I have entered yes to migrate the state file to the S3 remote backend and let the CDKTF manage the S3 bucket and DynamoDB table.

Run cdktf diff from the project root directory to compare the current state of the stack with the desired state of the stack. The output should be empty, which means there are no changes to be made and the state file is up to date.



cdktf diff

Note: you have to be in the project root directory to run cdktf diff.

Great! You have successfully migrated the local state backend to an S3 remote backend. Way to go, you have achieved another milestone! 🎉🎉🎉

Step 4: Learn How to Use Construct Hub and AWS Provider Submodules

Prior to digging into the AWS provider, let's first understand most commonly used terms in the CDKTF documentation:

Submodules is a collection of related resources. For example, the s3.S3Bucket construct is part of the s3 submodule. The s3.S3Bucket construct creates an S3 bucket. The s3 submodule contains other constructs such as s3.S3BucketPolicy, s3.S3BucketAcl, s3.S3BucketObject, etc.
Construct is another important term to understand. A construct is a class that represents a Terraform resource, data source, or provider. The s3 submodule contains classes that represent S3 constructs, classes and structs.
Stack is a collection of constructs. The MyStack class in the main.py file is a stack. The MyStack class in the main.py file is a stack. The MyStack class contains constructs that represent e.g. Terraform resources, data sources, and providers.

Scenario 1: S3 Bucket

Let's say we would like to create an S3 bucket but we don't know which construct to use. Let's head to the Python Construct Hub for the AWS provider and follow the steps below:

From the left hand side and under Documentation, click on Choose Submodule.
In the search box, type in s3 and then click on the result, which is s3.
Under Submodule:s3, you will see a list of Constructs and Structs. Click on S3Bucket
To create an S3 bucket, you will import the s3 submodule as shown under Initializers.
The construct to use is s3.S3Bucket.
We need to find the required configurations for the s3.S3Bucket construct. Scan the page and look for configurations marked Required. In this case, S3 bucket does not have any required configuration, not even a name. If you leave the name argument empty, the S3 bucket will be created with a random name. We can specify a name for the S3 bucket and other configurations, but this is not required.

We have to distinguish between required and optional configurations. Required configurations must be specified when creating a resource. Optional configurations can be specified when creating a resource.

This is the code snippet for creating an S3 bucket with minimal configurations:



my_3bucket= s3.S3Bucket(self, "s3_bucket")

Note: if you leave the name argument empty, the S3 bucket will be created with a random name.

Scenario 2: ECS Cluster

This time let's say we would like to create an ECS cluster. Let's head to the Python Construct Hub for the AWS provider and follow the steps below:

From the left hand side and under Documentation, click on Choose Submodule.
In the search box, type in ecs and then click on the result, which is ecs.
Under Submodule:ecs, you will see a list of Constructs and Structs. Click on EcsCluster
To create an ECS cluster, you will import ecs class as shown under Initializers.
The construct to use is ecs.EcsCluster as shown below.
We need to find the required configurations for the ecs.EcsCluster construct. The minimum required configurations to create an ECS cluster is just the name of the cluster. But, we can also specify other configurations such as, capacity_providers, default_capacity_provider_strategy, configuration, etc.



my_ecs_cluster = ecs.EcsCluster(self, "my_ecs_cluster",
name = "My_Cluster"
)

CDKTF Commands:

There are several CDKTF commands that we need to be familiar with. The below table shows the commands, their descriptions and the corresponding Terraform commands.

Commands	Description	Aliases
cdktf init	Create a new cdktf project from a template.
cdktf get	Generate CDK Constructs for Terraform providers and modules.
cdktf convert	Converts a single file of HCL configuration to CDK for Terraform. Takes the file to be converted on stdin.
cdktf deploy	Deploy the given stacks	[aliases: apply]
cdktf destroy	Destroy the given stacks
cdktf diff	Perform a diff (terraform plan) for the given stack	[aliases: plan]
cdktf list	List stacks in app.
cdktf login	Retrieves an API token to connect to Terraform Cloud or Terraform Enterprise.
cdktf synth	Synthesizes Terraform code for the given app in a directory.	[aliases: synthesize]
cdktf watch	[experimental] Watch for file changes and automatically trigger a deploy
cdktf output	Prints the output of stacks	[aliases: outputs]
cdktf debug	Get debug information about the current project and environment
cdktf provider	A set of subcommands that facilitates provider management
cdktf completion	generate completion script

To find out more about the cdktf commands, run cdktf [command] --help and replace [command] with the command you want to learn more about.

For example, to learn more about the cdktf deploy command, run cdktf deploy --help.

Step 5: Deploying a Lambda Function URL using CDKTF

CDKTF is a great tool to provision AWS resources. We have already created an S3 bucket and DynamoDB table in the previous section. In this section, I will show you how to create a Lambda function using CDKTF with function url enabled. The lambda will host a simple static web page, and well configure the function url as an output. The process requires creating an IAM role and attaching a policy to the role. I will also introduce you to multiple concepts in CDKTF.

In this section, I will cover the following topics:

How to create an IAM role for Lambda function
How to attach a policy to the IAM role
How to create a Lambda function
How to enable function url for the Lambda function
How to package a Lambda function from a local directory and python file
How to create an output for the Lambda function url

Buckle up, we are going to learn a lot in this section! 🚀🚀🚀

Follow the steps below to deploy a Lambda function URL using CDKTF:

Firstly, let's keep out project organized and create a new directory called lambda in the root directory of the project. This is where we will store our Lambda function code.
Create a new file called lambda_function.py in the lambda directory.
Copy the lambda_function.py code from my GitHub repository and paste it into the lambda_function.py file.
I will go over the main.py code and the final main.py file will provided at the end of the section.

In the main.py file, import the TerraformOutput, TerraformAsset and AssetType classes from the cdktf module. The Asset construct was introduced in CDK for Terraform v0.4+ and is used to package our local directory and python file into a zip file. The TerraformOutput construct is used to create an output for the Lambda function url. The AssetType is used to specify the type of asset.

The final import statements should look like this:



from constructs import Construct
from cdktf import App, TerraformStack, S3Backend, TerraformOutput, TerraformAsset, AssetType
from cdktf_cdktf_provider_aws import AwsProvider, s3, dynamodb, iam, lambdafunction
import os
import os.path as Path

Note, we have imported os and os.path as Path modules. We will use these modules to get the current working directory and to join the path to the lambda directory.

The TerraformAsset construct requires a path argument. The path argument is the path to the directory or file that you want to package. In this case, we will use the os module to get the current working directory and then join the path to the lambda directory. The AssetType.ARCHIVE is to specify that the asset should produce an archive. The final path argument should look like this:



asset = TerraformAsset(self, "lambda_file",
path = Path.join(os.getcwd(), "lambda"),
type = AssetType.ARCHIVE,
)

Creating a Lambda function requires creating an IAM role. Therefore, we will create an IAM role first. We will also attach the AWSLambdaBasicExecutionRole AWS managed policy to the IAM role. This policy allows the Lambda function to write logs to CloudWatch. Refer to AWS documentation for more information about the AWSLambdaBasicExecutionRole policy.

The assume_role_policy argument is the policy that grants permission to assume the IAM role, which requires a JSON string. The JSON string is the policy document that grants permission to assume the IAM role. The final snippet of code should look like this:



        lambda_role = iam.IamRole(self, "lambda_role",
        name="my-lambda-url-role",
        managed_policy_arns=[
            "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ],
        assume_role_policy="""{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Effect": "Allow",
                    "Sid": ""
                }
            ]
        }""",
        )

Now, we are ready to create a lambda function. The handler configuration is the name of the Python file that contains the lambda function. The runtime provides a language-specific environment that runs in an execution environment. The source_code_hash argument is the hash of the file that contains the lambda function. The source_code_hash argument is required to trigger a new deployment when the lambda function code changes. The filename argument is the path to the file that contains the lambda function.

The final snippet of code should look like this:



        my_lambda = lambdafunction.LambdaFunction(self, "my_lambda",
        function_name="my-lambda-url",
        handler="lambda_function.lambda_handler",
        role=lambda_role.arn,
        runtime="python3.9",
        source_code_hash = asset.asset_hash,
        filename=asset.path,
        )

We need to enable function url for the lambda function and define an authorization_type for the function url. The authorization_type argument is the type of authorization that is used to invoke the function url. The authorization_type argument can be set to NONE or AWS_IAM. The final snippet of code should look like this:



        my_lambda_url = lambdafunction.LambdaFunctionUrl(self, "my_lambda_url",
        function_name=my_lambda.function_name,
        authorization_type="NONE",
        )

Finally, we need to create an output for the Lambda function url. The value argument is the value of the output. The value argument can be a string, number, boolean, or a list. The final snippet of code should look like this:



        TerraformOutput(self, "lambda_url",
        value=my_lambda_url.invoke_url,
        )

The final main.py code should look like this:



from constructs import Construct
from cdktf import App, TerraformStack, S3Backend, TerraformOutput, TerraformAsset, AssetType
from cdktf_cdktf_provider_aws import AwsProvider, s3, dynamodb, iam, lambdafunction
import os
import os.path as Path

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str):
        super().__init__(scope, ns)

        AwsProvider(self, "AWS", region="us-east-1", profile="CDKTF")

        #S3 Remote Backend
        S3Backend(self,
        bucket="cdktf-remote-backend-2",
        key="first_project/terraform.tfstate",
        encrypt=True,
        region="us-east-1",
        dynamodb_table="cdktf-remote-backend-lock-2",
        profile="CDKTF",
        )

        # Resources
        s3_backend_bucket = s3.S3Bucket(self, "s3_backend_bucket",
        bucket="cdktf-remote-backend-2",
        )

        dynamodb_lock_table = dynamodb.DynamodbTable(self, "dynamodb_lock_table",
        name="cdktf-remote-backend-lock-2",
        billing_mode="PAY_PER_REQUEST",
        attribute=[
            {
                "name": "LockID",
                "type": "S"
            }
        ],
        hash_key="LockID",
        )

        # Asset for Lambda Function
        asset = TerraformAsset(self, "lambda_file",
        path = Path.join(os.getcwd(), "lambda"),
        type = AssetType.ARCHIVE,
        )

        # IAM Role for Lambda Function
        lambda_role = iam.IamRole(self, "lambda_role",
        name="my-lambda-url-role",
        managed_policy_arns=[
            "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
        ],
        assume_role_policy="""{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Effect": "Allow",
                    "Sid": ""
                }
            ]
        }""",
        )

        # Lambda Function
        my_lambda = lambdafunction.LambdaFunction(self, "my_lambda",
        function_name="my-lambda-url",
        handler="lambda_function.lambda_handler",
        role=lambda_role.arn,
        runtime="python3.9",
        source_code_hash = asset.asset_hash,
        filename=asset.path,
        )

        # Lambda Function Url
        my_lambda_url = lambdafunction.LambdaFunctionUrl(self, "my_lambda_url",
        function_name=my_lambda.function_name,
        authorization_type="NONE",
        )



        # Outputs for Lambda Function Url
        TerraformOutput(self, "lambda_url",
        value=my_lambda_url.function_url,
        )


app = App()
MyStack(app, "first_project")

app.synth()

Run cdktf deploy to deploy the stack. ```

cdktf deploy


>Note: you can also run `cdktf deploy --auto-approve` to deploy the stack without confirmation. However, I would suggest to refrain from using this option in production unless you are absolutely sure that you want to deploy the stack without confirmation.

Finally, grab the `lambda_url` output and paste it in your browser. If you see the dancing bananas, then you have successfully deployed your first lambda function using CDKTF.



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s58zzzwtfkb32h251l0u.png)


<br>

Congratulations! You have successfully deployed a lambda function with a function url using CDKTF. I'm sure you feel like you are on top of the world right now. Well done!

---


**To delete the stack, we need to follow the below steps:**

Note, we chose to allow CDKTF to manage the remote backend. This means that CDKTF will delete the remote backend (the S3 bucket and DynamoDB Table) when we delete the stack. There are many methods to delete the stack, but I find the below method to be the easiest. Let's go through the steps:


A. Add `force_destroy=True` to the `s3_backend_bucket` configurations. The S3 bucket cannot be deleted if it is not empty. This is the reason why we need to add `force_destroy=True`.

    s3_backend_bucket = s3.S3Bucket(self, "s3_backend_bucket",
    bucket="cdktf-remote-backend-2",
    force_destroy=True
    )


B. Run `cdktf deploy` to update the S3 bucket configurations:

cdktf deploy


C. Run `cdktf destroy` to delete the entire stack:

cdktf destroy


>Note: after running `cdktf destroy`, you will get an error message saying `failed to retrieve lock info`. This is expected due to the fact the dynamodb table is deleted. You can ignore this error message.

---


## Conclusion

In this tutorial, we have learned how to properly install and configure CDKTF, how to migrate from a local backend to an S3 remote backend. We have also learned how to deploy a lambda function with a function url using CDKTF. We have also briefly learned how to read and utilize the CDKTF documentation from the Construct Hub.

The most important thing to remember is that CDKTF is still in its early stages. I am confident that CDKTF will be a great tool for managing Terraform stacks in the near future.


Congratulations on completing this tutorial and overcoming several challenges. You have achieved many learning milestones. I hope this tutorial added value to your learning journey. Thank you for reading! 

<br>

**Omar A Omar**
Site Reliability Engineer
AWS Community Builder

Building a Patching Model using AWS Systems Manager - Patch Manager for Mutable Infrastructure

Omar Omar — Fri, 08 Jul 2022 14:24:59 +0000

Building a Patching Model using AWS Systems Manager - Patch Manager for Mutable Infrastructure
- AWS Systems Manager Patch Manager
  - What is the Patch Baseline?
  - How Patch Baseline Rules Work on Amazon Linux 2
- Patching Model Solution Architecture
  - Architecture Diagram
- How to Create a Patching Model for a Single Managed Amazon Linux 2 EC2 Instance - Step by Step
  - Prerequisites
  - How to Create SSM Documents
  - Create IAM Service Role for Maintenance Window
    - Create a policy
    - Create IAM resource role
  - Create and add inline policies to the EC2 IAM instance profile
  - Step 1: Create a Custom Patch Baseline
  - Step 2: Tagging the EC2 Instance with the Patch Group Key-Value
  - Step 3: Assigning a Patch Group to the Patch Baseline
  - Step 4: Creating a Maintenance Window
  - Step 5: Registering Targets to the Maintenance Window
  - Step 6: Assigning Tasks to the Maintenance Window
- Conclusion

July 8th, 2022

AWS Systems Manager Patch Manager

Patch Manager is a capability of AWS Systems Manager. It applies and automates the patching process of managed nodes for both security related and other types of updates, which makes it a powerful tool for mutable infrastructure model. It's capable of patching operating systems as well as applications. With the use of patch baseline, Patch Manager includes rules for auto-approving patches and for creating a list of approved or rejected patches. Patches can be installed individually or to a large groups of managed instances using tags.
To schedule patching, Patch Manager runs tasks of Maintenance Window, which is another capability of Systems Manager. Although using Patch Manager is not the only patching option, it is one of the most straightforward and practical approaches. This patching model tutorial is to provide a proof of concept for patching a single managed Amazon Linux 2 instance. The tutorial's goal is to equip cloud teams the know-how they need to start using the Patch Manager. As a result, the tutorial's expertise and knowledge are to be applied to a fleet of managed instances or on-premises servers.

Note: Patch Manager doesn't support upgrading major versions of operating systems.

What is the Patch Baseline?

Patch Manager provides predefined baselines for each supported operating systems. The service uses the native package manager to drive the installation of patches approved by the patch baseline. For Amazon Linux 2, the predefined baseline is to approve all operating systems patches classified as Security and have a severity level of Critical or Important. The patches are auto-approved 7 days after release. Moreover, all patches classified Bugfix are also auto-approved 7 days after release. The predefined patch baseline are not customizable. However, it's feasible to create a custom patch baseline to control patch classifications, approval/rejection and auto-approve days after release.

How Patch Baseline Rules Work on Amazon Linux 2

Below is the guidelines for the packages selected for update:

Security option	Equivalent Yum Command
Pre-defined default patch baselines provided by Amazon (non-security updates option is not selected)	`sudo yum update-minimal --sec-severity=critical,important --bugfix -y`
User Custom patch baselines (non-security updates option is selected)	`sudo yum update --security --bugfix -y`

Please, refer to AWS documentation for further information on how security patches are selected.

Patching Model Solution Architecture

The plan is to utilize Patch Manager to develop a patching model for a single managed Amazon Linux 2 EC2 instance by using tags. Based on a scheduled maintenance window, an SSM Agent, installed on the EC2 instance, receives command issued to commence a patching process. The agent validates the instance's Patch Group tag-value and queries Patch Manager for an associated patch baseline. Once Patch Manager confirms the patch baseline for the Patch Group tag-value, it notifies the SSM Agent to retrieve the patch baseline snapshot. Finally, the SSM Agent begins scanning and installing patches based on the rules defined in the patch baseline snapshot provided by Patch Manager.

TheAWS-RunPatchBaselineWithHooks SSM document will be used to orchestrate multi-step installing patches. It offers three optional hooks which allows running SSM documents at three points during the patching cycle (pre-install, post-patch and post-reboot). We will create three simple SSM documents, as a proof of concept, to be used during the patching cycle. To read more about AWS-RunPatchBaselineWithHooks SSM document , please refer to AWS blog.

Kernel Live Patching is another feature available for Amazon Linux 2. It allows applying patches without the need for an immediate reboot or any disruption to running applications. We will not be using this feature for our patching model but I would like to recommend considering it. For more information about Kernel Live Patching, please refer to AWS documentation. Please, bear in mind in case there is a need to run patching outside of the maintenance window for a fleet of instances, it's recommend by AWS, as best practice, to provide a Snapshot-ID. It ensures consistency among the targeted instances. Please, refer to AWS documentation for more information. In this patching model, we have allocated a Patch Group tag per instance; therefore, no snapshot id is needed for patching outside the maintenance window.

Note: it's highly recommended to test the patching model in a development environment prior to deploying to production. Also, for multi-account and multi-region patching, please refer to AW blog for more information.

Architecture Diagram - Patching Model using Patch Manager

This patching model should work as a proof of concept. A similar concept to this architecture can be applied to fleets of managed EC2 instances and on-premises servers.

Note: the lambda functions to Slack channel and to PagerDuty are not included in this tutorial.

How to Create Patching Model for Single Managed Amazon Linux 2 EC2 Instance - Step by Step

Prerequisites

SSM Agent version 3.0.502 or later (requirement for the AWS-RunPatchBaselineWithHooks SSM document)
Internet connectivity. The managed instance must have access to the source patch repositories.
Minimum operating system - Amazon Linux 2 2 - 2.0
IAM service role for Systems Manager to run Maintenance Window tasks
Optionally, adding policies to the IAM service role for SNS and CloudWatch logs
Optionally, a preconfigured S3 bucket to receive the patching command logs
Optionally, a preconfigured SNS topic for patching event notifications
Optionally, adding S3 and CloudWatch logs policies to the EC2 instance profile role

Step 1: Create a Custom Patch Baseline

Navigate to AWS Systems Manager console. Under Node Management, select Patch Manager.

Image 1

Select View predefined patch baselines and click on Create patch baseline. The, fill out the requirements as following:
1. Under Patch baseline details:
  1. Name: give the custom baseline a name such as, AmazonLinux2AllPatchesBaseline
  2. Description: optionally, add a description for the baseline such as, Amazon Linux2 All Patches Baseline
  3. Operating system: from the dropdown menu, select Amazon Linux 2
2. Under Approval rule for operating systems:
  1. Product: select All
  2. Classification: All
  3. Severity: All
  4. Auto-approval: leave the default selected option, Approve patches after a specified number of days
  5. Specify the number of days: leave the default 0 days
  6. Compliance reporting - optional: leave the default selected option, Unspecified
  7. Include nonsecurity updates: check the box to install nonsecurity patches
3. Under Patch exceptions:
  1. Rejected patches - optional: add system-release.* as shown below.
  2. Rejected patches action - optional: select Block
  Note: the purpose of this block is to reject patches to new Amazon Linux releases beyond the Patch Manager supported operating systems.
4. Under Patch sources: we will not add other source. Please, refer to AWS documentation on how to define alternative patch source repository
5. Under Manage tags: optionally define tags for the patch baseline

Image 2

Step 2: Assigning a Patch Group to the Patch Baseline

Navigate to AWS Systems Manager console. Under Node Management, select Patch Manager.
Click on Patch baseline tab and select the previously created custom baseline name AmazonLinux2AllPatchesBaseline.
Click on the Baseline ID which is leading to the baseline details page.
On Actions button, select Modify patch groups.
In the Patch groups textbox, type in the value we defined in step 2, which is WebServer-Prd. Then, click on Add button and Close.

Image 3 - 5

Step 3: Creating a Maintenance Window

Navigate to AWS Systems Manager console. Under Change Management, select Maintenance Windows.
Click on Create maintenance window and fill out the requirements as following:
1. Under Provide maintenance window details:
  1. Name: give the maintenance window a name such as, Patch-WebServer-Prd
  2. Description: optionally, add a description for the maintenance window such as, Patching Maintenance Window for Patching WebServer in production
  3. Unregistered targets: uncheck this option. If the option is selected, it allows to register instances that are not part of the targets.
2. Under Schedule:
  1. Specify with: select CRON/Rate expression. For our scenario, we will run patching on the first Wednesday of each month at 9:00pm CDT. Therefore, the CRON job is cron(0 21 ? * WED#1 *). We will define the timezone shortly. >Note: for testing, you can use Rate schedule to the run the patching job every hour or so. Also, refer to AWS documentation for more information about cron and rate expressions for Systems Manager.
  2. Duration: the number of hours the maintenance window will run. Type in 2.
  3. Stop initiating tasks: the number of hours before the end of the maintenance window that the systems should stop scheduling new tasks to run before the window closes. Type in 1.
  4. Window start date: select the starting date and time of this maintenance window such as, 7/6/2022 20:00pm GMT-05:00 (the GMT-05:00 is a conversion to US/Cental time).
  5. Window end date: you may define an end date for the maintenance window. For our scenario, leave it empty.
  6. Schedule timezone: select US/Central timezone.
  Note: the Central Daylight Time (CDT) is -5 hours from GMT;therefore, the maintenance window is in affect starting from the 6th of July at 8:00pm CDT.
  1. Schedule offset: leave it empty.
3. Under Manage tags: optionally define tags for the maintenance window
Finally, click on Create maintenance window to complete the process.

Image 6

Step 4: Registering Targets to the Maintenance Window

Navigate to AWS Systems Manager console. Under Change Management, select Maintenance Windows. Then, click View details.
Press the Actions button and select Register targets from the dropdown menu.
On the Register target screen under maintenance window target details, fill out the requirements as following:
1. Name: give the targets a name such as, WebServer-Prd
2. Description: optionally, add a description for the maintenance window such as, The target is the Web Server in production
3. Owner information: optionally, you may specify a name of the owner such as, The A-Engineering Team
Under the Targets section, for Target selection, select Specify instance tags. Then, enter the patch group key-value tag that we created previously and click Add as shown below.

Key	Value
Patch Group	WebServer-Prd

Finally, click on Register target to complete the process.

Image 7 - 9

Step 5: Assigning Tasks to the Maintenance Window

Navigate to AWS Systems Manager console. Under Change Management, select Maintenance Windows. Then, click View details.
Press the Actions button and select Register run command task from the dropdown menu.
On Register Run command task and Under Maintenance window task details section, fill out the requirements as following:
1. Name: give the task a name such as, PatchWebServerPrd
2. Description: optionally, add a description for the maintenance window such as, Run Command Task for patching Patching Web Server in production
3. New task invocation cutoff: check Enabled to prevent new task invocations when the maintenance window cutoff time is reached
Under Command document section, search for AWS-RunPatchBaselineWithHooks and then select it. Leave Document version at the default selection Default Version at runtime and leave the Task priority at default value of 1.

Note: AWS-RunPatchBaselineWithHooks is a wrapper document for AWS-RunPatchBaseline document. It divides the patching into two events, before reboot and after reboot for a total of three hooks to support custom functionality. Refer to About the AWS-RunPatchBaselineWithHooks SSM document for more information.

For the Targets section, select Selecting registered target groups which is the default selection. Then, from the list select the target group that we assigned to the maintenance window.
Under Rate control,
1. For Concurrency, leave it at default selection, targets, and type in 1
2. For Error threshold, leave it at default selection, errors, and type in 1
On the IAM service role, select maintenance-window-role role from the dropdown menu.

Note: if the IAM service role is not created yet, refer to Create IAM Service Role for Maintenance Window section.

Under Output options,
1. Optionally, check Enable writing to S3 and type the name of the preconfigured S3 bucket. In my case, it's patching-webserver.
Note: to capture the complete terminal output logs, configure an S3 bucket because only the last 2500 characters of a command output are displayed in the console.
1. Optionally, add an S3 key prefix such as, patching/webserver/prd
2. Check CloudWatch output and, optionally, type in a name for the Log group such as, patching/webserver/prd
Note: we will have to give the required permissions to the EC instance IAM profile role to put objects to S3 and put logs to CloudWatch.
For the SNS notifications section
1. Check Enable SNS notifications
2. Select the preconfigured IAM role for the SNS service. In our case, it's maintenance-window-role because we have added an SNS policy to it.
3. Paste the SNS topic ARN
4. For Event type, leave it at default selection which is All events
5. For Notification type, select Per instance basis notification when the command status on each instance changes
Under the Parameters section:
1. For Operation, select Install from the dropdown menu
2. Snapshot Id, leave it empty
3. Reboot Option: leave it at default value, RebootIfNeeded
4. Pre Install Hook Doc Name: type in the name of the pre install document
5. Post Install Hook Doc Name: : type in the name of the post install document
6. On Exit Hook Doc Name: type in the name of the on-exit document
7. Comment: optionally, add comments about the command
8. Timeout (seconds): leave it at default value, 600
9. Finally, click Register Run command task
Note: for details on how to create documents, refer to How to Create SSM Document section. If you don't want to use any of the three hooks, then leave it at default value, which is AWS-Noop.

Image 10 - 11

Step 6: Assigning Patch Group Tag to the Targeted Amazon Linux 2 EC2 Instance

Navigate to AWS Systems Manager console. Under Node Management, select Fleet Manager.
From the Managed nodes list tab, select the EC3 that is targeted for patching. Then, click on Node actions and select View details from dropdown menu.
From the Tags tab, click Edit to add the below tag key-value and then press the Save button:

Key	Value
Patch Group	WebServer-Prd

Image 12 - 14

How to Create SSM Documents

We will create three simple documents as a proof of concept. Please, refer to AWS documentation for more details about creating SSM documents.

Navigate to AWS Systems Manager console. Under Shared Resources, select Documents.
Click Create document button and select Command or Session from the dropdown menu.
On the Create document screen:
1. Name: give the document a name such as, Pre-patch-WebServer-Document
2. Target type: leave it empty
3. Document type: leave it at default which is Command document
4. Content: replace the default content with the below into the YAML editor
5. Document tags: optionally add key-value tag to the document
6. Finally, click Create document

Image 15

---
schemaVersion: '2.2'
description: Pre-patch Web Server Document
parameters: {}
mainSteps:
- action: aws:runShellScript
  name: configureServer
  inputs:
    runCommand:
    - sudo systemctl status httpd

Repeat the process from bullet #3 for Post-install-WebServer-Document and Post-reboot-WebServer-Document. The YAML contents are below.

---
schemaVersion: '2.2'
description: Post-install Web Server Document
parameters: {}
mainSteps:
- action: aws:runShellScript
  name: configureServer
  inputs:
    runCommand:
    - sudo systemctl stop httpd

---
schemaVersion: '2.2'
description: Post-reboot Web Server Document
parameters: {}
mainSteps:
- action: aws:runShellScript
  name: configureServer
  inputs:
    runCommand:
    - sudo systemctl start httpd
    - sudo systemctl status httpd

Image 16

Create IAM Service Role for Maintenance Window

As Systems Manager needs permissions to run maintenance window tasks, below are the steps to create the required IAM role:

Create a policy:

From the IAM console, navigate to Policies and click Create policy button.
Click on the JSON tab and after clearing the default content, paste the below policy and click Next:Tags.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:ListCommands",
                "ssm:ListCommandInvocations",
                "ssm:GetCommandInvocation",
                "ssm:ListTagsForResource",
                "ssm:GetParameters"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "resource-groups:ListGroups",
                "resource-groups:ListGroupResources"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "tag:GetResources"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sns:Publish"
            ],
            "Resource": [
                "arn:aws:sns:*:*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": [
                        "ssm.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

Optionally, add tag-key value pairs and click Next:Review.
Give the policy a name and description such as: a. Name: maintenance-window-policy b. Description: The policy allows Systems manager to run maintenance window tasks
Finally, click Create policy

Create IAM resource role:

From the IAM console, navigate to Roles and click Create role button.
For Select trusted entity page:
1. Trusted entity type: leave at default, AWS service
2. Use case - Use cases for other AWS services: search for Systems Manager and select it (NOT Systems Manager - Inventory and Maintenance Windows)
3. Click Next
On Add permissions screen, search for the previously created policy, maintenance-window-policy and select it. Click Next
Give the role a name such as, maintenance-window-role and click Create role.

Image 17 - 19

Create and add inline policies to the EC2 IAM instance profile:

The EC2 instance profile requires necessary permissions to write data to an S3 bucket and to put log events to CloudWatch. We will create inline policies to assign the required permissions to the EC2 instance profile.

From the IAM console, navigate to Roles and search for your EC2 instance profile role. Then, click on the Role name.
Under *Permissions tab:
1. From Add permissions button, select Create inline policy from the dropdown menu
2. Select the JSON tab and replace the default content with the below policy:
```
{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": "*"
    }
  ]
}
```
1. Click Review policy
2. On the Review policy page, give the policy a name such as, S3-Put-Instance-Profile
3. Finally, click Create policy
Repeat the process from bullet 2 to create an inline policy to allow the EC2 instance to create log stream and to put log events to CloudWatch. Use the below inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        }
    ]
}

Conclusion

Patch Manager is an effective and powerful tool. It assists cloud teams in managing security risks and addressing vulnerabilities by automating the patching process. By the end of the tutorial, we have learned how to schedule patches for a single managed Amazon Linux 2 instance using Patch Manager and its features. Now that we have this knowledge, we can use it to create a patching model that is more intricate for a fleet of managed instances or on-premises servers.

I hope this tutorial helps you learn more about Patch Manager and how to use it for your patching models.

Omar A Omar

How to Create a FREE Custom Domain Name for Your Lambda URL - A Step by Step Tutorial

Omar Omar — Sun, 10 Apr 2022 05:42:44 +0000

Introduction:

On the 6th of April 2022, AWS announced Lambda Function URLs with a Built-in HTTPS Endpoint for Single-Function Microservices. The new feature allows users to configure an HTTPS endpoint for a Lambda function without employing any additional resources such as AWS API Gateway or Application Load Balancer. AWS describes it as a highly available, scalable and secure HTTPS service. The newly added feature was well received by the community. I called it, "one small step for FaaS, one giant leap for Serverless".

There are several articles published to discuss and introduce the newly released feature. Aj Stuyvenberg, an AWS Community Builder, posted a well written article about the Lambda Function URLs titled, Introducing Lambda Function URLs. Another worth reading article was written by Serverless Framework. Moreover, AWS documentation at your fingertips to dig a bit deeper.

With that being said, it's known that the Lambda Function URLs feature does not support custom domain names out of the box; therefore, I have decided to write this step by step tutorial to walk you through how to create a free custom domain name for your Lambda URL. The solution utilizes AWS Route 53 hosted zone, AWS CloudFront distribution, AWS Certificate Manager and Freenom for a free custom domain name registration. In addition, the Lambda function is to host a static page for illustration purposes. To reiterate, the goal of the tutorial is to provide a step by step tutorial on how to create a custom domain name for a Lambda Function URL. Alright enough babbling, let's get at it.

NOTE: AWS Route 53 hosted zone costs $0.50/month.

The Solution Architecture Diagram:

Tutorial Steps:

Obtain a free domain name from Freenom
Create a Lambda function to host a static page
Create a hosted zone in AWS Route 53
Request a Public SSL Certificate from AWS Certificate Manager
Create an AWS CloudFront distribution for the Lambda URL
Create Route 53 A-record for the CloudFront distribution and confirm the custom domain is functioning
Do your victory dance 😉

NOTE: throughout the tutorial, leave default options as is unless instructed otherwise.

Step 1: Obtain a free domain name from Freenom

1- Navigate to Freenom and create a free account.
2- Select Register a New Domain as shown below.

3- Check the availability of a domain name of your choosing. In my case, it’s lambda.cf as it's available and free 😉. Now, We are ready to Checkout.

4- Choose the Period for your selected domain. It ranges from 1 to 12 months. Then, click Continue.
5- Read the Terms & Conditions and select it once you're done. Finally, click Complete Order to register the free domain name.

Congratulations. You have successfully registered a free domain name.

NOTE: it might take an hour or so for the domain registration to take effect.

2 Create a Lambda function to host a static web page

1- From the AWS Lambda console, click Create function.
2- Give your function a name and select Python 3.9 for a Runtime. Under Advanced settings, check Enable function URL and select None for Auth type. Then, click Create function.

3- Copy and paste the below Python code into the body of the lambda_function as shown below and then click Deploy.




#******************************************************************************************
# Author - Omar A Omar
# This lambda function will act as a static web page
#******************************************************************************************


def lambda_handler(event, context):
    response = {
        "statusCode": 200,
        "statusDescription": "200 OK",
        "isBase64Encoded": False,
        "headers": {
        "Content-Type": "text/html; charset=utf-8"
        }
    }

    response['body'] = """
    <html>
        <head>
            <title>Lambda URL</title>
            <style>
                html, body {
                background-color:rgb(22, 30, 43);
                margin: 10; padding: 10;
                font-family: arial; font-weight: 10; font-size: 1em;
                text-align: center;
                }
                html, h1 {
                color: white;
                font-family: verdana;
                font-size: 150%;
                }
                html, p {
                color: white;
                ont-size: 50%;
                }
            </style>
        </head>
        <body>

            <h1>Hello Friend!</h1>
            <p style="color:White;">I'm a static web page running on a Lambda function</p>
            <img src="https://media.tenor.com/YhKAJhNKFeoAAAAC/dance-dancing.gif" width="450" />
        </body>
    </html>
    """

    return response

4- Click on the Function URL to open the lambda static page.

Congratulations. Now, you have a Lambda function hosting a static page.

NOTE: If you're planning to implement weight traffic shifting and safe deployments, AWS recommends associating an alias with your Lambda function. Please, refer to Announcing AWS Lambda Function URLs: Built-in HTTPS Endpoints for Single-Function Microservices for more information.

3 Create a Hosted Zone in AWS Route 53

1- From AWS Route 53 console, click on Hosted zones. Then, click Create hosted zone.

2- Add the Domain name that we have previously obtained from Freenom. In my case, it's lambda.cf. Then, click Create hosted zone.

3- Now, we will need to copy the name servers (type NS) to Freenom for our domain name to point to this hosted zone. Let's copy the four name servers, as shown below, to Freenom.

4- On Freenom, click on Manage Domain and then on nameservers.

5- Select Use custom nameservers (enter below) and paste the four name servers from our Route 53 hosted zone one at a time. Finally, click Change Nameservers.

NOTE: Don't copy the period at the end of the name servers from our Route 53 hosted zone. At this point of time, we are done with Freenom, you can signout and close the tab.

4 Request a Public SSL Certificate from AWS Certificate Manager

1- From the AWS Certificate Manager console, click Request. Keep the default, Request a public certificate selected and click Next.

2- Under Fully qualified domain name, type in your domain name . Then, click Add another name to this certificate and add an asterisk and period *. as a wildcard to your domain name.

NOTE: Adding the wildcard to the second qualified domain name allows us to use the same certificate for other CNAMEs and A-records. If you don't know what a CNAME is and can't differentiate between types of DNS records, it's a great time to fill in the knowledge gap. I would highly recommend taking a look at the AWS documentation for more information.

No other changes are needed. Click on Request to submit the certificate request. Notice and as shown below, the Status is Pending validation. We still need to complete one more step which is validation.

3- Let's validate the certificate. Under Certificate ID, Click on the newly created certificate ID. We will be directed to the certificate status page and under Domains section, click Create records in Route 53.

4- I believe we had to add the validation manually but AWS added this easy to use feature. Now, we automatically add the DNS records to our hosted zone in Route 53 by clicking Create records.

NOTE: If you faced any hiccups adding the validation automatically, copy the CNAME name and CNAME value from under Domains section and use them to manually create a validation CNAME record in our hosted zone.

5- To confirm that the validation CNAME record has been added correctly, head back to our hosted zone and we should see a CNAME record added to the list of DNS records in our hosted zone as shown below.

NOTE: Don't forget to refresh the records in the hosted zone if the newly added validation CNAME record has not showed up . Also, the Status of the certificate in AWS certificate Manager might take a few minutes to become Issued. Therefore, take a break, grab a coffee, or even do a flip 😄

5 Create an AWS CloudFront Distribution for the Lambda URL

1- From the CloudFront console, click Create distribution.
2- The Origin domain is your Lambda Function URL.
3- Protocol: HTTPS only
4- Leave Origin path empty.

5- For the Name, you can change it to a name of your choosing instead of the default name created by CloudFront.
6- For Viewer, select Redirect HTTP to HTTPS.

7- Under Settings, click Add item and add your domain name. In my case, it's lambda.cf.

8- From Custom SSL certificate-optional, let's select our certificate from the drop down menu. Then, click Create distribution.

NOTE: Note, as shown below, CloudFront distribution might take a few minutes to propagate the deployment around the globe; therefore, take a break, grab a cup of coffee, or even do a flip, why not 😄

9- Once the deployment is completed, click on your distribution ID. Copy and paste the Distribution domain name into your browser. We should see our static page.

Congratulations. We now have our Lambda URL distributed to all AWS edge locations using the world's highest performant, secured and convenient content delivery network. Yep, it's CloudFront.

6 Create a Route 53 A-record for the CloudFront distribution

1- Let's head once more to Route 53 console and add an alias record. From our hosted zone, click Create record.

2- Keep Record types as an A-record. But, we need to toggle the Value to Alias. Then from the drop down menu, select Alias to CloudFront distribution. Here, we will paste our Distribution domain name from CloudFront console as shown below. Bear in mind, the Distribution domain name should show up as a selectable option. This is the distribution we have created for our Lambda URL in CloudFront. Finally, click Create records.

NOTE: If you're copying and pasting the CloudFront Distribution domain name, remove the protocol from the URL as shown below (remove https://).

Let's check whether or not our custom domain name for the Lambda URL works. It's the moment of truth 🙊 Let's paste our custom domain name into our browser. Wait for it...Wait for it...I hope you're doing your victory dance by now 🥳

Conclusion:

I might have congratulated you multiple times during the tutorial but it does not hurt to say it one more time, CONGRATULATIONS💥.

We have successfully implemented a solution that employs an AWS Route 53 hosted zone, AWS CloudFront distribution, AWS Certificate Manager and Freenom for a free custom domain name registration. Utilizing this solution, we have given our Lambda function URL a custom domain name.

I hope this tutorial adds value to your learning curve, and I can't wait to see what the future unpacks for Lambda Function URLs. Great things indeed.

Streaming to AWS Kinesis Data Streams using Kinesis Agent - Step by Step Tutorial

Omar Omar — Tue, 22 Mar 2022 05:06:27 +0000

Introduction:

The purpose of this tutorial is to establish a solid understanding of AWS Kinesis services and how to distinguish between AWS Kinesis Data Streams and AWS Kinesis Data Firehose. If you plan to take the AWS Solutions Architect Associate exam, it's very important to understand the differences and case uses of AWS Kinesis services.

During the tutorial, we will spin up an Amazon Linux 2 instance and will install Kinesis Agent. We will configure the Kinesis Agent to send randomly generated numbers from a Python code to AWS Kinesis Data Streams. The AWS Kinesis Data Firehose will route the ingested steam to an S3 bucket for retention and further analysis.

Achieving a high level of comprehension on how to create AWS Kinesis Data Streams, AWS Kinesis Data Firehose and putting data into streams is the goal of the tutorial. The concept; therefore, can be applied to application logs, IoT sensor data, system metrics, videos, audio, analytics and more.

Architectural Diagram:

What is AWS Kinesis Data Streams?

AWS Kinesis Data Streams is an AWS service that ingests and processes data records from streams in real time. It provides accelerated data feed intake for application logs, metrics, videos, website clickstreams and more. For more information about AWS Kinesis Data Streams, please refer to AWS documentation.

What is AWS Kinesis Data Firehose:

AWS Kinesis Data Firehose is an AWS managed service that delivers real-time streaming data records to variety of destinations. These destinations could be:

Examples of AWS Kinesis Data Firehose Destinations
S3
Splunk
New Relic
Datadog
OpenSearch
LogicMonitor
MongoDB
any custom HTTP endpoint
more options...

To read more about AWS Kinesis Data Firehose, please refer to AWS documentation.

What is AWS Kinesis Agent?

It's a standalone Java based application. It collects monitored files/data/logs and send them to Kinesis Data Streams. To read more about Kinesis Agent functionalities, please refer to AWS documentation.

Tutorial Steps:

Create an S3 bucket
Create a Kinesis data stream
Create a Kinesis data firehose
Spin up an AWS EC2 instance
Create an IAM role and attach it to the EC2 instance
SSH into the EC2 instance
Install an AWS Kinesis Agent
Create a folder for the randomly generated numbers Python code
Create a logfile.log to host the randomly generated numbers
Configure the AWS Kinesis Agent to monitor the logfile.log
Test the Python code and tail the logfile.log
Run the Kinesis Agent
Tail the Kinesis Agent log file
Monitor Kinesis Data Streams dashboard
Monitor Kinesis Data Firehose dashboard
Download and open the file from the S3 bucket
Do your victory dance 😉

Step 1: Create an S3 bucket

From the S3 console, click on Create bucket. Then, give a bucket a unique name and click on Create bucket.

NOTE: make sure your select us-east-1 for region.

2 Create a Kinesis Data Stream

From the AWS Kinesis Services, select Kinesis Data Streams. It should be selected by default and then click Create data stream.

Type in stream-1 for the name of the data stream and then leave it as default, On-demand.
Click Create data stream.

3 Create a Kinesis Data Firehose

While we are still on the AWS Kinesis Services console, let's select Delivery streams from the left hand side menu as shown below.

Click on Create delivery stream.
For the source, select Amazon Kinesis Data Streams, and for the Destination, select Amazon S3 as shown below.

Under Source settings, click Browse and choose the name of the Kinesis data stream, which is stream-1. Then, click Choose.

The Delivery stream name is generated randomly by AWS. Let's leave it as is.
Skip the Transformation and convert records.
On the Destination settings, click on Browse and then choose the S3 bucket that we created in step 1.
Leave Dynamic partitioning to the default selections.
Under Buffer hints, compression and encryption, lower the Buffer interval to 60 seconds instead of the 300 seconds default value.
Skip Advanced settings and click Create delivery stream.

4 Spin up an AWS Linux 2 instance

From the EC2 console, click on Launch Instance.

Step 1: Choose an Amazon Machine Image (AMI): select the first AMI on the list , Amazon Linux 2 AMI.
Step 2: Choose an Instance Type: keep the default selection which is t2.micro.
Step 3: Configure Instance Details: keep the default selections.
Step 4: Add Storage: keep the default selection.
Step 5: Add Tags: click on click to add a Name tag and type in a value, Kinesis-Agent.
Step 6: Configure Security Group:

A. Security group name: Kinesis-Agent-SG

B. Description: Opens port 22 to my local IP

C. For the existed SSH rule, change the source to My IP.

Step 7: Review Instance Launch: review and click on Launch.
On the modal screen, select create a new key pair and name it kinesis-agent. Then, click on Download Key Pair to save the key pair to your local device. Finally, click Launch instances to spin up the EC2 instance.

5 Create an IAM role

From IAM console, select Policies from the side menu.
Click on Create policy.
On Create policy screen, select the JSON tab and paste the below policy into the JSON editor. Then, click Next:Tags.



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "kinesis:PutRecords"
            ],
            "Resource": "*"
        }
    ]
}

NOTE: The required permission for Kinesis Agent to put data into a stream is PutRecords. The PutMetricsData is needed for CloudWatch to publish metric data points, if CloudWatch monitoring is enabled for the Kinesis Agent. It's best practice to follow principle of least privilege. Please, refer to AWS documentation for more information.

Skip the optional Add tags screen by clicking Next Review.
On Review policy: A. Name: Kinesis-agent-policy B. Description: This policy allows Kinesis agent installed on an EC2 to put data points into AWS Kinesis Data Streams. It also allows CloudWatch to publish metrics for the agent. C. Click on Create policy.

From the IAM console, click on Roles and click Create role.
On the Select trusted entity screen, choose AWS service and for Use case select EC2. Then, click Next.

On Add permissions screen, filter policies by typing Kinesis-agent-policy and hit enter. This is the policy that we have previously created. Select the policy as shown below and click Next.

On Name, review and create screen:

A. Role name: kinesis-agent-role

B. Description: This role allows the EC2 instance that has Kinesis agent installed to call AWS Kinesis Data Streams.

C. Scroll to the bottom and click on Create role.

Head to EC2 console to attach the newly created IAM role to the EC2 instance as shown on the below image.

From the IAM role drop down menu, select Kinesis-agent-role and click Save.

6 - 12: Install and Configure AWS Kinesis Agent

6- SSH into the EC2 instance:

Let's change the key pair permission. This is done for MacOS and Linux (not Windows). From within the folder, where you have saved the key:



chmod 400 Kinesis-agent.cer

Now, we are ready to SSH into the EC2 instance:



ssh -i Kinesis-agent.cer ec2-user@'instance-public-ip-address'

7- Install the AWS Kinesis Agent:

Before we install the Kinesis Agent, let's update the instance as best practice:



sudo yum update

Now, let's install the Kinesis Agent:



sudo yum install –y aws-kinesis-agent

8- Create a folder for the randomly generated numbers Python code:

Let's cd into the opt directory to create a folder which will host our Python code and then cd into the folder:



cd /opt/
sudo mkdir stream-1 
cd stream-1

We will open nano text editor to create a file for our Python code:



sudo nano stream-1.py

Copy the below Python code and paste it into the nano text editor:



import logging
import json
from datetime import datetime
import calendar
import random
import time

logging.basicConfig(filename='logfile.log', level=logging.DEBUG, format='%(asctime)s %(message)s')



def put_to_stream(id, value, timestamp):
    payload = {
                'random': str(value),
                'timestamp': str(timestamp),
                'id': id
              }

    response ='Putting to stream: ' + str(payload)
    logging.debug(response)


while True:
    value = random.randint(1, 100)
    timestamp = calendar.timegm(datetime.utcnow().timetuple())
    id = 'stream-1'

    put_to_stream(id, value, timestamp)

    # wait for 5 second
    time.sleep(5)

NOTE: The Python code will generate random numbers between 1 to 100, and will output the values every 5 seconds to logfile.log. We will create a logfile.log which has to be in the root folder with the code. The code will timestamp the output values and will tag them with an id = stream-1. Once we have the Kinesis Agent running, we will verify that our architecture is configured correctly by identifying the id = stream-1 from our stream destination which is our S3 bucket. Adding this tag to the data records makes the identification and distinction process efficient, especially if we have multiple producers to the stream.

To save the Python code using nano:

A. Hold down control and click x.
B. Click, y
C. Click, enter

9- Create a logfile.log to host our randomly generated numbers:

Now, we will create the logfile.log which our Python code will log to:



sudo touch logfile.log

Let's confirm that we have two files in the folder by running the ls Linux command:

ls

10- Configure the AWS Kinesis Agent to monitor the logfile.log:

The agent.json is the configuration file for Kinesis Agent. Let's use cat command to view the content of the file. The file resides in the following directory by default:



sudo cat /etc/aws-kinesis/agent.json

The file contains default values. These values were generated during the installation process of the agent. We will only need to have the following configurations on the json file; therefore, we delete the file first and then create a fresh file:



sudo rm /etc/aws-kinesis/agent.json 
sudo nano /etc/aws-kinesis/agent.json

Now, let's copy and paste the below json into the nano text editor and save it:



{
    "cloudwatch.emitMetrics":true,
    "kinesis.endpoint":"https://kinesis.us-east-1.amazonaws.com",
    "flows":[
       {
          "filePattern":"/opt/stream-1/logfile.log",
          "kinesisStream":"stream-1"
       }
    ]
 }

NOTE: If you have changed the location of the Python code, changed the name of the log file or changed the name of the stream, you will have to update the above agent.json file. However, if you have followed the names indicated on the tutorial, no modifications are needed.

11- Test the Python code and tail the logfile.log.

Prior to running the Kinesis agent, we need to change the owner of the /opt/stream-1 folder. By doing so, we are allowing the agent to monitor the log file in the folder, the logfile.log to be specific:



sudo chown aws-kinesis-agent-user:aws-kinesis-agent-user -R /opt/stream-1

Let's run ll command to ensure the ownership of the folder and the files belong to the agent.

Also, prior to running the agent, we will need to ensure that our Python code is logging data into the logfile.log. Let's run the Python code first. We will run the Python code in the background by running the following command;



sudo python /opt/stream-1/stream-1.py &

NOTE: capture the 'PID' number. It's the process ID assigned to the Python code process. We need this number to stop the code from running in the background. However, you could also use the following command to find the PID for the Python process running in the background, 'sudo ps -x | grep python'.

For right now, don't stop the code but if you would like to stop the code from running in the background, run the following command:



sudo kill -9 'add PID here'

Another useful linux command to find all running programs in the background:



ps -e

Let's tail the logfile.log to see if our code is logging every 5 seconds:



sudo tail -f /opt/stream-1/logfile.log

Note: you should see new row of data every 5 seconds as shown below. To exit out of the tail command, hold down control button and click c.

12- Run the Kinesis Agent.

Now, we are ready to run the Kinesis Agent. Let's run the below command to let the agent start monitoring logfile.log and transmitting data points to our stream:



sudo service aws-kinesis-agent start

Let's check the status of the agent:



sudo service aws-kinesis-agent status

NOTE: if the agent started properly, we should see similar output as shown on the below image.

Other important agent commands:



sudo service aws-kinesis-agent status
sudo service aws-kinesis-agent restart
sudo service aws-kinesis-agent stop

13- Tail the Kinesis Agent log file.

Let's tail the Kinesis Agent log. It should give us valuable information about the current status of the agent. We will inspect for errors or any permission messages.



sudo tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

NOTE: we are looking for a confirmation that the agent has started sending data records to our stream as shown on the below image. Notice, the agent log stated 26 records sent successfully to destination. This is a clear indication that AWS Kinesis Data Stream is ingesting our data records.

14- Monitor Kinesis Data Streams dashboard.

Let's head to AWS Kinesis Services console. From our Kinesis Data Streams, click on Monitoring and navigate to Get records - sum(Count) dashboard. As shown below, the data records have been ingested by the stream successfully.

15- Monitor Kinesis Data Firehose dashboard.

Now, let's head to our Delivery streams (AWS Kinesis Data Firehose) and click on Monitoring and navigate to Records read from Kinesis Data Streams (Sum) dashboard as well as Delivery to Amazon S3 success dashboard. As shown below, the firehose has routed the real-time data to our S3 bucket successfully.

16- Download and open the file from the S3 bucket.

Currently, we have confirmed that the data records are flowing into our stream and firehose, we should find the files in our S3 bucket. Let's download one of the file and open it on our local device. We should confirm that the logged data has stream-1 id. This is the id we have defined in our randomly generated numbers Python code.

Conclusion:

If you get this far, I would like congratulate you on your achievement. We have create an AWS Kinesis Data Stream and an AWS Kinesis Data Firehose. We have also employed a Python code to randomly generate numbers and tag records with timestamp and stream name at a 5 second interval. Then, we have installed a Kinesis Agent on an EC2 and configured it to monitor our log file. We have confirmed the agent is running successfully and data records are being ingested by our stream. Lastly, we have ensured that AWS Kinesis Data Firehose has routed the data records to our S3 bucket destination by downloading and inspecting the files for the stream id.

I hope the tutorial adds greatly to your learning curve. Having accomplished a great deal of understanding about AWS Kinesis Data Streams, AWS Kinesis Data Firehose and AWS Kinesis Agent is a triumph. I wish you all the best in applying these conceptual and practical knowledge to application logs, IoT sensor data, system metrics, videos, audio, analytics and more. Now, off you go, the sky is the limit.

How to Create and Send AWS CloudWatch Alarm Notifications to PagerDuty

Omar Omar — Wed, 16 Mar 2022 02:46:47 +0000

Introduction:

AWS CloudWatch provides monitoring and observability services for AWS resources. It is built for Site Reliability Engineers, DevOps Engineers and developers in mind. The CloudWatch service collects data and gains insights. It's capable of alerting and resolving operational issues for users. It also gives a system wide visibility into resource utilization. For more information about AWS CloudWatch, please refer to AWS documentation.

PagerDuty is an on-call management and incident response system. The main goal of this tutorial is to walk you through how to configure a CloudWatch alarm, set a threshold and how to send alarm notifications to create and resolve incidents in PagerDuty. I chose 'AutoScaling Group Terminating Instances' as an example for this tutorial. When an EC2 instance in an AutoScaling group is terminated due to any unforeseen reason, a CloudWatch alarm is triggered and ;as a result, an incident is created in PagerDuty for the on-call engineer to inspect and resolve.

With that being said, the same concept and steps can be applied to create other CloudWatch alarm notifications to PagerDuty for any CloudWatch metrics in AWS namespaces or custom namespaces. Please, refer to AWS documentation about AWS services that publish CloudWatch metrics.

Alright, since there are lots of materials/steps to cover, the tutorial is to the point, short and succinct. Enjoy it. 😉

The Architectural Diagram

Step 1: Creating SNS Topic

From the SNS console, select Topics. Then, click on Create topic.

Select Standard for the type of the SNS topic and give it a name. Then, click Create topic.

Now, we have successfully created an SNS topic for our AutoScaling Group Alarms (or any other alarm of your choosing). We will subscribe to the topic once we create an alarm.

Step 2: Creating a CloudWatch Alarm:

From the CloudWatch console, click on All alarms under Alarms section. Then, click Create alarm.

On Specify metric and conditions, click on Select metric.
In the search box under metrics, type in autoscaling and hit enter.

Select Auto Scaling > Group Metrics from the options.
Scroll down to select GroupTerminatingInstances for your AutoScaling group. In my case, the name of my AutoScaling group is myAutoScaling. Then, click on Select metric.

The Specify metric and conditions screen is where we would customize and configure the metric alarm by specifying thresholds, period and other conditions.

A. Change the period to 1 minute.

B. I would define the threshold value to 0. I would like to be alerted if an EC2 is terminated within my autoscaling group.

C. The Datapoints to alarm is 1 out of 1.

D. The Missing data treatment is Treat missing data as ignore... This for dev environment but it's not recommended for prod environment. No further changes are needed. Nevertheless, you may configure the alert however you like. Next, click Next.

On Configure actions screen, leave the Alarm state trigger to default which is In alarm.
Under Select an SNS topic, choose Select an existing SNS topic. If you insert the cursor into the Send a notification to textbox, you could select the SNS topic that we created in step 1. But, if it's not showing up for any reason, paste the Amazon Resource Number (ARN) for your SNS topic.

We will add a second Alarm state trigger by clicking on Add notification. Note, we will have one trigger to send notifications of the In alarm state and a second trigger to resolve the first alarm to the OK state. Therefore, select OK and select the same SNS topic as shown below. Then, click Next.

Give the alarm a name and description.

Finally, review and click on Create alarm. CloudWatch will start gathering more data about the metric/alarm as showing on the State. It will take a minute or two for the state to go to the OK state.

As of right now, we have successfully created a CloudWatch alarm for AutoScaling Group Terminating Instances. Any terminated EC2 instance within the specified AutoScaling group will trigger the alarm.

Step 3: Creating PagerDuty - CloudWatch Integration:

According to PagerDuty, there are two ways to integrate AWS CloudWatch:

A. Integrate with a PagerDuty Event Rule.

B. Integrate with a PagerDuty Service.

For this tutorial, we will utilize PagerDuty service as a method of CloudWatch integration.

On the PagerDuty console, click on services to locate the service you would like to add CloudWatch integration to. Then, click on the service name.

Select Integrations and click on Add an Integration.

Type in CloudWatch in the search box under Select the integration(s) you use to send alerts to this service. Once AWS CloudWatch is selected, click Add.

Copy the Integration URL. We will use this endpoint to subscribe to our SNS topic.

So far, we have successfully create a CloudWatch integration with our PagerDuty service. Now, it's time to head back to the AWS console and to SNS console to be specific.

Step 4: Subscribing PagerDuty-CloudWatch Integration to SNS topic:

On the AWS SNS console, click on the name of the SNS topic we have created in step 1.
Click on Create subscription

Select HTTPS as a protocol and paste the Integration URL/endpoint that we saved from step 3. Then, click Create subscription.

Note: make sure that Enable raw message delivery is unchecked. It should be unchecked by default.

Refresh the page until the Status is shown as Confirmed.

We have successfully created an SNS topic, a CloudWatch alarm and a PagerDuty integration for CloudWatch. We have also subscribed a PagerDuty integration endpoint to the SNS topic. Don't do your victory dance yet, let's test the alarm and the integration first. 😉

Step 5: Testing CloudWatch Alarm to PagerDuty Integration:

It has been journey, and I would like to congratulate you for getting this far. It's the moment of truth. The time has come to test our integration, fingers crossed.

In order for us to trigger a CloudWatch alarm for the AutoScaling Group Terminating Instances, we need to terminate an instance within our AutoScaling group. I'm sure you're not doing this in prod environment 😉 I will go ahead and terminate one of the two EC2 instances I have in the AutoScaling group.

Let's head to the CloudWatch console to monitor the alarm status and progress. As we see on the image below, the alarm status is still OK. The delay for the alarm to go off is due to the default Health check grace period, which is 300 seconds (5 mins) by default. We can lower the health check grace period to deem the EC2 unhealthy quicker. Another important term is scaling cooldown period, which prevents AutoScaling groups from launching or terminating instances before the effects of previous activities are visible. We can control the cooldown period by using different scaling policies. Note that the default cooldown period is 300 seconds (5 mins), which can be changed. If you would like to learn more about scaling cooldown period, please refer to AWS documentation. If the alarm did not go off, then lower the Health check grace period to 60 seconds instead of default 300 seconds for your AutoScaling group.

Once the instance was deemed unhealthy by the AutoScaling group, the CloudWatch alarm went off as shown below.

We should have also received a PagerDuty incident notification by now. This was an indication that we had completed the integration properly.

Note, the PagerDuty notification methods whether it's an SMS, an email or a phone call, they are depend on how we configured our escalation policy and notification preference in PagerDuty.

The last portion of testing is to wait for the AutoScaling group to recover this instance and for the CloudWatch alarm to send a notification for state OK which should resolve the PagerDuty incident automatically.

The below image shows that the alarm state went to the OK state.

The below images shows that the PagerDuty incident is resolved automatically.

Conclusion:

During this tutorial, we have successfully created an SNS topic, a CloudWatch alarm for AutoScaling Group Terminating Instances metric and integrated them with PagerDuty CloudWatch integration.
We have tested the configuration and confirmed the following:

Once an instance was terminated, a CloudWatch alarm went off and a PagerDuty incident was created. And, the engineer on-call freaked out. :laugh:
As soon as the AutoScaling group replaced the instance and deemed healthy, the CloudWatch alarm state went from In Alarm to OK state. As a result, the PagerDuty incident was automatically resolved and the engineer on-call went back to bed 😂

Thank you for taking this journey with me, and I hope it was beneficial to you. Oh, one last thing, now you can do your victory dance 😜

GitLab Runner on Raspberry Pi 4 (Build, Push Docker images to Docker Hub using GitLab Runner on GitLab)

Omar Omar — Mon, 28 Feb 2022 04:21:48 +0000

What is GitLab Runner

GitLab Runner is an agent that runs GitLab CI/CD (Continues Integration/Continuous Deployment) jobs in a pipeline. It's heavily utilized in the world of DevOps to provision and configure infrastructure. The GitLab Runner can be installed as a binary on Linux, MacOS or Windows. It can also be installed as a container.

On this tutorial, I will walk through installing and configuring GitLab Runner as a container using a Docker image on a RPI-4..yaay. I will make it very swift to get you started and won't feel a thing. I will not bore you with details, but if there are useful links for further study, I will definitely throw it in. The goal is to get you started with GitLab Runners and the rest is on you.

As a bonus, we will run our first job of building docker images and push them to Docker hub. Enough reading, let's get our hands dirty, I mean our keyboards 😉

To lean more about GitLab Runners, refer to GitLab official documentation.

Note: Sensitive information will be pixilated or erased. This should not alter the quality of the tutorial.

Run a GitLab Runner Container on a RPI-4

If you don't have Docker installed on your RPI-4, you may refer to my Docker on Ubuntu 20.04 Raspberry Pi 4 tutorial.

1- Let's create a persistent Docker volume for the runner. On your RPI-4 terminal, run the following command:

docker volume create gitlab-runner-volume

Note: you can change the volume name gitlab-runner-volume to any name of your chosen, but you should be consistent as we will use the volume name to bind the container to the RPI-4 local host.

2- Run the below commands to start GitLab Runner container:

docker run -d --name gitlab-runner --restart always --env TZ=US \
   -v  gitlab-runner-volume:/etc/gitlab-runner \
   -v /var/run/docker.sock:/var/run/docker.sock \
   gitlab/gitlab-runner:alpine

The only parameters you have the options to change are:

the name flag, gitlab-runner
the volume name, gitlab-runner-volume
the gitlab runner image, gitlab/gitlab-runner:alpine
the env flag, TZ=US

For more information about the installation process, here is the link GitLab's official documentation.

3- Capture the GitLab Runner token:

We do need to head to our GitLab account and grab the runner's token. If you don't have a GitLab account, you can create one for free. Here is the link.

Now, we need to create a repository to host our project. From GitLab, let's click on create New project. On the Create new project, select create blank project. Then, give the project a name and click Create project. In my case, I named my repo, GitLab-Runner-RPI-4.

From the repository, let's click on settings, CI/CD and then Expand on the Runners section as shown below.

Capture the token and save it on a note pad. We will need the token for the next step.
Disable Enable shared runners for this project.

4- Register the GitLab Runner:

Replace the token place holder after registration-token with the one from our note pad, and then run the following commands on your RPI-4 terminal:

docker run --rm -it -v gitlab-runner-volume:/etc/gitlab-runner gitlab/gitlab-runner:alpine register -n \
  --url https://gitlab.com/ \
  --registration-token GR1348941EDhyNWqfxPttukrGVKJd \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:20.10.12-dind-alpine3.15" \
  --docker-privileged \
  --docker-volumes "/certs/client"

If everything went smooth, you should not exhibit no errors as shown below.

To confirm that the GitLab Runner container is running, run the below Docker command:

docker ps -a

Alright, this is a great indication that we have successfully configured GitLab Runner on RPI-4. Now, we are ready to make some actions 💥

Build and Push Docker images to Docker Hub using GitLab Runner on GitLab

1- First of all, we need get a token from our Docker hub account to avoid using account password. The token is to allow GitLab Runner to authenticate and push Docker images to our Docker hub repository. If you don't know what Docker hub is, it is the world's largest library for container images.

On Docker hub account settings, click on Security, New Access Token and generate a Read, Write, Delete token. Docker hub free account allows for one active token. Capture the token and save to a note pad.

2- From GitLab repo, which have we have created previously:

Click on Clone and copy the Clone with HTTPS link.
On your RPI-4 terminal and in a folder of your chosen, run the below command. This is where the repo will be saved locally:

git clone `your-Clone-with-HTTPS-link`

Note: the git clone command will prompt you to enter your GitLab credentials for authentication.

3- cd into the repo and either use a text editor or the terminal to edit the .gitlab-ci.yml file. The file should only contain the following code. Be very careful with the indentation. Lastly, save the file:

image: docker:19.03.12

variables:
    IMAGE_NAME: "test:1.0.0"
    DOCKER_TLS_CERTDIR: "/certs"

services:
    - docker:19.03.12-dind
before_script:
  - docker info
build image:
    stage: build 
    script:    
        - docker build -t $REGISTRY_USER/$IMAGE_NAME .
        - docker login -u $REGISTRY_USER -p $DOCKER_HUB_TOKEN
        - docker push $REGISTRY_USER/$IMAGE_NAME

4- We will create a simple Dockerfile. The runner will use this Dockerfile to build a Docker image and push it to Docker hub. For right now, we will keep the Dockerfile VERY simple. Once the image is built from the Dockerfile, it will echo Hello World.
With your preferred text editor or terminal, create and name a file Dockerfile without any extension. Then save it in the same local repo.

FROM alpine
RUN echo "Hello World"

Note: In the local repo, we should have two files, Dockerfile and .gitlab-ci.yml. A README file might be there as well.

5- We will need to head back to GitLab repo to create two variables. On Settings, click on CI/CD, then expand Variables. We will click on Add variable to create two variables:

Key: REGISTRY_USER
Value: your Docker hub username NOT the email
Key: DOCKER_HUB_TOKEN
Value: the token which we generated from Docker hub

Note: the flags should be unchecked for simplicity for the two variables.

6- Now, we push the local repo to GitLab repo (remote repo) to start the GitLab Runner pipeline process. Yes, once the remote repo is updated, the runner will be triggered. Let's get back to our terminal on the RPI-4 and from within the local repo, run the following git commands:

git add -A
git commit -am "first test"
git push

If you head back to our GitLab repo and click on CI/CD under Pipelines, you would notice that the pipeline is running.

Moreover, if you click on the running status and you would be directed to the current stage, which is Build.

Here, click on build image, you shall see more details about the current build status. The goal is to see Job succeeded.

Once you see Job succeeded and passed status in green, you can start doing your victory dance 👯 👯

If we head back to Docker hub account, we should see that our Docker image test:1.0.0 has been successfully pushed to the repo.

Conclusion:

By the end of this tutorial, we have successfully configured a GitLab Runner on a RPI-4, created a GitLab repo and registered GitLab Runner to it. Finally, we created a Hello World Docker image from a Dockerfile and had the runner building the Docker image and pushing it to our Docker hub account.
Now, off you go and the sky is the limit.

A Free Cloudflare Tunnel running on a Raspberry Pi

Omar Omar — Tue, 11 May 2021 04:29:34 +0000

Cloudflare is a global network designed to make everything you connect to the Internet secure, private, fast, and reliable. Cloudflare offers a suite of services and Zero Trust Services are the services we will utilize in the following tutorials. Zero Trust Services consist of Teams, Access, Gateway and Browser Isolation.

Our main goal is to obtain a free domain from Freenom and connect our hosted applications on a Ubuntu 20.04 LTS Raspberry Pi 4 within our local home network via a Cloudflare Tunnel to the world wide web securely without any port-forwarding complications or altering firewall.

Tutorial Scenario:

Signup for a free Cloudflare for Teams.
Install and authenticate cloudflared on a Raspberry Pi 4.
Create a Cloudflare Tunnel.
Configure the Tunnel details.
Create DNS records to route traffic to the Tunnel.
Run and manage the Tunnel.
Add a Zero Trust policy.
Run Tunnel as a service.

Step 1: Signup for a free Cloudflare for Teams:

Navigate to link and signup for a free account. Cloudflare has a well documented Get started site to walk you through the setup process. For this step, you don't need to go beyond signing up.

Step 2: Install and authenticate Cloudflared on a Raspberry Pi 4:

First of all, if you’d like to check your device’s architecture, run the following command:

uname -a

Navigate to link site to download the proper package for your architecture. In my case, I will install the Cloudflared daemon on my RPI-4, which is an arm64 architecture.

arm64 architecture (64-bit Raspberry Pi 4):

wget -O cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-arm64
sudo mv cloudflared /usr/local/bin
sudo chmod +x /usr/local/bin/cloudflared
cloudflared -v

AMD64 architecture (Debian/Ubuntu):

sudo wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.deb
sudo apt-get install ./cloudflared-stable-linux-amd64.deb
cloudflared -v

armhf architecture (32-bit Raspberry Pi):

wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-arm.tgz
tar -xvzf cloudflared-stable-linux-arm.tgz
sudo cp ./cloudflared /usr/local/bin
sudo chmod +x /usr/local/bin/cloudflared
cloudflared -v

Once we have installed Cloudflared successfully, we will run the following command to authenticate the cloudflared daemon to our Cloudflare account.

cloudflared login

Running the above command will launch the default browser window and prompt you to login to your Cloudflare account. Then, you will be prompted to select a hostname site, which we have create previously in Part link.

As soon as you have chosen your hostname, Cloudflare will download a certificate file to authenticate Cloudflared with Cloudflare's network.

Notice:
The cert.pem gives Cloudflared the capabilities to create tunnels and modify DNS records in the account. Once you have created a named Tunnel, you no longer need the cert.pem file to run that Tunnel and connect it to Cloudflare’s network. However, hte cert.pem file is still required to create additional Tunnels, list existing tunnels, manage DNS records, or delete Tunnels.

Once authorization is completed successfully, your cert.pem will be download to the default directory as shown below.

If you're running a headless server (no monitor or keyboard), you could copy the authentication URL and paste it in a browser manually.

Note:
The credentials file contains a secret scoped to the specific Tunnel UUID which establishes a connection from cloudflared to Cloudflare’s network. cloudflared operates like a client and establishes a TLS connection from your infrastructure to Cloudflare’s edge.

Step 3: Create a Cloudflare Tunnel:

Now, we are ready to create a Cloudflare Tunnel that will connect Cloudflared to Cloudflare's edge. Utilizing the following command will create a Tunnel with tht name and generate an ID credentials file for it.

Prior to creating the Tunnel, you may need to exit the Command Line (CL). Next, let create the Tunnel.

Note: replace with any name of your choosing for the Tunnel.

cloudflared tunnel create <NAME>

Once the Tunnel is created, a credential file is generated. It's a JSON file that has the Universally Unique Identifier (UUID) assigned for the Tunnel.

Note: although the Tunnel is created, the connection is not established yet.

Step 4: Configure the Tunnel details:

Although we can configure the Tunnel run in an add hoc mode, we will go over creating a configuring the Tunnel to automatically run it as a service.

Cloudflare utilizes a configuration file to determine how to route traffic. The configuration file contains keys and values, which is written in YAML syntax. You may need to modify the following keys and values to meet your configuration file requirements:

Keys	Values
tunnel	Tunnel name or Tunnel UUID
credentials-file	location credentials file (JSON)
hostname	subdomain.hostname.xxx
service	url - http://localhost:8000
service	http_status:404
port of your app	80

By default, on Linux systems, Tunnel expects to find the configuration file in ~/.cloudflared, /etc/cloudflared and /usr/local/etc/cloudflared in that order.
Let's create our config file and save in the default expected directory for this tutorial.

sudo nano ~/.cloudflared/config.yml

Or,

sudo nano home/<username>/.cloudflared/config.yml

Then, we will paste our keys and values as shown below:

tunnel: 1082b601-bce9-45e4-b6ae-f19020e7d071
credentials-file: /root/.cloudflared/1082b601-bce9-45e4-b6ae-f19020e7d071.json

ingress:
  - hostname: test.mytunnel.ml
    service: http://localhost:80
  - service: http_status:404

Note:
If you don't have any application ready to test the Tunnel, I'd suggest installing NGINX web server and port mapping it to port 80 as I've done in the configuration file.

How to install NGINX web server on RPI-4:

sudo apt install nginx

Once the installation is completed, open a browser and type in: localhost:80. If the NGINX web server is installed properly, you shall see it running with its default index.html as shown below.

Let's make sure that we have all files in this directory:

ls -al

Now, we have configured all required files to run the Tunnel in the default directory.

Note, if you'd like to save the config.yml file in a different location ( we will refrain from using this method for this tutorial), you will have to point to that directory during the run command by using the following:
cloudflared tunnel --config path/config.yml run UUID or Tunnel Name

It's very import to specify --config to change default directory for the config file. For more information about the link.

Step 5: Create DNS records to route traffic to the Tunnel:

Cloudflare can route traffic to our Tunnel connection using a DNS record or a loud balancer. We will configure a DNS CNAME record to point to our Tunnel subdomain. There are two ways to acheive this mission:

A. Manually: navigate to the DNS tab on Cloudflare Dashboard, create a new CNAME record and add your subdomain of your Tunnel as follows:

Type: CNAME
Name: any subdomain name of your choosing.
Target: consists of two parts: <UUID> and <cfargotunnel.com> such as, <UUID.cfargotunnel.com>

B. Programmatically: run the following command from the command line. This command will generate a CNAME record that points to the subdomain of a specific Tunnel. The result is the same as creating a CNAME record from the dashboard as shown in step A.

cloudflared tunnel route dns <UUID or NAME> test.example.com

Note: unlike the previous Argo Tunnel architecture, this DNS record will not be deleted if the Tunnel disconnects.

Step 6: Run and manage the Tunnel:

The run command will connect cloudflared to Cloudflare's edge network using the configuration created in step 4. We will not specify a configuration file location so Cloudflared retrieves it from the default location, which is ~/.cloudflared/config.yml

cloudflared tunnel run <UUID> or <Tunnel Name>

If the config.yml file is not placed in the default directory, we need to pinpoint to its location to run the Tunnel:

cloudflared tunnel --config path/config.yml run <NAME> or <UUID>

We can review the list of Tunnels we have created by running the following command:

Cloudflared Commands:

Functions	Commands
Create a Tunnel	cloudflared tunnel run <NAME>
List Tunnels	cloudflared tunnel list
Stop Tunnel	cloudflared tunnel stop <NAME>
Restart Tunnel	cloudflared tunnel restart <NAME>
Delete Tunnel	cloudflared tunnel delete <NAME>
Force Delete Tunnel	cloudflared tunnel delete -f <NAME>
Show each Cloudflared info	cloudflared tunnel info <NAME>

Note: stopping Cloudflared will not delete the Tunnel or the DNS record created. Although Tunnel deletes DNS records after 24-48 hours of a Tunnel being unregistered, it does not delete TLS certificates on your behalf once the Tunnel is shut down. If you want to clean up a Tunnel you’ve shut down, you can delete DNS records in the DNS editor and revoke TLS certificates in the Origin Certificates section of the SSL/TLS tab of the Cloudflare dashboard.

To update Cloudflared:

sudo cloudflared update

To uninstall Cloudflared

sudo cloudflared service uninstall

Step 7: Add a Zero Trust policy:

Now, we are ready to head back to Teams dashboard to configure our application and create a Zero Trust Policy.

On Teams dashboard, navigate to the Application tab and click on Add an application.

Select Self-hosted.

Choose an application name, Session Duration, subdomain and Application domain. Then, click on Next.

Notice that the Tunnel duration ranges from 15 mins to 1 month.

Add a name to the rule and select Bypass as a Rule action. On Configure a rule, include Everyone. This rule allows everyone to view our NGINX site at test.mytunnel.ml

In the Advanced settings, enable automatic cloudflared authentication and browser rendering.

Finally, our application is now available in Cloudflare Access and is part of our Application list. We can navigate to a browser and type in our url test.MyTunnel.ml and if our Tunnel is established correctly, we shall see our NGINX web server running as shown below.

Step 8: Run Tunnel as a service:

By running the following command, the Tunnel can be installed as a system service which allows the Tunnel to run at boot automatically as launch daemon. By default, the Tunnel expects to find the configuration file in the default directory, ~/.cloudflared/config.yml but to run Tunnel as a service, we might need to move the config.yml file in ~/etc/cloudflared/.

We can employ the move mv command to do the job: mv <*path/config.yml> to </etc/cloudflared/*>

The below command is in my case with my RPI-4 and how I moved the config file to /etc/cloudflared/

sudo mv /home/p2/.cloudflared/config.yml /etc/cloudflared/

Now, we are ready to run Tunnel as a service utilizing the command below:

sudo cloudflared service install

Conclusion:

We have successfully established a secure Cloudflare Tunnel that links our locally hosted NGINX web server to Cloudflare's network without requiring any public IP address, port-forwarding or punching through a firewall. We have also configured the Tunnel as a service to start at boot, and now we have our NGINX web server associated and accessible via our domain name, test.MyTunnel.ml

Best of luck with you future project. Cheers!!

DEV Community: Omar Omar

AWS Cognito for Bedrock Storyteller running on Containerized Lambda Function with Web Adapter

Prerequisites

Step 1: Create a new AWS Cognito User Pool

Step 2: Add a new user to the User Pool

Creating a Lambda function to confirm new users

Adding the Lambda trigger to the User Pool

Adding a new user to the User Pool

Step 3: Update the Bedrock Storyteller Flask application to use AWS Cognito for authentication

Step 4: Deploy the application to a containerized AWS Lambda with Web Adapter using AWS SAM

The directory structure

Conclusion

Future Work

References

Alexa to Run Systems Manager Documents

Introduction:

Alexa Cloud Solution: Logic and Architecture:

Alexa Cloud-based Solution Architecture Diagram

Tutorial Sections:

Prerequisites:

1: Creating Alexa skill hosted on AWS Lambda function (AlexaSkill)

Step 1: Setting up the ASK SDK in a virtual environment on Linux or macOS

Step 2: Add skill source code

Step 3: Packaging and Creating Lambda Function (AlexaSkill function)

2: Configuring skill with Alexa service via the Alexa Developer Console

Step 1: Configuring Alexa service

Step 2: Configuring skill with Alexa Service

Step 3: Configuring Lambda function hosting Alexa skill with Alexa service

3: Creating SSM document to stop EC2 instances

Step 1: Customer created SSM document:

Step 2: Amazon managed SSM document:

4: Creating SNS topic to receive notifications from Systems Manager - Run Command

5: Creating IAM role to allow Systems Manager to send notifications to SNS

Step 1: Creating the IAM role:

Step 2: Attaching inline policy to the IAM role:

6: Provisioning on-demand DynamoDB table

Step 1: Creating DynamoDB table:

Step 2: Creating DynamoDB table items:

7: Creating Master Lambda function (MasterLambda) to send SSM commands to Run Command

Step 1: Creating Lambda function:

Step 2: Configuring communication between MasterLambda function and AlexaSkill Lambda function:

8: Creating Lambda function (SlackLambda) to send notifications to Slack (Optional)

Step 1: Creating Secrets Manager secret to store Slack webhook URL:

Step 2: Creating Slack lambda function:

Step 3: Subscribing SlackLambda function to the SNS topic:

9: Testing Alexa skill solution

Step 1: Tagging EC2 instances:

Step 2: Testing Alexa skill:

OmarCloud20 / alexa-runs-ssm-documents

Alexa to Run Systems Manager Documents

Introduction:

Conclusion

Omar OmarFollow

References

OmarCloud20 / alexa-runs-ssm-documents

Alexa to Run Systems Manager Documents

Introduction:

CDK for Terraform (CDKTF) on AWS: How to Configure an S3 Remote Backend and Deploy a Lambda Function URL using Python

Update (10/12/22):

Table of Contents

Introduction

Step 1: Required Prerequisites

Step 2: Initializing the First CDKTF Project using Python Template

Step 3: Configuring an S3 Remote Backend

Option 1: Utilize an existing S3 bucket and DynamoDB to configure the S3 Remote Backend.

Option 2: Create an S3 Bucket and DynamoDB Table for the S3 Remote Backend using CDKTF

Step 4: Learn How to Use Construct Hub and AWS Provider Submodules

Scenario 1: S3 Bucket

Scenario 2: ECS Cluster

CDKTF Commands:

Step 5: Deploying a Lambda Function URL using CDKTF

Building a Patching Model using AWS Systems Manager - Patch Manager for Mutable Infrastructure

Table of Contents

July 8th, 2022

AWS Systems Manager Patch Manager

What is the Patch Baseline?

How Patch Baseline Rules Work on Amazon Linux 2

Patching Model Solution Architecture

Architecture Diagram - Patching Model using Patch Manager

How to Create Patching Model for Single Managed Amazon Linux 2 EC2 Instance - Step by Step