DEV Community: Tom Spencer

Unspam Your AWS Service: Conquer Pristine Inbox Delivery

Tom Spencer — Mon, 08 Jan 2024 18:18:45 +0000

Problem

Was Completely frustated with emails landing in spam...
How? So I was meticulously setting up AWS SES to send emails from my movie booking application hosted on ECS and routed through Route53 domain, but rather reaching the eager customers, mails were getting unceremoniously dumped into spam folders.

But you relax, because I have figured out cause & effective solution. Just follow along !!!

Cause

So culprit here is email spoofing.
Have you ever opened an email claiming to be from a bank, only to find a suspicious offer or a phishing link? This could be the work of email spoofing, a deceptive technique where bad actors impersonate legitimate senders to deceive recipients.

Think of it like this: imagine someone forging your signature on a letter and sending it out under your name. Spoofing does the same thing in the digital world, manipulating email headers to make it appear as if the message originates from someone else.

Motto of email spoofing includes phishing, spamming and fraud etc. Thankfully, email servers and services today employ various authentication protocols to unmask spoofing attempts. These protocols act like digital gatekeepers, verifying the sender's identity before letting the email pass through.

Catching our problem here, that we too need to prove these gatekeepers the authenticity of our service.

Solution

Building a Fortress of Trust: The SPF, DKIM, and DMARC Trifecta

In the bustling world of email, trust is a precious commodity. To ensure that only legitimate emails reach their intended inboxes, a trio of authentication protocols stands guard: SPF, DKIM, and DMARC.

Here's how they work in coordination to create a dependable verification system:

1. SPF (Sender Policy Framework):

The Gatekeeper: SPF establishes a clear list of authorized mail servers for a domain.
Think of it as a VIP guest list: Only those with their names on it (i.e., approved IP addresses) can send emails on behalf of the domain.
How it works: Receiving servers check incoming emails against the SPF record in the domain's DNS. If the sender's IP doesn't match, the email may be flagged as suspicious.

2. DKIM (DomainKeys Identified Mail):

The Seal of Authenticity: DKIM adds a digital signature to emails, guaranteeing their integrity and sender verification.
Imagine a tamper-proof seal: It ensures the email hasn't been altered in transit and originates from the rightful sender.
How it works: The signature is generated using a private key known only to the domain owner. Receiving servers use the corresponding public key to verify the signature's authenticity.

3. DMARC (Domain-based Message Authentication, Reporting & Conformance):

The Orchestrator: DMARC builds upon SPF and DKIM, providing policy enforcement and reporting capabilities.
Think of it as a watchful manager: It instructs receiving servers on what actions to take when emails fail authentication checks.
How it works: Domain owners set DMARC policies to specify how to handle unauthenticated emails (e.g., quarantine or reject). They also receive reports on authentication results, aiding in identifying and addressing potential spoofing attempts.

Now as we know, lets implement above mechanism !!!

Implementation

Lets do it then, follow me:

SPF

Create a record set in the sender domain, as follows:
Value

"v={spf version} {mechanisms}{directive} {qualifier}all"

Type
TXT

So for me, all mails from SES are authorized to get soft pass(all servers except SES, will be considered as spam).

include:amazonses.com ensures authorizing SES
~all instructs soft pass

Done TADA... Lets go to, DKIM signatures.

DKIM

Follow along please:

1. Go to verified identities in SES

2. Get your domain verified

3. Enabling DKIM signature on domain, you will get public keys

4. Create the generated record sets in Route 53

Now, there will be in total 5 records to add,
3 will be having public key matching for reciever to read,
publicKey1._domainkey.yourhosteddomain
publicKey2._domainkey.yourhosteddomain
publicKey3._domainkey.yourhosteddomain

2 record sets for configuring MAIL FROM,
Value:subdomain.yourhosteddomain Type:MX
Value:subdomain.yourhosteddomain Type:TXT

Done! Done! Done! Be ready for last & least DMARC

DMARC

Set up a DMARC record set, with
Type: TXT
Value:

"v:DMARC1;p=reject;pct=100;rua=mailto:support@ahtcloud.com:fo=1;adkim=s;aspf=s;"

That's it! Now your domain is fully authorized to act as email sender...

How to connect securely to an RDS instance running in a private Subnet

Tom Spencer — Fri, 22 Dec 2023 16:30:23 +0000

In this article we will connect to an RDS instance locally using an SSH tunnel. This video was useful.

Goal

Need to connect to an RDS instance present within a private subnet, which lacks public accessibility.

Problem

A public subnet has a route to an internet gateway configured in its route table. This helps making a connection between the VPC and the internet.
A private subnet does not have any route to an internet gateway. We cannot access resources within the private subnet from outside the VPC.
We should not put our RDS instances in public subnets only for the purpose of accessing them because this leads to direct public access of our data which is a security concern.
However, at times, we may want to access our data locally for debugging purposes. Therefore, we need a mechanism to connect to our database in a private subnet.

Resolution

Get along, till end, you will find out.

Create an EC2 instance

We need to create an EC2 instance keeping the following features:

Locate the EC2 instance in a public subnet
Make sure it is the same VPC in which we have put the RDS instance
Don't forget to download the .pem file to access the EC2 instance
Associate the EC2 with a security group which has inbound access from anywhere (or at least your own IP)

Once you follow above steps, you have a pem file, and an EC2 host, with you, which we are going to utilize later using the universal ec2 user, which is ec2-user.

Testing local EC2 connection

To test the local connection to EC2, we are using jetbrains Intellij, and will be following below steps:

Go to data sources and drivers, then the SSH tunnel tab.
Set an SSH configuration, mentioning the
EC2 host
EC2 user (default user: ec2-user)
Select the authentication method as key pair, then locate the pem file we downloaded earlier for EC2 connection.
Now, check if correct, by testing connection.

Now, we have EC2 securely connected on our local, which we are going to use for connecting with database now.

Configuring local Database connection

To connect with DB, again we will use intelliJ IDE. For making connection with the DB, you need following details:

RDS instance host
DB username
DB password
DB Port
DB name

Go to the general tab, enter above details.

Once filled, test the connection, if you have followed above steps along with me, we will surely see below prompt.

Note

There is also a super easy way to connect to private RDS with 7777 and RDS

Spring Boot and AWS

Tom Spencer — Mon, 03 Apr 2023 08:21:14 +0000

Introduction

In this article we will learn about everything that is required for deploying a Spring Boot application
to AWS. We will deploy a Docker container and learn how to use AWS CloudFormation and the AWS Cloud Development kit (CDK)
to automate deployments and build a continuous deployment pipeline with Github Actions.

We will also learn about several AWS services that we can use for common tasks. We will build a user registration and login by
leveraging Amazon Cognito. We will connect our Spring Boot application with a relational database.
We will also build a feature to share todos between users fully managed by AWS.

We will then look at some of the important aspects for running an application in production. We will learn how to use Amazon CloudWatch
to view logs and metrics. By actively monitoring our application and creating alerts we will be able to detect failures quickly.

Useful links:
https://github.com/stratospheric-dev/stratospheric/tree/main/chapters
https://stratospheric.dev/

Spring Boot and AWS

Spring Boot is the leading framework for building applications with Java. With Spring Boot we can quickly build
production-ready software. AWS is the leading cloud platform and we can make use of a vast array of AWS cloud services
to help us architect, build and deploy software applications. Netflix and Atlassian use AWS and run all of their
Software as a Service applications on AWS infrastructure. This article will help you to integrate Spring Boot applications with
AWS. Operational work can seem daunting as a developer particularly since we are used to depending on operations teams to
deploy our code. However, the control that is offered by becoming a Devops engineer is empowering and learning a few AWS services
and tips and tricks can help take your devops skills to the next level.

In this article we will learn hands-on how to get a Spring Boot application into the cloud and operate it on the cloud.
We will build a continuous deployment pipeline, access AWS services from the Spring Boot application and learn how to monitor
the application on the cloud.

Deploying with AWS

This link is useful for setting up the aws cli:
https://docs.aws.amazon.com/cli/latest/reference/

These are commands to create a vpc:

aws ec2 create-vpc --cidr-block 10.0.0.0/16 --region ap-southeast-2
aws ec2 describe-vpcs --region ap-southeast-2
aws ec2 delete-vpc --region ap-southeast-2 --vpc-id=<VPC-ID>
aws ec2 describe-vpcs --region ap-southeast-2

This is an example vpc configuration using CloudFormation. This file contains a stack of resources.
This file describes a stack:

AWSTemplateFormatVersion: '2010-09-09'

Description: Deploys a VPC

Resources:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'

Outputs:

  VPCId:
    Description: The ID of the VPC that this stack is deployed in
    Value: !Ref 'VPC'

We can use these commands to create and delete a CloudFormation stack:

aws cloudformation deploy --template-file vpc-stack.yml --stack-name vpc-stack --region ap-southeast-2
aws cloudformation delete-stack --stack-name vpc-stack --region ap-southeast-2
aws cloudformation describe-stacks --region ap-southeast-2

We can also use the CDK to deploy a stack:


public class CdkApp {
    public static void main(final String[] args) {
        App app = new App();

        new CdkStack(app, "CdkStack", StackProps.builder()
                .env(Environment.builder()
                .account("221875718260")
                .region("ap-southeast-2")
                .build())
                .build());

        app.synth();
    }
}

This is the Stack for the CdkApp:

public class CdkStack extends Stack {
    public CdkStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public CdkStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        new Vpc(this, "vpc", VpcProps.builder()
                .cidr("10.0.0.0/24")
                .build());
    }
}

The above stack creates VPC. CDK calls the CloudFormation API. This allows us to build our own construct libraries.
This makes the configuration more useful. In order to run the CDK code we should use the CDK command line tool which we
can install with the following:

npm install -g aws-cdk             # install latest version

AWS allows us to use the services via the web console, AWS CLI, CloudFormation and CDK.
CloudFormation allows us to declare resources and deploy them with a single command. CDK allows
us to combine resources to "constructs" and share them as Java libraries.

Using AWS (setup)

aws --version
aws configure
aws configure --profile stratospheric
export AWS_PROFILE=stratospheric
aws ec2 describe-vpcs --profile stratospheric

This is how to check your configuration:

tom@tom-ubuntu:~/.aws$ aws configure --profile default
AWS Access Key ID [****************HJEB]: 
AWS Secret Access Key [****************mdFI]: 
Default region name [eu-west-2]: 
Default output format [None]:

We can update our cli with:

pip3 install awscli --upgrade --user

We should add our configuration for a different profile with the following:

aws configure --profile stratospheric

We then add the credentials and configuration we wish to use with the profile. The profile information
is kept in the .aws folder.
In this section we have looked at creating a "free" AWS account. We have installed the AWS CLI and we can use profiles
if we want to manage multiple AWS accounts with the AWS CLI.

Elastic Container Registry

Here we will learn how to wrap an app into a Docker image. We will also learn how to create a Docker repository with Amazon
Elastic Container Registry (ECR) and how to publish a Docker image to an Elastic Container Registry (ECR)
repository.

The application that we will deploy is a simple Hello World app. This is the Dockerfile for the deployment:

FROM eclipse-temurin:17-jre

ARG JAR_FILE=build/libs/*.jar
COPY ${JAR_FILE} app.jar

ENTRYPOINT ["java", "-jar", "/app.jar"]

This Dockerfile downloads Java 17, builds the application to a Docker image with an entry point to the jar file.
We build the jar file with:

./gradlew clean build

We then build the docker image:

docker build -t hello-world-app .

Here we give the docker image the name hello-world-app.
We check for our docker image with the follow grep on docker image ls:

tom@tom-ubuntu:~/Projects/stratospheric-dev$ docker image ls | grep hello-world-app
hello-world-app   latest    f1307dc16f7e   48 seconds ago   288MB

We can then run the application and expose the app to port 8080 on our host computer:

docker run -p 8080:8080 hello-world-app

We can now see the application on port 8080:

We are now going to create a docker repository on Amazon ECR. We now go to Amazon Elastic Container Registry on AWS.
We should enable tag immutability to avoid changing tags later.

We then create the repository.

For the moment it has no Docker images:

We now tag our image locally with the url of our new docker repository and the tags latest and v1:

tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ docker tag hello-world-app 706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app:latest
tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ docker tag hello-world-app 706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app:v1

We can check the tags by looking for our images in our local repository:

tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ docker image ls | grep hello-world-app
706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app   latest    e4d2957745e5   3 minutes ago   288MB
706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app   v1        e4d2957745e5   3 minutes ago   288MB
hello-world-app                                                latest    e4d2957745e5   3 minutes ago   288MB

We now have two images with the url of our ECR repository and tagged with latest and v1 respectively.
We now log into the docker repository to publish our image. We use the AWS cli and login to docker with our aws repository:

tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 706054169063.dkr.ecr.us-east-1.amazonaws.com 
WARNING! Your password will be stored unencrypted in /home/tom/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

We then push the latest image to our repository:

tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ docker push 706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app:latest

We can now see the image in our ECR repository:

We also push v1:

tom@tom-ubuntu:~/Projects/stratospheric/helloWorld$ docker push 706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app:v1
The push refers to repository [706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app]
8e12efddf5f4: Layer already exists 
5ff725d720ec: Layer already exists 
3b028c775252: Layer already exists 
f3a12c51479f: Layer already exists 
b93c1bd012ab: Layer already exists 
v1: digest: sha256:5b1075ab16f8bfdd00930963af1bfaca318fd73715aaccdb1e77c865d6dd1d40 size: 1372

We now have the tag of v1 and latest in our repository:

So far we have created a Docker repository with ECR. We create one Docker repository per app. We can also pull and push to the ECR repository
from the AWS CLI.

CloudFormation

CloudFormation is the AWS service for infrastructure as code. AWS creates the resources that we declare in our CloudFormation file.
We now look at resources and stacks. We will look at how to create a stack template in yaml and how to create a stack from
a template file. We will also find our stack on the AWS web console.

We declare the resources we need in a stack template. These might include a VPC, Load balancer and a database. We declare the
resources in a stack file and CloudFormation deploys the resources for us. The CloudFormation docs list all the resources
we can deploy with CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

For each of the resources we need to refer to the documentation for each resource. A stack is a group of resources that belong
together. All the resources in the stack are created updated and deleted together.
We are now going to create a CloudFormation stack to create an ECR repository:

We will create a Docker repository stack:

We first need to look at the CloudFormation docs and ECR repository in particular:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecr-repository.html

This shows us the options that we can use for our stack:

Type: AWS::ECR::Repository
Properties: 
  EncryptionConfiguration: 
    EncryptionConfiguration
  ImageScanningConfiguration: 
    ImageScanningConfiguration
  ImageTagMutability: String
  LifecyclePolicy: 
    LifecyclePolicy
  RepositoryName: String
  RepositoryPolicyText: Json
  Tags: 
    - Tag

Each of the options are documented on the page. We can use the configuration details given on the page to create ecr.yml page:

AWSTemplateFormatVersion: '2010-09-09'
Description: Deploys a Docker container registry.
Parameters:
  RegistryName:
    Type: String
    Description: The name of the container registry
Resources:
  Registry:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: !Ref 'RegistryName'
      LifecyclePolicy:
        LifecyclePolicyText: |
          {
              "rules": [
                  {
                      "rulePriority": 10,
                      "description": "Limit to 10 images",
                      "selection": {
                         "tagStatus": "any",
                         "countType": "imageCountMoreThan",
                         "countNumber": 10
                     },
                      "action": {
                          "type": "expire"
                      }
                  }
              ]
          }
      RepositoryPolicyText:
        Version: "2012-10-17"
        Statement:
          - Sid: AllowPushPull
            Effect: Allow
            Principal:
              AWS:
                - "arn:aws:iam::706054169063:user/tom-developer"
            Action:
              - "ecr:GetDownloadUrlForLayer"
              - "ecr:BatchGetImage"
              - "ecr:BatchCheckLayerAvailability"
              - "ecr:PutImage"
              - "ecr:InitiateLayerUpload"
              - "ecr:UploadLayerPart"
              - "ecr:CompleteLayerUpload"

The parameters section:

Parameters:
  RegistryName:
    Type: String
    Description: The name of the container registry

means that when we deploy the stack we can add a parameter with the registry name. This means we can
reuse the stack. We set the name of the repository (RepositoryName) with the input parameter we added earlier:

Resources:
  Registry:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: !Ref 'RegistryName'

Here the lifecycle policy sets a limit of 10 images and will delete images when the number of images is above that number:

LifecyclePolicy:
        LifecyclePolicyText: |
          {
              "rules": [
                  {
                      "rulePriority": 10,
                      "description": "Limit to 10 images",
                      "selection": {
                         "tagStatus": "any",
                         "countType": "imageCountMoreThan",
                         "countNumber": 10
                     },
                      "action": {
                          "type": "expire"
                      }
                  }
              ]
          }

The repository policy text specifies the allowed actions on the repository for the Principal users that we define.
The actions include sending images to the repository and downloading images.
We deploy the stack with the following command:

tom@tom-ubuntu:~/Projects/stratospheric/cloudformation$ aws cloudformation deploy \
> --stack-name=hello-world-docker-repository \
> --template-file ecr.yml \
> --parameter-overrides RegistryName=hello-world-app \
> --profile stratospheric

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - hello-world-docker-repository

This has created a Docker ECR repository in the eu-west-2 region:

If we use the same stack-name we can change the properties of the same stack.
We can also see the CloudFormation stack on the CloudFormation page:

We can also look at the hollow-world-docker-repository and the actions and events for the CloudFormation stack:

So far we have declared resources in a stack file. CloudFormation has taken care of the lifecycle of the resources that we
defined. We have parameterised the stack file so that we can use the stack file for multiple deployments with different parameters.

Deploying a Network Stack with Cloud Formation

In this section we will look at why we use separate CloudFormation stacks for Network deployments.
We will look at the networking resources AWS provides. We will also learn more about the CloudFormation stack file
syntax.

The above network stack contains all the resources we need so that our users can access the internet. It also includes
internal networking so that our application can access the database for example. We deploy a network stack which can be deployed
once. The application is deployed in the service stack which includes the compute resources that we require for our application.
There will be different requirements in different contexts.

The above is an overview of the main resources that our stack will contain. This is what the network stack yaml file will
look like:

AWSTemplateFormatVersion: '2010-09-09'
Description: A basic network stack that creates a VPC with a single public subnet and some ECS resources that we need to start a Docker container within this subnet.
Resources:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: {Ref: 'AWS::Region'}
      VpcId: !Ref 'VPC'
      CidrBlock: '10.0.1.0/24'
      MapPublicIpOnLaunch: true

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  GatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref 'VPC'
      InternetGatewayId: !Ref 'InternetGateway'

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref 'VPC'

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayAttachment
    Properties:
      RouteTableId: !Ref 'PublicRouteTable'
      DestinationCidrBlock: '0.0.0.0/0'
      GatewayId: !Ref 'InternetGateway'

  ECSCluster:
    Type: AWS::ECS::Cluster

  ECSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Access to the ECS containers
      VpcId: !Ref 'VPC'

  ECSSecurityGroupIngressFromAnywhere:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Allows inbound traffic from anywhere
      GroupId: !Ref 'ECSSecurityGroup'
      IpProtocol: -1
      CidrIp: 0.0.0.0/0

  ECSRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs.amazonaws.com]
            Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: ecs-service
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  # Rules which allow ECS to attach network interfaces to instances
                  # on your behalf in order for awsvpc networking mode to work right
                  - 'ec2:AttachNetworkInterface'
                  - 'ec2:CreateNetworkInterface'
                  - 'ec2:CreateNetworkInterfacePermission'
                  - 'ec2:DeleteNetworkInterface'
                  - 'ec2:DeleteNetworkInterfacePermission'
                  - 'ec2:Describe*'
                  - 'ec2:DetachNetworkInterface'
                Resource: '*'

  ECSTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: AmazonECSTaskExecutionRolePolicy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  # Allow the ECS Tasks to download images from ECR
                  - 'ecr:GetAuthorizationToken'
                  - 'ecr:BatchCheckLayerAvailability'
                  - 'ecr:GetDownloadUrlForLayer'
                  - 'ecr:BatchGetImage'

                  # Allow the ECS tasks to upload logs to CloudWatch
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'

Outputs:
  ClusterName:
    Description: The name of the ECS cluster
    Value: !Ref 'ECSCluster'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ClusterName' ] ]
  ECSRole:
    Description: The ARN of the ECS role
    Value: !GetAtt 'ECSRole.Arn'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSRole' ] ]
  ECSTaskExecutionRole:
    Description: The ARN of the ECS role
    Value: !GetAtt 'ECSTaskExecutionRole.Arn'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSTaskExecutionRole' ] ]
  VPCId:
    Description: The ID of the VPC that this stack is deployed in
    Value: !Ref 'VPC'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'VPCId' ] ]
  PublicSubnet:
    Description: Public subnet one
    Value: !Ref 'PublicSubnet'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnet' ] ]
  ECSSecurityGroup:
    Description: A security group used to allow ECS containers to receive traffic
    Value: !Ref 'ECSSecurityGroup'
    Export:
      Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSSecurityGroup' ] ]

This link is useful for considering the built in functions that we can use with AWS CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html
In the above stack we are using two CloudFormation intrinsic functions in particular


 and

 ```Fn::GetAZs```

.


```Fn::Select```

 returns a single object from a list of objects by index.

 ```Fn::GetAZs```

 returns
an array that lists Availability Zones for a specified region in alphabetical order. So the following section of the above
CloudFormation stack:


```yaml
        Fn::Select:
          - 0
          - Fn::GetAZs: {Ref: 'AWS::Region'}

selects the first Availability Zone returned from the region inputted by the command calling the above CloudFormation template.
This doc is useful for AWS::EC2::VPC.
The command specifies a virtual private cloud (VPC). In the above example we have specified a CIDR block for our VPC:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: '10.0.0.0/16'

This CIDR block contains IP addresses from 10.0.0.0 to 10.0.255.255. The 16 means that we can select any number between 0 and 256 on
the second to last and last sections of the 4 section IP address.
This doc is useful for AWS::EC2::Subnet

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: {Ref: 'AWS::Region'}
      VpcId: !Ref 'VPC'
      CidrBlock: '10.0.1.0/24'
      MapPublicIpOnLaunch: true

The


 specifies a subnet for the specified VPC. Here we have selected the same Availability Zone
as the above VPC and use the command line argument to set the id of the VPC. The number 24 means that the earlier sections are fixed
and we can only use a range from 10.0.1.0 to 10.0.1.255 for our IP address. The entry

 ```MapPublicIpOnLaunch```


indicates whether instances launched in the subnet receive a public IPv4 address. The default value is

 ```false```

.
Here we have selected true.

This doc is useful for [AWS::EC2::InternetGateway](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-internetgateway.html).
It allocates an internet gateful for use with our VPC. After creating the Internet Gateway we attach it to the VPC with VPCId 
entry. We also add an [AWS::EC2::RouteTable](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-routetable.html).
A route table is a set of rules, called routes, that determine where the network traffic from our subnet or gateway is directed.
Here we attach the Route Table to our VPC. We also use [AWS::EC2::AWS::EC2::SubnetRouteTableAssociation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnetroutetableassociation.html) to associate the subnet with the route table.
The [AWS::EC2::Route](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route.html) specifies the route table within the VPC.
Here we specify the

 ```DestinationCidrBlock```

 as 0.0.0.0/0. We also add the Public Gateway as a parameter on our command line argument.



```yaml


  InternetGateway:
    Type: AWS::EC2::InternetGateway

  GatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref 'VPC'
      InternetGatewayId: !Ref 'InternetGateway'

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref 'VPC'

  PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayAttachment
    Properties:
      RouteTableId: !Ref 'PublicRouteTable'
      DestinationCidrBlock: '0.0.0.0/0'
      GatewayId: !Ref 'InternetGateway'

We use the following command to deploy the above network stack:

aws cloudformation deploy \
--stack-name=dev-network \
--template-file network.yml \
--capabilities CAPABILITY_IAM \
--profile stratospheric

The Elastic Container Stack (ECS) cluster is the service we use to contain the resources we want to deploy. It is in the network stack
because we will only deploy the ECS cluster once. We run the above command and can see that the stack has been created:

We have created a network stack which will not need to change. We have added input parameters which we then use in the stack template.
We have also used the CloudFormation docs to explore the available AWS resources.

Deploying Service Stack with CloudFormation

Here we will learn about ECS basics including clusters, services and tasks. We will define ECS resources with CloudFormation
and we will learn about the interactions that are possible between CloudFormation stacks.

ECS Basics

Elastic Container Service (ECS) is an AWS self service container orchestration service. The applications to deploy are called tasks.
In the diagram above we have multiple tasks. Service A runs two instances of Task 1 and one of Task 2. The application may need more resources
for Task 1 so that is why we have included them within this service. The cluster is a wrapper around one or more services and binds the services
to a specific AWS region and VPC. Each task is a process. We can now look at how each Task is assigned a process:

The Task Definition tells us what Docker image to run and then we use the image within that Docker Repository. There are two
different launch types for ECS - Fargate and EC2. The Fargate launch type does work for us. Fargate spins a fleet of required EC2 instances and manages
them for us after we have specified which tasks to run and how many task instances we want to have running at all times. The EC2
launch type allows us to control the EC2 instances that are running our tasks and we have to manage this process. We can define
auto scaling rules but have to define our own auto-scaling group. We are going to use the Fargate Launch type.
We have defined ECS in our network.yml file:



  ECSCluster:
    Type: AWS::ECS::Cluster

  ECSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Access to the ECS containers
      VpcId: !Ref 'VPC'

  ECSSecurityGroupIngressFromAnywhere:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Allows inbound traffic from anywhere
      GroupId: !Ref 'ECSSecurityGroup'
      IpProtocol: -1
      CidrIp: 0.0.0.0/0

Here we define a security group that allows inbound traffic from anywhere. We set the IpProtocol to -1 which means any
ip protocol. Later in the file we add IAM role for permission for ECS and logging. We are using the same ECS cluster for all
services.

Deploying the Service Stack with CloudFormation

For our Service Stack we have added several parameters:

Parameters:
  NetworkStackName:
    Type: String
    Description: The name of the networking stack that
      these resources are put into.
  ServiceName:
    Type: String
    Description: A human-readable name for the service.
  ImageUrl:
    Type: String
    Description: The url of a docker image that will handle incoming traffic.
  ContainerPort:
    Type: Number
    Default: 80
    Description: The port number the application inside the docker container
      is binding to.
  ContainerCpu:
    Type: Number
    Default: 256
    Description: How much CPU to give the container. 1024 is 1 CPU.
  ContainerMemory:
    Type: Number
    Default: 512
    Description: How much memory in megabytes to give the container.
  DesiredCount:
    Type: Number
    Default: 1
    Description: How many copies of the service task to run.

These include the network stack name, service name, image url and container port for each service that we want to deploy
within the cluster. We have a Task section to define the task within our service:

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref 'ServiceName'
      Cpu: !Ref 'ContainerCpu'
      Memory: !Ref 'ContainerMemory'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn:
        Fn::ImportValue:
          !Join [':', [!Ref 'NetworkStackName', 'ECSTaskExecutionRole']]
      ContainerDefinitions:
        - Name: !Ref 'ServiceName'
          Cpu: !Ref 'ContainerCpu'
          Memory: !Ref 'ContainerMemory'
          Image: !Ref 'ImageUrl'
          PortMappings:
            - ContainerPort: !Ref 'ContainerPort'
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'ServiceName'
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Ref 'ServiceName'

We also define the service with a maximum percent deployment configuration for when we redeploy Tasks within the service:

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Ref 'ServiceName'
      Cluster:
        Fn::ImportValue:
          !Join [':', [!Ref 'NetworkStackName', 'ClusterName']]
      LaunchType: FARGATE
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 50
      DesiredCount: !Ref 'DesiredCount'
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups:
            - Fn::ImportValue:
                !Join [':', [!Ref 'NetworkStackName', 'ECSSecurityGroup']]
          Subnets:
            - Fn::ImportValue:
                !Join [':', [!Ref 'NetworkStackName', 'PublicSubnet']]
      TaskDefinition: !Ref 'TaskDefinition'

We will use the hello-world-app for deploying our service. We need the url of our hello-world-app.

This is the command for deploying the service:

aws cloudformation deploy \
--stack-name dev-service \
--template-file service.yml \
--profile stratospheric \
--parameter-overrides \
  NetworkStackName=dev-network \
  ServiceName=hello-world-app \
  ImageUrl=706054169063.dkr.ecr.us-east-1.amazonaws.com/hello-world-app:latest
  ContainerPort=8080

We can then see the created stack on CloudFormation Stacks:

We also now have one cluster running with one service and one running task:

If we check the Task for the running Service we see that we have a running task and we can also see a public IP address:

We can now type in the ip address at the port 8080:

and we can see our running task using the Docker image that we deployed to ECR.

We have looked at ECS tasks which run docker images that run in an ECS service and ECS cluster.
A task can run one or more Docker images and we can use resources across different CloudFormation stacks using output and input
parameters. In order to replace our ip address we will use a load balancer as a single point of entry for the stack with a fixed host name.

Using Cloud Development Kit (CDK)

With CloudFormation we declared the resources we want in a cloud yaml file. We will now use Infrastructure as Code with Java to describe
the cloud resources required. CDK is an abstraction on top of CloudFormation.
CDK introduces the idea of a construct to combine multiple resources. Level 1 constructs represent one CloudFormation Resource.
Level 2 Constructs represent a set of resources. Level 2 Constructs infer some of the options for us and set sensible defaults.
Level 3 Constructs create patterns for several resources.

We install the CDK CLI with:

tom@tom-ubuntu:~/Projects/stratospheric/cloudformation$ npm install -g aws-cdk
tom@tom-ubuntu:~/Projects/stratospheric/cloudformation$ cdk --version
2.70.0 (build c13a0f1)

We create a new cdk app with:

cdk init app --language=java

This creates an application with two main parts. The App:

public class CdkExampleApp {
    public static void main(final String[] args) {
        App app = new App();

        new CdkExampleStack(app, "CdkExampleStack", StackProps.builder()
                .build());

        app.synth();
    }
}

This class creates an App object and a stack object linked to the app. The synth command creates a synthesis
step to create a CloudFormation template from the Java code.

The Stack class is where we define the resources that we want to deploy:

public class CdkExampleStack extends Stack {
    public CdkExampleStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public CdkExampleStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);
    }
}

This is a maven project and we can use the following commands:

It is a [Maven](https://maven.apache.org/) based project, so you can open this project with any Maven compatible Java IDE to build and run tests.

## Useful commands

 * `mvn package`     compile and run tests
 * `cdk ls`          list all stacks in the app
 * `cdk synth`       emits the synthesized CloudFormation template
 * `cdk deploy`      deploy this stack to your default AWS account/region
 * `cdk diff`        compare deployed stack with current state
 * `cdk docs`        open CDK documentation

We need maven versions to be shared across different environments. This will help us to install the correct
maven version. We install the maven wrapper with the following command:

mvn wrapper:wrapper

This installs the .mvn file:

We can now use the command


 with the same maven commands as before.

We also change the cdk.json to refer to the mvnw command:


```yaml
{
  "app": "mvnw -e -q compile exec:java"
}

We now need to run the following command to create infrastructure for cdk to deploy apps to CloudFormation:

cdk bootstrap --profile stratospheric

This creates an S3 bucket and IAM roles for the deployment.
We can now deploy the application with:

cdk deploy --profile stratospheric

We can now see the CloudFormation stack in the CloudFormation web console:

At the moment the stack is empty. We can now create the ECS repository with CDK using the following doc:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecr.Repository.html

Level 2 Constructs are listed as Constructs. The level 1 Constructs are listed as Cfn... For the moment
we will stick with Level 2 Constructs. We get the correct methods for our CDK Stack with the following:
https://docs.aws.amazon.com/cdk/api/v2/java/index.html?software/amazon/awscdk/services/ecr/Repository.html

Our CdkExampleStack now looks like the following:

public class CdkExampleStack extends Stack {
    public CdkExampleStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public CdkExampleStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        Repository.Builder.create(this, "repository")
                .repositoryName("hello-world-repo")
                .imageScanOnPush(Boolean.TRUE)
                .lifecycleRules(Arrays.asList(LifecycleRule.builder()
                        .maxImageCount(10)
                        .build()))
                .imageTagMutability(TagMutability.IMMUTABLE)
                .build();
    }
}

We can now deploy our cdk app with:

cdk deploy --profile stratospheric

This has created a CloudFormation stack

and a hello-world-repo ECR repository:

For now this repository is empty. We will use the Stratospheric cdk construct maven dependency here
to add more to our repository configuration. We will add this configuration with the following maven dependency:

<dependency>
    <groupId>dev.stratospheric</groupId>
    <artifactId>cdk-constructs</artifactId>
    <version>0.1.13</version>
</dependency>

We can now reduce the code using the stratospheric cdk-constructs dependency:

public class CdkExampleStack extends Stack {
    public CdkExampleStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public CdkExampleStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        String accountId = "12345";
        Environment environment = Environment.builder()
                .account(accountId)
                .region("ap-southeast-2")
                .build();

        DockerRepository repo = new DockerRepository(this, "repo", environment,
                new DockerRepository.DockerRepositoryInputParameters(
                        "hello-world-repo",
                        accountId,
                        10
                ));
    }
}

This will do the same as above but also add push and pull permissions. We can delete the hello-world-repo
and create a new stack with the following command:

cdk deploy --profile stratospheric

This fails because we have the incorrect AccountID in /stratospheric/cdk-example/cdk.out/CdkExampleStack.template.json:

[
  "arn:",
   {
    "Ref": "AWS::Partition"
   },
  ":iam::12345:root"
]

We will look at how to fix this error in the next section.

In this section, we have abstracted some of CloudFormation's complexity with Constructs and used some of our knowledge of
CloudFormation resources. We have also used a shared library to share the construct across projects.

CDK Best Practices

We will now learn how to debug a CDK app. We will manage different environments with CDK, manage input parameters and
also manage multiple stacks.

First we correct the above code with an environment object:

public class CdkExampleApp {
    public static void main(final String[] args) {
        App app = new App();

        String accountId = (String) app.getNode().tryGetContext("accountId");
        String region = (String) app.getNode().tryGetContext("region");

        Environment environment = makeEnv(accountId, region);

        new CdkExampleStack(app, "CdkExampleStack", StackProps.builder()
                .env(environment)
                .build());

        app.synth();
    }

    static Environment makeEnv(String account, String region) {
        return Environment.builder()
                .account(account)
                .region(region)
                .build();
    }
}

The environment can then be used in the Stack directly:

public class CdkExampleStack extends Stack {
    public CdkExampleStack(final Construct scope, final String id) {
        this(scope, id, null);
    }

    public CdkExampleStack(final Construct scope, final String id, final StackProps props) {
        super(scope, id, props);

        Environment env = props.getEnv();

        DockerRepository repo = new DockerRepository(this, "repo", env,
                new DockerRepository.DockerRepositoryInputParameters(
                        "hello-world-repo",
                        env.getAccount(),
                        10
                ));
    }
}

We now need to pass the accountId into our command. We can call the above code with the following:

cdk deploy -c accountId=<ACCOUNT_ID> region=eu-west-2

This successfully creates the hello-world-repo on ECR.

AWS AppConfig Agent for Containers

What is AWS AppConfig?
Why did we develop an agent?
About the agent
Container image
Demo: Using the agent in ECS

What is AWS AppConfig?

AppConfig is changing the way software is developed released and run. It is a dynamic configuration service.
Configuration changes the way the software runs depending on the environment. For instance we may want to use different
databases for different environments. Before dynamic configuration we had static configuration.
For applications we have source code and configuration data. Static Configuration can be time consuming.
In order to change endpoints we would have to roll back the application.
Dynamic configuration allows us to deploy the configuration data from a remote service at runtime.
If we use AppConfig we can change configuration separately from the deployment.
There are deployment safety configurations we can use with AppConfig which allow us to deploy the configuration changes
gradually. AppConfig includes Feature Flags, Access Control Lists and throttling limits.
Throttling is a DDOS protection mechanism which allows a limit of requests to the service. You can allow overrides for
particular users.

AppConfig Agent

Before AppConfig Agent management logic including caching, background polling and session management needed to be contained
within AppConfig directly. The agent is a sidecar process that runs alongside the application. It allows us to add configuration
separately. Customers can then set their own config themselves. This is the image gallery:

These doc is useful for AWS AppConfig integration with Amazon ECS and Amazon EKS:
https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-containers-agent.html

We have now added two Apps to be deployed and deleted the plugin code relating to the default App:

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <mainClass>com.myorg.CdkExampleApp</mainClass>
                </configuration>
            </plugin>

This does mean that we have to add the App name as parameter when we deploy our App.

tom@tom-ubuntu:~/Projects/stratospheric/cdk-example$ cdk deploy --app "./mvnw -e -q compile exec:java -Dex
ec.mainClass=com.myorg.CdkExampleApp2" \
--profile stratospheric \
-c accountId=<ACCOUNT_ID> \
-c region=eu-west-2 \
-c repoName=foo

We can also add the arguments to the cdk.json file:

{
  "context": {
    "region": "eu-west-2",
    "accountId": "706054169063",
    "repoName": "my-cool-repo"
  }
}

This shortens our command:

cdk deploy --app "./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.CdkExampleApp2" --profile stratospheric

We can also create a package.json file to deploy and destroy our apps:

{
  "name": "hello-cdk",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "stack1:deploy": "cdk deploy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.CdkExampleApp \" --require-approval never --profile stratospheric",
    "stack1:destroy": "cdk destroy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.CdkExampleApp \" --require-approval never --profile stratospheric",
    "stack2:deploy": "cdk deploy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.CdkExampleApp2 \" --require-approval never --profile stratospheric",
    "stack2:destroy": "cdk destroy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.CdkExampleApp2 \" --require-approval never --profile stratospheric",
  },
  "devDependencies": {
    "aws-sdk": "2.43.1"
  },
  "engines": {
    "node": ">=16"
  }
}

We run npm install to initialize the npm project and can run:

npm run stack2:deploy

The parameters for the deploy are still kept in the cdk.json file:

{
  "context": {
    "region": "eu-west-2",
    "accountId": "706054169063",
    "repoName": "my-cool-repo"
  }
}

We could override the values in the cdk.json file from the command line.

So far we have looked at debugging CloudFormation scripts by using the cdk.out file.
We have tried to stick to the rule of using one stack per app and we have used the cdk.json
and npm to save retyping scripts and parameters.

Deploying a Network Stack with CDK

In this section we will build a CDK app to deploy a network. We will explore the Network construct and experiment with
sharing parameters between CDK constructs with SSM.
This is a cleaned up version of the DockerRepositoryApp:

public class DockerRepositoryApp {

    public static void main(String[] args) {
        App app = new App();

        String accountId = (String) app.getNode().tryGetContext("accountId");
        Validations.requireNonEmpty(accountId, "context variable 'accountId' must not be null");

        String region = (String) app.getNode().tryGetContext("region");
        Validations.requireNonEmpty(region, "context variable 'region' must not be null");

        String applicationName = (String) app.getNode().tryGetContext("applicationName");
        Validations.requireNonEmpty(applicationName, "context variable 'applicationName' must not be null");

        Environment awsEnvironment = makeEnv(accountId, region);

        Stack dockerRepositoryStack = new Stack(app, "DockerRepositoryStack", StackProps.builder()
                .stackName(applicationName + "-DockerRepository")
                .env(awsEnvironment)
                .build()
        );

        DockerRepository dockerRepository = new DockerRepository(
                dockerRepositoryStack,
                "DockerRepository",
                awsEnvironment,
                new DockerRepository.DockerRepositoryInputParameters(applicationName, accountId)
        );

        app.synth();

    }

    static Environment makeEnv(String account, String region) {
        return Environment.builder()
                .account(account)
                .region(region)
                .build();
    }
}

We then create the NetworApp:

public class NetworkApp {

    public static void main(String[] args) {
        App app = new App();

        String accountId = (String) app.getNode().tryGetContext("accountId");
        Validations.requireNonEmpty(accountId, "context variable 'accountId' must not be null");

        String region = (String) app.getNode().tryGetContext("region");
        Validations.requireNonEmpty(region, "context variable 'region' must not be null");

        String applicationName = (String) app.getNode().tryGetContext("applicationName");
        Validations.requireNonEmpty(applicationName, "context variable 'applicationName' must not be null");

        String environmentName = (String) app.getNode().tryGetContext("environmentName");
        Validations.requireNonEmpty(environmentName, "context variable 'environmentName' must not be null");


        Environment env = makeEnv(accountId, region);

        Stack networkStack = new Stack(app, "NetworkStack", StackProps.builder()
                .stackName(environmentName + "-Network")
                .env(env)
                .build()
        );

        Network network = new Network(
                networkStack,
                "Network",
                env,
                environmentName,
                new Network.NetworkInputParameters());

        app.synth();

    }

    static Environment makeEnv(String account, String region) {
        return Environment.builder()
                .account(account)
                .region(region)
                .build();
    }
}

We then add the NetworkApp to the package.json:


{
  "name": "cdk-example",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "repository:deploy": "cdk deploy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.DockerRepositoryApp \" --require-approval never --profile stratospheric",
    "repository:destroy": "cdk destroy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.DockerRepositoryApp \" --require-approval never --profile stratospheric"
    "network:deploy": "cdk deploy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.NetworkApp \" --require-approval never --profile stratospheric",
    "network:destroy": "cdk destroy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=com.myorg.NetworkApp \" --require-approval never --profile stratospheric"
  },
  "devDependencies": {
    "aws-cdk": "2.70.0"
  },
  "engines": {
    "node": ">=16"
  }
}

Here we are using the Network Construct to create the network:

public class Network extends Construct {
    public Network(Construct scope, String id, Environment environment, String environmentName, NetworkInputParameters networkInputParameters) {
        super(scope, id);
        this.environmentName = environmentName;
        this.vpc = this.createVpc(environmentName);
        this.ecsCluster = Builder.create(this, "cluster").vpc(this.vpc).clusterName(this.prefixWithEnvironmentName("ecsCluster")).build();
        this.createLoadBalancer(this.vpc, networkInputParameters.getSslCertificateArn());
        Tags.of(this).add("environment", environmentName);
    }

    private void createLoadBalancer(IVpc vpc, Optional<String> sslCertificateArn) {
        this.loadbalancerSecurityGroup = software.amazon.awscdk.services.ec2.SecurityGroup.Builder.create(this, "loadbalancerSecurityGroup").securityGroupName(this.prefixWithEnvironmentName("loadbalancerSecurityGroup")).description("Public access to the load balancer.").vpc(vpc).build();
        CfnSecurityGroupIngress ingressFromPublic = software.amazon.awscdk.services.ec2.CfnSecurityGroupIngress.Builder.create(this, "ingressToLoadbalancer").groupId(this.loadbalancerSecurityGroup.getSecurityGroupId()).cidrIp("0.0.0.0/0").ipProtocol("-1").build();
        this.loadBalancer = software.amazon.awscdk.services.elasticloadbalancingv2.ApplicationLoadBalancer.Builder.create(this, "loadbalancer").loadBalancerName(this.prefixWithEnvironmentName("loadbalancer")).vpc(vpc).internetFacing(true).securityGroup(this.loadbalancerSecurityGroup).build();
        IApplicationTargetGroup dummyTargetGroup = software.amazon.awscdk.services.elasticloadbalancingv2.ApplicationTargetGroup.Builder.create(this, "defaultTargetGroup").vpc(vpc).port(8080).protocol(ApplicationProtocol.HTTP).targetGroupName(this.prefixWithEnvironmentName("no-op-targetGroup")).targetType(TargetType.IP).deregistrationDelay(Duration.seconds(5)).healthCheck(HealthCheck.builder().healthyThresholdCount(2).interval(Duration.seconds(10)).timeout(Duration.seconds(5)).build()).build();
        this.httpListener = this.loadBalancer.addListener("httpListener", BaseApplicationListenerProps.builder().port(80).protocol(ApplicationProtocol.HTTP).open(true).build());
        this.httpListener.addTargetGroups("http-defaultTargetGroup", AddApplicationTargetGroupsProps.builder().targetGroups(Collections.singletonList(dummyTargetGroup)).build());
        if (sslCertificateArn.isPresent()) {
            IListenerCertificate certificate = ListenerCertificate.fromArn((String)sslCertificateArn.get());
            this.httpsListener = this.loadBalancer.addListener("httpsListener", BaseApplicationListenerProps.builder().port(443).protocol(ApplicationProtocol.HTTPS).certificates(Collections.singletonList(certificate)).open(true).build());
            this.httpsListener.addTargetGroups("https-defaultTargetGroup", AddApplicationTargetGroupsProps.builder().targetGroups(Collections.singletonList(dummyTargetGroup)).build());
            ListenerAction redirectAction = ListenerAction.redirect(RedirectOptions.builder().protocol("HTTPS").port("443").build());
            new ApplicationListenerRule(this, "HttpListenerRule", ApplicationListenerRuleProps.builder().listener(this.httpListener).priority(1).conditions(List.of(ListenerCondition.pathPatterns(List.of("*")))).action(redirectAction).build());
        }

        this.createOutputParameters();
    }
}

This Construct creates a VPC, ECSCluster and Load Balancer for us. The VPC Construct creates the Internet Gateway and Routing Table for us also.
CDK abstracts the details for us. For each construct the Network construct creates StringParameters which are accessible through AWS Systems Manager.

We can deploy the stack with:

npm run network:deploy -- -c environmentName=staging

We can also deploy the stack for production:

npm run network:deploy -- -c environmentName=prod

We can look on CloudFormation to see all the Network resources that have been deployed for us:

We can now see how Constructs can abstract a lot of the complexity of deploying a full stack application. Using the
We have also seen how the Construct creates entries in AWS Systems Manager for the parameters that are related to the resources
we have deployed.

Deploying a Service Stack with CDK

We will now use the Service Construct to create a service on AWS ECS.

The above image describes the main parts of the service app.

We now create a Service Stack:


public class ServiceApp {

    public static void main(String[] args) {
        App app = new App();

        String accountId = (String) app.getNode().tryGetContext("accountId");
        Validations.requireNonEmpty(accountId, "context variable 'accountId' must not be null");

        String region = (String) app.getNode().tryGetContext("region");
        Validations.requireNonEmpty(region, "context variable 'region' must not be null");

        String applicationName = (String) app.getNode().tryGetContext("applicationName");
        Validations.requireNonEmpty(applicationName, "context variable 'applicationName' must not be null");

        String environmentName = (String) app.getNode().tryGetContext("environmentName");
        Validations.requireNonEmpty(environmentName, "context variable 'environmentName' must not be null");

        String dockerRepositoryName = (String) app.getNode().tryGetContext("dockerRepositoryName");
        Validations.requireNonEmpty(dockerRepositoryName, "context variable 'dockerRepositoryName' must not be null");


        String dockerImageTag = (String) app.getNode().tryGetContext("dockerImageTag");
        Validations.requireNonEmpty(dockerImageTag, "context variable 'dockerImageTag' must not be null");



        Environment env = makeEnv(accountId, region);

        ApplicationEnvironment applicationEnvironment = new ApplicationEnvironment(
                applicationName,
                environmentName);

        Stack serviceStack = new Stack(app, "ServiceStack", StackProps.builder()
                .stackName(environmentName + "-Service")
                .env(env)
                .build()
        );

        Network.NetworkOutputParameters networkOutputParameters = Network.getOutputParametersFromParameterStore(serviceStack, applicationEnvironment.getEnvironmentName());

        Service service = new Service(
                serviceStack,
                "Service",
                env,
                applicationEnvironment,
                new Service.ServiceInputParameters(
                        new Service.DockerImageSource(dockerRepositoryName, dockerImageTag),
                        new HashMap<>()),
                networkOutputParameters
        );

        app.synth();

    }

    static Environment makeEnv(String account, String region) {
        return Environment.builder()
                .account(account)
                .region(region)
                .build();
    }
}

We are going to refer to the hello-world-app on ECR:

We can now deploy our service with:

npm run service:deploy -- -c environmentName=staging

We can now see all the Constructs created as part of our deployment:

We can view the logs of the deployment:

We can also see the app running on the loadbalancer associated with our target group:

So, we have deployed a SpringBoot application to the cloud using CDK. The CDK and CloudFormation did a lot of the hard work by spinning up the
Application Load Balancer and EC2 instances. We have built loosely coupled stacks. The Constructs have protected us from a lot of the
complexity of building the resources.

Continuous Deployment with CDK and Github Actions

We can now practice deploying our application to the cloud using Github Actions and build a CD pipeline.

Hello Stratospheric

This is the link for our Continuous Deployment github actions example:
https://github.com/TomSpencerLondon/hello-stratospheric

A Hello World app to showcase a continuous deployment pipeline with Spring Boot, CDK, and AWS.

Build app

./gradlew build

We have a Dockerfile for building the application:

FROM eclipse-temurin:17-jre

ARG JAR_FILE=build/libs/*.jar
COPY ${JAR_FILE} app.jar

ENTRYPOINT ["java", "-jar", "/app.jar"]

We can query our AWS credentials with:

aws sts get-caller-identity

Continuous Deployment

In the first lane of the diagram above we have the main pipeline which runs tests and builds for a jar file. This is then converted to an image that is published to ECR.
The deploy step deploys the application using the Service App from our repository. The ECS Service requires a VPC and an ECR
Repository. We manually create teh Docker Repository and Network before creating the Service. When we push to the git repository the app will be built and then published to the ECR repository. If the publish step is successful
the deploy step deploys the Docker image. The ECS Service requires a VPC and ECR Repository. The ECR Repository and the Network will not change
frequently so they are not included in the pipeline. We create the Docker Repository and Network manually.
In GitHub Actions we have a workflow which is a series of actions that are triggered for instance by a push to the main branch.
The Workflow contains one or more jobs and jobs contain one or more steps. Steps in a job always run in sequence.
We will start with a Bootstrap command pushing to the Docker repository.

We will have to enter the secrets in the github repository:

  AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}

We are keeping all the folders for AWS deployment with CDK in the CDK folder. The github workflows are kept in the .github folder.
As part of the manual section of the deploy we bootstrap the environment for the DockerRepository. Bootstrapping is the deployment of an
AWS CloudFormation template to a specific AWS environment (account and Region). The bootstrapping template accepts parameters that customize
some aspects of the bootstrapped resources. We can see the github actions on the Actions tab of our repository:

The instructions for the manual deploy are kept in 01-bootstrap.yml and 02-create-environment.yml. The build and deploy commands are kept in
03-build-and-deploy.yml. This runs every time there is a push to the main branch:

on:
  push:
    branches:
      - main

We also have a concurrency entry on deploy:

  deploy:
    runs-on: ubuntu-20.04
    name: Deploy Todo App (Demo)
    needs: build-and-publish
    timeout-minutes: 15
    if: github.ref == 'refs/heads/main'
    concurrency: hello-world-deployment
    steps:

This avoids conflicts if two people merge code in the same timeframe. This doc is
useful on concurrency. We can now see the staging-Service on CloudFormation:

If we click on the Load Balancer url in Resources we can see our app is up and running:

We have now wrapped the cdk commands in a GitHub workflow. We had two forms of workflow: manual triggers and on push.

Spring Boot & AWS

We are going to build an application with Spring Boot and deploy it to AWS Cloud.
This application includes:

Registration and login
CRUD: Viewing, adding and deleting Todos
Sharing Todos
Email notifications
Push notifications

The above architecture describes the features we will implement. We interact with a REST API for showing thymeleaf pages.
We use an RDS database on RDS and a DynamoDB database for storage. We will decouple sending emails with SQS. We will allow
push notifications for collaboration with Amazon MQ with a websocket broker relay.

Local Development

We will learn about the challenges of local cloud development.
We will also learn about LocalStack and Drop-in replacements for Amazon RDS and Amazon Cognito.
The goal is to make local development as easy as possible. LocalStack is a fully functional
AWS Cloud stack:
https://localstack.cloud/

We will use the localstack docker image:

docker run -p 4566:4566 -e SERVICES=s3 localstack/localstack

To use localstack we can use the following command:

aws s3api create-bucket --bucket toms-s3-bucket --endpoint-url http://localhost:4566 --create-bucket-configuration LocationConstraint=eu-west-1
{
    "Location": "http://toms-s3-bucket.s3.localhost.localstack.cloud:4566/"
}

With local development, we now need to connect to the LocalStack instance:

  cloud:
    aws:
      rds:
        enabled: false
      sqs:
        endpoint: http://localhost:4566
        region: eu-central-1
      mail:
        endpoint: http://localhost:4566
        region: eu-central-1
      credentials:
        secret-key: foo
        access-key: bar

We also hard code our credentials above. We then create our resources on LocalStack. We will use docker-compose.yml for this.
We use local-aws-infrastructure.sh to create our services:

awslocal sqs create-queue --queue-name stratospheric-todo-sharing

We use a local docker image for postgres to replace AWS RDS. We will also use KeyCloak to replace Amazon Cognito:

  keycloak:
    image: quay.io/keycloak/keycloak:18.0.0-legacy
    ports:
      - 8888:8080
    environment:
      - KEYCLOAK_USER=keycloak
      - KEYCLOAK_PASSWORD=keycloak
      - DB_VENDOR=h2
      - JAVA_OPTS=-Dkeycloak.migration.action=import -Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.file=/tmp/stratospheric-realm.json
    volumes:
      - ./src/test/resources/keycloak/stratospheric-realm.json:/tmp/stratospheric-realm.json

We will use docker compose before starting our application:

docker-compose up

We now have two properties application-dev and application-aws for local development.
We have learnt about the approaches for local cloud development, using the LocalStack and Docker and we have replaced
RDS and Cognito with a local postgres instance and KeyCloak.

Todo App Design

Building User Registration and Login with Amazon Cognito

We will now learn how to add User Registration and Login using AWS Cognito. We will first learn about Amazon Cognito.
We will then look at OAuth 2.0 and OpenID Connect and look at the OAuth Authorization Code Grant Flow.
Amazon Cognito is a managed service that provides authentication, authorization and user management for applications.
This allows us to delegate user management to AWS.

OAuth 2.0

Open Authorization is the industry standard for authentication and authorization. It provides a protocol for login for applications.
When we sign up we are asked to grant access to the site for specific tasks. We are redirected to another site to grant permissions.
Once we have granted permission the application has access to data shared by our identifying website. We will now learn about
Resource Owners (end users who own resources in a third party application), Resource Sever (provides protected resources),
Client for accessing resources and Authorization Server (a server dedicated to authorization). There are different Grant Types which
might be Authorization Codes, Client Codes and Refresh Tokens.

OpenID Connect 1.0

OpenID connect is a protocol for identity authentication. With OpenID Connect the end user entity becomes the protected resource.
This authentication mechanism is most commonly used with the Authorization Code grant type. When we see login with Google
the application is most commonly using OpenID Connect.

OAuth Authorization Code Grant Flow

Above we describe user management with OAuth2. Here the resource owner asks to access the application.
The application requests authorization for Github repos from the user. The user authenticates at Github and grants authorization.
The Authorization server then sends an authorization code. The application then exchanges the authorization code for an access token
which is then returned by the authorization server. The application then requests the protected resources (in this case Github repos) with
the access token. If the access token is correct then the resource server returns information about the resource.

Here we have learnt about OAuth and OpenID Connect and the Authentication and Authorization Flows. We will now learn about
Amazon Cognito.

Amazon Cognito

We will now learn about creating AWS Cognito resources with CDK. We will now look at some of the terms that Amazon Cognito uses.
A User Pool stores and manages User information. The User Pool App Client runs operations on a User Pool.
The Identity Pool works with IAM roles. For our application, we will create a single user pool to store all our users.
In the User Pool we will configure our password policy, define required and optional user attributes, enable password recovery and customize email notifications.
We will register our application as a User Pool app client to enable user login with OIDC and OAuth 2.0.
This is the setup for the CognitoApp:

public class CognitoApp {

  public static void main(final String[] args) {
    App app = new App();

    String environmentName = (String) app.getNode().tryGetContext("environmentName");
    Validations.requireNonEmpty(environmentName, "context variable 'environmentName' must not be null");

    String applicationName = (String) app.getNode().tryGetContext("applicationName");
    Validations.requireNonEmpty(applicationName, "context variable 'applicationName' must not be null");

    String accountId = (String) app.getNode().tryGetContext("accountId");
    Validations.requireNonEmpty(accountId, "context variable 'accountId' must not be null");

    String region = (String) app.getNode().tryGetContext("region");
    Validations.requireNonEmpty(region, "context variable 'region' must not be null");

    String applicationUrl = (String) app.getNode().tryGetContext("applicationUrl");
    Validations.requireNonEmpty(applicationUrl, "context variable 'applicationUrl' must not be null");

    String loginPageDomainPrefix = (String) app.getNode().tryGetContext("loginPageDomainPrefix");
    Validations.requireNonEmpty(loginPageDomainPrefix, "context variable 'loginPageDomainPrefix' must not be null");

    Environment awsEnvironment = makeEnv(accountId, region);

    ApplicationEnvironment applicationEnvironment = new ApplicationEnvironment(
      applicationName,
      environmentName
    );

    new CognitoStack(app, "cognito", awsEnvironment, applicationEnvironment, new CognitoStack.CognitoInputParameters(
      applicationName,
      applicationUrl,
      loginPageDomainPrefix));

    app.synth();
  }

  static Environment makeEnv(String account, String region) {
    return Environment.builder()
      .account(account)
      .region(region)
      .build();
  }

}

Here we add the domain name of our application with the following:


public class CognitoApp {

    public static void main(final String[] args) {
        App app = new App();
        // ...
        String applicationUrl = (String) app.getNode().tryGetContext("applicationUrl");
        Validations.requireNonEmpty(applicationUrl, "context variable 'applicationUrl' must not be null");

        String loginPageDomainPrefix = (String) app.getNode().tryGetContext("loginPageDomainPrefix");
        Validations.requireNonEmpty(loginPageDomainPrefix, "context variable 'loginPageDomainPrefix' must not be null");
    }
}

The application url parameter is the final url of our application. Stack sets up the application:

class CognitoStack extends Stack {

    private final ApplicationEnvironment applicationEnvironment;

    private final UserPool userPool;
    private final UserPoolClient userPoolClient;
    private final UserPoolDomain userPoolDomain;
    private String userPoolClientSecret;
    private final String logoutUrl;

    public CognitoStack(
            final Construct scope,
            final String id,
            final Environment awsEnvironment,
            final ApplicationEnvironment applicationEnvironment,
            final CognitoInputParameters inputParameters) {
        super(scope, id, StackProps.builder()
                .stackName(applicationEnvironment.prefix("Cognito"))
                .env(awsEnvironment).build());

        this.applicationEnvironment = applicationEnvironment;
        this.logoutUrl = String.format("https://%s.auth.%s.amazoncognito.com/logout", inputParameters.loginPageDomainPrefix, awsEnvironment.getRegion());

        this.userPool = UserPool.Builder.create(this, "userPool")
                .userPoolName(inputParameters.applicationName + "-user-pool")
                .selfSignUpEnabled(false)
                .accountRecovery(AccountRecovery.EMAIL_ONLY)
                .autoVerify(AutoVerifiedAttrs.builder().email(true).build())
                .signInAliases(SignInAliases.builder().username(true).email(true).build())
                .signInCaseSensitive(true)
                .standardAttributes(StandardAttributes.builder()
                        .email(StandardAttribute.builder().required(true).mutable(false).build())
                        .build())
                .mfa(Mfa.OFF)
                .passwordPolicy(PasswordPolicy.builder()
                        .requireLowercase(true)
                        .requireDigits(true)
                        .requireSymbols(true)
                        .requireUppercase(true)
                        .minLength(12)
                        .tempPasswordValidity(Duration.days(7))
                        .build())
                .build();

        this.userPoolClient = UserPoolClient.Builder.create(this, "userPoolClient")
                .userPoolClientName(inputParameters.applicationName + "-client")
                .generateSecret(true)
                .userPool(this.userPool)
                .oAuth(OAuthSettings.builder()
                        .callbackUrls(Arrays.asList(
                                String.format("%s/login/oauth2/code/cognito", inputParameters.applicationUrl),
                                "http://localhost:8080/login/oauth2/code/cognito"
                        ))
                        .logoutUrls(Arrays.asList(inputParameters.applicationUrl, "http://localhost:8080"))
                        .flows(OAuthFlows.builder()
                                .authorizationCodeGrant(true)
                                .build())
                        .scopes(Arrays.asList(OAuthScope.EMAIL, OAuthScope.OPENID, OAuthScope.PROFILE))
                        .build())
                .supportedIdentityProviders(Collections.singletonList(UserPoolClientIdentityProvider.COGNITO))
                .build();

        this.userPoolDomain = UserPoolDomain.Builder.create(this, "userPoolDomain")
                .userPool(this.userPool)
                .cognitoDomain(CognitoDomainOptions.builder()
                        .domainPrefix(inputParameters.loginPageDomainPrefix)
                        .build())
                .build();

        createOutputParameters();

        applicationEnvironment.tag(this);
    }
}

Here we have avoided using MFA and set a password policy for the user. We also set the configuration for the User Pool Client.
We also add a list of UserPoolClientIdentityProviders. We also create output parameters for the Spring application.

We have learnt about Cognito Resources and Infrastructure. This is a useful link for Amazon Cognito:
https://aws.amazon.com/cognito/
This is a useful link for more information about OAuth:
https://oauth.net/2/

Connecting to Database with Amazon RDS

In this section we will learn how to work with AWS RDS.
We will learn about RDS and also develop the infrastructure we will deploy.
AWS RDS offers supports PostgreSQL, MySQL, MariaDB, Oracle, Microsoft and Aurora.
It allows us to manage relational databases with tools such as AWS CLI, IAM, CloudFormation and CDK.

Above is a diagram of the database setup. We put a database instance into a private subnet. The application in our
Service stack would connect to the RDS database. The ECS Task represents a Docker image and the ECS Service wraps
several Docker images into a Service. The overall infrastructure runs on a VPC. We will use an Application Load Balancer
and Internet Gateway to enable access for the internet.

Here we have learnt about working with relational databases on AWS. In the next section we will learn about configuring IAM permissions
for RDS access. We will deploy a database related infrastructure and use the RDS database from our SpringBoot application.

Connecting to the Database with AWS RDS

Here we will look at setting up the required IAM permissions. We will then deploy RDS with CDK and then configure the RDS
databaes with our SpringBoot application.

Setting up the requisite IAM permissions

We need to add the RDSFullAccess policy for our developer user account:

The application will have access to the database through the CDK app. We will deploy a database infrastructure with CDK and use the
Postgres Database Construct. The

public class PostgresDatabase extends Construct {
    public PostgresDatabase(
            final Construct scope,
            final String id,
            final Environment awsEnvironment,
            final ApplicationEnvironment applicationEnvironment,
            final DatabaseInputParameters databaseInputParameters) {

        super(scope, id);

        this.applicationEnvironment = applicationEnvironment;

        // Sadly, we cannot use VPC.fromLookup() to resolve a VPC object from this VpcId, because it's broken
        // (https://github.com/aws/aws-cdk/issues/3600). So, we have to resolve all properties we need from the VPC
        // via SSM parameter store.
        Network.NetworkOutputParameters networkOutputParameters = Network.getOutputParametersFromParameterStore(this, applicationEnvironment.getEnvironmentName());

        String username = sanitizeDbParameterName(applicationEnvironment.prefix("dbUser"));

        databaseSecurityGroup = CfnSecurityGroup.Builder.create(this, "databaseSecurityGroup")
                .vpcId(networkOutputParameters.getVpcId())
                .groupDescription("Security Group for the database instance")
                .groupName(applicationEnvironment.prefix("dbSecurityGroup"))
                .build();

        // This will generate a JSON object with the keys "username" and "password".
        databaseSecret = Secret.Builder.create(this, "databaseSecret")
                .secretName(applicationEnvironment.prefix("DatabaseSecret"))
                .description("Credentials to the RDS instance")
                .generateSecretString(SecretStringGenerator.builder()
                        .secretStringTemplate(String.format("{\"username\": \"%s\"}", username))
                        .generateStringKey("password")
                        .passwordLength(32)
                        .excludeCharacters("@/\\\" ")
                        .build())
                .build();

        CfnDBSubnetGroup subnetGroup = CfnDBSubnetGroup.Builder.create(this, "dbSubnetGroup")
                .dbSubnetGroupDescription("Subnet group for the RDS instance")
                .dbSubnetGroupName(applicationEnvironment.prefix("dbSubnetGroup"))
                .subnetIds(networkOutputParameters.getIsolatedSubnets())
                .build();

        dbInstance = CfnDBInstance.Builder.create(this, "postgresInstance")
                .dbInstanceIdentifier(applicationEnvironment.prefix("database"))
                .allocatedStorage(String.valueOf(databaseInputParameters.storageInGb))
                .availabilityZone(networkOutputParameters.getAvailabilityZones().get(0))
                .dbInstanceClass(databaseInputParameters.instanceClass)
                .dbName(sanitizeDbParameterName(applicationEnvironment.prefix("database")))
                .dbSubnetGroupName(subnetGroup.getDbSubnetGroupName())
                .engine("postgres")
                .engineVersion(databaseInputParameters.postgresVersion)
                .masterUsername(username)
                .masterUserPassword(databaseSecret.secretValueFromJson("password").toString())
                .publiclyAccessible(false)
                .vpcSecurityGroups(Collections.singletonList(databaseSecurityGroup.getAttrGroupId()))
                .build();

        CfnSecretTargetAttachment.Builder.create(this, "secretTargetAttachment")
                .secretId(databaseSecret.getSecretArn())
                .targetId(dbInstance.getRef())
                .targetType("AWS::RDS::DBInstance")
                .build();

        createOutputParameters();

        applicationEnvironment.tag(this);

    }
}

First the construct creates a database security group. We also define subnets which will be used by the database instance. We also create a database secret
and then create the Postgres Database. We can then deploy and destroy the application with our package.json script:

{
  "scripts": {
    "database:deploy": "cdk deploy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=dev.stratospheric.todoapp.cdk.DatabaseApp\" --require-approval never",
    "database:destroy": "cdk destroy --app \"./mvnw -e -q compile exec:java -Dexec.mainClass=dev.stratospheric.todoapp.cdk.DatabaseApp\" --force --require-approval never"
  }
}

We run the command and create and instantiate the database and server that we will use.

npm run database:deploy

AWS will instantiate the Postgres server and the other resources we need. The above command runs the cdk deploy.
If we want to use a different profile we can use:

npm run database:deploy -- --profile stratospheric

Here the double dash tells npm not to evaluate what is after it so the --profile is sent on to the aws script.

We now have a database running so we can now setup our SpringBoot application. We do this with the ServiceApp inside the CDK folder:


public class ServiceApp {
    public static void main(final String[] args) {
        App app = new App();

        String environmentName = (String) app.getNode().tryGetContext("environmentName");
        Validations.requireNonEmpty(environmentName, "context variable 'environmentName' must not be null");

        String applicationName = (String) app.getNode().tryGetContext("applicationName");
        Validations.requireNonEmpty(applicationName, "context variable 'applicationName' must not be null");

        String accountId = (String) app.getNode().tryGetContext("accountId");
        Validations.requireNonEmpty(accountId, "context variable 'accountId' must not be null");

        String springProfile = (String) app.getNode().tryGetContext("springProfile");
        Validations.requireNonEmpty(springProfile, "context variable 'springProfile' must not be null");

        String dockerRepositoryName = (String) app.getNode().tryGetContext("dockerRepositoryName");
        Validations.requireNonEmpty(dockerRepositoryName, "context variable 'dockerRepositoryName' must not be null");

        String dockerImageTag = (String) app.getNode().tryGetContext("dockerImageTag");
        Validations.requireNonEmpty(dockerImageTag, "context variable 'dockerImageTag' must not be null");

        String region = (String) app.getNode().tryGetContext("region");
        Validations.requireNonEmpty(region, "context variable 'region' must not be null");

        Environment awsEnvironment = makeEnv(accountId, region);

        ApplicationEnvironment applicationEnvironment = new ApplicationEnvironment(
                applicationName,
                environmentName
        );

        // This stack is just a container for the parameters below, because they need a Stack as a scope.
        // We're making this parameters stack unique with each deployment by adding a timestamp, because updating an existing
        // parameters stack will fail because the parameters may be used by an old service stack.
        // This means that each update will generate a new parameters stack that needs to be cleaned up manually!
        long timestamp = System.currentTimeMillis();
        Stack parametersStack = new Stack(app, "ServiceParameters-" + timestamp, StackProps.builder()
                .stackName(applicationEnvironment.prefix("Service-Parameters-" + timestamp))
                .env(awsEnvironment)
                .build());

        Stack serviceStack = new Stack(app, "ServiceStack", StackProps.builder()
                .stackName(applicationEnvironment.prefix("Service"))
                .env(awsEnvironment)
                .build());

        PostgresDatabase.DatabaseOutputParameters databaseOutputParameters =
                PostgresDatabase.getOutputParametersFromParameterStore(parametersStack, applicationEnvironment);

        CognitoStack.CognitoOutputParameters cognitoOutputParameters =
                CognitoStack.getOutputParametersFromParameterStore(parametersStack, applicationEnvironment);

        MessagingStack.MessagingOutputParameters messagingOutputParameters =
                MessagingStack.getOutputParametersFromParameterStore(parametersStack, applicationEnvironment);

        ActiveMqStack.ActiveMqOutputParameters activeMqOutputParameters =
                ActiveMqStack.getOutputParametersFromParameterStore(parametersStack, applicationEnvironment);

        List<String> securityGroupIdsToGrantIngressFromEcs = Arrays.asList(
                databaseOutputParameters.getDatabaseSecurityGroupId(),
                activeMqOutputParameters.getActiveMqSecurityGroupId()
        );

        new Service(
                serviceStack,
                "Service",
                awsEnvironment,
                applicationEnvironment,
                new Service.ServiceInputParameters(
                        new Service.DockerImageSource(dockerRepositoryName, dockerImageTag),
                        securityGroupIdsToGrantIngressFromEcs,
                        environmentVariables(
                                serviceStack,
                                databaseOutputParameters,
                                cognitoOutputParameters,
                                messagingOutputParameters,
                                activeMqOutputParameters,
                                springProfile,
                                environmentName))
                        .withTaskRolePolicyStatements(List.of(
                                PolicyStatement.Builder.create()
                                        .sid("AllowSQSAccess")
                                        .effect(Effect.ALLOW)
                                        .resources(List.of(
                                                String.format("arn:aws:sqs:%s:%s:%s", region, accountId, messagingOutputParameters.getTodoSharingQueueName())
                                        ))
                                        .actions(Arrays.asList(
                                                "sqs:DeleteMessage",
                                                "sqs:GetQueueUrl",
                                                "sqs:ListDeadLetterSourceQueues",
                                                "sqs:ListQueues",
                                                "sqs:ListQueueTags",
                                                "sqs:ReceiveMessage",
                                                "sqs:SendMessage",
                                                "sqs:ChangeMessageVisibility",
                                                "sqs:GetQueueAttributes"))
                                        .build(),
                                PolicyStatement.Builder.create()
                                        .sid("AllowCreatingUsers")
                                        .effect(Effect.ALLOW)
                                        .resources(
                                                List.of(String.format("arn:aws:cognito-idp:%s:%s:userpool/%s", region, accountId, cognitoOutputParameters.getUserPoolId()))
                                        )
                                        .actions(List.of(
                                                "cognito-idp:AdminCreateUser"
                                        ))
                                        .build(),
                                PolicyStatement.Builder.create()
                                        .sid("AllowSendingEmails")
                                        .effect(Effect.ALLOW)
                                        .resources(
                                                List.of(String.format("arn:aws:ses:%s:%s:identity/stratospheric.dev", region, accountId))
                                        )
                                        .actions(List.of(
                                                "ses:SendEmail",
                                                "ses:SendRawEmail"
                                        ))
                                        .build(),
                                PolicyStatement.Builder.create()
                                        .sid("AllowDynamoTableAccess")
                                        .effect(Effect.ALLOW)
                                        .resources(
                                                List.of(String.format("arn:aws:dynamodb:%s:%s:table/%s", region, accountId, applicationEnvironment.prefix("breadcrumb")))
                                        )
                                        .actions(List.of(
                                                "dynamodb:Scan",
                                                "dynamodb:Query",
                                                "dynamodb:PutItem",
                                                "dynamodb:GetItem",
                                                "dynamodb:BatchWriteItem",
                                                "dynamodb:BatchWriteGet"
                                        ))
                                        .build(),
                                PolicyStatement.Builder.create()
                                        .sid("AllowSendingMetricsToCloudWatch")
                                        .effect(Effect.ALLOW)
                                        .resources(singletonList("*")) // CloudWatch does not have any resource-level permissions, see https://stackoverflow.com/a/38055068/9085273
                                        .actions(singletonList("cloudwatch:PutMetricData"))
                                        .build()
                        ))
                        .withStickySessionsEnabled(true)
                        .withHealthCheckPath("/actuator/health")
                        .withAwsLogsDateTimeFormat("%Y-%m-%dT%H:%M:%S.%f%z")
                        .withHealthCheckIntervalSeconds(30), // needs to be long enough to allow for slow start up with low-end computing instances

                Network.getOutputParametersFromParameterStore(serviceStack, applicationEnvironment.getEnvironmentName()));

        app.synth();
    }

    static Map<String, String> environmentVariables(
            Construct scope,
            PostgresDatabase.DatabaseOutputParameters databaseOutputParameters,
            CognitoStack.CognitoOutputParameters cognitoOutputParameters,
            MessagingStack.MessagingOutputParameters messagingOutputParameters,
            ActiveMqStack.ActiveMqOutputParameters activeMqOutputParameters,
            String springProfile,
            String environmentName
    ) {
        Map<String, String> vars = new HashMap<>();

        String databaseSecretArn = databaseOutputParameters.getDatabaseSecretArn();
        ISecret databaseSecret = Secret.fromSecretCompleteArn(scope, "databaseSecret", databaseSecretArn);

        vars.put("SPRING_PROFILES_ACTIVE", springProfile);
        vars.put("SPRING_DATASOURCE_URL",
                String.format("jdbc:postgresql://%s:%s/%s",
                        databaseOutputParameters.getEndpointAddress(),
                        databaseOutputParameters.getEndpointPort(),
                        databaseOutputParameters.getDbName()));
        vars.put("SPRING_DATASOURCE_USERNAME",
                databaseSecret.secretValueFromJson("username").toString());
        vars.put("SPRING_DATASOURCE_PASSWORD",
                databaseSecret.secretValueFromJson("password").toString());
        vars.put("COGNITO_CLIENT_ID", cognitoOutputParameters.getUserPoolClientId());
        vars.put("COGNITO_CLIENT_SECRET", cognitoOutputParameters.getUserPoolClientSecret());
        vars.put("COGNITO_USER_POOL_ID", cognitoOutputParameters.getUserPoolId());
        vars.put("COGNITO_LOGOUT_URL", cognitoOutputParameters.getLogoutUrl());
        vars.put("COGNITO_PROVIDER_URL", cognitoOutputParameters.getProviderUrl());
        vars.put("TODO_SHARING_QUEUE_NAME", messagingOutputParameters.getTodoSharingQueueName());
        vars.put("WEB_SOCKET_RELAY_ENDPOINT", activeMqOutputParameters.getStompEndpoint());
        vars.put("WEB_SOCKET_RELAY_USERNAME", activeMqOutputParameters.getActiveMqUsername());
        vars.put("WEB_SOCKET_RELAY_PASSWORD", activeMqOutputParameters.getActiveMqPassword());
        vars.put("ENVIRONMENT_NAME", environmentName);

        return vars;
    }

}

Here we create the SpringBoot Service with the correct environment variables for configuring the database connection.
The AWS specific database stack can then be used with the Spring application. Spring uses the environment variables to
configure the database connection on SpringBoot start.

We have seen how to configure IAM permissions for RDS access. We have deployed the database and are now using the RDS database from
our Spring application.

Sharing Todos with Amazon SQS and Amazon SES

Here we will integrate two new AWS services, AWS Simple Queue Service (SQS) and
AWS Simple Email Service (SES) so that we can share Todos with other users so that they
can collaborate. Users accept collaboration by email and then collaborate. When users share their todos we put the request
into an SQS queue and then send out emails to the requested other user.

Introduction to Amazon SQS

Amazon SQS is a fully managed messaging service for queueing messages to different parts of our application. SQS can
also decouple components in a microservices distributed architecture. We interact with SQS via an https API.
SQS can persist our messages for up to 14 days. The standard queue type delivers on a best effort ordering. FIFO SQS
guarantees messages are sent in the same order as they are sent. Each message remains on the queue until the consumer
acknowledges its delivery by deleting the message. We set up our messaging queue with the MessagingStack:

class MessagingStack extends Stack {

  private final ApplicationEnvironment applicationEnvironment;
  private final IQueue todoSharingQueue;
  private final IQueue todoSharingDlq;

  public MessagingStack(
    final Construct scope,
    final String id,
    final Environment awsEnvironment,
    final ApplicationEnvironment applicationEnvironment) {
    super(scope, id, StackProps.builder()
      .stackName(applicationEnvironment.prefix("Messaging"))
      .env(awsEnvironment).build());

    this.applicationEnvironment = applicationEnvironment;

    this.todoSharingDlq = Queue.Builder.create(this, "todoSharingDlq")
      .queueName(applicationEnvironment.prefix("todo-sharing-dead-letter-queue"))
      .retentionPeriod(Duration.days(14))
      .build();

    this.todoSharingQueue = Queue.Builder.create(this, "todoSharingQueue")
      .queueName(applicationEnvironment.prefix("todo-sharing-queue"))
      .visibilityTimeout(Duration.seconds(30))
      .retentionPeriod(Duration.days(14))
      .deadLetterQueue(DeadLetterQueue.builder()
        .queue(todoSharingDlq)
        .maxReceiveCount(3)
        .build())
      .build();

    createOutputParameters();

    applicationEnvironment.tag(this);
  }

  private static final String PARAMETER_TODO_SHARING_QUEUE_NAME = "todoSharingQueueName";

  private void createOutputParameters() {
    StringParameter.Builder.create(this, PARAMETER_TODO_SHARING_QUEUE_NAME)
      .parameterName(createParameterName(applicationEnvironment, PARAMETER_TODO_SHARING_QUEUE_NAME))
      .stringValue(this.todoSharingQueue.getQueueName())
      .build();
  }

  private static String createParameterName(ApplicationEnvironment applicationEnvironment, String parameterName) {
    return applicationEnvironment.getEnvironmentName() + "-" + applicationEnvironment.getApplicationName() + "-Messaging-" + parameterName;
  }

  public static String getTodoSharingQueueName(Construct scope, ApplicationEnvironment applicationEnvironment) {
    return StringParameter.fromStringParameterName(scope, PARAMETER_TODO_SHARING_QUEUE_NAME, createParameterName(applicationEnvironment, PARAMETER_TODO_SHARING_QUEUE_NAME))
      .getStringValue();
  }

  public static MessagingOutputParameters getOutputParametersFromParameterStore(Construct scope, ApplicationEnvironment applicationEnvironment) {
    return new MessagingOutputParameters(
      getTodoSharingQueueName(scope, applicationEnvironment)
    );
  }

  public static class MessagingOutputParameters {
    private final String todoSharingQueueName;

    public MessagingOutputParameters(String todoSharingQueueName) {
      this.todoSharingQueueName = todoSharingQueueName;
    }

    public String getTodoSharingQueueName() {
      return todoSharingQueueName;
    }
  }

}

Here we set up the queue and set the Deadletter queue to accept messages after four attempts. We also need to set up
SQS as a dependency in our build.gradle:

  implementation 'io.awspring.cloud:spring-cloud-aws-starter-sqs'

We then send the todo with a form on our dashboard:


<div class="dropdown-menu" aria-labelledby="dropdownMenuLink">
              <span class="dropdown-item" th:if="${collaborators.isEmpty()}">
                No collaborator available
              </span>
    <form th:method="POST"
          th:each="collaborator : ${collaborators}"
          th:action="@{/todo/{todoId}/collaborations/{collaboratorId}(todoId=${todo.id}, collaboratorId=${collaborator.id})}">
        <button
                th:text="${collaborator.name}"
                type="submit"
                name="submit"
                class="dropdown-item">
        </button>
    </form>
</div>

We also have a post endpoint on our controller to share todos:

@Controller
@RequestMapping("/todo")
public class TodoCollaborationController {

    private final TodoCollaborationService todoCollaborationService;

    public TodoCollaborationController(TodoCollaborationService todoCollaborationService) {
        this.todoCollaborationService = todoCollaborationService;
    }

    @Timed(
            value = "stratospheric.collaboration.sharing",
            description = "Measure the time how long it takes to share a todo"
    )
    @PostMapping("/{todoId}/collaborations/{collaboratorId}")
    public String shareTodoWithCollaborator(
            @PathVariable("todoId") Long todoId,
            @PathVariable("collaboratorId") Long collaboratorId,
            @AuthenticationPrincipal OidcUser user,
            RedirectAttributes redirectAttributes
    ) throws JsonProcessingException {
        String collaboratorName = todoCollaborationService.shareWithCollaborator(user.getEmail(), todoId, collaboratorId);

        redirectAttributes.addFlashAttribute("message",
                String.format("You successfully shared your todo with the user %s. " +
                        "Once the user accepts the invite, you'll see them as a collaborator on your todo.", collaboratorName));
        redirectAttributes.addFlashAttribute("messageType", "success");

        return "redirect:/dashboard";
    }
}

The TodoCollaboration controller takes collaboration requests and sharings them using the todoCollaborationService.
The Service then shares the request:

@Service
@Transactional
public class TodoCollaborationService {

    private final TodoRepository todoRepository;
    private final PersonRepository personRepository;
    private final TodoCollaborationRequestRepository todoCollaborationRequestRepository;

    private final SqsTemplate sqsTemplate;
    private final String todoSharingQueueName;

    private final SimpMessagingTemplate simpMessagingTemplate;

    private static final Logger LOG = LoggerFactory.getLogger(TodoCollaborationService.class.getName());

    private static final String INVALID_TODO_ID = "Invalid todo ID: ";
    private static final String INVALID_PERSON_ID = "Invalid person ID: ";
    private static final String INVALID_PERSON_EMAIL = "Invalid person Email: ";

    public TodoCollaborationService(
            @Value("${custom.sharing-queue}") String todoSharingQueueName,
            TodoRepository todoRepository,
            PersonRepository personRepository,
            TodoCollaborationRequestRepository todoCollaborationRequestRepository,
            SqsTemplate sqsTemplate,
            SimpMessagingTemplate simpMessagingTemplate) {
        this.todoRepository = todoRepository;
        this.personRepository = personRepository;
        this.todoCollaborationRequestRepository = todoCollaborationRequestRepository;
        this.sqsTemplate = sqsTemplate;
        this.todoSharingQueueName = todoSharingQueueName;
        this.simpMessagingTemplate = simpMessagingTemplate;
    }

    public String shareWithCollaborator(String todoOwnerEmail, Long todoId, Long collaboratorId) {

        Todo todo = todoRepository
                .findByIdAndOwnerEmail(todoId, todoOwnerEmail)
                .orElseThrow(() -> new IllegalArgumentException(INVALID_TODO_ID + todoId));

        Person collaborator = personRepository
                .findById(collaboratorId)
                .orElseThrow(() -> new IllegalArgumentException(INVALID_PERSON_ID + collaboratorId));

        if (todoCollaborationRequestRepository.findByTodoAndCollaborator(todo, collaborator) != null) {
            LOG.info("Collaboration request for todo {} with collaborator {} already exists", todoId, collaboratorId);
            return collaborator.getName();
        }

        LOG.info("About to share todo with id {} with collaborator {}", todoId, collaboratorId);

        TodoCollaborationRequest collaboration = new TodoCollaborationRequest();
        String token = UUID.randomUUID().toString();
        collaboration.setToken(token);
        collaboration.setCollaborator(collaborator);
        collaboration.setTodo(todo);
        todo.getCollaborationRequests().add(collaboration);

        todoCollaborationRequestRepository.save(collaboration);

        sqsTemplate.send(todoSharingQueueName, new TodoCollaborationNotification(collaboration));

        return collaborator.getName();
    }
}

We then listen for the request with our TodoSharingListener:


@Component
public class TodoSharingListener {

  private final MailSender mailSender;
  private final TodoCollaborationService todoCollaborationService;
  private final boolean autoConfirmCollaborations;
  private final String confirmEmailFromAddress;
  private final String externalUrl;

  private static final Logger LOG = LoggerFactory.getLogger(TodoSharingListener.class.getName());

  public TodoSharingListener(
    MailSender mailSender,
    TodoCollaborationService todoCollaborationService,
    @Value("${custom.auto-confirm-collaborations}") boolean autoConfirmCollaborations,
    @Value("${custom.confirm-email-from-address}") String confirmEmailFromAddress,
    @Value("${custom.external-url}") String externalUrl) {
    this.mailSender = mailSender;
    this.todoCollaborationService = todoCollaborationService;
    this.autoConfirmCollaborations = autoConfirmCollaborations;
    this.confirmEmailFromAddress = confirmEmailFromAddress;
    this.externalUrl = externalUrl;
  }

  @SqsListener(value = "${custom.sharing-queue}")
  public void listenToSharingMessages(TodoCollaborationNotification payload) throws InterruptedException {
    LOG.info("Incoming todo sharing payload: {}", payload);

    SimpleMailMessage message = new SimpleMailMessage();
    message.setFrom(confirmEmailFromAddress);
    message.setTo(payload.getCollaboratorEmail());
    message.setSubject("A todo was shared with you");
    message.setText(
      String.format(
        """
          Hi %s,\s

          someone shared a Todo from %s with you.

          Information about the shared Todo item:\s

          Title: %s\s
          Description: %s\s
          Priority: %s\s

          You can accept the collaboration by clicking this link: %s/todo/%s/collaborations/%s/confirm?token=%s\s

          Kind regards,\s
          Stratospheric""",
        payload.getCollaboratorEmail(),
        externalUrl,
        payload.getTodoTitle(),
        payload.getTodoDescription(),
        payload.getTodoPriority(),
        externalUrl,
        payload.getTodoId(),
        payload.getCollaboratorId(),
        payload.getToken()
      )
    );
    mailSender.send(message);

    LOG.info("Successfully informed collaborator about shared todo.");

    if (autoConfirmCollaborations) {
      LOG.info("Auto-confirmed collaboration request for todo: {}", payload.getTodoId());
      Thread.sleep(2_500);
      todoCollaborationService.confirmCollaboration(payload.getCollaboratorEmail(), payload.getTodoId(), payload.getCollaboratorId(), payload.getToken());
    }
  }
}

Here we have learnt about implementing collaboration features with SQS for decoupling events from behaviour.

Sharing Todos with Amazon SQS and Amazon SES

Here, we will use Amazon SES for sending emails to collaborators. Amazon SES is an easy to set up email service.
We interact with the SES service with CDK or an API. We could use this service for marketing, sign up or email news services.
Amazon SES is available in several regions. We allow sending emails with the following configuration:

public class ServiceApp {

    public static void main(final String[] args) {
        // ...
        PolicyStatement.Builder.create()
                .sid("AllowSendingEmails")
                .effect(Effect.ALLOW)
                .resources(
                        List.of(String.format("arn:aws:ses:%s:%s:identity/stratospheric.dev", region, accountId))
                )
                .actions(List.of(
                        "ses:SendEmail",
                        "ses:SendRawEmail"
                ))
                .build();
        // ...
    }
}

Spring defines two interfaces for sending emails: MailSender and JavaMailSender. We also add the ses dependency to our
build.gradle file:

  implementation 'io.awspring.cloud:spring-cloud-aws-starter-ses'

We send the email from our TodoSharingListener:

@Component
public class TodoSharingListener {
    // ...
  @SqsListener(value = "${custom.sharing-queue}")
  public void listenToSharingMessages(TodoCollaborationNotification payload) throws InterruptedException {
    LOG.info("Incoming todo sharing payload: {}", payload);

    SimpleMailMessage message = new SimpleMailMessage();
    message.setFrom(confirmEmailFromAddress);
    message.setTo(payload.getCollaboratorEmail());
    message.setSubject("A todo was shared with you");
    message.setText(
      String.format(
        """
          Hi %s,\s

          someone shared a Todo from %s with you.

          Information about the shared Todo item:\s

          Title: %s\s
          Description: %s\s
          Priority: %s\s

          You can accept the collaboration by clicking this link: %s/todo/%s/collaborations/%s/confirm?token=%s\s

          Kind regards,\s
          Stratospheric""",
        payload.getCollaboratorEmail(),
        externalUrl,
        payload.getTodoTitle(),
        payload.getTodoDescription(),
        payload.getTodoPriority(),
        externalUrl,
        payload.getTodoId(),
        payload.getCollaboratorId(),
        payload.getToken()
      )
    );
    mailSender.send(message);

    LOG.info("Successfully informed collaborator about shared todo.");

    if (autoConfirmCollaborations) {
      LOG.info("Auto-confirmed collaboration request for todo: {}", payload.getTodoId());
      Thread.sleep(2_500);
      todoCollaborationService.confirmCollaboration(payload.getCollaboratorEmail(), payload.getTodoId(), payload.getCollaboratorId(), payload.getToken());
    }
  }
}

The mailSender#send function hands the delivery of the email to Amazon SES. We handle the user's response to the email in
the TodoCollaborationController:

@Controller
@RequestMapping("/todo")
public class TodoCollaborationController {
    //...
  @GetMapping("/{todoId}/collaborations/{collaboratorId}/confirm")
  public String confirmCollaboration(
    @PathVariable("todoId") Long todoId,
    @PathVariable("collaboratorId") Long collaboratorId,
    @RequestParam("token") String token,
    @AuthenticationPrincipal OidcUser user,
    RedirectAttributes redirectAttributes
  ) {
    if (todoCollaborationService.confirmCollaboration(user.getEmail(), todoId, collaboratorId, token)) {
      redirectAttributes.addFlashAttribute("message", "You've confirmed that you'd like to collaborate on this todo.");
      redirectAttributes.addFlashAttribute("messageType", "success");
    } else {
      redirectAttributes.addFlashAttribute("message", "Invalid collaboration request.");
      redirectAttributes.addFlashAttribute("messageType", "danger");
    }

    return "redirect:/dashboard";
  }
}

We confirm the collaboration with the TodoCollaborationService:

@Service
@Transactional
public class TodoCollaborationService {
    // ...
  public boolean confirmCollaboration(String authenticatedUserEmail, Long todoId, Long collaboratorId, String token) {

    Person collaborator = personRepository
      .findByEmail(authenticatedUserEmail)
      .orElseThrow(() -> new IllegalArgumentException(INVALID_PERSON_EMAIL + authenticatedUserEmail));

    if (!collaborator.getId().equals(collaboratorId)) {
      return false;
    }

    TodoCollaborationRequest collaborationRequest = todoCollaborationRequestRepository
      .findByTodoIdAndCollaboratorId(todoId, collaboratorId);

    LOG.info("Collaboration request: {}", collaborationRequest);

    if (collaborationRequest == null || !collaborationRequest.getToken().equals(token)) {
      return false;
    }

    LOG.info("Original collaboration token: {}", collaborationRequest.getToken());
    LOG.info("Request token: {}", token);

    Todo todo = todoRepository
      .findById(todoId)
      .orElseThrow(() -> new IllegalArgumentException(INVALID_TODO_ID + todoId));

    todo.addCollaborator(collaborator);

    todoCollaborationRequestRepository.delete(collaborationRequest);

    String name = collaborationRequest.getCollaborator().getName();
    String subject = "Collaboration confirmed.";
    String message = "User "
      + name
      + " has accepted your collaboration request for todo #"
      + collaborationRequest.getTodo().getId()
      + ".";
    String ownerEmail = collaborationRequest.getTodo().getOwner().getEmail();

    simpMessagingTemplate.convertAndSend("/topic/todoUpdates/" + ownerEmail, subject + " " + message);

    LOG.info("Successfully informed owner about accepted request.");

    return true;
  }
}

For local testing with LocalStack we use the following definition in our docker-compose.yml:

  localstack:
    image: localstack/localstack:0.14.4
    ports:
      - 4566:4566
    environment:
      - SERVICES=sqs,ses,dynamodb
      - DEFAULT_REGION=eu-central-1
      - USE_SINGLE_REGION=true
    volumes:
      - ./src/test/resources/localstack/local-aws-infrastructure.sh:/docker-entrypoint-initaws.d/init.sh

To set up our local environment we use the following script in local-aws-infrastructure.sh:

#!/bin/sh

awslocal sqs create-queue --queue-name stratospheric-todo-sharing

awslocal ses verify-email-identity --email-address noreply@stratospheric.dev
awslocal ses verify-email-identity --email-address info@stratospheric.dev
awslocal ses verify-email-identity --email-address tom@stratospheric.dev
awslocal ses verify-email-identity --email-address bjoern@stratospheric.dev
awslocal ses verify-email-identity --email-address philip@stratospheric.dev

awslocal dynamodb create-table \
    --table-name local-todo-app-breadcrumb \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --key-schema AttributeName=id,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10 \

echo "Initialized."

For local development we override the url endpoint with our application-dev.yml:

  cloud:
    aws:
      rds:
        enabled: false
      sqs:
        endpoint: http://localhost:4566
        region: eu-central-1
      mail:
        endpoint: http://localhost:4566
        region: eu-central-1
      credentials:
        secret-key: foo
        access-key: bar

We set auto-confirm-collaborations to true to enable email success:

  auto-confirm-collaborations: true

This is then used in the autoConfirmCollaborations parameter in our TodoSharingListener:

public class TodoSharingListener {

    private final MailSender mailSender;
    private final TodoCollaborationService todoCollaborationService;
    private final boolean autoConfirmCollaborations;
    private final String confirmEmailFromAddress;
    private final String externalUrl;

    private static final Logger LOG = LoggerFactory.getLogger(TodoSharingListener.class.getName());

    public TodoSharingListener(
            MailSender mailSender,
            TodoCollaborationService todoCollaborationService,
            @Value("${custom.auto-confirm-collaborations}") boolean autoConfirmCollaborations,
            @Value("${custom.confirm-email-from-address}") String confirmEmailFromAddress,
            @Value("${custom.external-url}") String externalUrl) {
        this.mailSender = mailSender;
        this.todoCollaborationService = todoCollaborationService;
        this.autoConfirmCollaborations = autoConfirmCollaborations;
        this.confirmEmailFromAddress = confirmEmailFromAddress;
        this.externalUrl = externalUrl;
    }
}

Here we have learnt about sending and receiving emails with AWS SES and we have looked at implementing email functionality
in a Spring Boot application.

Production Readiness with AWS

We will now look at Amazon CloudWatch and send log data to this service. We will set up metrics for CloudWatch and alarms if
thresholds are breached. We will also make the application production-ready by securing it with HTTPS and hosting it on a custom
domain.

We will use Spring Boot Logback for logging:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>

  <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="de.siegmar.logbackawslogsjsonencoder.AwsJsonLogEncoder"/>
  </appender>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
  </root>
</configuration>

Here we will use centralized logging. We will use CloudWatch logging and learn how to send logs from ECS Fargate
to Amazon CloudWatch.

Above, we can see that we are running our application in an ECS Cluster. The Fargate Task runs our Docker images
and we will send logs to Amazon CloudWatch so that we can keep our logs in a centralized location.

Deploying the application

We first need to set up our cdk.json to the correct configuration for our account:

{
  "context": {
    "applicationName": "todo-maker",
    "region": "eu-west-2",
    "accountId": "706054169063",
    "dockerRepositoryName": "todo-maker",
    "dockerImageTag": "1",
    "applicationUrl": "https://app.drspencer.io",
    "loginPageDomainPrefix": "stratospheric-staging",
    "environmentName": "staging",
    "springProfile": "aws",
    "activeMqUsername": "activemqUser",
    "canaryUsername": "canary",
    "canaryUserPassword": "SECRET_OVERRIDDEN_BY_WORKFLOW",
    "confirmationEmail": "info@stratospheric.dev",
    "applicationDomain": "app.drspencer.io",
    "sslCertificateArn": "arn:aws:acm:eu-west-2:706054169063:certificate/ab9a0a58-cd09-493f-84cb-b42f9db0902a",
    "hostedZoneDomain": "drspencer.io",
    "githubToken": "SECRET_OVERRIDDEN_BY_WORKFLOW"
  }
}

We then bootstrap the resources:

npm run bootstrap -- --profile stratospheric

We then create an SSL Certificate for my domain:

npm run certificate:deploy -- --profile stratospheric

We then deploy the NetworkStack-dependent infrastructure:

npm run network:deploy -- --profile stratospheric
npm run database:deploy -- --profile stratospheric
npm run activeMq:deploy -- --profile stratospheric

Next we deploy the NetworkStack-independent infrastructure:

npm run repository:deploy -- --profile stratospheric
npm run messaging:deploy -- --profile stratospheric
npm run cognito:deploy -- --profile stratospheric

Then we run the command to route traffic from our custom domain to the ELB:

npm run domain:deploy -- --profile stratospheric

Build and Push the First Docker Image

First we go to the root of the application and run:

./gradlew build docker build -t <accountId>.dkr.ecr.<region>.amazonaws.com/<applicationName>:1

aws ecr get-login-password --region <region> --profile stratospheric | docker login \
--username AWS --password-stdin <accountId>.drk.ecr.

docker push <accountId>.drk.ecr.<region>.amazonaws.com/<applicationName>:1

Deploy the Docker image to the ECS Cluster

npm run service:deploy -- --profile stratospheric

Deploy monitoring Infrastructure

cd cdk
npm run monitoring:deploy -- --profile stratospheric

Deploy Canary stack

cd cdk
npm run canary:deploy -- --profile stratospheric

Destroy everything

npm run *:destroy -- --profile stratospheric

Run the above command with all the earlier scripts to ensure that all stacks are removed

This is what the app looks like after deploy:

CloudWatch Logging Terminology

Log Stream: stream of logs from the same source (e.g. a single Docker Container)
Log Group: An aggregation of log streams to group logs together
CloudWatch Log Insights: Service that provides UI and query language to search one or log groups

Example queries to CloudWatch could include:

fields @timestamp, @message
| sort @timestamp desc
| limit 20

Spring Framework DevOps on AWS

Tom Spencer — Mon, 27 Mar 2023 19:01:38 +0000

This is an overview of how to use Spring with AWS. We will look at some of the tools we can use and also provision ec2 instances
for each of the services. I have enjoyed creating this overview for you. The github for this repo is:
https://github.com/TomSpencerLondon/spring-boot-docker

Thanks again!

Externalising Properties

First we externalise properties to make our build portable.



@Configuration
@PropertySource("classpath:testing.properties")
public class ExternalPropsPropertySourceTestConfig {

    @Value("${guru.jms.server}")
    String jmsServer;

    @Value("${guru.jms.port}")
    Integer jmsPort;

    @Value("${guru.jms.user}")
    String jmsUser;

    @Value("${guru.jms.password}")
    String jmsPassword;

    @Bean
    public static PropertySourcesPlaceholderConfigurer properties() {
        PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer = new PropertySourcesPlaceholderConfigurer();

        return propertySourcesPlaceholderConfigurer;
    }

    @Bean
    public FakeJmsBroker fakeJmsBroker() {
        FakeJmsBroker fakeJmsBroker = new FakeJmsBroker();
        fakeJmsBroker.setUrl(jmsServer);
        fakeJmsBroker.setPort(jmsPort);
        fakeJmsBroker.setUser(jmsUser);
        fakeJmsBroker.setPassword(jmsPassword);

        return fakeJmsBroker;
    }
}

This refers to testing.properties in the test folder:

guru.jms.server=10.10.10.123
guru.jms.port=3330
guru.jms.user=Ron
guru.jms.password=Burgundy

The properties can be overridden by environment variables. Spring Boot lets you externalize your configuration so that you can work with the same application code in different environments. You can use properties files, YAML files, environment variables, and command-line arguments to externalize configuration. Property values can be injected directly into your beans by using the @Value annotation, accessed through Spring’s Environment abstraction, or be bound to structured objects through @ConfigurationProperties.
Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).
@TestPropertySource annotations on your tests.
properties attribute on your tests. Available on @SpringBootTest and the test annotations for testing a particular slice of your application.
Command line arguments.
Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).
ServletConfig init parameters.
ServletContext init parameters.
JNDI attributes from java:comp/env.
Java System properties (System.getProperties()).
OS environment variables.
A RandomValuePropertySource that has properties only in random.*.
Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).
Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).
Application properties outside of your packaged jar (application.properties and YAML variants).
Application properties packaged inside your jar (application.properties and YAML variants).
@PropertySource annotations on your @Configuration classes.
Default properties (specified by setting SpringApplication.setDefaultProperties).

Using Spring Profiles

We can use Spring Profiles to load the appropriate data source. If the bean does not have an active profile it is brought in.
Beans with a dev profile will get ignored in production.

public interface FakeDataSource {
    String getConnectionInfo();
}

We then have different profiles for each source:

@Component
@Profile("dev")
public class DevDataSource implements FakeDataSource{
    @Override
    public String getConnectionInfo() {
        return "I'm dev data source";
    }
}

We can then use one of those profiles in our test:


@ExtendWith(SpringExtension.class)
@ContextConfiguration(classes = DataSourceConfig.class)
@ActiveProfiles("dev")
public class DataSourceTest {
    private FakeDataSource fakeDataSource;

    @Autowired
    public void setFakeDataSource(FakeDataSource fakeDataSource) {
        this.fakeDataSource = fakeDataSource;
    }

    @Test
    void testDataSource() {
        System.out.println(fakeDataSource.toString());
    }
}

Setting the active profile at runtime

Spring offers a number of ways to set the active profile at runtime.

Using Databases

We are going to use AWS RDS for our production environment. We will have h2 for dev.
MySQL local for QA and in production we will use AWS. We will use a separate jar for our pom for mysql data source:


        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>

Here we have added h2 for dev and the mysql connector jar. Set up mysql locally and check the connection:

tom@tom-ubuntu:~/.m2/repository/org$ sudo mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.32-0ubuntu0.22.10.2 (Ubuntu)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database springguru
    -> ;
Query OK, 1 row affected (0.01 sec)

mysql>

Here we are using the root user. We then set the properties for our QA environment:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL8Dialect
spring.datasource.url=jdbc:mysql://localhost:3306/springguru
spring.datasource.username=root
spring.datasource.password=
spring.jpa.hibernate.ddl-auto=update

We then set the active profile to qa. You may need to change the permissions for connecting to mysql:

mysql > ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';
mysql > FLUSH PRIVILEGES;

Mysql Workbench connection issues on linux. On a terminal window, run the following commands:

# snap connect mysql-workbench-community:password-manager-service
# snap connect mysql-workbench-community:ssh-keys

Entity diagram of database

Creating a service account

We will create a user with restricted access. No create tables or moderate tables. This is standard practice in enterprise
applications.

CREATE USER 'springframework'@'localhost' IDENTIFIED BY 'guru';

GRANT SELECT ON springguru.* to 'springframework'@'localhost';
GRANT INSERT ON springguru.* to 'springframework'@'localhost';
GRANT DELETE ON springguru.* to 'springframework'@'localhost';
GRANT UPDATE ON springguru.* to 'springframework'@'localhost';

If we check in MySQL Workbench we see that our new user has limited privileges:

We can then add this user for testing with the qa profile:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL8Dialect
spring.datasource.url=jdbc:mysql://localhost:3306/springguru
spring.datasource.username=springframework
spring.datasource.password=guru
spring.jpa.hibernate.ddl-auto=update

It is now best practice to use a service account with restricted privileges.

Encrypting Properties

We can use Jasypt for encryption of the database password:

        <dependency>
            <groupId>com.github.ulisesbocchio</groupId>
            <artifactId>jasypt-spring-boot-starter</artifactId>
            <version>3.0.5</version>
        </dependency>

We can then encrypt the username and password:

 ./encrypt.sh input=springframework password=password

We then add the encrypted username and password to the application.properties:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL8Dialect
spring.datasource.url=jdbc:mysql://localhost:3306/springguru
jasypt.encryptor.password=password
jasypt.encryptor.iv-generator-classname=org.jasypt.iv.NoIvGenerator
jasypt.encryptor.algorithm=PBEWithMD5AndTripleDES
spring.datasource.username=ENC(3ZeqNvW+Rm0DgOkkVVXgPBxIEeyXsKQ6)
spring.datasource.password=ENC(1qHrTuI0RB4Qzl3zPE3FmQ==)
spring.jpa.hibernate.ddl-auto=update

Continuous Integration

We are now going to provision services in the cloud such as jenkins and a dns for the
jenkins instance. We will also set up Artifactory with docker so that we can deploy jenkins builds
to deploy maven artifacts to the Artifactory repository.

Introduction to AWS

We will now look in more detail at Amazon Web Services. AWS cloud compute was created because they were not using
servers during the year. This business has now grown to make them a true leader in the space. Companies like Netflix
use AWS for deployments. It is an incredible collection of resources:

For our application we will use EC2 virtual servers in the cloud to provision a server in the cloud to run our jenkins instance.
We will use Elastic Beanstalk for deploying our Tomcat instance. We will also use RDS to provision our mysql database managed
by Amazon. Amazon will handle all backups and software upgrades on the database. This is useful for small startups who want to
out source the management of their databases. We will also use Route 53 to host our domain and DNS. We will then point traffic
at this domain.

Linux distributions

We will use the Redhat flavour of linux. There are two branches: debian (ubuntu) and fedora. In enterprise applications
Redhat or Fedora is frequently used. The fedora linuxes are mostly standard in enterprise applications. Most often Redhat
enterprise linux is used.

Provisioning a server on AWS

We will now go to the AWS console to set up an AMI to use jenkins.

I have set up a Redhat linux instance and added a security group with a port range inbound traffic on port 8080.
We now install oracle jdk and set up jenkins. Our instance is a t2 micro and will reference the private IP address. We can
also use ssh to connect to the instance.

tom@tom-ubuntu:~/Desktop$ chmod 400 springguru.pem
tom@tom-ubuntu:~/Desktop$ ssh -i "springguru.pem" ec2-user@ec2-54-236-41-164.compute-1.amazonaws.com
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
[ec2-user@ip-172-31-52-103 ~]$

We are now inside the amazon instance. We will use wget to get the oracle instance of jdk.

> sudo su
> yum install wget
> wget https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz
> tar xzf amazon-corretto-11-x64-linux-jdk.tar.gz
> cp -r ./amazon-corretto-11.0.18.10.1-linux-x64/ /opt/
> cd /opt
> alternatives --install /usr/bin/java java /opt/amazon-corretto-11.0.18.10.1-linux-x64/bin/java 2
> alternatives config java

[root@ip-172-31-52-103 opt]# java -version
openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-11.0.18.10.1 (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.18.10.1 (build 11.0.18+10-LTS, mixed mode)

> alternatives --install /usr/bin/jar jar /opt/amazon-corretto-11.0.18.10.1-linux-x64/bin/jar 2
> alternatives --install /usr/bin/javac javac /opt/amazon-corretto-11.0.18.10.1-linux-x64/bin/javac 2
> alternatives --set jar /opt/amazon-corretto-11.0.18.10.1-linux-x64/bin/jar
> alternatives --set javac /opt/amazon-corretto-11.0.18.10.1-linux-x64/bin/javac

A new release is produced weekly to deliver bug fixes and features to users and plugin developers. It can be installed from the redhat yum repository.


sudo wget -O /etc/yum.repos.d/jenkins.repo \
https://pkg.jenkins.io/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo yum upgrade
sudo yum install jenkins

Add required dependencies for the jenkins package

https://www.jenkins.io/doc/book/installing/linux/

You can enable the Jenkins service to start at boot with the command:

sudo systemctl enable jenkins

You can start the Jenkins service with the command:


sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:


sudo systemctl status jenkins

We have now installed jenkins-2 on our EC2 instance and start the service.

> service jenkins start
> ps -ef | grep jenkins
root       11000    1029  0 02:17 pts/0    00:00:00 grep --color=auto jenkins

This link is very good for setting up jenkins:
https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/

This link is useful for font errors with headless java:
https://wiki.jenkins.io/display/JENKINS/Jenkins+got+java.awt.headless+problem

In particular this command is useful for fonts:

sudo yum install xorg-x11-server-Xvfb

We now get this page on http://ec2-100-26-253-161.compute-1.amazonaws.com:8080/login?from=%2F

We get the admin password by running:

[ec2-user ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword

We then install the suggested plugins and add our first user:

We now have a jenkins server for configuring builds:

How DNS works

stands for domain name services or domain name server
gets you an ip address associated with a human readable text address
springframework.guru is IP address 172.66.43.56

[ec2-user@ip-172-31-63-3 ~]$ nslookup springframework.guru

Non-authoritative answer:
Name:   springframework.guru
Address: 172.66.43.56
Name:   springframework.guru
Address: 172.66.40.200
Name:   springframework.guru
Address: 2606:4700:3108::ac42:28c8
Name:   springframework.guru
Address: 2606:4700:3108::ac42:2b38

This is the result for my own dns on route 53:

[ec2-user@ip-172-31-63-3 ~]$ nslookup drspencer.io

Non-authoritative answer:
Name:   drspencer.io
Address: 18.67.76.121
Name:   drspencer.io
Address: 18.67.76.122
Name:   drspencer.io
Address: 18.67.76.43
Name:   drspencer.io
Address: 18.67.76.113

How DNS works:

When your computer boots up it asks for an IP address. This is where it looks names up from. When we request
springframework.guru it works with top level domain. This request goes to find the dns server registered with the Internet
Server Provider. The route then goes to dns record sets which are then returned to your local machine. Cname records take
a url and convert it to springframework.guru. The DNS record then responds back with the IP address so that the computer
can find the server by the IP address. It can take time for address changes to propagate through all the global DNS servers.

Using Route 53

We are now going to set a domain and subdomain for our jenkins address using Route 53. We will point a subdomain at the ec2 instance
ip address. Now when we go to http://jenkins.drspencer.io:8080/login?from=%2F we can see the login page for jenkins.
Browsers run on port 80 but our jenkins server is running on 8080. We are overriding port 80 in the browser.
We now need to set up Apache to do port forwarding between 80 and 8080. Apache routes the war file running on 8080 to 80 so
that we can view the application directly in the browser.

[ec2-user@ip-172-31-63-3 ~]$ sudo yum install httpd
Last metadata expiration check: 1:42:53 ago on Tue Mar 21 00:26:29 2023.
Dependencies resolved.
===========================================================================================
 Package                  Arch        Version                       Repository        Size
===========================================================================================
Installing:
 httpd                    x86_64      2.4.55-1.amzn2023             amazonlinux       48 k
Installing dependencies:
 apr                      x86_64      1.7.2-2.amzn2023.0.2          amazonlinux      129 k
 apr-util                 x86_64      1.6.3-1.amzn2023.0.1          amazonlinux       98 k
 generic-logos-httpd      noarch      18.0.0-12.amzn2023.0.3        amazonlinux       19 k
 httpd-core               x86_64      2.4.55-1.amzn2023             amazonlinux      1.4 M
 httpd-filesystem         noarch      2.4.55-1.amzn2023             amazonlinux       15 k
 httpd-tools              x86_64      2.4.55-1.amzn2023             amazonlinux       82 k
 mailcap                  noarch      2.1.49-3.amzn2023.0.3         amazonlinux       33 k
Installing weak dependencies:
 apr-util-openssl         x86_64      1.6.3-1.amzn2023.0.1          amazonlinux       17 k
 mod_http2                x86_64      2.0.11-2.amzn2023             amazonlinux      150 k
 mod_lua                  x86_64      2.4.55-1.amzn2023             amazonlinux       62 k

Transaction Summary
===========================================================================================
Install  11 Packages

Total download size: 2.0 M
Installed size: 6.1 M
Is this ok [y/N]: y

[ec2-user@ip-172-31-63-3 ~]$ sudo service httpd start
Redirecting to /bin/systemctl start httpd.service

We now have apache up and running:

but we are not redirecting to jenkins yet. We need to go to configure the apache daemon:

[ec2-user@ip-172-31-63-3 ~]$ sudo service httpd start
Redirecting to /bin/systemctl start httpd.service
[ec2-user@ip-172-31-63-3 ~]$ cd /etc/httpd/conf
[ec2-user@ip-172-31-63-3 conf]$ ls
httpd.conf  magic

We now add our jenkins setup in the apache httpd.conf file:

# Jenkins setup
<VirtualHost *:80>
    ServerName jenkins.drspencer.io
    ProxyRequests Off
    ProxyPreserveHost On
    AllowEncodedSlashes NoDecode
    ProxyPass / http://localhost:8080/ nocanon
    ProxyPassReverse / http://localhost:8080/
    ProxyPassReverse / http://jenkins.drspencer.io
    <Proxy http://localhost:8080/* >
        Order deny,allow
        Allow from all
    </Proxy>
</VirtualHost>

This sets up a proxy for the server and reverses traffic from 8080. We now do an apache restart:

[ec2-user@ip-172-31-63-3 conf]$ service httpd restart
Redirecting to /bin/systemctl restart httpd.service

Failed to restart httpd.service: Access denied
See system logs and 'systemctl status httpd.service' for details.
[ec2-user@ip-172-31-63-3 conf]$ sudo service httpd restart
Redirecting to /bin/systemctl restart httpd.service
Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xeu httpd.service" for details.
[ec2-user@ip-172-31-63-3 conf]$ setsebool -P httpd_can_network_connect true
Cannot set persistent booleans, please try as root.
[ec2-user@ip-172-31-63-3 conf]$ sudo setsebool -P httpd_can_network_connect true

Because we are running on a Security-Enhanced Linux machine we have to make SE-Linux allow restart.
https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-with-jenkins/reverse-proxy-configuration-apache/#mod_proxy:~:text=If%20you%20are,P%20httpd_can_network_connect%20true
Our configuration is trying to connect to the tomcat instance running on 8080 and the Security settings are not allowing us.
This command allows apache to connect to the network:

setsebool -P httpd_can_network_connect true

We can now connect to jenkins via the domain jenkins.drspencer.io directly:

We can now block port 8080. We go to the jenkins file in /etc/sysconfig and open jenkins with vim.
We then use cntrl f to page down and add a Jenkins listen address:

JENKINS_PORT="8080"

## Type:        string
## Default:     ""
## ServiceRestart: jenkins
#
# IP address Jenkins listens on for HTTP requests.
# Default is all interfaces (0.0.0.0).
#
JENKINS_LISTEN_ADDRESS="127.0.0.1"

This blocks jenkins listening to the world on port 8080.
We then restart jenkins:

[root@ip-172-31-63-3 sysconfig]# service jenkins restart
Restarting jenkins (via systemctl):                        [  OK  ]

and check for the jenkins port run information:

[root@ip-172-31-63-3 sysconfig]# ps -f | grep jenkins
root       20945   20524  0 09:17 pts/0    00:00:00 grep --color=auto jenkins

Jenkins is now running on jenkins, its own user account.
This is a good security set up so it is locked to the outside world.
Jenkins is available on port 80 but no longer 8080.

This is not quite enough on aws linux. I had to add iptables to block all non-local access to port 8080:

iptables -A INPUT -p tcp --dport 8080 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j DROP

Now we can't access http://jenkins.drspencer.io:8080 from the browser. We can only access http://jenkins.drspencer.io

Creating ssh keys

When we are working with github and jenkins we can use a username and password for checking into git. This is not security
best practice. A great way to enable communication between github is to use ssh keys. Ssh is a great way to enable encrypted
communication between two servers and also to avoid sharing secrets. We are now going to create an ssh key from the linux
command line. We are currently under the username root:

[root@ip-172-31-63-3 sysconfig]# su jenkins
[root@ip-172-31-63-3 sysconfig]# whoami
root
[root@ip-172-31-63-3 sysconfig]# ps -ef | grep jenkins
jenkins    25804       1  0 10:18 ?        00:01:47 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
root       56019   20524  0 20:30 pts/0    00:00:00 grep --color=auto jenkins

Jenkins has non user account so we need to run the bash shell as jenkins:

[root@ip-172-31-63-3 sysconfig]# su -s /bin/bash jenkins
bash-5.2$ whoami
jenkins
bash-5.2$

We now go into the jenkins folder on our linux instance:

bash-5.2$ ls
acpid    console       i18n    man-db       nftables.conf  selinux
atd  cpupower      irqbalance  modules      rngd       sshd
chronyd  firewalld     jenkins     network      rpcbind    sysstat
clock    htcacheclean  keyboard    network-scripts  run-parts      sysstat.ioconf
bash-5.2$ cd
bash-5.2$ ls
config.xml                  nodeMonitors.xml
hudson.model.UpdateCenter.xml           nodes
hudson.plugins.git.GitTool.xml          plugins
identity.key.enc                queue.xml.bak
jenkins.install.InstallUtil.lastExecVersion secret.key
jenkins.install.UpgradeWizard.state     secret.key.not-so-secret
jenkins.model.JenkinsLocationConfiguration.xml  secrets
jenkins.telemetry.Correlator.xml        updates
jobs                        userContent
logs                        users
bash-5.2$ pwd
/var/lib/jenkins
bash-5.2$

We create an .ssh folder:

bash-5.2$ mkdir .ssh
bash-5.2$ cd .ssh/
bash-5.2$ pwd
/var/lib/jenkins/.ssh
bash-5.2$

We now want to generate an ssh key:

bash-5.2$ ssh-keygen -t rsa -C 'jenkins@example.com'
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/jenkins/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/jenkins/.ssh/id_rsa
Your public key has been saved in /var/lib/jenkins/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:3cre8EzGxywgkHxfchhxxceb0YEj5S6eRz4zAEan2Bs jenkins@example.com
The key's randomart image is:
+---[RSA 3072]----+
|         .ooo+ooo|
|     . .+ o=.o..+|
|      +..Eo +...+|
|       o.o+=.  o |
|        S.+o.o   |
|         o.+*o   |
|          +o=*+  |
|         . B.o+  |
|          . +    |
+----[SHA256]-----+

We add the public key to github:

We now need to install git on our linux instance:

> sudo yum install git
> git --version
git version 2.39.2

We are now going to setup the credentials in jenkins to use them for builds.

We also want to add maven to jenkins:

Build configuration

We are now going to create a build configuration in jenkins. We are now going to set up the build and get things cooking
in jenkins.

We also have to add our private key to the jenkins build configuration and a git webhook:

and set up the build:

We can check console output:

Now we have running builds in jenkins. We can now test the change by pushing to github.
We can test this with:

git commit --allow-empty -m "Empty commit"

on our repository.

Artifactory

We are now going to use Artifactory to store the jar builds. The advantage of Artifactory is that it is a private repository
where we can save our images. We are going to deploy Artifactory in a docker container.
We will create a new ec2 server on AWS and install docker.
To get Docker running on the AWS AMI will follow the steps below (these are all assuming you have ssh'd on to the EC2 instance).

Update the packages on our instance

[ec2-user ~]$ sudo yum update -y

Install Docker

[ec2-user ~]$ sudo yum install docker -y

Start the Docker Service

[ec2-user ~]$ sudo service docker start

Add the ec2-user to the docker group so you can execute Docker commands without using sudo.

[ec2-user ~]$ sudo usermod -a -G docker ec2-user

Containers

Have their own process space
Their own network interface
'Run' processes as root (inside the container)
Have their own disk space
(can share with host too)

Containers have no Guest operating system so we are running from the operating system of the original machine.
Docker containers are upto 30% more efficient than virtual machines.

Docker terminology

Docker image - the representation of a docker container. Like a jar file in Java
Docker container - standard runtime of Docker. Effectively a deployed and running Docker Image.
Docker Engine - The code which manages Docker stuff. Creates and runs Docker Containers

Here we see the way in which Docker is put together:

We are going to install Artifactory using volumes so that we can persist the maven repository using Artifactory.
This link was helpful:
https://medium.com/@raguyazhin/step-by-step-guide-to-install-jfrog-artifactory-on-amazon-linux-6b832dd8097b
and this one was excellent:
https://www.coachdevops.com/search?q=docker+artifactory
In particular, remember to allow ports 8081 and 8082 on your aws ec2 security group.

Resolving artifacts through Artifactory

In maven 3.9 we need https for artifactory to include these in our maven pom.
We first need to request an ssl certificate.
We will follow the instructions here:
https://levelup.gitconnected.com/adding-a-custom-domain-and-ssl-to-aws-ec2-a2eca296facd

The SSL certificate does take a few minutes to create so we do need to be patient:

Now the certificate is showing as issued:

We can now create a target group. I have a few target groups already:

We want to create one for our EC2 instance.
We then register our instance to the target group:

Next we need to create an application load balancer. We will choose the application load balancer.
We also need to set up a target group as described in the article above.
I have set up https and a DNS for the artifactory instance because we need to use
https with maven3. I added the following reverse proxy to


 on the ec2 ubuntu linux instance:


```bash
        ServerName jfrog.drspencer.io
        ProxyPass / http://localhost:8082/
        ProxyPassReverse / http://localhost:8082/

using Apache3.

Now we have working artifactory instance on https://jfrog.drspencer.io:

We can now use the correct settings in our settings.xml to deploy the jar:

Uploading to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/0.0.3/spring-core-devops-0.0.3.jar
Uploaded to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/0.0.3/spring-core-devops-0.0.3.jar (63 MB at 2.0 MB/s)
Uploading to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/0.0.3/spring-core-devops-0.0.3.pom
Uploaded to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/0.0.3/spring-core-devops-0.0.3.pom (3.9 kB at 6.2 kB/s)
Downloading from central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/maven-metadata.xml
Downloaded from central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/maven-metadata.xml (393 B at 1.5 kB/s)
Uploading to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/maven-metadata.xml
Uploaded to central: https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/maven-metadata.xml (345 B at 559 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  35.186 s
[INFO] Finished at: 2023-03-24T14:02:35Z
[INFO] ------------------------------------------------------------------------

We now see the release on our artifactory repository browser:

Jenkins with Artifactory

We now set up Jenkins to look at the maven home directory to get changes uploaded to artifactory.
We go to our operating system's home directory for jenkins:

/var/lib/jenkins

We first log into our ec2 instance and then change to jenkins user:

[ec2-user@ip-172-31-91-7 ~]$ sudo su -s /bin/bash jenkins
bash-5.2$
bash-5.2$ whoami
jenkins

We then add the same configuration from our settings.xml to /var/lib/jenkins/settings.xml. Now we return to jenkins and add a build.
We can now see that we are using our artifactory instance to download the required dependencies:

Artifactory is important so that we can host the jars that we are building. It also works as read cache to save dependencies.
Other tools on Artifactory such as Xray can help ensure that we are careful with open source software and have a history of what we
are using. Xray checks dependencies for security issues.

Virtualised cloud deployment

We will now set up a docker container for our mysql database. We will then deploy our application.
We will create a new ec2 instance and install docker with the same commands as before. We will then
add mysql with the following command:

sudo docker run -d --name mysql -p 3306:3306 -v /home/ec2-user/mysql_data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=password mysql:latest

We can now check our running docker container and connect to mysql:

# check running instances
sudo docker ps
# connect to container
sudo docker exec -it mysql bash
# connect to MySQL
mysql -p
# create database
CREATE DATABASE springguru;
create user 'spring_guru_owner'@'localhost' identified with mysq_native_password by 'password';
GRANT ALL PRIVILEGES ON springguru.* to 'spring_guru_owner'@'localhost';

Here we are connecting to mysql and creating the springguru database. We then create a new user account for interacting with the
springguru database. This is not appropriate on a production database. In a major application the application would not be
able to update the schema.

ec2 Docker application

We can run the application in the command line with properties so it is better to run with a file that can be kept on the
ec2 instance. We can set up a service with an external application.properties file on the operating system.
We put an application.properties file the same directory as the jar.

[ec2-user@ip-172-31-83-176 ~]$ ls
amazon-corretto-11-x64-linux-jdk.tar.gz  application.properties
amazon-corretto-11.0.18.10.1-linux-x64   spring-core-devops-0.0.3.jar

These are the contents:

spring.datasource.url=jdbc:mysql://54.82.50.187:3306/springguru
spring.datasource.username=spring_guru_owner
spring.datasource.password=GuruPassword
spring.jpa.hibernate.ddl-auto=update

If we wanted to add the same as environment variables we would do the following:

export SPRING_DATASOURCE_URL=jdbc:mysql://54.82.50.187:3306/springguru
export SPRING_DATASOURCE_USERNAME=spring_guru_owner
export SPRING_DATASOURCE_PASSWORD=GuruPassword

We would then run the application with:

java -jar ./spring-core-devops-0.0.3.jar

Instead we are going to run the application with systemd to ensure we keep environment variables each time.

Running Spring ec2 with persistent env

We then add a service file on the spring boot ec2 instance:

change to root and go to the system file on the linux instance

sudo su
cd /etc/systemd/system

Save the following to spring.service:

[Unit]
Description=Spring Boot Service
After=syslog.target
[Service]
User=ec2-user
# set di to location of application.properties and springboot jar
WorkingDirectory=/home/ec2-user
ExecStart=/usr/bin/java -jar spring-core-devops-0.0.3.jar
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target

The above is in /etc to run services. We run the jar as ec2-user and tell the application where java is
and run the jar file.

Reload service definitions:

systemctl daemon-reload

Add start on boot:

systemctl enable springboot.service

Start the application:

systemctl start springboot

We can now see the service is running:

[root@ip-172-31-83-176 system]# ps -ef | grep java
ec2-user   29225       1 99 12:21 ?        00:00:12 /bin/java -jar spring-core-devops-0.0.3.jar
root       29250   28901  0 12:21 pts/1    00:00:00 grep --color=auto java

In order to view the spring logs we can run the following:

tail -f /var/log/messages

If we want to remove services we can run the following:

systemctl stop spring
systemctl disable spring
rm /etc/systemd/system/spring
rm /etc/systemd/system/spring # and symlinks that might be related
rm /usr/lib/systemd/system/spring
rm /usr/lib/systemd/system/spring # and symlinks that might be related
systemctl daemon-reload
systemctl reset-failed

We can check the systemctl services running:

systemctl | grep spring

We can see the logs for the service:

journalctl -u service-name.service

You can also stop overrun for the logs with


.
You
We can see the application running at the public address for the ec2 instance (port 8080):
![image](https://user-images.githubusercontent.com/27693622/227719004-83520eb4-97b8-4d52-aa8d-3cfb31e1e602.png)
We can also view logs for a certain time frame:


```bash
journalctl -u springboot.service --since "2023-03-25 13:36:00" | less

Amazon RDS

We are now going to get our feet wet with Amazon RDS. We will use RDS to manage our mysql database.
Amazon backs up the database, applies patches and allows us to offload the management of the database to Amazon.
Amazon RDS allows us not to have to manage the database administration. AWS also helps us with security and patching.
We will provision an Amazon database on RDS and then update our configuration for our application to use the RDS database.
We will leverage the properties for the new database. Everything should work the same.

Provision Mysql database on RDS

We will start by provisioning a database on Amazon RDS.

This takes a while to set up the database:

We can now connect to the database using the following configuration:

guru.springframework.profile.message=This is mysql rds Profile
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL8Dialect
spring.datasource.url=jdbc:mysql://<YOUR MYSQL INSTANCE URL>:3306/springgurudb
spring.jpa.hibernate.ddl-auto=update
spring.datasource.username=spring
spring.datasource.password=password

Our application is now running against the rds database locally.
We now deploy our application using artifactory and then pull the new jar onto our ec2 instance.

We can now pull the jar to our ec2 instance using wget:

sudo wget --user=tom --password=AKCp8nzqSiJFSBYhrCe4q14oDSSsJUH7dhivt4q2Y8ym8igR3fRxdJ3tGSnebXX68aq6CfvKM https://jfrog.drspencer.io/artifactory/libs-release-local/guru/springframework/spring-core-devops/0.0.4/spring-core-devops-0.0.4.jar

We then run the application jar:

java -jar spring-core-devops-0.0.4.jar

Our application is now running on port 8080 on our ec2 instance:

I hope you find this intro to Spring with AWS helpful!

Top Tips for the AWS Certified Cloud Practitioner Exam!

Tom Spencer — Sat, 20 Mar 2021 13:53:33 +0000

I was really pleased to receive the AWS Cloud Practitioner exam this month. It was also great fun to take the opportunity to talk about this shared achievement with Brian Hough, Shubham and Parth. In this video we discussed best approaches for the Cloud Practitioner.

Resources:

A Cloud Guru.
A Cloud Guru is a nice mix of practical and theory and this was very useful for revising for the exam.
Qwik Labs
I really like working on the Qwiklabs training which includes labs on a wide range of AWS Cloud resources including Serverless Design and Big Data. Very useful for slightly more practical experience using Cloud without the worry of destroying resources.
Digital Cloud
This was a really nice overview of the exam. I used the Exam Cram sessions for last minute revision before the exam.
AWS Power Hour

AWS Power Hour was a very useful overview of the fundamentals of AWS.

Free Code Camp
I really liked this video from Andrew Brown which gave a really useful high level view of the Cloud services.
Whizlabs
Whizlabs was very useful for past papers and questions for AWS Cloud Practitioner.

The Cloud Practitioner has increased my confidence immensely and I have plans to move onto the AWS Cloud Developer exam. Go AWS!

AWS S3: A great way to share files through your browser!

Tom Spencer — Fri, 19 Feb 2021 19:59:59 +0000

This article forms part of a series of articles about the range of services that are available on AWS! In an earlier article I described why AWS Power Hour is such a great resource for learning about the wide range of AWS services. In this article I am continuing the theme of AWS dopeness by using S3 to share files.

WTF is S3?

Amazon Simple Storage Service (Amazon S3) is a safe, secure and highly scalable solution for object storage in the cloud. It is used for backup and storage, application or media hosting, high traffic website hosting or software delivery. To really understand S3 we need to clarify a few simple concepts. So let's get some points clear before we start:

AWS S3 stores data as objects within buckets. An object contains a file and optionally any metadata that describes the file.
In order to store objects in S3 you need to upload to a bucket. When you upload to your bucket you can set permissions on each of the objects within your buckets as well as any metadata.
You can also control access to your bucket, defining who can create, delete and list objects in the bucket.

Why is S3 good for me?

Everyone deserves a good series of holiday pics after busting their guts over Christmas.

Send it through S3!

Publishing data on Amazon S3

You are on holiday and want to share some of your pictures with the world. How would you share if you forgot your web server?

Amazon S3 is a perfect means by which to store and retrieve any files from anywhere on the internet. Amazon S3 is a web server for your data but save you from paying for keeping a web server. When you store pictures or data on S3 you are only charged for data storage and requests and data transfer.
In this article we are going to show you how to share your holiday pics on Amazon S3!

In this article I presume you have set up your own AWS account. If you haven't done this check out the instructions here.

Create an AWS S3 Bucket

First create your bucket

Then upload files:

Grant access to the bucket

Go into your bucket and choose edit under block public access. Deselect the Block all public access option and then leave all the options unselected.
You will also need to add a bucket policy under bucket policy:

You should add the correct configuration for your bucket then save the changes.

Configure static website hosting

Next you should choose edit on static website hosting and enable static website hosting. You will then have options to configure the index document and error document.

Enable Cross-Origin Resource Sharing

Finally, you should enable CORS with the following policy under CORS configuration:

AWS Power Hour: Great series of lessons to help with AWS Cloud Practitioner exam

Tom Spencer — Sat, 09 Jan 2021 20:08:40 +0000

AWS Power Hour is a great course to help learn AWS!

AWS Power Hour

It includes quizzes, videos, links to important resources and all in a six episode series that encapsulates everything you need to know for your Cloud Practitioner exam. And what's more... it's free! I want to take a moment to explain why I love this course.

The course is wide ranging but also the best aspect was the inclusive approach that the AWS Technical Evangelists brought to the twitch series that accompanied AWS Power Hour.

AWS

Diagrams

One aspect of the teaching during the AWS Power Hour course was the emphasis on improving our diagrams. Our homework included a task to draw a highly available web application running on Amazon EC2 within a region.

My diagram used an Elastic Load Balancer, Virtual Private Cloud and EC2 instances running in separate regions and public subnets. Not great for security but highly available at least!

Another homework asked us to draw a highly available and automatically scaled

This is a great course and I look forward to sharing progress.

Andrew Brown's Cloud Practitioner Overview

Another useful video is Andrew Brown's epic overview of the AWS Cloud Practitioner exam:

Top Ten AWS Services

Tom Spencer — Mon, 07 Dec 2020 22:50:46 +0000

There are a huge amount of AWS services with more being added and updated all the time as re:Invent shows. In this article we will give some context by showing you the top ten AWS services.

1. AWS EC2

The first service allows businesses to do away with physical services. Instead, developers and operations teams can use virtual machines with Amazon EC2 while managing other server features such as ports, security, and storage. Amazon EC2 is the vanilla option for servers on AWS and is extremely popular.

2. Amazon RDS

Amazon Relational Database Service (RDS) allows developers to create dedicated database instances within minutes. These instances can then support database engines such as SQL, PostgresQL.

3. Amazon RDS

Amazon Simple Storage Service (S3) offers highly secure and redundant file storage. With Amazon S3 we can get flexibility with low latency.

4. Amazon CloudFront

CloudFront works as a Global content delivery service (CDN) to improve end user content delivery. CloudFront helps to increase web page loading speed and can even pull website static files from data centres for efficient delivery.

5. Amazon VPC

With VPC (Virtual Private Cloud) we can create a private virtual network that cannot be accessed by anyone or anything except the people and systems we choose.

We can create a VPC with our AWS Management Console and then immediately access route tables, security groups, IP ranges and subnets. Some extras include security features such as network access control lists and security groups.

6. Amazon SNS

Amazon SNS is an event-driven system that alerts subscribers to perform tasks in response to specified triggers. SNS allows us to send notifications to users on any platform.

7. AWS Beanstalk

With Elastic Beanstalk allows developers to quickly deploy and manage applications in the AWS Cloud without having to learn about the infrastructure that runs those applications. Elastic Beanstalk reduces management complexity but also allows builders to retain choice and control. Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.

8. AWS Lambda

AWS Lambda is a compute service that allows builders to run code without provisioning or managing servers. The service only runs code when it is needed and scales automatically from a few requests a day to thousands a second. Lambda runs on high availability compute infrastructure and performs server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.

9. AWS Autoscaling

AWS Auto Scaling lets developers build scaling plans that automate how groups of different resources respond to changes in demand. AWS Auto Scaling automatically creates all scaling policies and sets targets for you based on your preference regarding cost or availability.

10. AWS IAM

AWS Identity Access Management fortifies sensitive data and AWS resources. It can be used together with 2FA and MFA and provides a useful extra layer of security. IAM allows users to manage users, roles and groups and decide how much or how little access to provide. Developers can also use service linked roles to delegate permissions to AWS services that create and manage AWS resources on their behalf.

AWS Community Builders react to AWS re:Invent 2020🤯

Tom Spencer — Sat, 05 Dec 2020 17:39:32 +0000

👉👉 Jump to the reactions 👈👈

Intro

What is re:Invent?

Packed with keynote announcements, training opportunities, and more than 2,500 technical sessions, re:Invent is a chance for the global cloud computing community to find out the latest news from AWS HQ, brush up on their skills, and take valuable insight back to their teams and businesses.

Why is re:Invent so important?

AWS re:Invent 2020 is the Amazon Web Services annual user conference that focuses on:

Cloud strategies
Cloud operations
Security and developer productivity
IT architecture and infrastructure

What is different about re:Invent this year?

Like everything in 2020, this year’s AWS re:Invent will be different. But unlike everything else about this year, there could be a massive upside because AWS re:Invent 2020 is free and entirely virtual.

What is an AWS Community Builder?

Community Builders are individuals who are passionate about using AWS and share what they have learned through blog posts, online articles like this one, teaching others, and more. The idea is the community builder shares their experience to help others either not make the same mistakes, or look at a problem differently and consider other solutions.

Why write this article?

I wanted to give a short overview of each of the AWS announcements with some expert reactions in order to assess the advantages of new developments in the AWS stack.

AWS Community Builder Reactions and Analysis

Andy Jassy Keynote, AWS Re:Invent 2020

The following are the list of key service updates announced by Andy Jassy in his keynote speech today (December 1).

Instances

Habana Gaudi-based Amazon EC2 instances

ML training powered by new processors from Intel.

AWS Trainium

ML Training Chip custom designed by AWS to deliver the most cost-effective training in the cloud. Available on EC2 or Sage Maker (read full article)

AWS re:Invent 2020 - Andy Jassy Keynote Highlights

Ajay Kumar S ・ Dec 1 '20 ・ 4 min read

#aws #cloud #serverless #reinvent

Whats New In Data: re:invent Andy Jassy Keynote

It's a different experience this year. The chat with my teammates is a mixture of discussion about new features and pictures of good times in Vegas from previous re:Invent conferences.

Andy Jassy has finished the first keynote of 2020 and I was not disappointed. Lots of great new features that we have use cases for.

Here are my favourite data related features announced during the Andy Jassy re:Invent keynote.

Whats New In Data: re:invent Andy Jassy Keynote

Matt Houghton ・ Dec 1 '20 ・ 5 min read

#datascience #analytics #database #aws

AWS Proton

The process of defining a service template involves the definition of cloud resources, continuous integration and continuous delivery (CI/CD) pipelines, and observability tools. AWS Proton will integrate with commonly used CI/CD pipelines and observability tools such as CodePipeline and CloudWatch. It also provides curated templates that follow AWS best practices for common use cases such as web services running on AWS Fargate or stream processing apps built on AWS Lambda.

Infrastructure teams can visualize and manage the list of service templates in the AWS Management Console.

This is what the list of templates looks like. (read full article)

AWS Proton Tutorial - Sample NodeJS App

Mubbashir Mustafa ・ Dec 4 '20 ・ 6 min read

#aws #devops #docker #tutorial

OWASP Top Ten

Tom Spencer — Thu, 01 Oct 2020 00:00:00 +0000

In this article, we discuss why the Open Web Application Security Project Top Ten is important in a developer context. The Open Web Application Security Project is a non-profit foundation that works to improve the security of software. The OWASP Top Ten raises vital issues of software security regularly based on most recent security breaches in the industry. Security can no longer be an afterthought, developers and business leaders need to bake in security to their applications and be ready to explain to senior management how to protect the company.

Application security is of paramount importance for companies. Breaches of application security can lead to regulatory fines, identity theft and fraud. Poor security can damage an organisation, its finances and personnel. In 2017 the Wannacry virus hit a perhaps overly complacent Talk Talk and reportedly cost the NHS £92 million, impacting the provision of care to patients. The CEO of Target resigned after a serious data breach. Other data breaches include Capital One which was fined $80 million for a data breach affecting up to 100 million users.

In this article, we will consider the Top Ten application vulnerabilities as laid out by the Open Web Application Security Project (OWASP).

OWASP

The Open Web Application Security Project is an online community that produces freely available articles on cyber security. The OWASP foundation also releases a regular update on the top ten security threats. The OWASP Top Ten is a regularly updated catalogue of app security incidents and vulnerabilities, first published in 2003. The mission of the OWASP foundation is to raise awareness about application security by identifying the current most critical risks companies are facing. This article reviews these vulnerabilities and also explores possible solutions for each of them.

1. Injection

SQL Injection

SQL injection involves the contamination of ordinary user input in order to turn one SQL statement into more than one. The archetypal example is the Bobby Tables attack.
In this amusing tale, the boy’s first name is “Robert’);” and his last name is “DROP TABLE Students;-- ”. The records application at Bobby’s school has concatenated strings in order to make queries to the database. The bracket in Bobby’s first name prematurely terminates the original query. Then his last name does its damage. The double hyphen at the end of Bobby’s surname acts as a comment so that the database will ignore the remainder of the input and the originally intended query.

In order to avoid SQL injection developers should avoid using a plain SQL statement in Java:

            String query = "SELECT * FROM STUDENT WHERE NAME = '" + name + "‘;"

Instead, developers should rather use a prepared statement:

            String query = “SELECT * FROM STUDENT WHERE NAME = ?;”;
            PreparedStatement stmt = connection.prepareStatement(query);
            stmt.setString(1, name);
            ResultSet results = stmt.executeQuery();

Here we prepare the SQL statement and turn the SQL into parameterized code without danger of injecting arbitrary SQL into the program.

Tips:

Use positive or “whitelist” server-side input validation
Escape special characters for inputs

2. Broken Authentication and Session Management

Authentication is the process of verifying that a user, or application, is who or what they claim to be. Authentication is usually carried out through the user inputting private information.

Session management is the process whereby the server maintains the state of the entity it is interacting with. This is necessary because HTTP is a stateless protocol meaning that it does not keep context information from earlier sent packets of information. Each transmission via HTTP can be understood in isolation without reference to other sent packets. Websites requiring sign-in use sessions in order to persist the user’s interaction with the site.

Sessions are maintained on the server through a session identifier which is passed back and forth with the client when transmitting and receiving requests. As a first precaution application should not include session IDs in their URLs.

“Session hijacking” is a major vulnerability for applications. In this form of attack, a session ID in plain text might be duplicated by the attacker. An attacker might have found a session identifier in the front end in order to carry out this attack, so ensuring that session identifiers are hidden is a good best practice.

There are many areas to consider when preparing a business case for a modernisation project. Still, most of the time, you can be quite safe if you focus on these three factors: the audience, what they need to know, and what they pay off is.

Tips:

Session IDs should be randomly generated and regenerated each time the user authenticates.
Avoid allowing unlimited authentication attempts as this can be an invitation for hackers.

3. Sensitive Data Exposure

Sensitive data can be loosely defined as valuable information that can be stolen or used against the victim. Sensitive data includes credit card details, medical records, email and even purchasing records.

Sensitive data exposure is often headline-worthy and can be incredibly damaging for companies. Here, it is useful to differentiate between data at rest and data in transit. Data in transit is data actively moving from one location to another. Data at rest is data stored on any device or network. Attackers are more likely to raid data in transit than attempt to break cryptographic protections. An attack might take place at points in the network where there is no TLS and information is passed through REST or HTTP (perhaps internally). An attacker could also target a stolen laptop with unencrypted information.

Teams should apply mutual authentication between microservices using TLS. This has the added advantage of encrypting data transmission between microservices.
It is also important to ensure sensitive data is encrypted in the database and to decrypt data based on the user’s authorization as opposed to that of the server. Avoid storing passwords in plain text.

Tips:

Classify and identify sensitive data in applications according to privacy laws or business needs
Don’t store sensitive data unnecessarily
Encrypt all sensitive data at rest
Encrypt all data in transit with secure protocols such as TLS

4. XML External Entities (XXE)

XML or External Entity Injection is a web vulnerability which allows the attacker to interfere with the processing of XML data. Here is an example of Java code that is open to an XML injection attack:

            File xmlFile = new File(“c://input.xml”);
            DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
            Document doc = dBuilder.parse(xmlFile);

The referenced XML file contains a further reference to a potentially malicious external (secret.txt):

            <?xml version=”1.0”?>
            <DOCTYPE test [
                <!ENTITY xxe SYSTEM “c://secret.txt” >
                ]>
            <test>&xxe;</test>

The insertion of the secret.txt file could allow the attacker to embed malicious code within the XML parser code. In order to protect against this form of attack we should add the following to our Java code:

            DocumentBuilderFactor dbFactory = DocumentBuilderFactory.newInstance();
            dbFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);

Here the secure processing feature is set to true in order to ensure that our XML is safe from injection attacks.

Tips:

Wherever possible use less complex data formats such as JSON and avoid serialization of sensitive data
Upgrade all XML processors and libraries in use. Use SOAP 1.2 or higher
Disable XML external entity in all XML parsers

5. Broken Access Control

Broken access control is when a user can access a resource or perform an action that they are not authorized to access. Broken access control can occur when attackers are able to directly access objects on the server. The following code uses unverified data from a request in an SQL query that is accessing information from the database:

            stmt.setString(1, request.getParameter(“#123”);
            ResultSet results = stmt.executeQuery();

The attacker is, therefore, able to modify the URL in the browser to query any account they want:

http://mybank.com/app/accountInfo?acct=#456

There are two possible means by which this form of attack might be mitigated.

The first, and perhaps more obvious protection against broken access control, would be to avoid including sensitive information in URLs. A further extension of this approach would be to use session specific mapping from random IDs to real IDs or use generic URLs that are session sensitive.

The second, more general, rule to follow would be that if a caller is not authorized to see the contents of a resource it should be as if the resource does not exist. In Java this is relatively easy to implement with @Preauthorize:

                @PreAuthorize(“hasAuthority(‘Admin’)”)
                @RequestMapping(“/restricted”)
                @ResponseBody
                public String restricted() {
                    return “restricted”;
                }

The @Preauthorize annotation ensures that an attacker would get a 403 response if they tried to access the resource.

Tips:

JWT tokens should be invalidated on the server after logout
Log access control failures and alert admins for repeated failures
With the exception of public resources, deny by default

6. Security Misconfiguration

Security misconfiguration occurs when developers fail to implement all the security controls for a server or web application, or worse, implement security controls with errors. In 2017 an unknown group of hackers were able to gain access to thousands of MongoDB databases by using the default passwords. The lesson is clear, forbid default passwords. Default passwords are a serious hazard.

Another metric for security configuration is an overexposed "attack surface", meaning the sum of IP addresses ports and protocols open to attackers. This vulnerability can be easily reduced by splitting internal traffic into its own interface separate from public facing traffic. It is also important to have a clear inventory of applications that are exposed on production. A sample server application released inadvertently to production can be an attractive target for attackers particularly since some of them are well known and are never updated. Reducing the “attack surface” is also a responsibility of the wider team in ensuring that release processes are robust in order to monitor and test non-production hardened applications.

Tips:

Automate processes to verify effectiveness of configurations in all environments
Keep platforms minimal without unnecessary features, components, documentation and samples
Remove or do not install unused features and frameworks

7. Cross Site Scripting (XSS)

Cross site Scripting, or XSS, is when an attacker deceives the client into running JavaScript in the victim’s browser. Reflected XXS occurs when attackers manipulate the DOM in order to add their own data. Vulnerability to this kind of attack is even possible in modern JS applications. Vulnerable code would look like this:

                <script>window.__STATE__ = $(JSON.stringify({data})</script>

Here, JSON.stringify() will turn data into a string as long as it is valid JSON for rendering on the page. Accordingly, attackers might add scripts to usernames or bios:

                {
                    username: “LOL”,
                    bio: “</script><script>alert(“XSS attack!”)</script>
                }

One solution to this vulnerability is to use the serialize-javascript npm module to escape the rendered JSON:

                > npm install --save serialize-javascript

The window variable can then be wrapped in the method serialise():

                <script>window.__STATE__ = ${ serialize( data, { isJSON: true } ) }</script>

Serializing and deserializing data is a common source of vulnerability and we will revisit this issue later in the article.

Tips:

Use frameworks that automatically escape XSS by design
Escape untrusted HTTP request data
Apply context sensitive encoding to protect against DOM XSS

8. Insecure deserialization

Serialization, aka marshalling in some languages, involves the conversion of objects to byte streams for transmission so that the object can then be reverted to a copy of the object. In this process, objects are converted into a format that can be saved to the database or transferred over a network. The reverse, deserialization, is when a serialized object is read from a file or network and converted back into an object.

Insecure deserialization can result in arbitrary code execution, granting attackers a wide range of capabilities. It occurs when attackers are able to manipulate serialized objects and authenticate as someone they are not. If an application uses insecure deserialization malicious users might even be able to embed code snippets in the object for execution during deserialization. Generally, deserialization of user input should be avoided unless absolutely necessary. Attacks involving insecure deserialization can also be used to inflict a DoS attack, execute code, whether through SQL injection, XSS.

Tips:

Enforce strict type constraints during deserialization before object creation
Log deserialization exceptions and failures
Monitor deserialization and alert if a user deserializes constantly

9. Using components with Known vulnerabilities

Known vulnerabilities are publicly disclosed security bugs. Attackers use these public vulnerabilities to target applications so it is important to ensure that security is up to date. Snyk is a good resource for checking vulnerabilities. For node applications Snyk can be installed via npm and authenticated on the command line. We can then use the command ‘snyk test’ to check the project for known vulnerabilities. Snyk is available in many other languages including Java.

It is extremely important for teams to stay ahead of potential vulnerabilities. Apache Struts 2 was a particularly notorious example of a third party vulnerability that was exploited by attackers. Apache Struts 2 is an open-source Java web application framework.

Tips:

Remove unused dependencies, unnecessary features, components, files and documentation
Continuously inventory versions of client side and server side components
Monitor for unmaintained libraries and components without security patches for older versions

10. Insufficient logging and monitoring

A good definition of monitoring was laid out by Greg Poirier put it at the Monitorama 2016 conference:

Monitoring is the action of observing and checking the behavior and outputs of a system and its components over time.

Exploiting insufficient logging and monitoring is the bedrock of nearly every major data breach or security incident. Attackers rely on the lack of monitoring and timely response to avoid detection. Unfortunately, there is no magic silver bullet to resolve lack of monitoring in your organisation.
For organisations running large infrastructure paying monitoring servers is not sufficient but needs to be combined with monitoring network infrastructure and applications.

Many software attacks use rootkits in order to gain access level to a computer or network. Rootkits are programs that apply concealment techniques to malicious code and activities in order to avoid detection. Rootkits can be difficult to detect. Rkhunter is one example of a security monitoring tool which scans for rootkits and other vulnerabilities. This can be a useful addition to your monitoring toolkit.

Tips:

Establish effective monitoring and alerting to detect and respond to suspicious activities quickly and efficiently
Ensure logs are generated in a format that is easily consumed by centralized log management solutions
Ensure an audit trail for high value transactions

Conclusion

The wide range of companies that have fallen victim to malicious breaches and the widely publicised mega breaches of recent years shows the importance of making security assessment part of the everyday work of developers, architects and business leaders. Possible team wide initiatives to improve security awareness could include a bug bounty, gamifying clearer developer understanding through regular email challenges or even a monthly cybersecurity quiz night. The most important point is to build a learners’ mindset amongst teams regarding cybersecurity and the OWASP Top Ten is a first port of call for better understanding in order to plan and defend against possible threats. The findings regularly remind us of the importance of baking security into the development process in order to manage security risks effectively.

For developers interested in developing awareness of the OWASP top ten amongst business leaders we have another article directed at business leaders.

If you wish to learn more about refactoring and improving security for your business please fill out the form below to access the handout.