DEV Community: mydeveloperplanet

How to Deploy a Spring Boot App on AWS ECS Cluster

mydeveloperplanet — Wed, 15 Sep 2021 15:42:12 +0000

In this post, you will learn how to setup an AWS ECS (Elastic Container Service) Cluster. You will create the Cluster and deploy a Docker image containing a Spring Boot App. Enjoy!

1. Introduction

Amazon Elastic Container Service is a managed container orchestration service which allows you to deploy and scale containerized applications. An overview of the features and pricing can be found at the AWS website.

ECS consists out of a few components:

Elastic Container Repository (ECR): A Docker repository to store your Docker images (similar as DockerHub but now provisioned by AWS).
Task Definition: A versioned template of a task which you would like to run. Here you will specify the Docker image to be used, memory, CPU, etc. for your container.
ECS Cluster: The Cluster definition itself where you will specify how many instances you would like to have and how it should scale.
Service: Based on a Task Definition, you will deploy the container by means of a Service into your Cluster. This is basically everything you need to know in order to get started with this hands-on tutorial. The sources being used in this post are available at GitHub.

You will create a Docker image for a basic Spring Boot Application, upload it to ECR, create a Task Definition for the image, create a Cluster and deploy the container by means of a Service to the Cluster. At the end, you will also do something similar but including an Application Load Balancer.

At the time of writing, a new UI is being developed. For most of the actions described in this post, the New ECS Experience toggle has been enabled.

2. Create the App

The Spring Boot App is a basic application with a Hello Rest endpoint which returns a hello message including the host where the application is running. The app is used before in a previous post. The controller is the following:

@RestController
public class HelloController {

    @GetMapping("/hello")
    public String hello() {
        String message = "Hello AWS!";
        try {
            InetAddress ip = InetAddress.getLocalHost();
            message += " From host: " + ip;
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
        return message;
    }

}

In order to be able to create a Docker image, you need to add the dockerfile-maven-plugin to the pom file.

<properties>
    <java.version>11</java.version>
   <docker.image.prefix>mydeveloperplanet</docker.image.prefix>
</properties>
...
<build>
    <plugins>
        ...
        <plugin>
            <groupId>com.spotify</groupId>
            <artifactId>dockerfile-maven-plugin</artifactId>
            <version>1.4.12</version>
            <executions>
                <execution>
                    <id>default</id>
                    <goals>
                        <goal>build</goal>
                        <goal>push</goal>
                    </goals>
                </execution>
            </executions>

            <configuration>
                <repository>${docker.image.prefix}/myawsplanet</repository>
                <tag>${project.version}</tag>
                <buildArgs>
                    <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
                </buildArgs>
            </configuration>
        </plugin>
    </plugins>

</build>

You also need to add a Dockerfile to the root of the repository. The Dockerfile is following some best practices from a previous post.

FROM openjdk:11-jdk
VOLUME /tmp

RUN useradd -d /home/appuser -m -s /bin/bash appuser
USER appuser

HEALTHCHECK --interval=5m --timeout=3s CMD curl -f http://localhost:8080/actuator/health/ || exit 1

ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

Run the build in order to create the jar file and the corresponding Docker image.

$ mvn clean verify

Verify whether the Docker image is available in your local Docker repository.

$ docker images
REPOSITORY                                          TAG                 IMAGE ID            CREATED             SIZE
mydeveloperplanet/myawsplanet                       0.0.1-SNAPSHOT      765984f7cfc2        24 seconds ago      666MB

3. Upload Image to ECR

Now that you have created the Docker image, you need to upload it to ECR, the AWS Docker repository. Navigate in AWS to the ECS Service and select in the left menu the Repositories section. First thing to do, is to create a repository by clicking the Create repository button.

Give the repository a recognizable name like mydeveloperplanet/myawsplanet and click the Create repository button.

In order to see how you can push the Docker image to the repository, click the View push commands button which is available in the repository overview.

Execute step 1, which will provide you temporary credentials in order to be able to gain access to the repository. The <account ID> will be replaced with your AWS account ID.

$ aws ecr get-login-password --region eu-west-3 | docker login --username AWS --password-stdin <account ID>.dkr.ecr.eu-west-3.amazonaws.com

Skip step2, building the Docker image is already executed by means of the Maven build. In step 3, adjust the command in order to use version 0.0.1-SNAPSHOT instead of latest for identifying the local Docker image.

$ docker tag mydeveloperplanet/myawsplanet:0.0.1-SNAPSHOT <account ID>.dkr.ecr.eu-west-3.amazonaws.com/mydeveloperplanet/myawsplanet:latest

In step 4, you push the local Docker image to the remote AWS Docker repository.

$ docker push <account ID>.dkr.ecr.eu-west-3.amazonaws.com/mydeveloperplanet/myawsplanet:latest

After successful upload, the Docker image is added to the repository.

4. Create Task Definition

Now that the Docker image is available in ECR, next thing to do is to create a Task Definition by creating the Create new Task Definition button in the Task Definitions section (left menu).

In step 1, choose for an EC2 self managed task and click the Next step button.

In step 2, give the task definiton the name myawsplanet.

Select a Task execution role in the Task execution IAM role section. This role is necessary for pulling the Docker image. If the role does not exist yet, select Create new role in the dropdown list. The Task memory and Task CPU fields are optional, leave them empty for now. In the screenshot below, the container is already added. See below the screenshot how to accomplish this.

Click the Add container button. Give the container the name myawsplanet. Fill in the Image to be pulled. This should be the Image URI of the Docker image from the ECR repository followed by the tag. Set the Memory Limits to 300 and add a Port mappings entry for host port 8080 to the container port 8080 (the Spring Boot application runs on port 8080 inside the container). Finally, click the Add button in order to add the container to the Task Definition.

The only thing left to do is to finalize step 2 by clicking the Create button at the bottom of the page.

5. Create Cluster

Navigate in the left menu to the Clusters section and click the Create cluster button.

In step 1, choose EC2 Linux + Networking and click the Next step button.

In step 2, give the Cluster the name myawsplanet and choose the t2.micro as EC2 instance type. This will allow you to remain in the Free Tier. Leave the other options as default. This will launch 1 EC2 instance into your Cluster.

In the Networking section, choose the default VPC and select all available subnets. Choose Create a new security group as Security group. You can also choose a Key pair if you want to be able to SSH to the EC2 instances. In the screenshot below, SSH access is not enabled.

The new security group must allow all traffic for port 8080 otherwise the Rest endpoint will not be accessible. This is configured in the Security group inbound rules section. Next, choose a previously created Container instance IAM role or choose Create new role when this is the first time you are going to use this. Click the Create button.

After a few minutes, the ECS Cluster is running. Navigate to the EC2 service and notice that the Auto Scaling group has been created and one EC2 instance is running.

The EC2 instance:

6. Create Service

You have created a Cluster and a Task Definition. Now it is time to deploy the Task Definition containing the configuration for the Docker container into the Cluster. Navigate to the Cluster and click the Deploy button.

Expand the Compute configuration (advanced) section and choose EC2 as Launch type.

In the Deployment configuration section, choose the Task Definition myawsplanet as Family and choose the latest revision. As you can see, task definitons are version controlled. Give the Service the name myawsplanetservice and create the Service.

Navigate to the EC2 service and copy the public URL of the EC2 instance. Use this URL in order to verify whether the endpoint is accessible.

$ curl http://ec2-13-36-172-189.eu-west-3.compute.amazonaws.com:8080/hello
Hello AWS! From host: a035a951c3c8/172.17.0.2

7. Create Service With ALB

With the previous configuration it is possible to run more tasks with more EC2 instances. However, it is not possible to route the traffic for port 8080 to different containers running on 1 EC2 instance. In other words, you are limited to run 1 task at each EC2 instance. Solution for this is to create a Service with an Application Load Balancer (ALB). It is, however, not possible to edit an existing Service to use an ALB. Therefore, you need to create a new Service. The Task Definition itself can be reused.

First, you will remove the existing Service. Navigate to the Service and click the Edit button. Set the Desired tasks to 0. This is necessary before you can remove the Service (do not mind the Revision number in the screenshot below, I just messed around while creating this blog). Click the Update button and remove the Service.

Navigate to the Task Definition and click the Create new revision button. Navigate to the container definition and set the host port to 0. This way, the ALB will choose a random port for the communication between the ALB and each Docker container. Click the Update button in order to update the container definiton and click the Create button in order to create the new Task Definition revision.

Create a new Service just like before, but in the Load balancing section choose for Application Load Balancer. Give the ALB the name MyAwsAlb and define port 8080 where the ALB should listen to. Give Target group the name ecs2containers and choose HTTP as protocol. Finally, click the Deploy button at the bottom of the page.

After a few minutes, the Service is created. Verify whether the Rest endpoint is accessible using the public IP of the ALB.

$ curl http://MyAwsAlb-4611708.eu-west-3.elb.amazonaws.com:8080/hello
curl: (28) Failed to connect to MyAwsAlb-4611708.eu-west-3.elb.amazonaws.com port 8080: Connection timed out

A time-out occurs, meaning that something is wrong with the Security Groups.

Navigate to the EC2 service to the Security Groups section. Create a new Security Group for the ALB where inbound traffic for port 8080 is allowed.

Attach this Security Group to the ALB via the Details tab in the Security Group section.

Navigate to the EC2 service to the EC2 Instance. Click the Security Group in the Security tab. Click the Edit inbound rules button in the Inbound rules tab and add an All Traffic rule with the ALB Security Group as source. Remove the existing 8080 rule.

Verify again whether the end point is accessible and now it works!

$ curl http://MyAwsAlb-555143254.eu-west-3.elb.amazonaws.com:8080/hello
Hello AWS! From host: 9a71b51064ee/172.17.0.2

The goal was to be able to deploy the image to multiple containers running on a single EC2 instance. Navigate to the Service and change the number of tasks to 2. Verify again with curl and you will notice that the IP is changing. Two tasks are running at one EC2 instance. It is possible that a 502 Bad Gateway occurs after updating the tasks, but this should stop after a few seconds.

Finally, let’s activate 2 EC2 instances by updating the ASG in the EC2 service.

In the ECS Service you can set the number of tasks to 4.

In the classic view (disable the New ECS Experience toggle), you can see that 2 EC2 instances are running with each 2 tasks.

Running curl again shows you that the four tasks are interrogated one after another.

$ curl http://MyAwsAlb-555143254.eu-west-3.elb.amazonaws.com:8080/hello
Hello AWS! From host: 5e97f1d8c823/172.17.0.3
$ curl http://MyAwsAlb-555143254.eu-west-3.elb.amazonaws.com:8080/hello
Hello AWS! From host: ef015b5e2050/172.17.0.2
$ curl http://MyAwsAlb-555143254.eu-west-3.elb.amazonaws.com:8080/hello
Hello AWS! From host: e70d69bb7d02/172.17.0.2
$ curl http://MyAwsAlb-555143254.eu-west-3.elb.amazonaws.com:8080/hello
Hello AWS! From host: 1b2af349eb95/172.17.0.3

8. Cleanup

Execute the following steps in order to remove all resources:

Set the number of tasks to 0 at the ECS Service;
Remove the Cluster, this is at the moment of writing only available in the classic view;
Click the Task Defintion, select all revisions and via the Actions menu, click the Deregister item;
Remove the ECR repository;
Remove the ALB, the Target Group and the Security Group my-ecs-alb-sg;
In the CloudFormation service, remove the Stack which was created. You did not create this by yourselves, but the ECS Service is using CloudFormation in the background in order to create the necessary resources.

9. Conclusion

In this blog, you learned how to create and configure an ECS Cluster and deployed a Docker image to it containing a Spring Boot Application. You also learned how to scale the EC2 instances and how to run multiple tasks on multiple EC2 instances using an ALB.

How to Create an AWS ALB and ASG

mydeveloperplanet — Tue, 29 Jun 2021 19:41:03 +0000

In this post, you will learn how to create an AWS Application Load Balancer (ALB) for your EC2 instances running a Spring Boot application. You will also create an Autoscaling Group (ASG) which will simplify the setup and will automatically scale-in and scale-out.

1. Introduction

A load balancer will make it possible to distribute the workload across multiple EC2 instances. A client application will connect to the load balancer without knowing which EC2 instance will handle the request. Because of this, EC2 instances can come and go without impacting your client requests. It is transparent because of the load balancer. It is easy to scale-out and thus to add more EC2 instances when traffic increases, or to scale-in and reduce the number of EC2 instances in order to reduce costs when traffic gets low.

In this post, you will learn how to setup an Application Load Balancer (note that there are other types as well, dependent on your need) which distributes traffic between several EC2 instances running a basic Spring Boot application. Next, you will learn how to use an Autoscaling Group, which will simplify things quite a bit.

In case that you do not have any knowledge how to create EC2 instances, read a previous blog. The sources used in this post are available at GitHub. The resources being used in this post will not cost you a thing, only Free Tier is used.

2. Sample Application

The basic application contains of a simple Hello endpoint which returns a message containing the host name of the machine which handled the request.

@RestController
public class HelloController {

    @GetMapping("/hello")
    public String hello() {
        String message = "Hello AWS!";
        try {
            InetAddress ip = InetAddress.getLocalHost();
            message += " From host: " + ip;
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
        return message;
    }

}

Also the health endpoint of Spring Actuator is added which you can use in order to enable the health check of the EC2 instance.

In case you want to run the application locally, just run the following command:

$ mvn spring-boot:run

Verify whether the endpoints are available:

$ curl http://localhost:8080/hello
Hello AWS! From host: <name of your machine>/127.0.1.1
$ curl http://localhost:8080/actuator/health
{"status":"UP"}

Create the jar file:

$ mvn clean verify

The jar file is uploaded to GitHub as a release in order to be able to download it to the EC2 instance.

3. Create the EC2 Instances

Create 2 EC2 instances in two different availability zones with the following User Data:

#!/bin/bash
yum -y install java-11-amazon-corretto-headless
wget https://github.com/mydeveloperplanet/MyAWSPlanet/releases/download/v0.0.1-alpha/MyAWSPlanet-0.0.1-SNAPSHOT.jar
java -jar MyAWSPlanet-0.0.1-SNAPSHOT.jar

During startup of the EC2 instance, Java 11 is downloaded and installed, the jar file is downloaded and the jar file is started.

Create a new Security Group and leave the default SSH access inbound rule for now. Name the Security Group ec2-sg. Now launch the EC2 instances.

4. Create the ALB

In the left menu, navigate to Load Balancers in the Load Balancing section and click the Create Load Balancer button. Here you can choose the type of load balancer you want to use. Choose Application Load Balancer by clicking the Create button.

In Step 1, you give the load balancer the name MyFirstLoadBalancer.

Set the listener to port 8080.

You also enable the availability zones for the load balancer. Check in which availability zones your EC2 instances are running and enable the same availability zones. Click the Next: Configure Security Settings button.

In Step 2, just click the Next: Configure Security Groups button.

In Step 3, create a new Security Group alb-sg for your ALB allowing HTTP traffic to port 8080. Click the Next: Configure Routing button.

In Step 4, you need to create a Target Group. A target group can be a number of EC2 instances the ALB will send traffic to. Name the target group MyFirstTargetGroup, set the port to 8080 and set the health check path to /actuator/health. Click the Next: Register Targets button.

In Step 5, you need to add the EC2 instances you want to include in the target group. Select both EC2 instances and click the Add to registered button. Click the Next: Review button.

At the end, you are able to review all the settings and click the Create button. You need to wait some time before the ALB is active.

Try to invoke the URL with the DNS Name of the load balancer.

$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

The time-out is expected. The EC2 instances only allow SSH traffic and no HTTP traffic. In the left menu, navigate to Security Groups in the Network & Security section. Select the ec2-sg security group and click the Edit inbound rules button of the Inbound rules tab.

Add a rule which allows HTTP traffic over port 8080. As a source, you choose the security group alb-sg of your ALB. This means that your EC2 instances cannot be reached directly over HTTP but only via the load balancer. Click the Save rules button.

Try to invoke the URL again and now the Hello message is returned. It will be more or less equally divided between the two machines when you invoke it several times.

$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-2-55.eu-central-1.compute.internal/172.31.2.55
$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-2-55.eu-central-1.compute.internal/172.31.2.55
$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-2-55.eu-central-1.compute.internal/172.31.2.55
$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-2-55.eu-central-1.compute.internal/172.31.2.55
$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-21-171.eu-central-1.compute.internal/172.31.21.171
$ curl http://MyFirstLoadBalancer-1212513449.eu-central-1.elb.amazonaws.com:8080/hello
Hello AWS! From host: ip-172-31-21-171.eu-central-1.compute.internal/172.31.21.171
...

Navigate to the Target Groups in the Load Balancing section and take a look at the health of the instances. Here you can see that both EC2 instances are healthy based on the configured health check.

In order to cleanup everything, you need to delete the load balancer, the target group, terminate the EC2 instances, delete the EC2 security group and finally delete the ALB security group. If you want to continue to follow this blog, only terminate the EC2 instances, you will need the other items in the next section.

5. Create the ASG

Creating an ALB is easy and has its advantages, but you still need to manually add and remove instances. This problem is solved with an Auto Scaling Group (ASG). Based on a template, scale-out and scale-in will be done automatically based on triggers you define.

In the left menu, navigate to Auto Scaling Groups in the Auto Scaling section and click the Create Auto Scaling Group button.

Give the ASG the name MyFirstASG. Next thing to do, is to create a Launch Template. The Launch Template will provide information about how to create EC2 instances. Click the Create a launch template link.

A new browser tab is opened where you first give the template the name MyFirstLaunchTemplate.

Fill in the AMI, the instance type and the key pair to be used just like you did while creating the EC2 instance.

In the Network settings, choose the ec2-sg security group.

Leave the Network settings, Storage, Resource tags and Network interfaces as default. Unfold the Advanced details section and scroll all the way down. Copy the User Data which you used for creating the EC2 instances before into the User data field. Finally, click the Create launch template button.

Return to the create ASG wizard, click the refresh button next to the Launch template field and select the just created launch template. Click the Next button.

In Step 2, select the three subnets and click the Next button.

In Step 3, choose to attach an existing load balancer target groups and select MyFirstTargetGroup.

In the Health Checks section, you enable ELB. This way, the ELB will automatically restart when it is unhealthy. Click the Next button.

In Step 4, you define the Group size with a Desired capacity of 2, a Minimum capacity of 2 and a Maximum capacity of 3.

With the Scaling policies, you can define how the autoscaling should take place when you choose Target tracking scaling policy. For now, just choose None, but it is good to take a look at the different options to automatically scale. Click the Next button.

In Step 5 and Step 6, just click the Next button. It is possible to add notifications and tags, but you will not use it in this tutorial. In Step 7, you can review everything and at the bottom of the page you click the Create auto scaling group button. It may take a few minutes before everything is up-and-running. You will see the following:

In the Auto Scaling section:
- The Auto Scaling Group
In the Instances section:
- Two EC2 instances which have been created because the minimum and desired capacity was set to 2
- The created Launch Template

Verify again by means of curl whether the Hello URL can be accessed.

Navigate to the Auto Scaling Group and the Details tab and click the Edit button.

Increase the desired and minimum capacity and click the Update button.

Automatically, a new instance is provisioned. Wait a few minutes until the instance is marked as being healthy in the Target Group and after this, the traffic is distributed over three EC2 instances.

In order to cleanup everything, you need to delete the Auto Scaling Group (this can take a while), the load balancer, the target group, the EC2 security group and finally delete the ALB security group.

6. Conclusion

In this post, you learned how to create an ALB as a single point of access for your EC2 instances. Problem with this setup was that you needed to manually add or remove instances when traffic got high or low. The ASG will solve this problem. It will launch or terminate EC2 instances based on several scaling policies (which you did not configure in this post, but just took a look at).

How to Create an AWS EC2 VM

mydeveloperplanet — Wed, 16 Jun 2021 16:48:44 +0000

In this post, you will learn how easy it is to create a virtual machine with AWS EC2. You will learn the basics and as an example, you will deploy and run a basic Spring Boot application. Enjoy!

1. Introduction

Amazon EC2 (Elastic Compute Cloud) is one of the most popular AWS services. It allows you to create cloud computing instances in no time. You have the choice out of predefined templates (AMI) for several Operating Systems. Pricing is dependent on what kind of purpose the instance needs to serve. Whether you want to have an instance continuously running, or only at a scheduled interval, or you need bare metal, or a dedicated instance, etc. Many options are available for different purposes. Besides that, a lot of different configurations are available to choose from. You will need to choose the right one for your application. In the Free Tier, you only use the micro instance. All of this information is available at the Amazon website and therefore only links to these pages are provided. The offerings may change and you should refer to the Amazon website for the details.

As a prerequisite for following this blog yourself, you will need to create an AWS account. You can create a Free Tier AWS account, which will allow you to experiment with most of the AWS services for free. See a previous blog how to setup the free account.

The sources used in this blog are available at GitHub.

2. Your First EC2 Instance

Login to the AWS Management Console and search for the EC2 service. This will bring you to the EC2 Dashboard. In the screenshots, the New EC2 Experience is enabled, so it might look a bit different on your side when you did not enable this new layout.

In the left menu, click on Instances and in order to start creating your first EC2 instance, click the Launch Instances button at the top right corner.

A wizard is started which will guide you through the creation process. In step 1, you need to select an AMI. You choose the Amazon Linux 2 AMI which is available in the Free Tier by clicking the Select button.

In step 2, you need to choose the Instance Type dependent on your needs concerning CPU, memory, storage and networking capacity. Being in the Free Tier, the choice is easy because there is only one Instance Type available. Choose the t2.micro and click the Next: Configure Instance Details button.

In step 3, you have the possibility to configure more items for your instance, e.g. the number of instances you want to create. Leave the defaults and click the Next: Add Storage button.

In step 4, you can configure the storage for your EC2 instance. Leave the defaults and click the Next: Add Tags button.

In step 5, you can add tags. You can add for example a Name tag by clicking the click to add a Name tag link.

Name it MyFirstEC2Instance and click the Next: Configure Security Group button.

In step 6, you need to select or configure a Security Group. A Security Group is like a firewall in front of your EC2 instance. You have several options here, it is advised to only allow the traffic which is absolutely necessary. In this case, you will create a new Security Group MyFirstSG for SSH access from your machine. Go to WhatIsMyIP.com and fill the IPv4 address followed with /32 as a source instead of 0.0.0.0/0. Click the Review and Launch button.

Finally, in step 7, you can review everything and when this is ok, you click the Launch button.

Before being launched, a popup window is show where we need to select an existing key pair or create a new one. This is necessary for accessing the instance by means of a SSH key. Create a new key pair EC2Blog and download the key pair. A EC2Blog.pem file is downloaded to your machine. Finally, click the Launch Instances button.

A Launch Status page is displayed, click the View Instances button.

If everything went OK, you first EC2 instance is running now.

3. SSH to Instance

Now that the EC2 instance is running, you can actually do something with it. Let’s see if you can SSH to the EC2 instance from your machine (note: the instructions are written for Linux). Open a terminal window from the directory where you saved the key pair .pem file. In the EC2 AWS console, you need the Public IPv4 address of your instance which is located at the right top corner of the Details page.

In the terminal, you enter the following command with your public IP of course.

$ ssh -i EC2Blog.pem ec2-user@3.66.155.101
The authenticity of host '3.66.155.101 (3.66.155.101)' can't be established.
ECDSA key fingerprint is SHA256:/5EorRulTwyFKUJLfTvNPmUlHS9Mt1eTffPD4+9tcwU.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '3.66.155.101' (ECDSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for 'EC2Blog.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "EC2Blog.pem": bad permissions
ec2-user@3.66.155.101: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

OK, it seems that something is wrong with the permissions of the .pem file. Let’s check the file permissions.

$ ll EC2Blog.pem 
-rw-rw-r-- EC2Blog.pem

The permissions are indeed a bit too open, so let’s first fix this issue.

$ chmod 400 EC2Blog.pem
$ ll EC2Blog.pem 
-r-------- EC2Blog.pem

Retry the SSH command and now it is successful and you can enter commands at your EC2 instance.

$ ssh -i EC2Blog.pem ec2-user@3.66.155.101

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-172-31-44-246 ~]$

Another way to connect via SSH is via the EC2 console. Select the instance and click the Connect button.

Leave the defaults and click the Connect button.

You will, however, not be able to connect because you restricted SSH access to the IP address of your machine only. Therefore, you need to allow the IP address where Instance Connect will connect from. Go to the page where the Amazon IP ranges are defined and search for EC2_INSTANCE_CONNECT for the region you are in. The IP below is for the eu-central-1 (Frankfurt) region.

{
  "ip_prefix": "3.120.181.40/29",
  "region": "eu-central-1",
  "service": "EC2_INSTANCE_CONNECT",
  "network_border_group": "eu-central-1"
}

Navigate to the Security Group section (left menu), select the Security Group, the Inbound rules tab and click the Edit inbound rules button.

Add a SSH inbound rule for the IP address and click the Save rules button.

Retry connecting via Connect Instance and you will have access to your EC2 instance.

4. Install Java

The goal is to install a Java Spring Boot application to the EC2 instance. First thing to do, is to install Java. The installation instructions can be found at the Amazon website. Execute the following command in the SSH terminal.

$ sudo yum install java-11-amazon-corretto-headless

Verify whether Java is available.

$ java --version
openjdk 11.0.11 2021-04-20 LTS
OpenJDK Runtime Environment Corretto-11.0.11.9.1 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.11.9.1 (build 11.0.11+9-LTS, mixed mode)

Problem now is that when you want to launch similar instances (right-click the instance Image and template – Launch more like this), you will need to install Java for each new instance. This can be solved by adding user data during the instance creation.

Exit the terminal and terminate the EC2 (select the instance, right-click and choose Terminate instance).

Create the instance again, just like you did before. In step 3, however, you scroll down to the bottom of the page and add User data with installation of java in the Advanced Details section. The User data will be executed as root, therefore there is no need to add sudo before the command. Also, add the argument -y to the yum command, otherwise the installation will fail because the user will be prompted to continue the installation.

In step 6, you can select the existing Security Group MyFirstSG. There is also no need to create a new key pair, just choose the existing one EC2Blog.

When the instance has been started, connect via SSH to the instance. Beware that the public IP has changed!

Verify whether Java is installed. When something went wrong, checkout the file cloud-init-output.log. This will give you more information about any errors occurred during startup.

$ sudo cat /var/log/cloud-init-output.log

5. Create and Install Spring Boot App

As Spring Boot Application, you can use one with a simple Hello World endpoint.

@RestController
public class HelloController {

    @GetMapping("/hello")
    public String hello() {
        String message = "Hello AWS!";
        try {
            InetAddress ip = InetAddress.getLocalHost();
            message += " From host: " + ip;
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
        return message;
    }

}

Run the following command in order to generate the jar file.

$ mvn clean verify

Run the application locally.

$ java -jar MyAWSPlanet-0.0.1-SNAPSHOT.jar

Verify whether you can access the URL.

$ curl http://localhost:8080/hello
Hello AWS! From host: <computer name>/127.0.1.1

Copy the jar file to the remote EC2 instance.

$ scp -i EC2Blog.pem MyAWSPlanet-0.0.1-SNAPSHOT.jar ec2-user@18.194.164.14:

Via the SSH connection, start the Spring Boot application.

The URL will not be accessible yet, therefore you need to add an extra inbound rule to the Security Group. Since the application is running at port 8080, you create a custom TCP rule for port 8080 which can be accessed from anywhere.

Verify whether you can access the URL. You need to use the public IP address of your EC2 instance.

$ curl http://18.194.164.14:8080/hello
Hello AWS! From host: ip-172-31-36-30.eu-central-1.compute.internal/172.31.36.30

Success! Do not forget to terminate the EC2 instance after you are done experimenting with it and to delete the Security Group if you do not need it anymore.

6. Conclusion

In this post, you learned how to create and configure an EC2 instance at Amazon. This is amazingly easy to do. You also learned how to access the remote instance from your laptop and how to install Java on it. Finally, you successfully started a Spring Boot application and accessed it from your machine. All of this has been executed in the Free Tier, so it did not cost you anything!

Automated Visual Testing With Robot Framework

mydeveloperplanet — Tue, 01 Jun 2021 17:48:16 +0000

A common problem when automating tests are tests which require a visual comparison to a previous state. This can be a very time consuming task when you need to execute many of these testing tasks. But not anymore, the Robot Framework DocTest library comes to the rescue!

1. Introduction

Even if you do not know what Robot Framework is, it is really worth the effort to continue reading. The DocTest library might save you a lot of testing time, but it requires some knowledge of Robot Framework. In case you do not know what Robot Framework is, read some of the previous posts in order to get a good introduction.

When trying to test your application in an automated way, some tests just cannot be automated. Mainly because these tests require a visual check. Let’s assume that your application makes use of maps which require a regular update, or reports need to be generated and must be verified, etc. These visual tests are probably also very time consuming and there is no guarantee that it is verified correctly. The visual check is done by humans and humans make mistakes. Wouldn’t it be great when you could automate the visual check or at least get notified when something changed compared to a previous version and the output of the test would show you the differences? All of this is now possible with the Robot Framework DocTest Library. It was presented at Robocon 2021. The talk is available online and it will take you only half an hour to watch it.

Enough for the introduction, let’s see how it works. The sources used in this post are available at GitHub.

2. Installation

The installation instructions below are executed with Ubuntu 20.04.2 LTS version. Installation is quite easy using pip:

$ pip install --upgrade robotframework-doctestlibrary

DocTest makes use of other applications, it is necessary to install these otherwise things can go wrong.

$ apt-get install imagemagick
$ apt-get install tesseract-ocr
$ apt-get install ghostscript
$ apt-get install libdmtx0b

If you do not install the above applications, using the DocTest keyword Compare Images will raise the following error:

[ ERROR ] Error in file '/<path>/MyDocTestPlanet/visual_test.robot' on line 2: Importing library 'DocTest.VisualTest' failed: ImportError: Unable to find dmtx shared library

If you want to compare PDF files, ImageMagick might raise the following error from the library:

File could not be converted by ImageMagick to OpenCV Image: testdata/sample_1_page.pdf

Trying to convert a PDF to a JPG file using ImageMagick by means of CLI will give you a more clear error:

$ convert sample.pdf sample.jpg 
convert-im6.q16: attempt to perform an operation not allowed by the security policy

ImageMagick makes use of a policy.xml file which contains by default the following lines which is the cause of the issue:

<!-- disable ghostscript format types -->
<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />

In the root of the DocTest repository, a policy.xml file is present. Replace the file in /etc/ImageMagick-6/policy.xml file with the one from the repository. The conversion should work now.

3. In Practice

In this section, you will learn how to use the library. Note that the GitHub repository of the library also provides some good examples, which will show you how to use the library. Clone the repository and run the test from the root of the repository:

$ robot atest/Compare.robot

Nevertheless, it is also good to verify whether this will work with items you created yourself. As a base image, some kind of map is created containing a timestamp. All of the images are located in the images directory. The original drawings *.odg files are present and the saved *.png files which will be used in the examples. The base image is base-image.png.

3.1 Comparison With Small Difference

The first comparison you will make, is one with a small difference in one of the triangles (base-image-with-triangle-difference.png). When you would like to compare this visually, you would have a hard time finding the difference because it is such a small one. So, how can we do so by means of the DocTest library? The corresponding test is:

| *** Settings *** |
| Library | DocTest.VisualTest

| *** Test Cases *** |
| 01 | [Documentation] | Compare Two Images With Triangle Difference
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png

You import the library and the only thing you need to do, is to run the Compare Images keyword with the reference and canditate image. Run the test:

$ robot visual_test.robot

The test obviously fails because there is a difference. You will notice that a directory screenshots has been created where the diff images are located. But, it is easier to open the log.html file with Robot Framework results.

This result is very useful, you immediately notice where the differences are. As expected, the triangle is indicated as a difference, but also the timestamp. Next to this screenshot, a combined difference image is provided. The results are available in the GitHub repository.

3.2 Comparison With Small Difference Ignoring Timestamp

There is however a problem with the previous comparison. When there are no differences in the map and the screenshots are taken at different points in time, the comparison will still mark the two maps as different. This is probably not what you want. The library provides a solution for this by means of masks. You can define masks in a JSON file where you define which items should be ignored during the comparison. In this case, you create a masks/mask-date.json file as follows:

[
    {
    "page": "all",
    "name": "Date Pattern",
    "type": "pattern",
    "pattern": ".*[0-9]{2}-[0-9]{2}-[0-9]{4}"
    }
]

The pattern being used is a simple regular expression. The test case only needs one extra placeholder_file argument.

| 02 | [Documentation] | Compare Two Images With Triangle difference and Time Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png | placeholder_file=masks/masks-time.json

When you run the test now, it still fails because the triangle has a difference, but in the comparison screenshot, the date is marked with blue, meaning that it has been ignored during the comparison.

Now, let’s verify whether the comparison is successful when the images are identical besides the time.

| 03 | [Documentation] | Compare Two Images With Time Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-time-difference.png | placeholder_file=masks/masks-time.json

And yes, the test is now successful although the time in the images is different.

This is great, you now have the possibility to only compare what you want in the image and ignore those items which will be different anyhow.

3.3 Comparison With Small Difference Ignoring Area

Besides a pattern, it is also possible to define an area to be ignored. Create a masks/mask-bottom.json file containing the following content:

[
    {
        "page": "all",
        "name": "Header Area",
        "type": "area",
        "location": "bottom",
        "percent":  10
    }
]

This time, you specify the bottom area and a percentage. The test case is now:

| 04 | [Documentation] | Compare Two Images With Bottom Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png | placeholder_file=masks/masks-bottom.json

Running this test shows you that the bottom part of the image is ignored.

3.4 Comparison With Small Move

When an item has moved slightly, the difference will be hard to spot. Besides the expected and actual result image, the Robot Framework log also shows us a combined result. The test is:

| 05 | [Documentation] | Compare Two Images With Small Move
|    | Compare Images  | images/base-image.png | images/base-image-with-small-move.png

In the combined result you can see the difference much better.

What if you would like to accept these kind of small moves? In other words, you do not want to fail the test when an item has moved a little bit. In that case, you can add the move_tolerance argument and specify the amount of pixels you tolerate. The previous test becomes the following:

| 06 | [Documentation] | Compare Two Images With Small Move And Move Tolerance
|    | Compare Images  | images/base-image.png | images/base-image-with-small-move.png | move_tolerance=10

The test now passes successfully and within the result, it is indicated that a small move has been found.

3.5 PDF Compare With Small Difference

Just like images, it is also possible to compare PDF files. This can be extremely useful when you have reports to compare. A small report is available in pdf/base-report.pdf (the original base-report.odt is also available). A report pdf/base-report-with-small-difference.pdf contains a different amount for the screw driver and a different date. The test is similar as for the images, this time you provide the get_pdf_content argument and set it to true.

| 07 | [Documentation] | Compare Two PDF Files With Small Difference
|    | Compare Images  | pdf/base-report.pdf | pdf/base-report-with-small-difference.pdf | get_pdf_content=${true}

Again, the log file shows you exactly the difference:

Masks also work with this kind of comparison. Let’s apply the date mask.

| 08 | [Documentation] | Compare Two PDF Files With Small Difference and Date Mask
|    | Compare Images  | pdf/base-report.pdf | pdf/base-report-with-small-difference.pdf | placeholder_file=masks/masks-date.json | get_pdf_content=${true}

As expected, only the amount difference is shown and the date is marked with blue indicating that it has been ignored.

4. Conclusion

The Robot Framework DocTest library is a really powerful library to compare differences in images and reports. It is simple in its use and it works just fine. This will definitely become a big time saver when you apply this during your testing efforts and it will reduce the chance for testing errors.

Automate ZAP With Docker

mydeveloperplanet — Tue, 18 May 2021 18:14:12 +0000

In the previous posts, you learned how to use ZAP with the Desktop client and via the command line with ZAP CLI. This post, you will learn how to use the Docker images which are provided by OWASP. This will even make it easier to automate ZAP, especially in a CI/CD pipeline.

1. Introduction

It is strongly advised to read the two previous posts about ZAP before starting with this one. You will need some files which were created in the previous posts. If you already have experience with ZAP, you can continue reading and use the files from the GitHub repository from directory zap2docker. The generated reports will also be available in this repository. This way, you will be able to compare your results.

In the previous posts, you were shown how to use the ZAP Desktop client and how to use ZAP CLI in order to automate the penetration test. However, OWASP also provides some Docker images which can be used for an automated scan.

You will again use WebGoat as vulnerable web application. If you followed the previous posts, it is better to start from scratch again and remove the Docker container you created.

$ docker rm goatandwolf

Since the application under test is running in a Docker container and ZAP will also run in a Docker container, it is necessary to create a Docker network. Otherwise it will not be possible to access WebGoat from within the ZAP Docker container.

$ docker network create zapnet

Next, create the WebGoat container within the just created network zapnet.

$ docker run --name goatandwolf -p 8080:8080 -p 9090:9090 -d --net zapnet webgoat/goatandwolf

Navigate to the WebGoat URL and create the user mydeveloperplanet with password password. This user will be used for authentication during the scan.

2. ZAP Docker Full Scan

The ZAP Docker image provides several scan possibilities. One of them is a Baseline Scan which will scan your application passively. The active scan, however, will give you better results and this can be accomplished with the Full Scan.

You will need the IP address of WebGoat within the zapnet network. This can be achieved with the following command. In the example below, 172.22.0.2 is the IP address where WebGoat can be accessed.

$ docker inspect goatandwolf | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.22.0.2",

First, you will scan the application without any user information. The complete list op options can be found here, below the used options are explained:

--net: in order to add ZAP to the network together with WebGoat
-v: this will map your current directory to the Docker image work directory
-I: do not return failure on warning
-j: run the AJAX spider in addition to the classic one
-m 10: the number of minutes to spider for (just a safeguard, the spider takes less time than 10 minutes)
-T 60: limit the total scan to 60 minutes
-t: the URL to scan for
-r: the name of the report for the results

$ docker run --net zapnet -v $(pwd):/zap/wrk/:rw \
  -t owasp/zap2docker-stable zap-full-scan.py -I -j -m 10 -T 60 \
  -t http://172.22.0.2:8080/WebGoat \
  -r 20210417-zap-full-scan-without-user.html

Just as you noticed when running the scan with ZAP CLI in the previous post, this scan will give you less results than expected. The spider does some work, but not enough and since you did not provide any user credentials, a large part of the application is not scanned.

In order to provide the user credentials, you can provide the context Webgoat.context you created last time. The only thing you need to do, is to replace localhost with the IP address in the entire file. Move the context file to the current directory in order that it will be accessible in the ZAP work directory inside the Docker container. You add the following two extra options to the command:

-n: The context file
-U: The user to use

$ docker run --net zapnet -v $(pwd):/zap/wrk/:rw \
  -t owasp/zap2docker-stable zap-full-scan.py -I -j -m 10 -T 60 \
  -t http://172.22.0.2:8080/WebGoat \
  -r 20210417-zap-full-scan.html \
  -n Webgoat.context \
  -U mydeveloperplanet

Running this command, results in the following error. It states that the URL is not in the context, but it is. Even if this would work, it is doubtful whether the spider would have found all of the URLs of the application. You have noticed in the previous posts that a manual exploration of the website together with a spider gave much more URLs to scan.

20432 [ZAP-ProxyThread-11] WARN  org.zaproxy.zap.extension.api.API - Bad request to API endpoint [/JSON/ajaxSpider/action/scanAsUser/] from [127.0.0.1]:
org.zaproxy.zap.extension.api.ApiException: url_not_in_context
    at org.zaproxy.zap.extension.spiderAjax.AjaxSpiderAPI.startScan(AjaxSpiderAPI.java:337) ~[?:?]
    at org.zaproxy.zap.extension.spiderAjax.AjaxSpiderAPI.handleApiAction(AjaxSpiderAPI.java:202) ~[?:?]
    at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:507) [zap-2.10.0.jar:2.10.0]
    at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:497) [zap-2.10.0.jar:2.10.0]
    at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:333) [zap-2.10.0.jar:2.10.0]
    at java.lang.Thread.run(Thread.java:834) [?:?]

3. ICTU ZAP Docker Full Scan

ICTU, a Dutch IT organisation of the government has extended the ZAP Docker images with a webhook for authentication. It would be interesting to find out whether this way you can scan the application including authentication. Notice that the Docker image is now taken from the ICTU DockerHub page. Two extra options are added compared to the full scan without user authentication:

--hook: the link to the Python script which will take care of the authentication
-z: some extra parameters needed for the authentication

$ docker run --rm -v $(pwd):/zap/wrk/:rw --net zapnet -t ictu/zap2docker-weekly zap-full-scan.py -I -j -m 10 -T 60 \
  -t http://172.22.0.2:8080/WebGoat \
  --hook=/zap/auth_hook.py \
  -r 20210417-ictuzap-full-scan.html \
  -z "auth.loginurl=http://172.22.0.2:8080/WebGoat/login \
      auth.username="mydeveloperplanet" \
      auth.password="password" \
      auth.auto=1"

This seems to do its work. However, less results are found compared to the ZAP CLI scan. Most likely due to the spider again.

4. ZAP CLI With Docker

The good news is that ZAP CLI is also shipped in the ZAP Docker image. Good results were achieved with ZAP CLI, so let’s see whether this also applies when you run it from within the ZAP Docker container. You run the Docker container again with a volume mapping to your current directory and with option -i in order to start the container in interactive mode. This will allow you to execute commands inside the Docker container.

$ docker run --name zap-cli --net zapnet -v $(pwd):/zap/wrk/:rw -i owasp/zap2docker-stable

As a test, you can verify whether WebGoat is accessible from within the ZAP Docker container with a wget.

$ wget http://172.22.0.2:8080/WebGoat
--2021-04-18 12:02:03--  http://172.22.0.2:8080/WebGoat
Connecting to 172.22.0.2:8080... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://172.22.0.2:8080/WebGoat/ [following]
--2021-04-18 12:02:03--  http://172.22.0.2:8080/WebGoat/
Reusing existing connection to 172.22.0.2:8080.
HTTP request sent, awaiting response... 302 Found
Location: http://172.22.0.2:8080/WebGoat/login [following]
--2021-04-18 12:02:03--  http://172.22.0.2:8080/WebGoat/login
Reusing existing connection to 172.22.0.2:8080.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘WebGoat’

     0K .                                                      94.3M=0s

2021-04-18 12:02:03 (94.3 MB/s) - ‘WebGoat’ saved [1877]

You will follow the exact same steps as in the previous post. The only difference is that you will execute the commands from within the Docker container. First thing to do is to start ZAP. For simplicity, you will disable the API key. Remember that the API key was necessary to access the ZAP API. You can retrieve the API key if you want via the webswing ZAP UI.

$ zap-cli --log-path wrk/ start --start-options '-config api.disablekey=true'
[INFO]            Starting ZAP daemon

Import the context. Remember that you changed localhost in the context file to the IP address where WebGoat can be accessed.

$ zap-cli -v context import /zap/wrk/Webgoat.context
[INFO]            Imported context from /zap/wrk/Webgoat.context

In the previous post, you exported the manually explored URLs in a file webgoat-exported-urls.txt. Copy this file to your current directory and find/replace localhost with the WebGoat IP address.

Also, copy the open-urls.sh script to your current directory and change the path to the text file.

#!/bin/bash
input="wrk/webgoat-exported-urls.txt"
while IFS= read -r line
do
  zap-cli open-url "$line"
done < "$input"

Execute the script, this will take approximately 10 minutes.

$ ./wrk/open-urls.sh
[INFO]            Accessing URL http://172.22.0.2:8080
[INFO]            Accessing URL http://172.22.0.2:8080/WebGoat
[INFO]            Accessing URL http://172.22.0.2:8080/WebGoat/
[INFO]            Accessing URL http://172.22.0.2:8080/WebGoat/AuthBypass.lesson.lesson
...
[INFO]            Accessing URL http://172.22.0.2:8080/sitemap.xml

Start the classic spider.

$ zap-cli -v spider -c Webgoat -u mydeveloperplanet "http://172.22.0.2:8080/WebGoat"
[INFO]            Running spider...
[DEBUG]           Spidering target http://172.22.0.2:8080/WebGoat...
[DEBUG]           Running spider in context 1 as user 2230
[DEBUG]           Started spider with ID 0...
[DEBUG]           Spider progress %: 0
[DEBUG]           Spider #0 completed

Start the active scan, this will take approximately 15 minutes.

$ zap-cli -v active-scan --recursive -c Webgoat -u mydeveloperplanet "http://172.22.0.2:8080/WebGoat"
[INFO]            Running an active scan...
[DEBUG]           Scanning target http://172.22.0.2:8080/WebGoat...
[DEBUG]           Scanning in context 1 as user 2230
[DEBUG]           Started scan with ID 0...
...
[DEBUG]           Scan progress %: 98
[DEBUG]           Scan #0 completed

Generate the report.

$ zap-cli -v report -o /zap/wrk/report-zap-full-scan.html -f html
[DEBUG]           Generating HTML report
[INFO]            Report saved to "/zap/wrk/report-zap-full-scan.html"

As you can see, this gives you similar results as in the previous post.

Save the session for next use.

$ zap-cli -v session save /zap/wrk/webgoat-20210418-active-scan.session
[DEBUG]           Saving the session to "/zap/wrk/webgoat-20210418-active-scan.session"

Shutdown ZAP.

$ zap-cli -v shutdown
[INFO]            Shutting down ZAP daemon
[DEBUG]           Shutting down ZAP.
[DEBUG]           ZAP shutdown successfully.

Finally, type exit to exit the interactive shell and shutdown the Webgoat Docker container.

$ docker stop goatandwolf

5. Conclusion

It is great that OWASP provides Docker images with ZAP pre-installed. This simplifies installation and makes it easier to integrate it into your CI/CD pipeline. The default scans which are provided did not give good enough results. Luckily, ZAP CLI is also provided and this did the job. Also note that ZAP CLI will be replaced in the near future with the Automation Framework.

Automated Pen Testing With ZAP CLI

mydeveloperplanet — Wed, 05 May 2021 18:45:00 +0000

In the previous post, you learnt how to execute an automated penetration test by means of Zed Attack Proxy (ZAP). This time, you will learn how to execute the test via a Command Line Interface (CLI) which will make it possible to add the test to your CI/CD pipeline.

1. Introduction

In the previous post, the different steps were explained how to execute an automated penetration test. The application under test being used was WebGoat, a vulnerable application developed by OWASP in order to learn security vulnerabilities. This application will be used in this post also. The steps to be executed for a penetration test with ZAP are:

Setup the context and session (especially login credentials when a part of the application is behind a login);
Manually explore all parts of the application;
Spider the application;
Execute the automated scan;
Inspect the results.

Beware that an automated scan will not find all vulnerabilities! It is always necessary to execute a manual penetration test. The automated scan will however give you a good indication about the state of security of your application.

The files being used in this post are available at GitHub in directory zap-cli.

2. ZAP CLI

ZAP contains an API for controlling ZAP. The ZAP CLI tool is a tool which wraps the API in order that commands can be executed via the command line. In this section, you basically will perform the same or similar actions as in the previous post, except that you will not use the ZAP Desktop this time. A complete list of the commands of ZAP CLI can be found at the GitHub website.

2.1 Installation and Preparation

As a prerequisite, you must have ZAP already installed of course. Next to ZAP, you need to install ZAP CLI, which is quite simple. Just execute the following command:

$ pip install --upgrade zapcli

Before you continue, it is very important to set the following environment variables:

export ZAP_PORT=8090
export ZAP_PATH=/usr/local/bin/zap.sh
export ZAP_API_KEY=<your API key>

The ZAP_API_KEY can be found in ZAP Desktop. Therefore, start ZAP Desktop and choose Tools – Options… in the menu. In the API section, the API key is shown and needs to be used for the environment variable (but do not yet set the environment variable until it is mentioned to do so in the next section). You also have the possibility to disable the usage of the key, but this is not advised.

First, close the ZAP Desktop tool before you continue. Let’s see what happens when you try to start ZAP:

$ zap-cli start
[INFO]            Starting ZAP daemon
Traceback (most recent call last):
  File "/home/<user>/.local/bin/zap-cli", line 8, in <module>
    sys.exit(cli())
  File "/home/<user>/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/<user>/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/<user>/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/<user>/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/<user>/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/<user>/.local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/<user>/.local/lib/python3.8/site-packages/zapcli/cli.py", line 58, in start_zap_daemon
    zap_helper.start(options=start_options)
  File "/home/<user>/.local/lib/python3.8/site-packages/zapcli/zap_helper.py", line 88, in start
    with open(log_path, 'w+') as log_file:
PermissionError: [Errno 13] Permission denied: '/usr/local/bin/zap.log'

It seems that you do not have sufficient rights to write to the zap.log. There are several solutions for this, but the easiest one is to set the log path to a directory where you have sufficient permissions. Navigate to this directory and execute the following command:

$ zap-cli --log-path . start
[INFO]            Starting ZAP daemon

This seems to have started the ZAP daemon. Let’s verify this:

$ zap-cli status
[INFO]            ZAP is running

Allright, ZAP is running. Let’s try to shutdown the ZAP daemon:

$ zap-cli shutdown
[INFO]            Shutting down ZAP daemon
...
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/core/action/shutdown/?apikey= (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))

Why is this not working? The reason for the exception is quite vague because it mentions ‘Cannot connect to proxy’ and ‘Remote end closed connection without response’. But why? The answer is shown in the preceding URL http://zap/JSON/core/action/shutdown/?apikey=. As you can see, the parameter apikey is empty. There are numerous questions on the internet about this exception, but very few will give you the answer. The solution is to set the ZAP_API_KEY environment variable.

$ export ZAP_API_KEY=<your API key>
$ zap-cli . shutdown
[INFO]            Shutting down ZAP daemon

And now it is possible to shut down the ZAP daemon. It appears that you do not need the API key for every API call, but when you set the environment variables, you will not need to worry about this.

2.2 Context Preparation

Remember from the previous post, that it was important to set user credentials in the Context and the manual exploration of the website. In this post, you will reuse this information in order to feed it to ZAP CLI.

Open the previously stored session via File – Open Session… and export the context via File – Export Context… Select the context, the path where you want to save it and click the Save button.

The manually explored URLs of the application are also present in the Session. You will need those URLs for ZAP CLI. Right-click in the Sites section on http://localhost:8080/ and choose Export Selected URLs to File… and name the file webgoat-exported-urls.txt. You will need this file in the next paragraph.

Close ZAP Desktop.

2.3 Scan With ZAP CLI

At this moment, you have done all the necessary preparation in order to get started to scan your application with ZAP CLI. Identical steps as during the ZAP Desktop scan will be performed, but this time via the ZAP CLI and you will make use of the Context and the exported URLs which have been created in the previous post.

Before you continue, ensure that the WebGoat application is running. Otherwise, it is a good moment to start the application now.

$ docker start goatandwolf

Start the ZAP daemon.

$ zap-cli --log-path . start
[INFO]            Starting ZAP daemon

Import the context. Ensure to include the complete path to the context file.

$ zap-cli -v context import /<path>/Webgoat.context 
[INFO]            Imported context from /<path>/Webgoat.context

When you list the contexts, you will notice that the list seems to be empty, but it isn’t. It is probably a bug that the name is not correctly displayed.

$ zap-cli context list
[INFO]            Available contexts: []

Next step is to explore the website, you will use the exported URLs file you created in the previous paragraph. What you will do, is to open each URL which is present in this file with the zap-cli open-url command. Create a bash script open-urls.sh for this with the following contents.

#!/bin/bash
input="webgoat-exported-urls.txt"
while IFS= read -r line
do
  zap-cli open-url "$line"
done < "$input"

Give the bash script executable permissions.

$ chmod +x open-urls.sh

Execute the bash script, this can take some time dependent on how many URLs you have in the text file.

$ ./open-urls.sh
[INFO]            Accessing URL http://localhost:8080
[INFO]            Accessing URL http://localhost:8080/WebGoat
...

Run the spider. Notice that the context is provided and also the user mydeveloperplanet which will allow you to login to the application.

$ zap-cli -v spider -c Webgoat -u mydeveloperplanet "http://localhost:8080/WebGoat"
[INFO]            Running spider...
[DEBUG]           Spidering target http://localhost:8080/WebGoat...
[DEBUG]           Running spider in context 1 as user 2230
[DEBUG]           Started spider with ID 0...
[DEBUG]           Spider progress %: 0
[DEBUG]           Spider #0 completed

Do not run the AJAX spider. It is good to note that with the argument --help, you can view the command arguments available for this command.

$ zap-cli ajax-spider --help
Usage: zap-cli ajax-spider [OPTIONS] URL

  Run the AJAX Spider against a URL.

Options:
  --help  Show this message and exit.

As you can see, only the URL is available as a parameter. This is strange because in the Desktop version it is possible to provide a context and a user. It also seems that it negatively influences the end results (less alerts were reported). Therefore, do not run the AJAX spider.

Start the active scan. This will take some time (about 10 to 15 minutes).

$ zap-cli -v active-scan --recursive -c Webgoat -u mydeveloperplanet "http://localhost:8080/WebGoat"
[INFO]            Running an active scan...
[DEBUG]           Scanning target http://localhost:8080/WebGoat...
[DEBUG]           Scanning in context 1 as user 2230
[DEBUG]           Started scan with ID 0...
[DEBUG]           Scan progress %: 0
[DEBUG]           Scan progress %: 1
[DEBUG]           Scan progress %: 3
...
[DEBUG]           Scan progress %: 97
[DEBUG]           Scan progress %: 99
[DEBUG]           Scan #0 completed

Generate the report.

$ zap-cli -v report -o /<path>/report-zap-cli-first-scan.html -f html
[DEBUG]           Generating HTML report
[INFO]            Report saved to "/<path>/report-zap-cli-first-scan.html"

Save the session. This will allow you to open it in ZAP Desktop for example.

$ zap-cli -v session save /<path>/webgoat-20210410-active-scan.session
[DEBUG]           Saving the session to "/<path>/webgoat-20210410-active-scan.session"

And shutdown the ZAP daemon.

$ zap-cli -v shutdown
[INFO]            Shutting down ZAP daemon
[DEBUG]           Shutting down ZAP.
[DEBUG]           ZAP shutdown successfully.

3. Conclusion

ZAP CLI gives you the opportunity to execute an automated penetration test via the command line. This will make it also possible to include it into your CI/CD pipeline. Beware that it will give you only an indication about the security of your application and it does not replace a complete penetration test. Also note that the results are not identical to the results from the previous post. But when you execute the same steps as in this post starting from a clean session in ZAP Desktop, the results will be quite similar. Reason for the difference is probably the execution of the AJAX spider or the manually exploration of the website from within ZAP Desktop.

Automated Pen Testing With Zed Attack Proxy

mydeveloperplanet — Wed, 21 Apr 2021 17:24:42 +0000

In this post, you will learn how to execute penetration tests with OWASP Zed Attack Proxy (ZAP). ZAP is a free web app scanner which can be used for security testing purposes.

1. Introduction

When you are developing an application, security must be addressed. It cannot be ignored anymore nowadays. Security must be taken into account starting from initial development and not thinking about it when you want to deploy to production for the first time. Often you will notice that adding security to your application at a later stage in development, will take a lot of time. It is better to take security into account from the beginning, this will save you from some painful headaches. You probably have some security experts inside of your company, so let them participate from the start when a new application needs to be developed. Nevertheless, you will also need to verify whether your developed application is secure. Penetration tests can help you with that. OWASP Zed Attack Proxy (ZAP) is a tool which can help you execute penetration tests for your application. In this post, you will learn how to setup ZAP and execute tests with the desktop client of ZAP. You will also need a preferably vulnerable application. For this purposes, Webgoat of OWASP will be used. In case you do not know what Webgoat is, you can read a previous post first. It might be a little bit outdated because Webgoat has been improved since then, but it will give you a good impression of what Webgoat is. It is advised to disconnect from the internet when using Webgoat because it may expose your machine to attacks.

2. Start the Application

First thing to do, is to start Webgoat. The easiest way is to run it as a Docker container. The Docker image contains the applications Webgoat and Webwolf, but you will only use Webgoat in this post. You give the container the name goatandwolf (this will make it easy to start and stop the container) and you run it in detached mode.

$ docker run --name goatandwolf -p 8080:8080 -p 9090:9090 -d webgoat/goatandwolf

After the container has started, verify whether you can access Webgoat via the browser at URL http://localhost:8080/WebGoat/. The login page should be shown.

That’s it for now, you are ready now to start with ZAP.

3. Zed Attack Proxy

3.1 Installation

Installation instructions for ZAP are dependent of your OS. For Linux, you download the file ZAP_2_10_0_unix.sh and execute it.

$ ./ZAP_2_10_0_unix.sh

Start ZAP, leave the default persistence setting and click Start.

3.2 Quick Start Scan

The quickest way to start a scan is to use the Quick Start menu and start an automated scan. Click the Automated scan button in this menu.

Fill in the URL you want to attack, enable Use ajax spider and click the Attack button. Do not think too much about all the options at this moment, they will become more clear later on in this post.

Some interesting things can be noted after running the scan. Let’s take a look at the Sites section and unfold it so you can see which URL’s did participate to the scan.

The spiders try to explore your website and do find some useful things, but in this case, almost your entire website is located after a login page. The scan was executed without logging in, so the major part of your website is not scanned. The scan did find some alerts but not as many as expected.

The automated scan is a nice way for obtaining some quick results, but nothing more than that. It can definitely not be seen as a good scan of your application, certainly when the larger part of your application needs a login.

3.3 Explore Your Application

One way or the other, you will need to let ZAP know how your application looks like. So, you need to manually explore your website and click all links, buttons, fill in all available forms, even navigate to maybe hidden URL’s, etc. You need to do so for every role your application has, in the case of WebGoat, you will only explore the site for a regular user in this post. Let’s start doing so!

Go to the Quick Start menu again, this time choose Manual Explore. Fill in the URL if not already done and click the Launch Browser button.

A browser window is opened and the login page is shown, just as we saw before when we accessed WebGoat. The difference is, that ZAP is now in between the browser and the application which makes it possible for ZAP to intercept all traffic and follow what we are doing onto the website.

Let’s continue with creating a user. Click the Register new user link and create a user. In the example below, the user mydeveloperplanet with password password (the password must be between 6 and 10 characters) is created. Agree with the terms and conditions and click the Sign up button.

Now it is time for the most tedious work, you have to explore the website as good as possible. When you do so, you will notice that the Sites section will continue to grow with new URL’s.

Close the browser and save this session via File – Persist Session…

3.4 Create Context

Before you can continue, you must provide login information to ZAP which it can use during scanning. This information has to be stored in the context. Double-click Default Context in the Sites section.

The Session Properties window opens where settings concerning the session can be changed. You can for example change the session name to a more convenient one than Untitled Session. You name it for example WebGoat mydeveloperplanet.

More importantly is the Contexts section. Change the name Default Context to e.g. Webgoat.

Add the WebGoat website and underlying URL’s by means of a regular expression to the Include in Context section.

In the Authentication section, choose Form-based Authentication (this will be dependent of your application) and fill in the URL for the login page.

And finally, add the created user in the Users section. Fill in the user name and password as previously created. This will allow ZAP to login to the application.

Do not forget to click the OK button in order to save the changes.

3.5 ZAP Modes

ZAP can act in four different modes:

Safe: no potentially dangerous operations are permitted;
Protected: only perform potentially dangerous operations on URL’s which are in scope;
Standard: you can do anything;
ATTACK: new nodes that are in scope are actively scanned as soon as they are discovered.

You must be aware of the fact that you can only run ZAP against a website you own. It is therefore recommended that you use Protected mode to ensure that you only attack your own sites.

3.6 Start Spider

Although you have done your best clicking on every link, button, you can think of, a spider can help you to reveal the things you might have missed. You can run a traditional spider and/or an AJAX spider. The latter will also crawl dynamically built links. Remember that you also enabled these spiders during the Quickscan.

In the Sites section, right-click the WebGoat URL and choose Attack – Spider…. Select the user mydeveloperplanet and click the Start Scan button.

The Spider finishes quickly and in the results you will notice that a certain amount of nodes have been added (37 in the screenshot).

Do the same with the AJAX spider. Right-click the WebGoat URL and choose Attack – AJAX Spider…, fill the necessary items and click the Start Scan button.

Again some new URL’s have been added.

3.7 Active Scan

At this moment, you have done all the preparation work and now it is time for the real action. Right-click the WebGoat URL, select Attack – Active Scan…, fill the necessary items and click the Start Scan button.

ZAP will do its work now, so be patient, this can take some time (it took approximately 15 minutes for the scan to complete). At the end, the results can be viewed in the Alerts section.

Clicking an alert will show more information about the vulnerability and how to solve it.

3.8 Export Results

Several export functions are available in the Reports menu for generating reports. You can for example generate a HTML report with Generate HTML Report….

4. Next Steps

The active scan will give you a first indication about vulnerabilities. Beware however that the active scan can only find certain types of vulnerabilities. In addition to the active scan, manual penetration tests should always be performed. The OWASP Top 10 website gives you good information about whether a vulnerability can be found with an automated scan or not.

In addition to the active scan, you can also use the Forced Browse Site which will try to find undiscoverable links and the Fuzz which will send in random data to your site.

Do not forget to stop the WebGoat Docker container when you are finished testing.

$ docker stop goatandwolf

5. Conclusion

In this blog you learnt how to use Zed Attack Proxy. It is advised to experiment with it, try to solve the issues, check which other information is available in the tool in order to get more acquainted with it. It is for example also possible to intercept a request and change items in the request.

Improve Your Robot Framework Tests With Robocop

mydeveloperplanet — Tue, 06 Apr 2021 17:24:52 +0000

Testing software will tell you something about the quality of your software. But how do you ensure that your tests have a high quality? A static analysis tool can help you in order to improve the quality. Robocop is a such a static analysis tool for Robot Framework tests. In this blog, you will learn how to use and configure Robocop for your purposes.

1. Introduction

Static code analyzers are commonly used during software development. SonarQube is a popular tool used for this purposes. The intention is to scan the code and verify whether it complies to a certain set of rules. The rules to check against should be agreed upon within your company and serve as best practices. The purpose of the scan is to learn, to standardize, to improve readability, etc.

Robocop is such a static analysis tool for Robot Framework tests. In case you need a basic overview of the capabilities of Robot Framework, you can read some previous blogs as an introduction:

The Robocop documentation gives you a good overview of all capabilities. In the sections below, you will learn how to use the tool from a practical point of view. This way, it should be quite easy for you to apply it to your own Robot Framework test cases. As an example, you can use the GitHub repository from the previous blogs which contain a set of test cases and resource files. The master branch contains non-analyzed tests, the feature/robocop branch contains fixes for the Robocop warnings.

First of all, you will need to install Robocop. At the time of writing, Robocop v1.5 is the latest release.

$ pip install robotframework-robocop

2. Run a Scan

Let’s see what happens when you run a Robocop scan from within the root of the repository. Robocop will search for test cases and resource files in the current directory and in the subdirectories and will run with default settings. The output shows you all the files where violations against the rules have been found. The output format gives you the following information:

The file name where the violation has been found;
The line number where the violation occured;
The column where the violation has been detected;
The severity of the violation (E for error, W for warning, I for information);
The ID of the rule;
The description of the rule.

$ robocop
<path>/test/employee.robot:1:0 [W] 0704 Ignored data found in file
<path>/test/employee.robot:4:0 [W] 1003 Invalid number of empty lines between sections (1/2)
<path>/test/employee.robot:7:0 [W] 1003 Invalid number of empty lines between sections (1/2)
<path>/test/employee.robot:9:13 [W] 1007 Line is over-indented
<path>/test/employee.robot:11:15 [W] 0906 Redundant equal sign in variable assignment
...

In order to limit the number of violations, it is possible to limit the scan to one specific file. This way, only the results for this particular file will be shown.

$ robocop test/employee.robot

Robocop also provides the option to generate a report of the issues found. It will give you a summary which provides a nice overview.

$ robocop --report all test/employee.robot
<path>/test/employee.robot:1:0 [W] 0704 Ignored data found in file
...
Processed 1 file(s) from which 1 file(s) contained issues

Issues by IDs:
W1007 (uneven-indent)                : 6
W0906 (redundant-equal-sign)         : 5
W1003 (empty-lines-between-sections) : 3
W0302 (not-capitalized-keyword-name) : 3
W0704 (ignored-data)                 : 1
W0508 (line-too-long)                : 1
W1002 (missing-trailing-blank-line)  : 1

Found 20 issue(s): 20 WARNING(s).

Scan took 0.006s

3. More About the Rules

You now have an overview of the issues being found, but what are the rules being checked against? This is well documented in the official documentation. Another way of showing this list, is by means of the following command:

$ robocop --list
Rule - 0201 [W]: missing-doc-keyword: Missing documentation in keyword (enabled)
Rule - 0202 [W]: missing-doc-testcase: Missing documentation in test case (enabled)
Rule - 0203 [W]: missing-doc-suite: Missing documentation in suite (enabled)
...

It might be the case that you do not agree with the severity of a certain rule. Let us assume that you do not agree with the severity of the line-too-long rule. This rule has a severity Warning, but you want to change it to severity Information. With Robocop it is possible to change this behaviour by means of the configuration argument. You just need to use the ID or message name of the rule, followed by argument severity, followed by the new value.

$ robocop -c line-too-long:severity:i --report all test/employee.robot
...
<path>/test/employee.robot:34:0 [I] 0508 Line is too long (138/120)
...
Issues by IDs:
...
I0508 (line-too-long)                : 1
...

In the output you can now see that the issue is still reported, but now with severity Information.

Not only the severity can be changed. A lot of rules have configurable parameters which can be changed according to your preferences. In order to know what these parameters are, you can take a look at the documentation. But it is also possible to show this information by means of the list-configurables command followed by the ID or message name of the rule.

$ robocop --list-configurables line-too-long
Rule - 0508 [W]: line-too-long: Line is too long (%d/%d) (enabled)
    Available configurable(s) for this rule:
        severity
        line_length

Besides the severity, it is also possible to change the line length of the line-too-long rule. Just like the severity, you change this parameter to e.g. 160. Running the scan with this setting, no issue is raised now because the line length is within the limits you specified.

$ robocop -c line-too-long:line_length:160 --report all test/employee.robot
...
Found 19 issue(s): 19 WARNING(s).

As you can see, one issue less is shown in the total number of issues and in your output, no line-too-long warning is shown anymore.

It might be that you do not agree with a certain rule at all. In this case, you can disable a rule from being checked. You can do so by means of the exclude argument. Again, this will result in 19 warnings for the employee.robot test case.

$ robocop --exclude 0508 --report all test/employee.robot
...
Found 19 issue(s): 19 WARNING(s).

Listing the rules, you notice that rule 0508 is now indicated as being disabled.

$ robocop --exclude 0508 --list
...
Rule - 0508 [W]: line-too-long: Line is too long (%d/%d) (disabled)
...

In another case, you might want the rule to be enabled by default, but you want to disable an issue. You have noticed the issue and you are ok with it and do not want to be triggered about this part again in the future. You can disable a certain part of your test case for a specific rule. It is not advised to use this regularly, it will not improve the readability of your test case. It is better to fix the issue or to disable the rule or to reconfigure the rule for your needs.

# robocop: disable=line-too-long
| | ${rc}                       | ${output} =     | Run and Return RC and Output | ${APPLICATION} add_employee ${first_name} ${last_name}
# robocop: enable

4. Use a Configuration File

Instead of tweaking the configuration of rules in the command line, you can add the configuration in a configuration file. This way you can ensure that everyone is using the same configuration, certainly when you put it under version control. The contents of this file is for example:

--exclude 0508
--report all

Now we run robocop with the argumentfile argument followed by the file containing the arguments, robocop_args.txt in this case.

$ robocop --argumentfile robocop_args.txt  test/employee.robot

Robocop also offers the possibility to read the configuration file by default. This is restricted under the following conditions:

The argument file must be named .robocop
Robocop must be run without any other arguments. Note that it may not contain any other argument, also no file to analyze for example. It only works when only the robocop command is being used.

The .robocop argument file has the following content when you follow the above example.

--exclude 0508
--report all
test/employee.robot

Now it is sufficient to use the robocop command. In the output you can see that the default configuration file has been loaded.

$ robocop 
Loaded default configuration file from '<path>/.robocop'
Processed 1 file(s) from which 1 file(s) contained issues

Issues by IDs:
No issues found

Found 0 issues

Scan took 0.005s

5. Fix Issues

In this section, some experiences are shared when fixing the issues in the test cases in the repository.

In the employee.robot file, all lines containing a Documentation tag raise the over-indented warning.

employee.robot:11:4 [W] 1007 Line is over-indented

Tried a lot of things in order to fix this, but this was not successful. Raised an issue for this and received a very quick response from the Robocop maintainers. It seems that the rule is ok, but when fixing it, another warning is shown (under-indented warning). This last one is not correct, but is already fixed in a Pull Request and will be available in a next release.

In the data_driver.robot file, a warning is raised about a missing documentation in the test case. In this case, it is ok to disable the rule, because this is the way the Data Driver library works. You cannot fix this.

*** Test Cases ***
# robocop: disable=missing-doc-testcase
| Add Employee ${first_name} ${last_name}
# robocop: enable

Before starting using Robocop, take a good look at the rules you want to enable or disable. Not every rule will be suitable for you and it is better to only enable those rules you really want to check upon. When too many rules are enabled which do not increase your overall quality, but are reported every time, people will tend to not using it anymore.

6. Conclusion

Robocop is a great initiative which can be of good help when you want to increase the quality of your Robot Framework scripts. Beware of which rules you enable, you do not want to be overflooded by warnings you do not care about.

Why Start a Technical Blog

mydeveloperplanet — Tue, 23 Mar 2021 18:33:50 +0000

About 3,5 years ago, I started this blog. Why have I done this and has it brought to me what I expected it to be. In this blog, I will look back at the past 3,5 years and share my experiences. It will also containe some tips and tricks when you want to start blogging yourself. Enjoy!

1. The Beginning

Since the start of my career, I have always been interested in learning new things. About 15 years ago, I learnt myself programming in Java and wanted to find a job as a Java software developer. It was difficult, because I almost always got rejected because of lack of experience. In those days, I started my first website where I would write about the books I read, the small applications I wrote, etc. This has definitely helped me getting my first Java job. For numerous reasons, I quit maintaining my website although I did keep on reading books, exploring new technologies, etc.

It occured to me, that in conversations with colleagues, I often could tell that I already did something with a certain piece of technology. However, it also often happened that I did not know anymore how I did something because I did not write it down. Also, sometimes I could recall how things needed to be done, but then I needed to write it down in an email, which costed me extra time and only the one colleague I mailed it to, had the benefit of it. At such a moment, 3,5 years ago, I thought about my first tech website, about sharing knowledge to more than 1 person, about writing down things I learnt in order to remember better how I accomplished things, and so on. That was the moment this blog was born!

2. The Why

In everything you do, it is important to know why you are doing it (Remember the Golden Circle of Simon Sinek). So, I first thought about what I wanted to accomplish with my blog. The most important thing for me was to learn new things. I was already doing that of course, but it was not very steady. I spent effort during some weekends and then I would stop again for some weeks. Sometimes spending a lot of time finding out where I left off the last time. A blog and a promise to myself for posting every x weeks, should solve that problem. Another motivation is that I like to share knowledge. If I spend time writing something down, I will in the first place remember what I have done and secondly, someone else can also take benefit of it. To conclude, I have three major reasons which will keep me motivated for writing a blog:

Learn new things;
Have a more steady commitment;
Share knowledge.

I do not think that a motivator should be to earn money with your blog. It would merely be a side effect, but it should not be one of your main drivers. However, you can find a lot of articles on the internet about this topic. There are other motivators which can be of significant importance, but which I consider also as nice-to-have side effects:

Appreciation: I do not merely mean Likes but rather personal comments. People who thank you for writing a blog or find it useful. This is a great motivator.
Number of visitors: an increasing number of visitors to your website is also motivating. However, I do not know how to judge this because it changes rapidly or slowly without knowing the cause of it.
Visibility: it certainly increases your visibility outside of your company. I am sure that opportunities will pass by more easily.

3. The How

Let us assume that you got motivated yourself for writing a blog. How do you get started? I can explain how I started and my advice is: just start. Below some tips and tricks how to get started.

3.1 Where to Post

You need a website where you can post your blog. The best option is to have your own website and your own domain name. When you post to your own website, your blogs will always be your content and you do not need to give away the ownership of what you have written. An own domain name will be better for search results and it will be easier to move your website to another provider when necessary. I did not want to spend a lot of time for maintaining the website and I chose to publish at WordPress. I started with the Free Plan but I shifted some time ago to the Personal Plan. It costs a small amount of money and in return you can disable advertisements and you receive customer support, which, in my experience, has been good up till now. I started also with acquiring a domain name, but switched to another registrar later on. I did not know that my personal information was published publicly at the first registrar I used. This resulted in an increasing amount of spam to my mail account. So, take a registrar where your personal details can be kept anonymous for the main public (the so called whois information).

If you do not want to maintain your own website, then you can choose websites where you can post your blogs. I can recommend DZone and dev.to. Medium is also a popular website for blogs, but your content cannot be published again outside of Medium. So for me, this is a no-go. I also publish at DZone and dev.to, but I will explain this in one of the next sections.

3.2 How to Attrack Visitors

We definitely want visitors to our website. We do want our blogs to be read by as many people as possible. Once you created your website, it must be found one way or the other, otherwise you will not have very much visitors. I must admit that this is not my strongest skill. However, I have gained some experience with it the past years.

The most easiest way is to share it via social media. Whenever you have a new blog, post it at LinkedIn, Twitter, … whatever medium you want. Do not expect a large increase in visitors when you have posted something, but it will increase your visibility and people will get to know your website.

Most of the visitors to my blog are originated by Google’s search results. Thus, it is important that your articles get ranked high in Google’s search results. You need therefore clicks, other websites linking to yours, etc. Links from other websites is one of the reasons I also publish at DZone and dev.to. But ensure that you set the canonical URL to your website. For both blog platforms, you have the possibility to set the canonical URL. You will find opinions that you should not publish at other locations because they will take away traffic from your website. I have done an experiment by not publishing anymore at other websites and after a few months, traffic to my website drastically decreased. After starting publishing again, it increased again and it all had to do with Google’s search results. Besides that, you will reach other people that learn to know your website.

Search for newsletters which are created by others and ask to include a link to your blog.

There are probably many other things you can do, but as said before, I am not an expert in website marketing.

3.3 Writing Style

Which writing style should you use? I have experimented with several writing styles and I am still searching and trying to improve it. When it is a tutorial like one, I try to use the second person (you) in the blog. When it is more an opinion, I try to make it more personal and will use the first person (I) in the blog. A good starting point for writing technical documentation, blogs can be found at this GitHub page. The style guides can give you some tips. You can take a look at the style guides, read a part of it now and then and improve your writing skills ongoing.

4. The What

So, what are you going to write about? Do not think too much about it whether you have something valuable or useful to write about, just write and share it. I have noticed that in a lot of blogs, assumptions are made about the knowledge of the reader. That should be ok when it is an in-depth blog, but for getting started guides, it is often disappointing when someone writes some vague text what you should do without a step-by-step guide. E.g. someone writes to startup a Docker container without writing down the command which you should use.

I try to create a mixed content of what I already know and what I would like to learn. Investigating and experimenting with something new costs a significant amount of time and sometimes things just do not work from the first time. Besides that, I want to decide myself of what I am going to write about, otherwise I will not stay motivated. I write in my spare time, so it must not become a task to do it, it must be fun.

At the start, you probably can come up with quite some topics to write about. But at a certain moment, inspiration can become a problem. I follow people at Twitter, read other blogs, websites, watch tech talks, attend conferences, etc. Whenever I read something interesting, I write it down in a todo list. Sometimes it is sufficient information to write a blog right away, most often it is the starting point for reading more about the topic first before I can write about it. I generally have no problem finding a topic, my todo list contains currently about 40 topics. The list is alive, topics get on the list, sometimes topics get off the list without writing a blog about it.

5. Some More Tips

In this last section, I will list some tips which I want to share but did not really fit in one of the previous sections:

Publish at a fixed interval. One that suits you. I originally started with a one week interval and noticed that this was too optimistic. I quickly changed it to a two week fixed interval.
Take some time off. I take two breaks each year. One break in July and August and one break from mid December up to mid January. It allows me to do some other things and to work in advance. I try to have two finished blogs into the pipeline.
Decide how much time you want to spend to your blog. I do not keep time when writing a blog, but I think it takes me 8 to 16 hours for writing a blog, dependent on the topic.

6. Conclusion

In this blog, I wanted to share my experiences about maintaining a blog. I do hope it will help someone to get started or that someone gets inspired by it.

How to Monitor a Spring Boot App

mydeveloperplanet — Wed, 10 Mar 2021 18:39:44 +0000

In this blog you will learn how to monitor a Spring Boot application. You will make use of Spring Actuator, Micrometer, Prometheus and Grafana. Seems a lot of work, but this is easier as you might think!

1. Introduction

When an application runs in production (but also your other environments), it is wise to monitor its health. You want to make sure that everything is running without any problems and the only way to know this, is to measure the health of your application. When something goes wrong, you hopefully will be notified before your customer notices the problem and maybe you can solve the problem before your customer notices anything. In this post, you will create a sample Spring Boot application which you can monitor with the help of Spring Actuator, Micrometer, Prometheus and Grafana. This is visualized in the overview below, where Spring Actuator and Micrometer are part of the Spring Boot App.

The purpose of the different components is explained briefly:

Spring Actuator: supplies several endpoints in order to monitor and interact with your application. See Spring Boot Actuator in Spring Boot 2.0 for more information.
Micrometer: an application metrics facade that supports numerous monitoring systems, Spring Boot Actuator provides support for it.
Prometheus: a timeseries database in order to collect the metrics.
Grafana: a dashboard for displaying the metrics.

Every component will be covered in the next sections. The code used in this post can be found at GitHub.

2. Create Sample App

First thing to do is to create a sample application which can be monitored. Go to Spring Initializr, add dependency Spring Boot Actuator, Prometheus and Spring Web. The sample application will be a Spring MVC application with two dummy endpoints.

Create a RestController with the two endpoints. The endpoints only return a simple String.

@RestController
public class MetricsController {

    @GetMapping("/endPoint1")
    public String endPoint1() {
        return "Metrics for endPoint1";
    }

    @GetMapping("/endPoint2")
    public String endPoint2() {
        return "Metrics for endPoint2";
    }

}

Start the application:

$ mvn spring-boot:run

Verify the endpoints are working:

$ curl http://localhost:8080/endPoint1
Metrics for endPoint1
$ curl http://localhost:8080/endPoint2
Metrics for endPoint2

Verify the Spring Actuator endpoint. The endpoint returns the information in json. In order to format the response so that it is readable, you can pipe the output of the actuator endpoint to mjson.

$ curl http://localhost:8080/actuator | python -mjson.tool
...
{
   "_links":{
      "self":{
         "href":"http://localhost:8080/actuator",
         "templated":false
      },
      "health":{
         "href":"http://localhost:8080/actuator/health",
         "templated":false
      },
      "health-path":{
         "href":"http://localhost:8080/actuator/health/{*path}",
         "templated":true
      },
      "info":{
         "href":"http://localhost:8080/actuator/info",
         "templated":false
      }
   }
}

By default, the above information is available. Much more information can be provided by Spring Actuator, but you need to enable this. In order to enable the Prometheus endpoint, you need to add the following line into the application.properties file.

management.endpoints.web.exposure.include=health,info,prometheus

Restart the application and retrieve the data from the Prometheus endpoint. A large bunch of metrics are returned and available. Only a small part of the output is displayed because it is a really long list. The information which is available at this endpoint, will be used by Prometheus.

$ curl http://localhost:8080/actuator/prometheus
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",} 2.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",} 0.009
...

As mentioned before, Micrometer is also needed. Micrometer can be seen as SLF4J, but then for metrics. Spring Boot Actuator provides autoconfiguration for Micrometer. The only thing you need to do is to have a dependency on micrometer-registry-{system} in your runtime classpath and that is exactly what we did by adding the prometheus dependency when creating the Spring Boot app.

The metrics Actuator endpoint can also be accessed when you add it to the application.properties file.

management.endpoints.web.exposure.include=health,info,metrics,prometheus

Restart the application and retrieve the data from the metrics endpoint.

$ curl http://localhost:8080/actuator/metrics | python -mjson.tool
...
{
    "names": [
        "http.server.requests",
        "jvm.buffer.count",
        "jvm.buffer.memory.used",
...

Each individual metric can be retrieved by adding it to the URL. E.g. the http.server.requests parameter can be retrieved as follows:

$ curl http://localhost:8080/actuator/metrics/http.server.requests | python -mjson.tool
...
{
    "name": "http.server.requests",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 3.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.08918682
        },
...

3. Add Prometheus

Prometheus is an open source monitoring system of the Cloud Native Computing Foundation. Since you have an endpoint in your application which provides the metrics for Prometheus, you can now configure Prometheus to monitor your Spring Boot application. The Spring documentation for doing so can be found here.

There are several ways to install Prometheus as described in the installation section of the Prometheus documentation. In this section, you will run Prometheus inside a Docker container.

You need to create a configuration prometheus.yml file with a basic configuration to add to the Docker container. The minimal properties are:

scrape_interval: how often Prometheus polls the metrics endpoint of your application
job_name: just a name for the polling job
metrics_path: the path to the URL where the metrics can be accessed
targets: the hostname and port number. Replace HOST with the IP address of your host machine

global:
  scrape_interval:     15s

scrape_configs:
  - job_name: 'myspringmetricsplanet'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['HOST:8080']

If you have difficulties finding out your IP address on Linux, you can use the following command:

$ ip -f inet -o addr show docker0 | awk '{print $4}' | cut -d '/' -f 1

Start the docker container and bind-mount the local prometheus.yml file to the one in the docker container. The above prometheus.yml file can be found in the git repository in directory prometheus.

$ docker run \
    -p 9090:9090 \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

After successful startup of the Docker container, first verify whether Prometheus is able to gather the data via url http://localhost:9090/targets.

It seems that Prometheus is not able to access the Spring Boot application running on the host. An error context deadline exceeded is mentioned.

This error can be solved by adding the Docker container to your host network which will enable Prometheus to access the URL. Therefore, add --network host as a parameter. Also remove the port mapping as this has no effect when --network is being used. Finally, give your container a name, this will make it easier to start and stop the container. The -d parameter will run the container in detached mode.

$ docker run \
    --name prometheus \
    --network host \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    -d \
    prom/prometheus

Verify again the Prometheus targets URL, the state indicates UP which means the prerequisite of accessing the metrics endpoint is fullfilled now.

It is now possible to display the Prometheus metrics. Navigate to http://localhost:9090/graph, enter http_server_requests_seconds_max in the search box and click the Execute button. Access a couple of times the endPoint1 URL in order to generate some traffic. This parameter will give you the maximum execution time during a time period of a request.

4. Add Grafana

The last component to add is Grafana. Although Prometheus is able to display the metrics, Grafana will allow you to show the metrics in a more fancy dashboard. Grafana also supports several ways for installing it, but you will run it in a Docker container, just like you did with Prometheus.

$ docker run --name grafana -d -p 3000:3000 grafana/grafana

Navigate to URL http://localhost:3000/, the URL where Grafana is accessible.

The default username/password is admin/admin. After clicking the Log in button, you need to change the default password. Google Chrome will also warn you about the default username/password.

Next thing to do, is to add a Data Source. Click in the left sidebar the Configuration icon and select Data Sources.

Click the Add data source button.

Prometheus is at the top of the list, select Prometheus.

Fill the URL where Prometheus can be accessed, set HTTP Access to Browser and click the Save & Test button at the bottom of the page.

When everything is OK, a green notification banner is shown indicating that the data source is working.

Now it is time to create a dashboard. You can create one of your own, but there are also several dashboards available which you can use. A popular one for displaying the Spring Boot metrics is the JVM dashboard.

In the left sidebar, click the + sign and choose Import.

Enter the URL https://grafana.com/grafana/dashboards/4701 where the JVM dashboard can be found and click the Load button.

Enter a meaningful name for the dashboard (e.g. MySpringMonitoringPlanet), select Prometheus as Data Source and click the Import button.

At this moment, you have a cool first Grafana dashboard at your disposal. Do not forget to scroll down, there are more metrics than shown in the screenshot. The default range is set to 24 hours, this is maybe a bit large when you just started the application. You can change the range in the top right corner. Change it to f.e. Last 30 minutes.

It is also possible to add a custom panel to the dashboard. At the top of the dashboard, click the Add panel icon.

Click Add new panel.

In the Metrics field, you enter http_server_requests_seconds_max, in the Panel title field in the right sidebar, you can enter a name for your panel.

Finally, click the Apply button at the top right corner and your panel is added to the dashboard. Do not forget to save the dashboard by means of the Save dashboard icon next to the Add panel icon.

Set some load to the application and see what happens to the metrics on the dashboard.

$ watch -n 5 curl http://localhost:8080/endPoint1
$ watch -n 10 curl http://localhost:8080/endPoint2

5. Conclusion

In this post, you have learnt how you can set up some basic monitoring for a Spring Boot application. It is necessary to use a combination of Spring Actuator, Micrometer, Prometheus and Grafana, but these are quite easy to set up and configure. This is of course just a starting point, but from here on, it is possible to expand and configure more specific metrics for your application.

Getting Started With RSocket Part 2

mydeveloperplanet — Wed, 24 Feb 2021 18:37:42 +0000

In this blog, you will continue where you left off after Part 1. You will explore the RSocket communication models Fire-and-Forget, Request-Stream and Channel. For all of these models, you will create the server, client and a unit test.

1. Introduction

In Part 1, you learnt the basics of the RSocket communication protocol. It is advised to read Part 1 first before continuing with Part 2. Remember that RSocket provides 4 communication models:

Request-Response (a stream of 1)
Fire-and-Forget (no response)
Request-Stream (a stream of many)
Channel (bi-directional streams)

You covered Request-Response in Part 1, the others will be covered in Part 2.

The source code being used in this post is of course available at GitHub.

2. Fire-and-Forget Model

The Fire-and-Forget model is quite similar to the Request-Response model. The only difference is that you do not expect a response to your request.

2.1 The Server Side

In the RSocketServerController you create a method fireAndForget. Because the request does not return anything, the return type of the method is void. Again, with the annotation @MessageMapping you define the name of the route. Just as with the Request-Response example, the server receives a Notification message. In order to see something happening when the message is received, you just log the Notification message.

@MessageMapping("my-fire-and-forget")
public void fireAndForget(Notification notification) {
    logger.info("Received notification: " + notification);
}

2.2 The Client Side

In the RSocketClientController you create a method fireAndForget. The implementation is identical to the Request-Response example except for the expected return type. Here you use retrieveMono(Void.class) instead of retrieveMono(Notification.class).

@GetMapping("/fire-and-forget")
public Mono<Void> fireAndForget() {
    Notification notification = new Notification(CLIENT, SERVER, "Test the Fire-And-Forget interaction model");
    logger.info("Send notification for my-fire-and-forget: " + notification);
    return rSocketRequester
            .route("my-fire-and-forget")
            .data(notification)
            .retrieveMono(Void.class);
}

Start both the server and the client and invoke the URL:

$ http://localhost:8080/fire-and-forget

As you can see, no response is returned. In the logging of client and server, you can verify the sending and receiving messages.

Client:

Send notification for my-fire-and-forget: Notification{source='Client', destination='Server', text='Test the Fire-And-Forget interaction model'}

Server:

Received notification: Notification{source='Client', destination='Server', text='Test the Fire-And-Forget interaction model'}

2.3 The Test Side

The test is again quite similar to the client code and the Request-Response example. In order to validate whether the Mono does not emit any data, it is sufficient to call verifyComplete. You do not need to call consumeWithNext. If the Mono does emit data, the test should fail. However, replacing the route my-fire-and-forget into my-request-response f.e., does not fail the test. It is unclear why this will not fail the test. If anyone has any suggestions or a solution, please add it into the comments of this blog.

@Test
void testFireAndForget() {
    // Send a fire-and-forget message
    Mono<Void> result = rSocketRequester
            .route("my-fire-and-forget")
            .data(new Notification(CLIENT, SERVER, "Test the Fire-And-Forget interaction model"))
            .retrieveMono(Void.class);

    // Assert that the result is a completed Mono.
    StepVerifier
            .create(result)
            .verifyComplete();
}

3. Request-Stream Model

With the Request-Stream model, you send a request to the server and you will receive a Stream of Notification messages.

3.1 The Server Side

In the RSocketServerController you create a method requestStream. This time, the server will return a Flux of Notification messages. Again, with the annotation @MessageMapping you define the name of the route. In this example, upon receipt of a Notification message, a Flux is returned which emits a new Notification every 3 seconds.

@MessageMapping("my-request-stream")
Flux<Notification> requestStream(Notification notification) {
    logger.info("Received notification for my-request-stream: " + notification);
    return Flux
            .interval(Duration.ofSeconds(3))
            .map(i -> new Notification(notification.getDestination(), notification.getSource(), "In response to: " + notification.getText()));
}

3.2 The Client Side

In the RSocketClientController you create a method requestStream. The implementation is identical to the Request-Response example except for the expected return type. Here you use retrieveFlux(Notification.class) instead of retrieveMono(Notification.class).

@GetMapping("/request-stream")
public ResponseEntity<Flux<Notification>> requestStream() {
    Notification notification = new Notification(CLIENT, SERVER, "Test the Request-Stream interaction model");
    logger.info("Send notification for my-request-stream: " + notification);
    Flux<Notification> notificationFlux = rSocketRequester
            .route("my-request-stream")
            .data(notification)
            .retrieveFlux(Notification.class);
    return ResponseEntity.ok().contentType(MediaType.TEXT_EVENT_STREAM).body(notificationFlux);
}

Start both the server and the client and invoke the URL:

$ curl http://localhost:8080/request-stream
data:{"source":"Server","destination":"Client","text":"In response to: Test the Request-Stream interaction model"}

data:{"source":"Server","destination":"Client","text":"In response to: Test the Request-Stream interaction model"}

data:{"source":"Server","destination":"Client","text":"In response to: Test the Request-Stream interaction model"}

data:{"source":"Server","destination":"Client","text":"In response to: Test the Request-Stream interaction model"}
...

As you can see, the server emits every 3 seconds a Notification. In the logging of client and server, you can verify the sending and receiving messages.

Client:

Send notification for my-request-stream: Notification{source='Client', destination='Server', text='Test the Request-Stream interaction model'}

Server:

Received notification for my-request-stream: Notification{source='Client', destination='Server', text='Test the Request-Stream interaction model'}

3.3 The Test Side

The test is again similar to the client code. During verification, you verify the first message received, then verify whether 5 messages are received and finally verify the last message.

@Test
void testRequestStream() {
    // Send a request message
    Flux<Notification> result = rSocketRequester
            .route("my-request-stream")
            .data(new Notification(CLIENT, SERVER, "Test the Request-Stream interaction model"))
            .retrieveFlux(Notification.class);

    // Verify that the response messages contain the expected data
    StepVerifier
      .create(result)
      .consumeNextWith(notification -> {
         assertThat(notification.getSource()).isEqualTo(SERVER);
         assertThat(notification.getDestination())
                                             .isEqualTo(CLIENT);
         assertThat(notification.getText()).isEqualTo("In response to: Test the Request-Stream interaction model");})
      .expectNextCount(5)
      .consumeNextWith(notification -> {
         assertThat(notification.getSource()).isEqualTo(SERVER);
         assertThat(notification.getDestination())
                                             .isEqualTo(CLIENT);
         assertThat(notification.getText()).isEqualTo("In response to: Test the Request-Stream interaction model");})
      .thenCancel()
      .verify();
}

4. Channel Model

The Channel model is a bit more complicated than the other models you have seen so far. Here you will send a Flux and as a response a Flux will be returned. This provides you the ability to send messages back and forth like within a chat conversation for example.

4.1 The Server Side

In the RSocketServerController you create a method channel. You will upon receipt of a Notification increment a counter and every one second the result of the counter notificationCount will be sent to the client. In order to be able to follow what is happening, you add logging at receiving the Notification and when the result is returned.

@MessageMapping("my-channel")
public Flux<Long> channel(Flux<Notification> notifications) {
    final AtomicLong notificationCount = new AtomicLong(0);
    return notifications
        .doOnNext(notification -> {
            logger.info("Received notification for channel: " + notification);
            notificationCount.incrementAndGet();
         })
        .switchMap(notification -> 
Flux.interval(Duration.ofSeconds(1)).map(new Object() {
                private Function<Long, Long> numberOfMessages(AtomicLong notificationCount) {
                    long count = notificationCount.get();
                    logger.info("Return flux with count: " + count);
                    return i -> count;
                }
         }.numberOfMessages(notificationCount))).log();
}

4.2 The Client Side

In the RSocketClientController you create a method channel. You need to create a Flux. In order to accomplish this, you create 3 Mono Notification items, one with a delay of 0 seconds (notification0), one with a delay of 2 seconds (notification2) and one with a delay of 5 seconds (notification5). You create the Flux notifications with a combination of the Mono‘s you just created. Every time the Flux emits a Notification, you log this in order to be able to follow what is happening. Finally, you send the Flux to the RSocket channel and retrieve the response as a Flux of Long and return it to the caller.

@GetMapping("/channel")
public ResponseEntity<Flux<Long>> channel() {
    Mono<Notification> notification0 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model"));
    Mono<Notification> notification2 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model")).delayElement(Duration.ofSeconds(2));
    Mono<Notification> notification5 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model")).delayElement(Duration.ofSeconds(5));

    Flux<Notification> notifications = Flux.concat(notification0, notification5, notification0, notification2, notification2, notification2)
            .doOnNext(d -> logger.info("Send notification for my-channel"));

    Flux<Long> numberOfNotifications = this.rSocketRequester
            .route("my-channel")
            .data(notifications)
            .retrieveFlux(Long.class);

    return  ResponseEntity.ok().contentType(MediaType.TEXT_EVENT_STREAM).body(numberOfNotifications);
}

Start both the server and the client and invoke the URL:

$ curl http://localhost:8080/channel
data:1
data:1
data:1
data:1
data:3
data:3
data:4
data:4
data:5
data:5
data:6
data:6
...

The result is as expected. First the notification0 is sent, after 5 seconds (notification5) the following Notification is sent together with a notification0, 2 seconds later a notification2, 2 seconds later a new one, and finally 2 seconds later the last one. After the last result, the Flux will keep on transmitting a count of 6. In the logging of client and server, you can verify the sending and receiving messages. This time including the timestamps, the complete logging contains even more information which is left out for brevity purposes. You should take a look at it more closely when running the examples yourself. Important to notice are the onNext log statements which occur each second and correspond with the response the server is sending.

Client:

17:01:19.820 Send notification for my-channel
17:01:24.849 Send notification for my-channel
17:01:24.879 Send notification for my-channel
17:01:26.881 Send notification for my-channel
17:01:28.908 Send notification for my-channel
17:01:30.935 Send notification for my-channel

Server:

17:01:19.945 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:19.947 Return flux with count: 1
17:01:20.949 onNext(1)
17:01:21.947 onNext(1)
17:01:22.947 onNext(1)
17:01:23.947 onNext(1)
17:01:24.881 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:24.882 Return flux with count: 2
17:01:24.884 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:24.885 Return flux with count: 3
17:01:25.886 onNext(3)
17:01:26.886 onNext(3)
17:01:26.909 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:26.909 Return flux with count: 4
17:01:27.910 onNext(4)
17:01:28.910 onNext(4)
17:01:28.936 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:28.937 Return flux with count: 5
17:01:29.937 onNext(5)
17:01:30.937 onNext(5)
17:01:30.963 Received notification for channel: Notification{source='Client', destination='Server', text='Test the Channel interaction model'}
17:01:30.964 Return flux with count: 6
17:01:31.964 onNext(6)
17:01:32.964 onNext(6)
...

4.3 The Test Side

The test is similar as the client code. You send the messages to the channel and verify the resulting counts. The repeating elements in the test are left out for brevity, the complete test is available at GitHub.

@Test
void testChannel() {
    Mono<Notification> notification0 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model"));
    Mono<Notification> notification2 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model")).delayElement(Duration.ofSeconds(2));
    Mono<Notification> notification5 = Mono.just(new Notification(CLIENT, SERVER, "Test the Channel interaction model")).delayElement(Duration.ofSeconds(5));

    Flux<Notification> notifications = Flux.concat(notification0, notification5, notification0, notification2, notification2, notification2);
    // Send a request message
    Flux<Long> result = rSocketRequester
            .route("my-channel")
            .data(notifications)
            .retrieveFlux(Long.class);

    // Verify that the response messages contain the expected data
    StepVerifier
            .create(result)
            .consumeNextWith(count -> {
                assertThat(count).isEqualTo(1);
                })
...
            .consumeNextWith(count -> {
                assertThat(count).isEqualTo(6);
                })
            .thenCancel()
            .verify();
}

5. Conclusion

You have learnt how to create a server, client and a unit test for the RSocket communication models Fire-and-Forget, Request-Streams and Channel. By now, you will have the basic knowledge in order to do some exploring, experimentation yourself.

Getting Started With RSocket Part 1

mydeveloperplanet — Wed, 10 Feb 2021 18:30:35 +0000

In this blog, you will learn the basics of RSocket, a binary application protocol which supports Reactive Streams. After the introduction, you will learn how to use RSocket in combination with Spring Boot. Enjoy!

1. Introduction

RSocket is a binary protocol to be used on top of TCP or WebSockets. RSocket is a communication protocol which embraces the Reactive principles. This means that RSocket uses asynchronuous communication. It is also suited for push notifications. When using HTTP for example, there will be a need for polling in order to check whether new messages are available. This causes unnessary network load. RSocket provides a solution for this. There are 4 communication models which can be used:

Request-Response (a stream of 1)
Fire-and-Forget (no response)
Request-Stream (a stream of many)
Channel (bi-directional streams)

RSocket is situated at the OSI layer 5/6 and therefore at the Application Layer of the TCP/IP Model.

In the next sections, you will find examples for each communication model: the server side, client side and a unit test. The source code being used in this post is of course available at GitHub.

2. Reference Documentation

Before getting started, it is useful to know where some interesting documentation can be found. During writing this blog, it appeared that reference documentation, examples, etc. cannot easily be found. This list should give you a flying start when taking your first steps with RSocket.

All information about the protocol, the specification, implementations can be found at the official RSocket website.

The Spring Framework’s support for the RSocket protocol.

The Spring Boot reference section for the RSocket protocol.

Ben Wilcock has written some awesome blogs about the RSocket protocol. The complete list can be found at GitHub.

3. Request-Response Model

The Request-Response model will allow you to send one request and receive one response in return. First thing to do, is to set up a basic Spring Boot application. Navigate to the Spring Initializr website, add dependency RSocket and create the project which you can open in your favorite IDE. When checking the pom, you notice that the spring-boot-starter-rsocket dependency and the dependency reactor-test are added. The first one will enable RSocket support in your Spring Boot application, the second one is needed for testing purposes.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-rsocket</artifactId>
</dependency>
...
<dependency>
    <groupId>io.projectreactor</groupId>
    <artifactId>reactor-test</artifactId>
    <scope>test</scope>
</dependency>

The source code at GitHub is divided into two Maven modules, one for the server and one for the client.

In order to exchange information between client and server, you create a Notification data class which will be the item to transport via RSocket. The Notification class contains a Source, a Destination and some free Text. The toString implementation will be used for logging purposes.

public class Notification {
    private String source;
    private String destination;
    private String text;

    public Notification() {
        super();
    }

    public Notification(String source, String destination, String text) {
        this.source = source;
        this.destination = destination;
        this.text = text;
    }

    public String getSource() {
        return source;
    }

    public String getDestination() {
        return destination;
    }

    public String getText() {
        return text;
    }

    @Override
    public String toString() {
        return "Notification{" +
                "source='" + source + '\'' +
                ", destination='" + destination + '\'' +
                ", text='" + text + '\'' +
                '}';
    }
}

3.1 The Server Side

You create a RsocketServerController and annotate it with @Controller. In order to create your first RSocket Request-Response example, you just add a method requestResponse which takes a Notification, logs the received Notification and returns a new Notification where you swap the received source and destination and add a simple text to it. In order to make it a RSocket request, you need to annotate the method with @MessageMapping and give it a name, e.g. my-request-response.

@Controller
public class RsocketServerController {

    Logger logger = LoggerFactory.getLogger(RsocketServerController.class);

    @MessageMapping("my-request-response")
    public Notification requestResponse(Notification notification) {
        logger.info("Received notification for my-request-response: " + notification);
        return new Notification(notification.getDestination(), notification.getSource(), "In response to: " + notification.getText());
    }
}

In order to ensure that the RSocket server is started, you also need to add the port to the application.properties file:

spring.rsocket.server.port=7000

Start the server:

$ mvn spring-boot:run

In the logging you notice that the Netty Webserver has started. Netty is the reactive counterpart of a Jetty Webserver.

Netty RSocket started on port(s): 7000

3.2 The Client Side

The client side is a little bit more complex. You will again create a Spring Boot application which will send a Notification message to the server. Sending the message will be invoked by means of an http call. Therefore, you add the dependency spring-boot-starter-webflux to the client pom. Beware that you cannot use spring-boot-starter-web, you need to use the reactive webflux variant.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Ensure that you do not define the port in the application.properties, otherwise a RSocket server will be started and that is not what is needed for your client. When you do so, the following error will appear in your console.

2021-01-02 12:04:58.853 ERROR 19058 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'rSocketServerBootstrap'; nested exception is reactor.netty.ChannelBindException: Failed to bind on [0.0.0.0:7000]
...
Caused by: reactor.netty.ChannelBindException: Failed to bind on [0.0.0.0:7000]
    Suppressed: java.lang.Exception: #block terminated with an error
...

You create a RsocketClientController and annotate it with @RestController. Next, you need to create a RSocketRequester instance in order to be able to connect to the RSocket server. In the requestResponse method, you create the Notification message (for ease of use, just copy the Notification class from the server module and make sure that it is also present on the client side) and with the rSocketRequester instance, you specify the route you want the message to be sent to (name equals the name as specified with the @MessageMapping annotation on the server side), the data you want to send and finally the response you expect. The response will be a Mono which means that you expect one response from the server and the response needs to be a Notification message. The message itself is returned to the caller.

@RestController
public class RsocketClientController {

    private static final String CLIENT = "Client";
    private static final String SERVER = "Server";

    private final RSocketRequester rSocketRequester;

    Logger logger = LoggerFactory.getLogger(RsocketClientController.class);

    public RsocketClientController(@Autowired RSocketRequester.Builder builder) {
        this.rSocketRequester = builder.tcp("localhost", 7000);
    }

    @GetMapping("/request-response")
    public Mono<Notification> requestResponse() {
        Notification notification = new Notification(CLIENT, SERVER, "Test the Request-Response interaction model");
        logger.info("Send notification for my-request-response: " + notification);
        return rSocketRequester
                .route("my-request-response")
                .data(notification)
                .retrieveMono(Notification.class);
    }
}

Start both the server and the client and invoke the URL:

$ curl http://localhost:8080/request-response
{"source":"Server","destination":"Client","text":"In response to: Test the Request-Response interaction model"}

As you can see, the server response is returned. In the logging of client and server, you can verify the sending and receiving messages.

Client:

Send notification for my-request-response: Notification{source='Client', destination='Server', text='Test the Request-Response interaction model'}

Server:

Received notification for my-request-response: Notification{source='Client', destination='Server', text='Test the Request-Response interaction model'}

3.3 The Test Side

Creating a test for the server code is quite similar as creating the client code. A RSocketRequester needs to be created in order to setup the connection. Sending a message is identical to the client code, only this time you put the response into a result variable of type Mono<Notification>. You can use this result variable in a StepVerifier in order to validate the response which is received. With a StepVerifier you can verify reactive responses in your unit test.

@SpringBootTest
class MyRsocketServerPlanetApplicationTests {

    private static final String CLIENT = "Client";
    private static final String SERVER = "Server";

    private static RSocketRequester rSocketRequester;

    @BeforeAll
    public static void setupOnce(@Autowired RSocketRequester.Builder builder, @Value("${spring.rsocket.server.port}") Integer port) {
        rSocketRequester = builder.tcp("localhost", port);
    }

    @Test
    void testRequestResponse() {
        // Send a request message
        Mono<Notification> result = rSocketRequester
                .route("my-request-response")
                .data(new Notification(CLIENT, SERVER, "Test the Request-Response interaction model"))
                .retrieveMono(Notification.class);

        // Verify that the response message contains the expected data
        StepVerifier
                .create(result)
                .consumeNextWith(notification -> {
                    assertThat(notification.getSource()).isEqualTo(SERVER);
                    assertThat(notification.getDestination()).isEqualTo(CLIENT);
                    assertThat(notification.getText()).isEqualTo("In response to: Test the Request-Response interaction model");
                })
                .verifyComplete();
    }
}

4. Conclusion

You have learnt the basics of the RSocket application protocol and explored how to create a server, client and unit test for the Request-Response communication model. In the next blog, you will learn how to create a server, client and unit test for the remaining three communication models.