Effective Testing of Cloud Components with Quarkus and LocalStack

Eric Rosetti Lessa — Fri, 08 Mar 2024 15:03:02 +0000

In this tutorial, we’ll understand how to code and test applications that need to store data, such as images and documents, on Amazon S3 and similar cloud services. We’ll build a Java application to integrate with the S3 API and use Quarkus for testing with containers. In the end, we’ll deploy the entire application stack to Amazon Web Services and see it live. Our goal is to practice and learn how easy it can be to code using local tests with TestContainers.

You can find all the code in the following repository:

https://github.com/ericrlessa/s3-quarkus-sample

This sample project shows 3 Rest services to put, get, and generate a pre-signed URL for an image stored in S3. In the project, there are separate test classes for each type of integration using the “REST Assured” library and test containers. Also, there is an Amazon CloudFormation template to deploy the stack to your AWS account.

Dependencies

First of all, the S3 quarkiverse extension should be added to the project. If you don’t have a quarkus project yet, or the Quarkus CLI installed, check the “Getting Started” guide and come back. The S3 extension can be added with this command, in your project directory:

$ quarkus ext add io.quarkiverse.amazonservices:quarkus-amazon-s3

As a result, the following dependencies are added to the project:
```
  <dependency>
    <groupId>${quarkus.platform.group-id}</groupId>
    <artifactId>quarkus-amazon-services-bom</artifactId>
    <version>${quarkus.platform.version}</version>
    <type>pom</type>
    <scope>import</scope>
 </dependency>

 <dependency>
     <groupId>io.quarkiverse.amazonservices</groupId>
     <artifactId>quarkus-amazon-s3</artifactId>
 </dependency>
```


      You also need to add one HTTP client library as a dependency, let’s use url-connection-client.

     ```


      <dependency>
         <groupId>software.amazon.awssdk</groupId>
         <artifactId>url-connection-client</artifactId>
      </dependency>

Docker containers

If you build the sample project with the above dependencies, with the $ quarkus build command, you can observe that the necessary containers are started automatically.

Let’s start the quarkus in dev mode with the following command: $ quarkus dev

Then, open a new terminal and verify the containers running with $ docker ps:

Three containers should be running:

localstack/localstack -> LocalStack is a service emulator that lets you test cloud applications locally, without incurring in network and service costs. Also it enables applications to test cloud clientes and APIs in Continuous Integration pipelines. Learn more at https://docs.localstack.cloud/getting-started/

mariadb -> Container running a mariadb database. We will use to store some metadata and a reference to S3 objects.

testcontainers/ryuk -> Testcontainers is an open source framework for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container.

There is no explicit dependency for testcontainers and localstack. We only need to add the extensions for S3 and MariaDB. With those dependencies present, quarkus automatically initializes a local environment during test and dev mode. In dev mode, the containers are started up without any configuration. In test mode, when you use @QuarkusTest annotation in your class test, you tell Quarkus to prepare the local environment before starting the tests. Everything is integrated without configuring one single property in the application. Learn more about other DevServices in the quarkus documentation.

But what if we want to avoid the localstack running by quarkus? What can be done if you want to run the S3 localcontainer separately? You can override the localcontainer host using an application property:

$ quarkus.s3.endpoint-override=http://localhost:4566

Let’s configure this property to verify it’s effect. In the sample project, from the github repository, open the file “application-dev.properties” file, under “src/main/resources”, and uncomment the line with this property. After this, enter in “quarkus dev” mode and run “docker ps” in another terminal:

Where is the localstack container? Quarkus did not start the localstack because you explicitly set the host, saying that you would take care of this by yourself. You could, for example, run your localstack directly by a docker command instead of quarkus:

$ docker run -it --publish 4566:4566 -e SERVICES=s3 -e START_WEB=0 localstack/localstack:3.0.1

Remember that this is only necessary to run localstack with extensive configuration. You can use the default DevService just as well. This demonstrates the two alternatives to deal with AWS local environment, with and without quarkus devservices.

To see the impact in tests, do the same thing in the application.properties located in the src/test/resources folder of the test project. Uncomment the line with “quarkus.s3.endpoint-override”, and try to run the test or build the project with “$ quarkus build”. You will receive an error where the stack trace shows the problem:
```
  SdkClientException: Unable to execute HTTP request
```


     You just received a connection error because quarkus did not run a localstack container before running your tests. Let’s fix that with a S3 integration.

3. **Java implementation S3 integration**

     Now that we understand the environment of our AWS S3 local integration, we can go to the Java code. How quarkinverse extension helps coding and S3 integration?
With this extension, you can use the @Inject annotation from jakarta CDI, to get a S3 client and do whatever you want with your bucket, either in production or test code:


     ```java


       @Inject
       private S3Client s3;

 To isolate the interaction with external aws library, I created one interface to abstract this process:

 ```java

   public interface ImageDataRepository {
      public void save(Path file,
                     Image image);
      public byte [] find(Image image);
      public void delete(Image image);
      public String createPresignedGetUrl(Image image);       
      public String getBucketName();
   }


     And the S3Client is injected into a concrete class called S3BucketImage, that invokes the API services.

     To make transfers to S3 safer, we can use pre-signed urls. Those are temporary addresses that lets your application securely share a link with clients and transfer content directly to them.
In our application, there are three rest services (JAX-RS resources) where it is possible to save, get and generate a presigned URL to your S3 object:

     ```java


        @Path("/image")
        public class ImageResource {
           @Inject
           private ImageDataRepository imageDataRepository;

           @POST
           @Consumes(MediaType.MULTIPART_FORM_DATA)
           @Transactional
           public void saveImage(ImageFormData imageFormData){
                Image image = ImageBuilder.newInstance()                                 .withBucket(imageDataRepository.getBucketName())
               .withFileName(imageFormData.fileName)
               .withMimeType(imageFormData.mimeType)
               .addTag("fileName",  imageFormData.fileName).build();
               image.persist();
           imageDataRepository.save(imageFormData.file.toPath(), image);
           }

           @Path("/{id}")
           @GET
           @Produces(MediaType.APPLICATION_OCTET_STREAM)
           public byte[] getImage(@PathParam("id") Long id){
              return imageDataRepository.find(Image.findById(id));
           }

           @Path("/{id}/presignedGetUrl")
           @GET
           public String createPresignedGetUrl(@PathParam("id") Long id){
              return imageDataRepository.createPresignedGetUrl(Image.findById(id));
           }
         }

 The ImageDataRepository implementation provides the S3 operations:

 ```java

    @ApplicationScoped
    public class S3BucketImage implements ImageDataRepository {
      @ConfigProperty(name = "bucket.name")
      private String bucketName;
      @ConfigProperty(name = "presigned.url.duration.in.minutes")
      private Integer presignedUrlDurationInMinutes;
      @Inject
      private S3Client s3;
      @Inject
      private S3Presigner presigner;

      public String getBucketName(){
         return this.bucketName;
      }

      public void save(Path file,
                Image productImage) {
         List<Tag> tagsS3 = getTags(productImage);
         s3.putObject(
         PutObjectRequest.builder()
           .bucket(productImage.bucket())
           .key(productImage.key())
           .contentType(productImage.mimeType())
          .tagging(Tagging.builder().tagSet(tagsS3).build())
           .build(),
           RequestBody.fromFile(file));
      }

      public void delete(Image productImage) {
           DeleteObjectRequest deleteRequest = DeleteObjectRequest.builder()
           .bucket(productImage.bucket())
           .key(productImage.key())
           .build();

           s3.deleteObject(deleteRequest);
      }

      public byte[] find(Image productImage) {
         try {
             return s3.getObject(GetObjectRequest.builder()
                  .bucket(productImage.bucket())
                  .key(productImage.key())
                  .build()).readAllBytes();
          } catch (IOException e) {
               throw new RuntimeException(e);
          }
      }

      public String createPresignedGetUrl(Image productImage) {
          GetObjectRequest objectRequest = GetObjectRequest.builder()
           .bucket(productImage.bucket())
           .key(productImage.key())
           .build();

          GetObjectPresignRequest presignRequest =  GetObjectPresignRequest.builder()
         .signatureDuration(Duration.ofMinutes(presignedUrlDurationInMinutes))
           .getObjectRequest(objectRequest)
           .build();

          PresignedGetObjectRequest presignedRequest = presigner.presignGetObject(presignRequest);


          return presignedRequest.url().toExternalForm();
      }

      private List<Tag> getTags(Image productImage) {
          List<Tag> tagsS3 = productImage.tags().stream().map(
           t -> parseTagS3(t)
           ).collect(Collectors.toList());
             return tagsS3;
      }

      private Tag parseTagS3(s3sample.domain.Tag t) {
          return Tag.builder().key(t.key()).value(t.value()).build();
      }

      @PostConstruct
      private void createBucket() {
          try {
             try {
                 HeadBucketRequest headBucketRequest = HeadBucketRequest.builder()
                   .bucket(bucketName)
                   .build();
           s3.headBucket(headBucketRequest);
           } catch (NoSuchBucketException e) {
                CreateBucketRequest bucketRequest = CreateBucketRequest.builder()
                   .bucket(bucketName)
                   .build();
                s3.createBucket(bucketRequest);
           }
         } catch (Exception e) {
            //FIXME add log
         }
    }
  }


     Notice that the method createBucket() creates the bucket only if does not already exists.

4. **Tests**

     The tests were created using REST Assured to POST and GET the image, and to GET a  pre-signed URL to the image. The tests were separated into two classes, one for get operations, and another to save the image.

     ```java


        @QuarkusTest
        public class ImageGetTest extends ImageTest {
             Image productImage;

             @BeforeEach
             @Transactional
             void startProductFixture(){
                  productImage = ImageBuilder.newInstance()
                    .withFileName(fileName)
                    .withBucket(bucketName)
                    .withMimeType(mimetype)
                    .addTag("fileName", fileName)
                    .addTag("city", "Vitoria")
                    .addTag("country", "Brasil")
                    .build();
                  productImage.persist();
                  imageDataRepository.save(file, productImage);
             }

             @Test
             public void testGetS3ProductImage() {
                    byte [] file = given()
                      .when()
                      .get("/api/image/" + productImage.id)
                      .then()
                      .statusCode(HttpStatus.SC_OK)
                      .extract()
                      .body()
                      .asByteArray(); // Adjust the expected     status code as needed
                    assertThat(file, notNullValue());

                   //saveFileLocalToVerifyManually(file);
             }

             @Test
             public void testPresignedGetUrlS3() {
                  String preAssignedUrl = given()
                     .when()
                     .get("/api/image/%d/presignedGetUrl".formatted(productImage.id))
                     .then()
                     .statusCode(HttpStatus.SC_OK)
                     .extract()
                     .body()
                     .asString();

                  byte [] file = given()
                     .baseUri(preAssignedUrl)
                     .get()
                     .then()
                     .statusCode(HttpStatus.SC_OK)
                     .extract()
                     .body()
                     .asByteArray();

                 assertThat(file, notNullValue());

                //   saveFileLocalToVerifyManually(file);
            }


            private void saveFileLocalToVerifyManually(byte [] file){
                String localFilePath = "/tmp/test.jpg";

                try (FileOutputStream fos = new  FileOutputStream(localFilePath)) {
                    fos.write(file);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }

          }

 Test to upload the image to S3:

 ```java

     @QuarkusTest
     public class ImagePostTest extends ImageTest {
         @Test
         public void testUpdateS3ProductImage() {
              Integer id = given()
              .multiPart("file", file)
              .multiPart("fileName", fileName)
              .multiPart("mimeType", mimetype)
              .when()
              .post("/api/image")
              .then()
              .statusCode(HttpStatus.SC_OK)
              .extract().path("id");

              Image img = Image.findById(id);
              assertThat(img, notNullValue());
              assertThat(img.bucket(), equalTo(bucketName));
              assertThat(img.id, notNullValue());
              assertThat(img.fileName(), equalTo(fileName));
              assertThat(img.mimeType(), equalTo(mimetype));

              byte [] imageFileData = imageDataRepository.find(img);
              assertThat(imageFileData, notNullValue());

              imageDataRepository.delete(img);
         }
     }


     The abstract superclass supply common resources to the tests:

     ```Java


        public abstract class ImageTest {
             String fileName = "Vitoria_ES_Brasil.jpg";
             String mimetype = "image/jpeg";
             Path file = Paths.get("src/test/resources/s3sample/domain/core/product/" + fileName);
             @ConfigProperty(name = "bucket.name")
             String bucketName;
             @Inject
             ImageDataRepository imageDataRepository;
        }

 As you can see, the interface ImageDataRepository provides the implementation to operate on S3. In the end, quarkus will inject an S3Client pointing to the local services running in a docker container.

Deploy Cloud formation stack

Important: The following steps use AWS services that might incur in costs. There is no cost for opening an AWS account and there is a free tier for most services in the first year. You can open your AWS account at https://aws.amazon.com/free. Remember to delete all resources after finishing this tutorial to avoid unnecessary charges.

After coding and testing everything locally, it is time to deploy the lambda function in the AWS environment. To deploy the stack, let’s use AWS Serverless Application Model (SAM):

$ sam deploy --stack-name cfn-fn --template-file infrastructure.cfn.yaml --resolve-s3 --capabilities "CAPABILITY_IAM" "CAPABILITY_AUTO_EXPAND" "CAPABILITY_NAMED_IAM"

SAM will create the resources through a CloudFormation Stack that can be observed in service console. For the purposes of this demonstration, the relevant resources are the lambda function with the execution role and their policies, allowing access to the S3 bucket.

After the deployment, it is possible to use Postman or any other tool to execute the http requests to get and post the image from the Bucket.
```
  1. Copy the Service HTTP URI generated in the outputs of the recently created stack.
  2. Define the request body:
        - file: binary data 
        - fileName: example.jpg
        - mimeType: image/jpeg
  3. Send the request to Http Api URL, running in AWS.
```
In Postman, you can use a POST request body of type “form-data” to send your file:

To get the file, just change the host of your API Gateway and make a GET request on browser, curl, postman, or any tool of your preference:

https://e5fasnm3ak.execute-api.us-east-1.amazonaws.com/api/image/1

To get the pre-signed URL, just adjust the URI:

https://e5fasnm3ak.execute-api.us-east-1.amazonaws.com/api/image/1/presignedGetUrl

The server will return the pre-signed URL. Just copy and paste on the browser to see the image.
Delete the stack

Before deleting the stack, empty the Image Bucket and delete it. After that, you can run the following command or delete the stack on the cloudformation console:

$ aws cloudformation delete-stack --stack-name cfn-fn

Confirm the stack has been deleted

$ aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"
Conclusion

In this article, we demonstrated how applications can use and test object storage, and other cloud services, with Quarkus and TestContainers. That helps applications to be more reliable, without resorting to “mocks” in testing. Containers make it more productive to reproduce and test scenarios and improve the overall developer experience.

You can put those techniques to practice and learn more in our open-source project EcoMarkets. Also, you’ll be helping many families to live more sustainably and eat more healthly. Join the project repository activity at https://github.com/CaravanaCloud/ecomarkets
References

Quarkus extension for amazon S3:

https://quarkus.io/extensions/io.quarkiverse.amazonservices/quarkus-amazon-s3/

Quarkiverse doc:

https://docs.quarkiverse.io/quarkus-amazon-services/dev/amazon-s3.html

Localstack:

https://docs.localstack.cloud/getting-started/
Testcontainers:

https://testcontainers.com/

DEV Community: Eric Rosetti Lessa

Effective Testing of Cloud Components with Quarkus and LocalStack