AWS SQS with spring boot & Localstack with Junit Testing

Preface

Building microservices architecture often involves creating microservices communicating using a message bus or any means of loosely coupled means such as AWS Simple Queue Service, dearly called AWS SQS.

What are we building

Complete Code: spring-boot-localstack
Here is step-by-step guide of setting up a simple spring-boot web application talking to AWS SQS using localstack to mock the AWS environment.

This includes bare minimum configurations required to create a web app communicating using SQS only.

Basic Definitions:

1. localstack: Simply a tool to mock AWS Cloud Provider in your local environment, to help develop cloud applications.
2. Junit5: A testing framework for Java application based on Java8.
3. awaitability: A tool to express expectations for asynchronous system in an easy and concise manner.
4. Docker: Run any process or application in a containerised manner.

Pre-requisites:

1. Basic knowledge of Java and spring-boot.
2. Environment setup for running Docker e.g. docker for mac, or just a happy linux system
3. Can setup AWS CLI to play around with application. A command line utility to interact with AWS services.
4. A Familiar IDE.

Setup Basic Project

1. Get on to the second-best website on internet. Spring Initializer
2. Create a spring project preferably with 2.3 Spring boot version and following dependencies
   • Spring web
   • Lombok (java utility to avoid writing boilerplate code)
   • AWS Simple Queue Service
3. Also add following dependencies externally in pom.xml

<dependency>
    <groupId>org.awaitility</groupId>
    <artifactId>awaitility</artifactId>
    <version>3.1.3</version>
    <scope>test</scope>
</dependency>

Create simple event and data models to send and receive messages:

@Data
@AllArgsConstructor
@NoArgsConstructor
@Builder
public class SampleEvent {

  private String eventId;
  private String version;
  private String type;
  private ZonedDateTime eventTime;
  private EventData data;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
@Builder
public class EventData {

  private String name;
  private int age;
  private String description;
  private EventType eventType;

  public enum EventType {
    CREATED, PROCESSED
  }
}

Create a simple controller annotated with @SqsListener to listen to a queue.

@SqsListener(value = "${cloud.aws.sqs.incoming-queue.url}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
  private void consumeFromSQS(SampleEvent sampleEvent) {
    log.info("Receive message {}", sampleEvent);
    //do some processing
    sampleEvent.setEventTime(ZonedDateTime.now());
    sampleEvent.getData().setEventType(EventData.EventType.PROCESSED);
    amazonSQSAsync.sendMessage(outgoingQueueUrl, mapper.writeValueAsString(sampleEvent));
  }

Spring property configurations:

setup application.yml with aws sqs properties, such as:

localstack:
  host: localhost

cloud:
  aws:
    credentials:
      access-key: some-access-key
      secret-key: some-secret-key
    sqs:
      incoming-queue:
        url: http://localhost:4576/queue/incoming-queue
        name: incoming-queue
      outgoing-queue:
        name: outgoing-queue
        url: http://localhost:4576/queue/outgoing-queue
    stack:
      auto: false
    region:
      static: eu-central-1

logging:
  level:
    com:
      amazonaws:
        util:
          EC2MetadataUtils: error

Notes:
a. Aws credentials can also be set up environmental variables or in a .aws/credentials file, (further read 😉)
b. define name and url of queues so application can listen to and write to queue. Localstack runs SQS service on port number: 4576
c. The below logging property is to avoid having multiple lines of error where application tries to connect to EC2 instance of localstack. (workaround 😏)

AWS Local SQS configs:

Inside the java configuration for SQS we create some beans to allow our application to talk to SQS service provided by localstack. You could add a profile for each config when deploying app in production i.e. actual AWS environment.

@Bean
//endpoint config for connecting to localstack and not actual aws environment.
  public AwsClientBuilder.EndpointConfiguration endpointConfiguration(){
    return new AwsClientBuilder.EndpointConfiguration("http://localhost:4576", region);
  }

  @Bean
  @Primary
//This bean will be used for communicating to AWS SQS
  public AmazonSQSAsync amazonSQSAsync(final AwsClientBuilder.EndpointConfiguration endpointConfiguration){
    AmazonSQSAsync amazonSQSAsync = AmazonSQSAsyncClientBuilder
        .standard()
        .withEndpointConfiguration(endpointConfiguration)
        .withCredentials(new AWSStaticCredentialsProvider(
            new BasicAWSCredentials(awsAccesskey, awsSecretKey)
        ))
        .build();
    createQueues(amazonSQSAsync, "incoming-queue");
    createQueues(amazonSQSAsync, "outgoing-queue");
    return amazonSQSAsync;
  }
//create initial queue so our application can talk to it
  private void createQueues(final AmazonSQSAsync amazonSQSAsync,
                            final String queueName){
    amazonSQSAsync.createQueue(queueName);
    var queueUrl = amazonSQSAsync.getQueueUrl(queueName).getQueueUrl();
    amazonSQSAsync.purgeQueueAsync(new PurgeQueueRequest(queueUrl));
  }

We can use QueueMessagingTemplate for sending and receiving messages from AWS SQS

 @Bean
  public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSQSAsync){
    return new QueueMessagingTemplate(amazonSQSAsync);
  }

Also setup QueueMessageHandlerFactory so it can convert incoming messages from SQS as String to the actual object you want, in this case Simple Event, using objectmapper.
You can configure objectmapper separately. Add your custom deserialiser o such by registering your module or add custom datetime conversion.

@Bean
  public QueueMessageHandlerFactory queueMessageHandlerFactory(MessageConverter messageConverter) {

    var factory = new QueueMessageHandlerFactory();
    factory.setArgumentResolvers(singletonList(new PayloadArgumentResolver(messageConverter)));
    return factory;
  }

  @Bean
  protected MessageConverter messageConverter(ObjectMapper objectMapper) {

    var converter = new MappingJackson2MessageConverter();
    converter.setObjectMapper(objectMapper);
    // Serialization support:
    converter.setSerializedPayloadClass(String.class);
    // Deserialization support: (suppress "contentType=application/json" header requirement)
    converter.setStrictContentTypeMatch(false);
    return converter;
  }

Finally add this docker compose yaml asking docker to create a localstack container:

version: '3.0'

services:
  localstack:
    image: localstack/localstack:0.10.7
    environment:
      - DEFAULT_REGION=eu-central-1
      - SERVICES=sqs
    ports:
      - "4576:4576"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

Starting the application

1. start the localstack using: docker-compose up (in same directory as docker file)
2. run application using: mvn spring-boot:run
3. send a message to sqs using AWS CLI for e.g.:

aws --endpoint="http://localhost:4576" --region=eu-central-1 sqs send-message --queue-url http://localhost:4576/queue/incoming-queue --message-body '{
  "eventId": "some-event-id-1",
  "eventTime": "2016-09-03T16:35:13.273Z",
  "type": "some-type",
  "version": "1.0",
  "data": {
    "name": "Dev. to",
    "age": 20,
    "description": "User created",
    "eventType": "CREATED"
  }
}'

1. You should see log messages of event received and forwarded.
2. Also added a Junit Test which uses same configuration as for local run.

Thank you!!