DEV Community: Ivonne Roberts

How to choose a database on AWS

Ivonne Roberts — Mon, 09 Aug 2021 10:35:36 +0000

How do I choose a database on AWS? How do I choose between NoSQL (Amazon DynamoDB) vs SQL (Amazon RDS)? When building a serverless microservice on AWS, I will walk you through how I narrow down what the best database on AWS is for my microservice.

Database options on AWS

When it comes to storing data in AWS, at the time of writing, there are over 15 different options. The options for our microservice’s persistent store are the following:

Amazon Aurora
Amazon RDS
- Aurora
- PostgreSQL
- MySQL
- MariaDB
- Oracle
- SQLServer
Amazon Redshift
Amazon DynamoDB
DocumentDB
Amazon Keyspaces

Not included here are the services for graph data and memory caching as it is not applicable for our use case.

Why choosing the right database is important

When choosing a database, it is important to make an informed decision to avoid issues down the road caused by cost, performance, or even worse, having to migrating to another database to support our business model. There are a few questions we need to ask ourselves when choosing a database.

Do we need a NoSQL or SQL option?

Here is the JSON object we need to be able to support. If you want to understand how we arrived at this JSON object, read How to design a RESTful API on AWS.

{
   "status": "ACTIVE",
   "public": true,
   "id": 1234556,
   "title": "Weird Animal Facts",
   "description": "Tune in to learn about how horses make the neighing sound",
   "dateTime": "2021-06-30T08:00:00+01:00",
   "duration": {
     "length":"1",
     "unit":"HOUR",
   },
   "locations": [
     {
       "type": "ONLINE",
       "url": "https://ivonneroberts.com/meetingURL"
     },
     {
       "type": "VENUE",
       "address": {
         "streetAddress1": "1 5th Avenue",
         "streetAddress2": null,
         "city": "New York",
         "state": "NY",
         "zip": 10001
       }
     }
   ]
 }

A normalized form of this data could be represented by an Event table, a Schedule table and a Location table. Let’s analyze these entities to be able to determine which type of database on AWS we could use, NoSQL or SQL.

The Event table could have a 1:many relationship with Schedule. You can imagine a scenario where an event could have a repeating schedule. That Schedule table could have multiple entries for a given event. To decide NoSQL versus SQL, let’s look at the update query. How many event records would need to be updated, if an instance needs to be changed? We would need to update one record. This makes it a candidate for NoSQL database.

The Event table could also have a 1:many relationship with Location. An event could be both online, as well as, at a venue. Asking the same NoSQL versus SQL, let’s look at the update query. If a location needs to be updated how many event records would need to be updated? The answer is n events that use this same venue. You would have to query the table to find how many events have that venue and then update each one. This is not a candidate for NoSQL. However, the attributes of a location, have a lot of details or nuances that have nothing to do with an event. In fact, I would argue Location should be in its own microservice. The Event REST API response should instead only include a location id, and that information stored as part of another microservice.

With those questions answered, the remaining data in our microservice now could be stored in a NoSQL database.

How the data will be queried

For these first two urls, a NoSQL implementation (specifically Amazon DynamoDB) would be a scan on the full table. As the data grows, scans are not cost efficient, nor performant. We can mitigate this problem by introducing a range/sort key in addition to our primary/partition key, for example category. As a client, I am subscribed to the YouTube category. These APIs could use your authentication to determine your eligible categories, query the DB for them and then further filter either by date or other attributes of the event.

https://api.mydomain.com/events/v1/events/
https://api.mydomain.com/events/v1/events/search?date[gt]=2021-07-01&status=CANCELLED

This next API is straight forward. We can query our NoSQL database by the event id (partition key in DyanmoDB).

https://api.mydomain.com/events/v1/events/{eventId}/

Costs associated with databases on AWS

The next question we need to ask is what are the cost consideration. When choosing a database there is both the cost of the actual database/license, as well as, the people cost to maintain and fine tune that database. For some of the databases on AWS you pay for what you use/store, most of the heavy lifting is abstracted from you. Other databases require EC2 instances, additional licenses in addition to the service. Also, depending on the database, you might need a database administrator to update, manage and fine tune that database. This can be cost prohibitive for a startup.

Depending on where you are in your journey it is acceptable to start with the minimum requirements, and then at a later date take the downtime and migrate to another database that better meets your needs.

Finalized JSON structure

Now that we have reviewed the considerations for choosing our database, and we have finalized on a NoSQL option. Let’s update our JSON payload to reflect the changes we called out. First we have added a category attribute, this will be our range/sort key. Second, we have added a schedule attribute that is a list of all our instances of this event. Lastly we have updated the location object to now simply refer to an ID that references a resource in a separate location microservice.

{
   "status": "ACTIVE",
   "id": 1234556,
   "category": "DEVWIDGETS_YOUTUBE",
   "title": "Weird Animal Facts",
   "description": "Tune in to learn about how horses make the neighing sound",
   "schedules": [{
     "dateTime": "2021-06-30T08:00:00+01:00",
     "duration": {
       "length":"1",
       "unit":"HOUR",
     }
   }],
   "locations": [
     {
       "id": 456
     },
     {
       "id": 789
     }
   ]
 }

Conclusion

So after much deliberation we have finalized our JSON payload and have decided to use NoSQL. On AWS our options are Amazon DynamoDB, Amazon DocumentDB, and Amazon Keyspaces. For this microservice I am going to use Amazon DynamoDB as it is a fully managed service. It is a fairly robust database that abstracts a lot of the heavy lifting associated with managing a database. Most importantly it is serverless which means I don’t need to concern myself with EC2 pricing, patching and troubleshooting. And, to name a few more key features:

Fully managed
multi-region
multi-active
built-in security
backup and restore
in-memory caching
configurable autoscaling

We now have walked through how to choose a database on AWS Cloud. Next week, I will continue to build on this design and implement some of these APIs on AWS. Check back weekly for new content. Feel free to comment below any questions you may have or reach out to me on twitter at @ivlo11

Happy Coding!

How to design a RESTful API on AWS

Ivonne Roberts — Mon, 26 Jul 2021 15:07:08 +0000

Have you ever asked yourself what is a REST API? How do I design a RESTful API on AWS Cloud? How do I write a RESTful java microservice? As a software engineer building on AWS, I’ll walk you through designing a REST API.

What is a REST API?

A REST API is a way for clients to get/store information with an application through HTTP. This is typically what websites and mobile apps use to communicate with backends services. For example: What’s today’s weather in Paris? A website makes a HTTP call to a weather service with Paris as an input. In response, the weather service returns a message with the details that that website would need to show that information.

The text format that the client and the backend service exchange is usually in JSON structure. However, you could choose to use others like HTML, XML, plain text and so on. The RESTful specification does not define what format you must use unlike something like SOAP which uses XML.

REST APIs should also be stateless. When calling the same API multiple times, nothing is stored in the backend causing it to change its response. For example, with pagination, calling an API will return a “page” (or subset) of items. Every time i call it, it will return the same subset, regardless of if I had already called that API. However, if I call that API give me the next 20 items starting an index 40, it will then return something different.

Another characteristic of REST APIs is that they can be cached. When requesting the item, with an id of 123, multiple times, the first request would go to the microservice. The second time I request that item, with a short window, it is unlikely that it has changed. At that point I should instead just receive a cached response.

Business Use Case to Design a REST API for

So let’s put some of what we have learned to practice. We will be designing the APIs for the following business use case. An application needs a way to get the list of upcoming events. The consumers for this API could be a mobile application that displays the events that you are registered for. This API could also be used an a website that advertises upcoming events.

Designing the RESTful API

When you name an api you want to use something short and simple. It should accurately convey the service or data it is providing and indication of what you will receive. You don’t want to use long string like events-i-signed-up-for. Instead you want things like that to show up in an hierarchy of paths like events/search?registered=true.

The next thing you want is versioning. When creating an API, you define a contract with you consumers. When you need to make a breaking change, they may not be able to update their code base at the precise minute that you deploy. Instead, you would deploy a new version in addition to your existing version. This gives clients time to migration from the previous version to the new one. The key here is to not make breaking changes, but when you have to, use versioning.

A simple path could be as follows. The name of your microservice, the version number and then the resource that you are requesting. In the below example, the microservice is called events, the version is v1 and the resources are events and locations respectively.

https://api.mydomain.com/events/v1/events/
https://api.mydomain.com/events/v1/locations/

These APIs would return a list indicated by the plural resource in the path and brackets in the response

[
  {
    "id": 123
  },
  {
    "id": 234
  }
]

A path to return a specific event would be structured as follows. The name of your microservice, the version number, the resource that you are requesting and the id of the specific item you want.

https://api.mydomain.com/events/v1/events/{eventId}/
https://api.mydomain.com/events/v1/events/{eventId}/locations

You can also do complicated searching on resources with something like the below pattern. The path is the name of your microservice, the version number, the resources that you are searching, and that fields that should match the returned resources. For example in the below paths, it searches for events where the date is greater than 2021-07-01 and the status is CANCELLED. The second path searches for events that have a duration of longer than 1 hour

https://api.mydomain.com/events/v1/events/search?date[gt]=2021-07-01&status=CANCELLED
https://api.mydomain.com/events/v1/events/search?dateAndTime.duration.length[gt]=1&dateAndTime.duration.unit=HOUR

JSON Structure of our RESTful API

Now that we have an idea of what our paths should look like let us look at the JSON structure that our responses will look like. JSON objects are surrounded by curly braces {}. Then the contents are key value pairs where the keys are surrounded in quotes. The values can be the following data types: a number, a string, a boolean, an object, a list or simply null.

When choosing which data types to use you want to keep flexibility in mind. Considering that this is a contract with your clients, you want to choose data types that will allow you to extend. For example status is a property that would not need multiple fields. However, the location could be multiple places, so a list would allow us to represent that instead of having a flat structure like location1, location2, etc. Another example is date and time. It could have multiple attribute like time, date, and duration. This might be better represented with another JSON object.

{
   "status": "ACTIVE",
   "public": true,
   "id": 1234556,
   "title": "Weird Animal Facts",
   "description": "Tune in to learn about how horses make the neighing sound",
   "dateAndTime": {
     "date": "2021-06-30",
     "time": "08:00:00",
     "timeZone": "GMT",
     "duration": {
       "length":"1",
       "unit":"HOUR",
     }
   },
   "locations": [
     {
       "type": "ONLINE",
       "url": "https://ivonneroberts.com/meetingURL"
     },
     {
       "type": "VENUE",
       "address": {
         "streetAddress1": "1 5th Avenue",
         "streetAddress2": null,
         "city": "New York",
         "state": "NY",
         "zip": 10001
       }
     }
   ],
   "cost": {
     "currency": "USD",
     "price": 20,
     "unit": "PER_PERSON"
   }
 }

Conclusion

We now have walked through how I design a RESTful API, the path and JSON structure, on AWS Cloud. I continue to build on this design in How to Choose a Database on AWS. Check back weekly for new content. Feel free to comment below any questions you may have or reach out to me on twitter at @ivlo11

Happy Coding!

Add Security Headers to your Serverless Static Site

Ivonne Roberts — Wed, 07 Jul 2021 12:50:32 +0000

Being able to create a static website hosted on AWS S3 and fronted by Amazon CloudFront has become the serverless standard these days. Both of these AWS services facilitate a lot of the heavy lifting for you by giving you performant web hosting, with features like caching at edge locations closer to your end users, and security features like mitigating DDoS attacks.

One thing that required a little more effort then I’d prefer was the ability to add security headers to responses. That used to have to be done through Lambda@Edge which, while still serverless, ideally it should be reserved for more computationally involved origin resolution or response manipulation. Today you can now do that with Amazon CloudFront Functions.

Amazon CloudFront Functions provide short lived simple JavaScript functions that can manipulate your response/request. Some use cases could be url rewrites/redirects, access authorization and what I will show you here, header manipulation. See Danilo Poccia’s blog post for more details around Amazon CloudFront Functions.

Keep reading to see how we add the following security headers to our requests using AWS Cloudformation and Amazon CloudFront Function

Strict-Transport-Security
Content-Security-Policy
X-Content-Type-Options
X-Frame-Options
X-XSS-Protection

Create your Serverless function

In CloudFormation there are a few things you need to define. 1) The name of your function, in my case add-security-headers. 2) that you want this to be published automatically. 3) Your function configuration which includes what run time it will use, which at the time of writing this only includes cloudfront-js-1.0. And lastly 4) your actual code. See the complete resource definition below

addSecurityHeadersFunction:
  Type: AWS::CloudFront::Function
  Properties:
    Name: add-security-headers
    AutoPublish: true
    FunctionConfig:
      Comment: Adds security headers to the response
      Runtime: cloudfront-js-1.0
    FunctionCode: |
      function handler(event) {
          var response = event.response;
          var headers = response.headers;

          // Set HTTP security headers
          // Since JavaScript doesn't allow for hyphens in variable names, we use the dict["key"] notation
          headers['strict-transport-security'] = { value: 'max-age=63072000; includeSubdomains; preload'};
          headers['content-security-policy'] = { value: "default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'"};
          headers['x-content-type-options'] = { value: 'nosniff'};
          headers['x-frame-options'] = {value: 'DENY'};
          headers['x-xss-protection'] = {value: '1; mode=block'};

          // Return the response to viewers
          return response;
      }

Link your CloudFront Distribution

Now you can update your existing CloudFront Distribution by associating the function to your CacheBehavior as below and deploy your cloudformation. In my example below I’m importing the ARN from a separate template that includes all my CloudFront Functions.

FunctionAssociations:
  - EventType: viewer-response
    FunctionARN: !ImportValue
      'Fn::Sub': ${CloudFrontStackName}-AddSecurityHeadersFunction

Test your implementation

Once completed head over to your terminal window and use curl to test out our headers:

 ivonne@my-machine ~ % curl -i https://static.ivonneroberts.com
 HTTP/1.1 200 OK
 Content-Type: text/html
 Content-Length: 87
 Connection: keep-alive
 Date: Fri, 28 May 2021 23:59:19 GMT
 Last-Modified: Fri, 28 May 2021 23:06:46 GMT
 Etag: "7875df45e56965139099615f6c5c907b"
 Accept-Ranges: bytes
 Server: AmazonS3
 Via: 1.1 3dc5af024af63cc0e8b9cf31fd852ecf.cloudfront.net (CloudFront)
 Content-Security-Policy: default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'
 Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
 X-Xss-Protection: 1; mode=block
 X-Frame-Options: DENY
 X-Content-Type-Options: nosniff

As you can see our 5 security headers are now displayed:

 Content-Security-Policy: default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'
 Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
 X-Xss-Protection: 1; mode=block
 X-Frame-Options: DENY
 X-Content-Type-Options: nosniff

Conclusion

Adding a Amazon CloudFront function is pretty simple and you can see the updates almost immediately. If you would like to see the full CloudFormation templates head on over my github (link below). Feel free to modify it to fit your use case. As always if you have any questions reach out!

Github: [https://github.com/ivlo11/serverless-patterns/tree/main/static-site-with-cloudfront-function]
Documentation: [https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html]
CloudFormation Documentation: [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudfront-function.html]
CloudFront Function Sample Code: [https://github.com/aws-samples/amazon-cloudfront-functions]

Happy Coding!

AWS SQS Events On AWS Lambda

Ivonne Roberts — Wed, 26 Aug 2020 19:18:35 +0000

In 2018, AWS has announced support for SQS triggered events on AWS Lambda. For companies embracing a serverless architecture this opened up new possibilities for event-driven architecture, streamlining batch infrastructure and much more.

Before the feature launched, if you were a serverless shop that needed to process SQS messages, the only option was to use CloudWatch to trigger a Lambda function, that polled for messages and then either fanned out workers or chewed threw batches of SQS messages. While this worked, it was prone to error. A Lambda function can only live for 5 minutes, at which point it could either spin up another Lambda function and pass the torch or simply wait for CloudWatch to trigger another Lambda function. Depending on how long the Lambda function would run, you either, lost time waiting for the next CloudWatch trigger or had a high number of messages with multiple receives as the Lambda function timed out.

At Edelman Financial Engines, we put a stake in the ground to embrace two things: event-driven architecture and serverless architecture. These two designs have allowed us to create more scalable applications as well as focus on the things that are important to us and have a direct impact on our clients. With the recent Lambda and SQS announcement, we now have a new option to process our queues in a serverless fashion: simply configure an event source mapping on a Lambda function for a queue.

Event-Driven Architecture

When a client or a planner does something on our site, a cascade of decisions and processes are run. An event-driven architecture enables us to do that work in a reasonable amount of time. Take, for example, a client logging in. When a client presses that login button, we have to do a series of things; get the client’s latest holdings from the provider, validate the client’s information, generate a snapshot of the client’s progress, run Monte Carlo simulations and the list goes on.

With event-driven architecture, as soon as the login happens, an event is sent to an event-bus where consumers are standing by waiting to process certain events. In our use case, an analytics consumer could process the login event and trigger a client engagement metric for reporting. Soon after, another event is published saying the client’s latest holdings are updated. Multiple consumers of that event can then pre-calculate the clients current progress and run Monte Carlo simulations simultaneously and cache the results. Now, as the client navigates to our site landing page, their results show up almost instantaneously instead of the 10+ seconds it would normally take if we did this all sequentially.

Having services like SNS and SQS greatly facilitates this architecture model. We can have consumers (AWS Lambda, SQS queue, HTTP endpoint, etc) set as endpoints to SNS topics that do asynchronous heavy lifting on the inputs. For SQS, queues can be processed by a consumer (formerly only EC2 instances, but now, Lambda functions as well) chewing through the messages in a batch fashion.

Serverless Architecture

At Financial Engines, we have embraced a DevOps culture where decisions are shifted to the functional development team level providing greater choice of tools and technology to match various problems at hand. However, what we don’t want is development teams to be bogged down with tasks like OS patching, auto-scaling servers, and so on. To that end, we have set a standard for teams to use managed services when possible.

Teams can spin up complete microservices using AWS Lambda, DynamoDB, S3, etc., without provisioning or managing any server and OS patching, and still get the power they need. In addition to ease of use, we have had significant cost savings from switching to serverless computing. (Checkout this Financial Engines AWS Case Study for more details on that.)

Using SQS & AWS Lambda to Pre-Calculate Progress

For an example of this new feature/architecture, I am going to dive deep into one of the scenarios mentioned above. When a client’s holdings have been updated, whether the client triggered the update or a job ran to update all client holdings, an event is published with a few details on that client and their holdings. We then have consumers of that event pre-calculate the client’s current progress and cache the results.

There are a couple considerations we needed to make.

If the client triggered the update (e.g. the client logged in), then it is likely the progress score will be consumed soon and the pre-calculation needs to happen as soon as possible.
If a job ran to update all client holdings, then the possibility of that progress score being consumed soon is fairly low and therefore we can take our time doing the pre-calculation.

To accomplish the requirements we created the following architecture.

Our event bus publishes events to SNS topics. From there, consumers decide how to process the messages. For our client-triggered updates, we have an AWS Lambda function set as an endpoint to the holdings updated topic. We also configured message attribute filtering so that this Lambda function only receives client-triggered events. For our job-triggered events, we used an SQS queue to consume filtered events. We then used the newly-created SQS Event on AWS lambda to process that queue.

To follow, I’ll walk you through how that SQS-to-AWS Lambda connection was configured.

Step 1:

The first thing you need is a lambda that accepts SQS Events

For our Lambda function we need the following dependencies via gradle:

compile 'com.amazonaws:aws-java-sdk-lambda:1.11.321'
compile 'com.amazonaws:aws-java-sdk-sqs:1.11.321'

At the very basic level, the handler code in Java 8 is the configured something like this:

package com.fngn.samples;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.fngn.samples.api.dto.EventDto;
import com.fngn.samples.api.dto.ResponseDto;
import com.fngn.samples.application.service.EventConsumer;
public class QueueConsumerLambda implements RequestHandler<SQSEvent, Void> {
  @Override
  public Void handleRequest(SQSEvent event, Context context) {
    try {
      for (SQSEvent.SQSMessage message : event.getRecords()) {
        EventDto eventDto = null;
        ResponseDto responseDto = null;
String input = message.getBody();
        eventDto = unmarshalEventBody(input);
        responseDto = eventConsumer.processEvent(eventDto);
        handleFailures(responseDto, message);
      }
    } catch (Exception ex) {
      logger.error("Exception handling batch seed request.", ex);
      throw ex;
    }
    return null;
  }
  ...
}

Lastly, we bring this together with the following CloudFormation definition for our Lambda function and our execution role (for bonus points, we have added dead letter queue configuration):

queueConsumerLambda:
  Type: 'AWS::Lambda::Function'
  Properties:
    Handler: 'com.fngn.samples.QueueConsumerLambda::handleRequest'
    Role: !Sub "arn:aws:iam::*:role/${queueConsumerExecutionRole}"
    Description: !Sub "Cloud formation lambda for ${projectName}"
    FunctionName: !Sub "${projectName}-queue-consumer"
    MemorySize: 512
    Timeout: 30
    Code:
      S3Bucket: "com.fngn.samples"
      S3Key: !Sub "${projectName}/${lambdaCode}"
    Runtime: java8
    TracingConfig:
      Mode: Active
    Environment:
      Variables:
        REGION: !Ref "AWS::Region"
        APP_ID: !Sub ${projectName}-event-consumer
        ENV_REALM: !Sub ${accountType}
  DependsOn:
    - queueConsumerExecutionRole
queueConsumerExecutionRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Action: 'sts:AssumeRole'
        Principal:
          Service: lambda.amazonaws.com
        Effect: Allow
        Sid: ''
    Policies:
    - PolicyName: !Sub "${projectName}-queue-consumer-policy"
      PolicyDocument:
        Version: '2008-10-17'
        Statement:
        ...
        - Action:
          - sqs:ChangeMessageVisibility
          - sqs:DeleteMessage
          - sqs:GetQueueAttributes
          - sqs:ReceiveMessage
          - sqs:SendMessage
          Effect: Allow
          Resource:
          - !GetAtt sqsQueue.Arn
          - !GetAtt sqsQueueDLQ.Arn
          Sid: 'SQSPermissions'

Step 2:

We can now create an SQS queue via CloudFormation that subscribes to our SNS topic.

sqsQueue:
  Type: 'AWS::SQS::Queue'
  Properties:
    QueueName: !Sub '${projectName}-ConsumerQueue'
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt sqsQueueConsumerDLQ.Arn
      maxReceiveCount: 5
    VisibilityTimeout: 600

policyQueueConsumer:
  Type: 'AWS::SQS::QueuePolicy'
  Properties:
    PolicyDocument:
      Id: !Sub '${projectName}-ConsumerQueuePolicy'
      Version: '2012-10-17'
      Statement:
      - Sid: 'AllowConsumerSnsToSqsPolicy'
        Effect: 'Allow'
        Principal:
          AWS: '*'
        Action:
          - 'sqs:SendMessage'
        Resource: !GetAtt sqsQueue.Arn
        Condition:
          ArnEquals:
            aws:SourceArn: !Sub 'arn:aws:sns:*:*:holding-events']]
    Queues:
      - !Ref sqsQueue

Step 3:

Now that those two are in place we can configure the trigger:

sqsEventTrigger:
  Type: "AWS::Lambda::EventSourceMapping"
  Properties:
    BatchSize: 5
    Enabled: true
    EventSourceArn: !GetAtt sqsQueue.Arn
    FunctionName: !Sub "${projectName}-queue-consumer"
  DependsOn:
    - queueConsumerLambda
    - queueConsumerExecutionRole

Step 4:

Send messages to your SQS queue or, in our case, SNS topic and sit back and watch the Lambda function chew through messages as they come in. Honestly, it is as simple as that.

With this configuration, here’s what’s happening. AWS Lambda takes over polling and invoking concurrent Lambda functions to chew through the queue with chunks of messages up to the configured batch size (In our case we configured the batch size to be 10 messages). The Lambda function processes each message in the SQSEvent. If any of those messages fail, the Lambda function fails and the messages stay on the queue. If all the messages are successful then AWS Lambda deletes the message from the queue. If the queue has a dead letter queue (redrive policy above) configured, then AWS Lambda will try to process those messages until it reaches the max receive count of 5. At which point, if the visibility timeout expires and the message is still on the queue, SQS will delete that message from the queue and move it to the dead letter queue.

Monitoring

You can use the dashboards in the AWS Lambda and SQS service consoles as well as configure your own CloudWatch dashboards to monitor the progress. We also employed the use of AWS X-Ray which really helped us early on to identify issues with downstream resources in our microservice.

In the AWS Lambda dashboard, you can see quickly see how the service is invoking your Lambda function pretty consistently until the queue is empty. In the below case, we limited our execution to 200 “workers.” So, you can also see throttles as AWS Lambda enforced the limit on concurrent executors.

In the SQS dashboard, you can see the rate of messages coming into the queue, rate of receives, and other interesting tidbits. As you can see in the below chart, our job was generating events faster that our 200 concurrent executors were processing them, by noticing the age of messages go up and then drop precipitously as the Lambda functions finish processing all the messages.

Here in X-Ray, you can see all the components that make up this microservice. The Lambda function that processes the SNS messages, the Lambda function that process the SQS queue, and the Lambda function that serves up the cached response.

Advance Configuration

Concurrency and EventSourceMapping Config

With our use case, we took configuration a bit further. For example, if a job ran to update all client holdings, then the rate of events per minute spikes fairly high in a short period of time. Downstream resources (e.g. database) used to calculate the progress score might not be able to support that load.

In this case, what we did was create a schedule with subscription details given the day of the week or time of day. We then configured a CloudWatch event that triggers a Lambda function hourly. Based on the schedule’s subscription details, that Lambda function updates either the SQS consumer Lambda function config or its SQS event mapping.

{
  "MON-FRI_09:00-04:00_11:00-04:00": {
    "queueEnabled":false
  },
  "MON-FRI_11:00-04:00_22:00-04:00": {
    "queueEnabled":true,
    "workers":15,
    "batchSize":10
  },
  "MON-FRI_22:00-04:00_09:00-04:00": {
    "queueEnabled":true,
    "workers":100,
    "batchSize":10
  }
}

This level of control proved to be quite powerful. During the day, at peak times of our load, it was prudent to not risk stressing downstream resources while they were simultaneously serving client traffic. We could either completely disable processing (i.e. disable the EventSourceMapping) or configure the Lambda function execution concurrency to a low number (ex: 15). However, in the evenings, we can ramp up the number of concurrent executions to 100. On the weekends, we could pump that up even higher to, say, 200 concurrent executions.

It took a cross-functional team effort and several runs to determine our concurrency limit. With resources both in the cloud and on-premises, we needed to closely monitor them as we kept ramping up the numbers. At the end of the day, our on-premises relational database ended up limiting the microservice to not exceed 200 concurrent executions, roughly equaling 5000 messages per minute and 1.4m database calls.

Filtering SNS messages

As mentioned previously, our use case had 2 flavors of the holdings updated event. There were events triggered by users and events triggered by jobs. Those SNS jobs have their event metadata also configured in the SNS message attributes field.

In this case, you can use the SDK to configure filtering the messages for a given endpoint. For our SNS-to-AWS Lambda, flow we could have it only receive events that had appType=WEB. For our SNS-to-SQS flow we configured it to only receive events that had appType=JOB.

Subscription subscription = getSubscriptionForArn(environment.getenv("BATCH_SNS_ENDPOINT"));
if (batchSubscription != null) {
  String jsonPolicy = "{\"applicationType\": [\"JOB\"]};
  setFilterPolicy(subscription.getSubscriptionArn(), jsonPolicy);
}

We took it a step further and also filtered the messages on the financial institutions or employers that were enabled. This helped us significantly reduce cost on message processing, as well as simplified our code, as the Lambda function no longer had to do its own filtering.

String getBatchSubscriptionFilterPolicy() throws IOException {
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  JsonGenerator generator = (new JsonFactory()).createGenerator(baos, JsonEncoding.UTF8);

  generator.writeStartObject();
  generator.writeFieldName("applicationType");
  generator.writeStartArray();
  generator.writeString("JOB");
  generator.writeEndArray();

  Set<String> financialInstitutions = getFinancialInstitutions();
  if (financialInstitutions.isEmpty() == false) {
    generator.writeFieldName("financialInstitutionId");
    generator.writeStartArray();
    for (String financialInstitution : financialInstitutions) {
      generator.writeString(financialInstitution);
    }
    generator.writeEndArray();
  }

  Set<String> employers = getEmployers();
  if (employers.isEmpty() == false) {
    generator.writeFieldName("employerId");
    generator.writeStartArray();
    for (String employer : employers) {
      generator.writeString(employer);
    }
    generator.writeEndArray();
  }

  generator.writeEndObject();
  generator.close();

  return baos.toString();
}

Notes/Tips/Features Requests

While this is a fairly strong feature, there are a few things we would like to see in the future.

For example, when processing the dead letter queue with this feature, you actually have to manually delete failed messages, or they very quickly become “poison pills.” If you don’t manually delete the messages, then you need to create a dead letter queue for your dead letter queue so that you can set the max receive count to 2 for example, and just ignore what gets into that final queue. It’s a little more heavy lifting than I had expected, when I thought I could just re-use the same Lambda function processing the main queue. It would be nice if AWS would allow you to set a redrive policy without specifying another dead letter queue.

The other issue was the rate of throttling in the Lambda function. At times, we saw several hundred invocations a minute getting throttled. While the documentation says that concurrent execution is honored, we assumed that the rate of Lambda functions the AWS Lambda would invoke was also going to be limited to the number of concurrent executors. I imagine that, long-term, this can get pricey and somewhat wasteful if you limit concurrent executions. I’d like to see AWS address this or try to minimize the number of throttled lambdas.

Lastly, while not really an issue that AWS could address, we struggled with configuring downstream resources’ auto scaling. Since, as mentioned before, jobs can start at any time of day and cause a sudden significant spike of publish events in a short period of time. Downstream services like DynamoDB’s read/write auto scaling always lagged to respond for the first few minutes. This would create a spike in throttles and therefore failed messages, another reason why configuring a dead letter queue is important. (Note that the dashboard shows a per minute average so that doesn’t always equate to actual throttles.)

Conclusion

I hope that through this article you can see how powerful this new feature is and how it can greatly reduce code/management of infrastructure to support queue processing.

Feel free to ask any questions and reach out with any comments you may have. Also, come back and check out any new content we publish. We are always playing around with some of the latest and greatest features with the goal of figuring out all the nuances before teams start using the features.