Micronaut 4 application on AWS Lambda- Part 3 Reducing Lambda cold starts with SnapStart and DynamoDB request priming

Introduction

In the part 1 of our series about how to develop, run and optimize Micronaut web application on AWS Lambda, we demonstrated how to write a sample application which uses the Micronaut framework, AWS Lambda, Amazon API Gateway and Amazon DynamoDB. We also made the first Lambda performance (cold and warm start time) measurements and observed quite a big cold start time.

In the part 2 of the series, we introduced Lambda SnapStart and measured how its enabling reduces the Lambda cold start time by far more than 50% depending on the percentile. We also clearly observed the impact of the AWS SnapStart Snapshot tiered cache in all our measurements so far.

In this part of our article series, we'll introduce how to apply Lambda SnapStart priming techniques by starting with DynamoDB request priming with the goal to further improve the performance of our Lambda functions.

Sample application with the activated AWS Lambda SnapStart with using DynamoDB request priming

We'll re-use the same sample application introduced in the part 1 of our series.

Activating Lambda SnapStart is also a prerequisite for this method.

Globals:
  Function:
    CodeUri: target/aws-lambda-micronaut-4.9-1.0.0-SNAPSHOT.jar
    Runtime: java21
    SnapStart:
     ApplyOn: PublishedVersions    
....

This can be done in the global’s section of the Lambda functions, in which case SnapStart applies to all Lambda functions defined in the AWS SAM template, or you can add the 2 lines

SnapStart:
 ApplyOn: PublishedVersions

to activate SnapStart only for the individual Lambda function.

You can read more about the concepts behind the Lambda SnapStart in the part 2.

SnapStart and runtime hooks offer you new possibilities to create your Lambda functions for low startup latency. With the pre-snapshot hook, we can prepare our Java application as much as possible for the first call. We load and initialize as much as possible which our Lambda function needs before the snapshot is created. This technique is known as priming.

In this method I will introduce you to the priming of DynamoDB request, which is implemented in the AmazonDynamoDBPrimingResource class.

@Singleton
public class AmazonDynamoDBRequestPrimingResource implements OrderedResource
{

  @Inject
  private ProductDao productDao;

  @Override
  public void beforeCheckpoint(Context<? extends Resource> context) throws Exception {
    this.productDao.getProduct("0");
 }

 @Override
 public void afterRestore(Context<? extends Resource> context) throws Exception {
}

We use Lambda SnapStart CRaC runtime hooks here. To do this, we need to declare the following Micronaut CRaC dependency in pom.xml:

<dependency>
   <groupId>io.micronaut.crac</groupId>
   <artifactId>micronaut-crac</artifactId>
</dependency>

AmazonDynamoDBPrimingResource class is annotated with jakarta.inject.Singleton annotation and implements io.micronaut.crac.OrderedResource. The priming itself happens in the method where we search for the product with the ID equal to 0 in the DynamoDB table. beforeCheckpoint method is a CRaC runtime hook that is invoked before creating the microVM snapshot. We are not even interested in the result of the call productDao.getProduct("0"), but with this call all required classes are loaded and instantiated and the expensive one-time initialization of HTTP Client (default is Apache HTTP Client) and Jackson Marshallers (for the purpose of converting Java objects to JSON and vice versa) is carried out. As this is done during the deployment phase of the Lambda function when SnapStart is activated and before the snapshot is created, the snapshot will already contain all of this. After the fast snapshot restore phase during the Lambda invoke, we will gain a lot in performance in case of cold start by priming this way (see measurements below). We therefore prime the DynamoDB request.

To ensure that only this priming takes effect, please temporarily delete the following AmazonAPIGatewayProxyRequestPrimingResource class, otherwise additional priming will take place which we'll cover in the part next part.

Measurements of cold and warm start times of our application with Lambda SnapStart and DynamoDB request priming

In the following, we will measure the performance of our GetProductByIdFunction Lambda function, which we will trigger by invoking curl -H "X-API-Key: a6ZbcDefQW12BN56WEM49" https://{$API_GATEWAY_URL}/prod/products/1.

The results of the experiment are based on reproducing more than 100 cold starts and about 100,000 warm starts with the Lambda function GetProductByIdFunction (we ask for the already existing product with ID=1) for a duration of about 1 hour. We give Lambda function 1024 MB memory, which is a good trade-off between performance and cost. We also use (default) x86 Lambda architecture. For the load tests I used the load test tool hey, but you can use whatever tool you want, like Serverless-artillery or Postman.

We will measure with tiered compilation (which is default in Java 21, we don't need to set anything separately) and compilation option XX:+TieredCompilation -XX:TieredStopAtLevel=1. To use the last option, you must set it in template.yaml in JAVA_OPTIONS environment variable as follows:

Globals:
  Function:
    ...
    Environment:
      Variables:
        JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"

Please also note the effect of the AWS SnapStart Snapshot tiered cache. This means that in the case of SnapStart activation, we get the largest cold starts during the first measurements. Due to the tiered cache, the subsequent cold starts will have lower values. For more details about the technical implementation of AWS SnapStart and its tiered cache, I refer you to the presentation by Mike Danilov: "AWS Lambda Under the Hood". Therefore, I will present the Lambda performance measurements with SnapStart being activated for all approx. 100 cold start times (labelled as all in the table), but also for the last approx. 70 (labelled as last 70 in the table), so that the effect of Snapshot Tiered Cache becomes visible to you. Depending on how often the respective Lambda function is updated and thus some layers of the cache are invalidated, a Lambda function can experience thousands or tens of thousands of cold starts during its life cycle, so that the first longer lasting cold starts no longer carry much weight.

To show the impact of the SnapStart with DynamoDB request priming, we'll also present the Lambda performance measurements without SnapStart being activated from the part 1 and with SnapStart being activated but without applying the priming techniques as measured in the part 2.

Cold (c) and warm (w) start time with tiered compilation in ms:

Scenario Number	c p50	c p75	c p90	c p99	c p99.9	c max	w p50	w p75	w p90	w p99	w p99.9	w max
No SnapStart enabled	4948	5038	5155	5387	5403	5404	5.37	6.01	7.10	16.01	52.05	1535
SnapStart enabled but no priming applied, all	1926	1981	3213	3232	3242	3245	5.33	5.96	6.93	14.43	38.76	2617
SnapStart enabled but no priming applied, last 70	1900	1959	2001	2063	2063	2063	5.29	5.91	6.93	14.66	37.84	1588
SnapStart enabled and DynamoDB request priming applied, all	743	787	879	1300	1798	1798	5.42	6.10	7.27	14.90	36.08	1095
SnapStart enabled and DynamoDB request priming applied, last 70	730	787	878	1301	1301	1301	5.37	6.10	7.33	15.02	33.85	433

Cold (c) and warm (w) start time with -XX:+TieredCompilation -XX:TieredStopAtLevel=1 compilation in ms:

Scenario Number	c p50	c p75	c p90	c p99	c p99.9	c max	w p50	w p75	w p90	w p99	w p99.9	w max
No SnapStart enabled	4993	5145	5392	5697	5852	5856	5.33	5.91	6.88	15.50	52.47	1616
SnapStart enabled but no priming applied, all	1895	1947	2025	2154	3368	3369	5.55	5.82	6.72	14.86	104.68	2609
SnapStart enabled but no priming applied, last 70	1891	1923	1989	2066	2066	2066	5.13	5.73	6.61	14.17	35.01	1637
SnapStart enabled and DynamoDB request priming applied, all	696	755	1611	1632	1632	1632	5.21	5.92	7.16	15.09	43.72	952
SnapStart enabled and DynamoDB request priming applied, last 70	663	693	759	826	826	826	5.13	5.82	7.05	14.39	35.57	370

Conclusion

In this part of the series, we introduced how to apply Lambda SnapStart priming techniques by starting with DynamoDB request priming with the goal to further improve the performance of our Lambda functions. We saw that by doing this kind of priming by writing some additional simple code we could significantly further reduce (depending on the percentiles by more than 50%) the Lambda cold start times compared to simply activating the SnapStart. Moreover we could significantly reduce the maximal value for the Lambda warm start times by preloading classes (as Java lazily loads classes when they are required for the first time) and doing some pre-initialization work (by invoking the method to retrieve the product from the DynamoDB table by its ID) which will only happen once during the first warm execution of the Lambda function.

We also saw that choosing the -XX:+TieredCompilation -XX:TieredStopAtLevel=1 java compilation led to much lower lower cold start times compared to the tiered compilation for this type of priming.

We also clearly observed the impact of the AWS SnapStart Snapshot tiered cache in our measurements.

In the next part of our article series, we'll introduce another Lambda SnapStart priming technique which is API Gateway Request Event priming. We'll then measure the Lambda performance by applying it and compare the results with other already introduced approaches.