My motivation for benchmarking all the compiler, deployment, and runtime options has been to feed my curiosity. I've seen various blog posts recommending one setting over another, but they never provided a justification.
Unfortunately, we can't use the outstanding BenchmarkDotNet tool with AWS Lambda. So, I built a benchmarking harness to collect data for all the deployment options, and hopefully, determine an "optimal" combination.
As the first blog post of this series mentioned, there are two distinct cases we can optimize for: "Minimize Cold Start Duration" or "Minimize Operating Cost". Now that the groundwork has been laid, we can formally capture what that means.
Optimal Strategies
To Minimize Cold Start Duration, we need to minimize the INIT and the first INVOKE phase. The optimal configuration yields the lowest duration, measured in milliseconds (ms), to process the request. For this measurement, we rely on the data reported by AWS Lambda in the logs. This is the same data that is used for billing and is the most accurate we have access to.
To Minimize Execution Cost, we need to minimize the sum of all INVOKE phases (cold and warm) while taking into account the Lambda memory configuration and CPU architecture. The optimal configuration yields the lowest execution cost for 1 cold start followed by 100 warm invocations. AWS Lambe execution has an extremely low unit cost. To make the number more intuitive, I opted to report execution cost in millionth of a dollar, or micro-Dollars (ยต$).
Benchmarked Options
To leave no stone unturned, the benchmarking harness executes all possible permutations of the following options.
- Tiered Compilation: On and Off
- ReadyToRun: On and Off
- Lambda Memory: 128 MB, 256 MB, 512 MB, 1024 MB, 1769 MB, and 5120 MB
- CPU Architecture: x86-64 and ARM64
- .NET Runtime: .NET Core 3.1 and .NET 6
- Pre-JIT .NET: On and Off
These options produce 192 unique combinations that are benchmarked.
Benchmarking Approach
The Lambda function for each project is measured performing 100 cold starts. Each cold start is followed by 100 warm invocations. The results are then averaged.
I debated using the median or a percentile value instead of the average value. The challenge is when these values are further combined. For example, summing the p99 value for the INIT phase with p99 value of the first INVOKE phase doesn't seem right. Furthermore, outliers happen in real life. I'm hoping that 100 cold and warm invocations are sufficient to fairly represent what should be expected in real world situations.
That said, the raw measurements have been captured for each benchmark. That way, alternative analyses can be done without having to collect the data again.
Benchmarked Projects
My interest was in studying how the options are impacting the compute aspect of Lambda functions. Therefore, I opted to only benchmark projects that do not perform I/O operations.
Minimal Baseline
The Minimal project establishes a baseline for all projects. It has no business logic and only includes required libraries.
JSON Serializers
JSON serialization is necessary for virtually all Lambda functions. With .NET 6, there are 3 common approached to handle this task.
NewtonsoftJson: using Newtonsoft JSON.NET
SourceGeneratorJson: using .NET 6 source generators for JSON parsing
SystemTextJson: using System.Text.Json
AWS SDK
Most Lambda functions will interact with other AWS services via the AWS SDK. The AwsSdk project is used to benchmark the cost of initializing the SDK.
Top-Level Statements
Available starting in .NET 6, Lambda functions can use top-level statements instead of declaring a class. What is the performance impact when doing so?
AwsNewtonsoftJson: using AWS .NET SDK and Newtonsoft JSON.NET
SampleAwsNewtonsoftTopLevel: using AWS .NET SDK, Newtonsoft JSON.NET, and Top-Level Statements
SampleAwsSystemTextJsonTopLevel: using AWS .NET SDK, System.Text.Json, and Top-Level Statements
Minimal API
Also new in .NET 6 is a new approach to express ASP.NET routes using top-level statements. This sample was taken from the .NET 6 support on AWS Lambda announcement blog post.
- SampleMinimalApi: using ASP.NET Core Minimal API
Benchmark Explorer
The results from the benchmarks have been compiled into an interactive Google spreadsheet. Feel free to explore the data anyway you like and draw your own conclusions. Any feedback on improving the analysis or visualization is welcome.
What's Next
Finally, we can dive into the results and see what new insights we can gain! First up is the baseline performance measurement. While this is not critical for production code, it gives us a foundation to work on.
Top comments (0)