Lambda Unleashed: Mastering Massive Data Responses

#aws #middleware #serverless #node

In the realm of serverless architecture, AWS Lambda has emerged as a cornerstone, offering scalability, flexibility, and efficiency. However, even in this advanced cloud environment, we encounter certain constraints — one being the limit on Lambda response sizes, traditionally capped at 6MB. At first glance, this seems reasonable. After all, APIs are generally not designed to handle or return voluminous amounts of data. The principle of keeping responses lean and efficient underlines much of our best practices in API design.

But what happens when the unexpected occurs? When, despite our best efforts and designs, the need arises for Lambdas to return larger payloads? It's not always about wanting to push the boundaries; sometimes, it's about adapting to them. In this blog post, I aim to delve into the nuances of scenarios where larger Lambda responses are not just beneficial but necessary.

Large Response Middleware

This middleware allows a service to log and save large responses to an S3 bucket, enabling developers to investigate the causes of such large responses. Furthermore, this middleware accepts a special header that allows the rewriting of the response with a $payload_ref pointing to the large payload stored in S3, enabling clients to recover gracefully.

Use Case

An internal application of this middleware at epilot involves a service that constructs a Programmatic Runtime containing a large {} variable context. This context is then used to generate sophisticated Document Templates. Our clients leverage these templates to dispatch Emails or Attachments to their end-customers. Although the resulting context may be substantial, its necessity is unquestionable for thorough exploration and real-time evaluation of the compiled template against the context.

Limitations

This work is a great all-in-one solution for one specific lambda use case:

API Gateway with Lambda Proxy Integration

Although, while we at @epilot use this pattern a lot, it's by no means the only way Lambda is used.

The same 6MB response limit applies in all Lambda use cases. Just to give some examples that are not covered by this middleware:

Lambda function URLs
Lambda in Step Functions
Direct Invoke via Lambda API
API Gateway Lambda service integration
Lambda@Edge

Since the 6MB limit applies anywhere, we could build and publish something more generic that works for all Lambda use cases (not just this one).

The good news for the community is that, we've an open issue for that aws-lambda-utility-middlewares/issues/1. And we may put some more work on it during the near future. Contributions are also more than welcomed 🙌

How it works

For handling a Large Response, clients must request with the HTTP Header Accept: application/large-response.vnd+json. This custom MIME type signals consent to receive a large response payload when required. The format of the response body for this MIME type is:

{
  "$payload_ref": "http://<s3 file reference link>"
}

Should the client specify this MIME type, the Lambda bypasses logging an error with Log.error. Instead, it restructures the original response, incorporating a reference to the substantial payload now in S3. The modified response also carries the HTTP header Content-Type set to application/large-response.vnd+json.

Absent this MIME type, the Lambda logs an error using Log.error, leading to a response failure due to the oversized body.

Configuration Options

Supported Parameters:

Parameter	Type	Description
thresholdWarn	`number`	Warning threshold level (percentage of `sizeLimitInMB`), e.g: 0.80
thresholdError	`number`	Error threshold level (percentage of `sizeLimitInMB`), e.g: 0.90
sizeLimitInMB	`number`	Maximum allowed size limit in MB, e.g 6
outputBucket	`string`	Identifier or name of the output S3 bucket
groupRequestsBy	`function - mapper`	Function to group requests, based on API Gateway event V2. Defaults to 'all'

Example Usage:

withLargeResponseHandler({
  thresholdWarn: 0.85, // 85% of the limit = 5.1MB
  thresholdError: 0.9, // 90% of the limit = 5.4MB
  sizeLimitInMB: 6,
}),