Creating S3 Object Lambda with CDK for C#

The moment I learnt that S3 Object Lambda was out, I knew I’d want to experiment with it. Why? For two reasons really – at work we have quite a few scenarios when the same objects in S3 need to be presented in different shapes or forms, data extracted, or content transformed. So I’ve volunteered to speak about it and its use cases. The second reason is to practise using AWS CDK for C# more – I’ve mentioned a few times that there are very few examples in C#, and I thought it’s be a good idea to provide one more and hopefully make someone’s life easier.

In a nutshell, S3 Object Lambda allows you to amend the data that you usually get by using S3 Get requests. The main characters in the story are:
An S3 Bucket where we drop files we want to transform:

var bucket = new Bucket(this, "xmlBucket", new BucketProps
{
    BucketName = "oxies-xml-bucket",
    RemovalPolicy = RemovalPolicy.DESTROY
});

A Lambda function which will do the transformation. My Lambda function does a simple XML transformation:

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
namespace TransformXML.Lambda
{    public class Handler
    {
        protected async Task<HttpResponseMessage> Transform(JObject request, ILambdaContext context)
        {
            try
            { 
                var s3Client = new AmazonS3Client();

                var input3Url = request["getObjectContext"]["inputS3Url"].ToString();
                var reqRoute = request["getObjectContext"]["outputRoute"].ToString();
                var token = request["getObjectContext"]["outputToken"].ToString();                

                using var httpClient = new HttpClient();
                var original = await httpClient.GetAsync(input3Url);

                var content = await original.Content.ReadAsStringAsync();

                var receivedXml = XDocument.Parse(content);
                var transformedXml = new XElement("article", receivedXml.Root.Element("body").Value);

                var toSend = new WriteGetObjectResponseRequest()
                {
                    Body = ToStream(transformedXml),
                    RequestRoute = reqRoute,
                    RequestToken = token
                };
                var response = await s3Client.WriteGetObjectResponseAsync(toSend);
            }
            catch (Exception ex)
            {
                context.Logger.Log($"ERROR: {ex.Message}; {ex.StackTrace}");              
            }
            return new HttpResponseMessage() { StatusCode = System.Net.HttpStatusCode.OK };            
        }

        private Stream ToStream(XElement onlyBodyXML)
        {
            return new MemoryStream(Encoding.UTF8.GetBytes(onlyBodyXML.ToString()));
        }
    }
}

What is worth noting here is that:

• The request that is sent from the Lambda contains getObjectContext property that contains inputs3Url which you use to get the original object
• You need to use a new WriteGetObjectResponseRequest method that is used to include the transformed content. WriteGetObjectResponseAsync sends the transformed object when Object Lambda Access Points are called.
• RequestToken allows the Lambda to connect the response with the caller.

Before we start looking at the main part of the CDK stack, I’m going to use namespace aliases to save some typing:

using S3ObjectLambdaCfnAccessPoint = Amazon.CDK.AWS.S3ObjectLambda.CfnAccessPoint;
using S3ObjectLambdaCfnAccessPointProps = Amazon.CDK.AWS.S3ObjectLambda.CfnAccessPointProps;

We then define the resources in the stack:

var function = new Function(this, "XMLTransformBody", new FunctionProps
{
    Runtime = Runtime.DOTNET_CORE_3_1,
    Code = Code.FromAsset("./TransformXMLLambda/bin/Release/netcoreapp3.1/publish"),
    Handler = "TransformXML.Lambda::TransformXML.Lambda.Handler::Transform",
    FunctionName = "XMLTransform",
    Timeout = Duration.Minutes(1)
});

We also need to give the Lambda execution role appropriate permissions like so (if you don't, ERROR: Forbidden will be returned from WriteGetObjectResponseAsync):

var policy = new PolicyStatement(new PolicyStatementProps
{
    Effect = Effect.ALLOW,
    Actions = new[] { "s3-object-lambda:WriteGetObjectResponse" },
    Resources = new[] { "*" }
});

function.AddToRolePolicy(policy);

And finally, the S3 Object Lambda Access Point. This is the access point that should be used in the application when making a GetObject request:

var objectLambdaAccessPoint = new S3ObjectLambdaCfnAccessPoint(this, "S3ObjectLambdaAccessPoint", new S3ObjectLambdaCfnAccessPointProps
{
    Name = "transformxml",
    ObjectLambdaConfiguration = new S3ObjectLambdaCfnAccessPoint.ObjectLambdaConfigurationProperty()
    {
        CloudWatchMetricsEnabled = true,

        SupportingAccessPoint = supportingAccessPoint,

        TransformationConfigurations = new object[]
        {
            new S3ObjectLambdaCfnAccessPoint.TransformationConfigurationProperty()
            {
                Actions = new string[] { "GetObject" },

                ContentTransformation = new Dictionary<string, object>()
                {
                    { 
                        "AwsLambda", new Dictionary<string, string>()
                        {
                            {"FunctionArn", function.FunctionArn }
                        } 
                    }
                }
            }
        }
    }
});

A few things here:

• The Actions array will always have only one element "GetObject" since it's the only operation supported with the Object Lambda.
• With CDK in C# you need to use a Dictionary when there are Javascript arrays or untyped objects. I must say this bit is so much simpler in Typescript!

Here are some other findings:

1. This error occurred when I changed the name of the Access Point to contain some capital letters or hyphens. So I just left it in lowercase.
2. In the ContentTransformation container you can also send FunctionPayload, and customize the behaviour of the function based on that payload.
3. To use the Object Lambda Access Point all you need to do is to replace the BucketName value of GetObjectRequest with the ARN of the Object Lambda Access Point:

GetObjectRequest request = new GetObjectRequest
{
    BucketName = "arn:aws:s3-object-lambda:us-east-1:<account-id>:accesspoint/transformxml",
    Key = "example.xml"
};

The full example is in my Github repo
It took me a while to build the stack since I’ve not worked with L1 Constructs much before. A huge thanks to Petra Novandi and the CDK team for giving me a hand, helping me to learn how S3 Object Lambda works, improving my knowledge of the CDK and enabling me to share my learnings with the world.

Useful resources:
Transforming objects with S3 Object Lambda
Demo - S3 Object Lambda | AWS Events
Working with CDK in C#