Writing a Lambda in Rust using aws-lambda-rust-runtime and aws-sdk-rust

Avi Illouz — Mon, 16 Aug 2021 23:44:35 +0000

Last week I had the opportunity to experiment with Rust and AWS lambda service. I learned to write an HTTP triggered lambda function written in Rust and compared it to a similar lambda function written in node.js.

What's the plan?

In short, I followed the READMEs on aws-lambda-rust-runtime and aws-sdk-rust to have a lambda that can handle HTTP events and get an object from s3. The main steps are:

Install rust toolchain (and if you're using mac also musl-cross so we can cross-compile things from mac to run natively on Amazon linux).
Follow awslabs aws-lambda-rust-runtime hello-http example to write a lambda function that can handle HTTP event.
Add awslabs aws-sdk-rust as a cargo dependency to use s3 client api.
Write lambda logic that receives an email, and gets the corresponding user's data from s3.
Wire up things together in AWS environment.
Compare initialization and execution duration with a similar node.js lambda function.

Install Rust

It's not covered in the examples but if you need to - Rustup is the recommended way to install Rust, and will allow us to easily add new build target.

Follow aws-lambda-rust-runtime README to compile an example and run it on AWS

This section just follows aws-lambda-rust-runtime README with small modifications because I wanted the hello-http example and added the aws-sdk-rust dependency:

added a new toolchain target:

rustup target add x86_64-unknown-linux-musl

Install and configure musl cross compiler

brew install filosottile/musl-cross/musl-cross
mkdir ~/.cargo
echo $'[target.x86_64-unknown-linux-musl]\nlinker = "x86_64-linux-musl-gcc"' > .cargo/config

Compile the hello-http example and wrap it for AWS

  cargo build -p lambda_http  --example hello-http --release --target x86_64-unknown-linux-musl

Rename and zip the executable to fit what AWS lambda custom runtime expects:

  cp ./target/x86_64-unknown-linux-musl/release/examples/hello-http ./bootstrap && zip lambda.zip bootstrap && rm bootstrap

Create a new lambda function in AWS. I did it using AWS Console, choosing "Custom runtime > Provide your own bootstrap on Amazon Linux 2" and uploading the zip file there.

Provide your own bootstrap on Amazon Linux 2" on AWS Console"/>

I've also created an API gateway so I can test it with HTTP requests (you can test your lambdas without this in AWS Console).

Add aws-sdk-rust s3 client as a dependency

aws-sdk-rust is a new AWS SDK for Rust that is under development and only "alpha" released. I've used just the s3 client from it, so all I had to add is:

[dependencies]
...
s3 = {git = "https://github.com/awslabs/aws-sdk-rust", tag = "v0.0.15-alpha", package = "aws-sdk-s3"}

I had an issue with compiling that I suspect is related to aws-sdk-rust/s3 using the ring crate somehow, and me trying to cross compile things for musl (both on mac and on linux desktops)

  error: failed to run custom build command for `ring v0.16.20`
  ...
  No such file or directory (os error 2)', /Users/user/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9

on mac, adding a TARGET_CC environment variable solved it for me (I saw it in some github issue but I can't find it now, on a linux machine the solution was to install the musl package)

  export TARGET_CC=x86_64-linux-musl-gcc

And, finally, it complies :)

  Finished release [optimized] target(s) in 2m 01s

Our Rust `get_user_data` lambda handler function

The code isn't that long so I can comment and go thorough it right here.

(note that it's a toy example without authentication, so it's not secure to use this with real data)

// lambda_http imports
use lambda_http::{

    // runtime related imports
    handler,
    lambda_runtime::{self, Context, Error},

    // imports that define the signature of our lambda
    IntoResponse, Request, RequestExt,
};

// used to calculate sha2 of user's email
use sha2::{Digest, Sha256};

// used to get user data from s3
use s3::Client;

#[tokio::main]
async fn main() -> Result<(), Error> {
    lambda_runtime::run(handler(get_user_data)).await?;
    Ok(())
}


// this is our lambda 
// get_user_data is a lambda that returns user data given it's email in query parameters (assuming the user authenticated somewhere else!)
// from the signature you can see that it handles `Request` objects and returns things that can turn `IntoResponse`
async fn get_user_data(event: Request, _: Context) -> Result<impl IntoResponse, Error> {

    // get email from query string params
    let params = event.query_string_parameters();
    let email = params.get("email").unwrap();

        // hash it and encode
    let hash = Sha256::new().chain(email).chain("some-salt").finalize();
    let hash = base64::encode(hash);

    // calculate key of s3 object with the hash above
    let key = format!("user-data/{}/some.json", hash);

    // use s3 API to get this object from s3
    let s3 = Client::from_env();
    let result = s3
        .get_object()
        .bucket("my-bucket")
        .key(key)
        .response_content_type("application/json")
        .send()
        .await?;

    // return the content as a response
    let data = result.body.collect().await?;
    let response = String::from_utf8(data.into_bytes().to_vec())?.into_response();

    Ok(response)
}

// TODO - handle errors
// TODO - do something smarter than from_utf8(data.into_bytes().to_vec())
// TODO - JWT authentication

// Please comment below with suggestions/feedback

Wire up things together in AWS environment

After you've compiled, zipped and uploaded the executable to aws, you should allow the lambda to access s3 (otherwise you'll get Access Denied responses when testing it).

What I did was to create my test bucket and objects on s3, and then from the lambda UI add a new role under "permissions", with a policy that gives S3 read access just for my test objects.

Comparing to node.js implementation

A similar logic in node.js would be something like:

//  a lambda that returns user data given it's email in query parameteres (assuming the user authenticated somewhere else!)
const S3 = require("aws-sdk/clients/s3");
const crypto = require("crypto");

exports.handler = async (event) => {
  const email = event.queryStringParameters.email;

  const s3 = new S3();
  const hash = crypto
    .createHash("sha256")
    .update(email)
    .update("some-salt")
    .digest("base64");

  const params = {
      Bucket: "my-bucket",
      Key: `user-data/${hash}/some.json`,
    }

  const data = await s3
      .getObject({
        Bucket: "my-bucket",
        Key: `user-data/${hash}/some.json`,
      })
      .promise();

  const data = data.Body.toString("utf-8");

  const response = {
        statusCode: 200,
        body: data,
    };
    return response;
};

AWS Lambda supports node.js runtime so you can actually create a new node function and edit the code directly in the console (you'll have to admit it's simpler than cross compiling Rust like we just did 🙃).

Running both lambdas with the same policies and test setup few times (with and without waiting for lambda to sleep between runs):

# each line is a new run.
# first run in each block is after few minutes of inactivity]
# followed by 4 consecutive runs

# Rust
Duration: 358.57 ms Billed Duration: 393 ms Memory Size: 128 MB Max Memory Used: 31 MB  Init Duration: 33.60 ms 
Duration: 39.76 ms  Billed Duration: 40 ms  Memory Size: 128 MB Max Memory Used: 31 MB  
Duration: 52.98 ms  Billed Duration: 53 ms  Memory Size: 128 MB Max Memory Used: 31 MB  
Duration: 49.17 ms  Billed Duration: 50 ms  Memory Size: 128 MB Max Memory Used: 31 MB  
Duration: 50.71 ms  Billed Duration: 51 ms  Memory Size: 128 MB Max Memory Used: 31 MB  

# node.js
Duration: 915.67 ms Billed Duration: 916 ms Memory Size: 128 MB Max Memory Used: 81 MB  Init Duration: 236.67 ms
Duration: 90.40 ms  Billed Duration: 91 ms  Memory Size: 128 MB Max Memory Used: 81 MB
Duration: 331.29 ms Billed Duration: 332 ms Memory Size: 128 MB Max Memory Used: 81 MB
Duration: 320.97 ms Billed Duration: 321 ms Memory Size: 128 MB Max Memory Used: 81 MB
Duration: 267.81 ms Billed Duration: 268 ms Memory Size: 128 MB Max Memory Used: 81 MB

If I get it right:

actual memory usage lower in rust (both ran on the smallest 128Mb runtime)
initialisation duration is lower in rust (maybe I'm doing something wrong in my node.js implementation?)
execution duration is lower in rust (although in some tests node-lambda get quite close)

Side note: In the toy example above we don't actually need to read the data inside the lambda, so a reasonable approach could be to pre sign url to the object and return just the url to the user:

const params = {
      Bucket: "my-bucket",
      Key: `user-data/${hash}/user.json`,
      Expires: 60
}  

const url = s3.getSignedUrl('getObject', params);

This improves execution time significantly, but it's not implemented yet in aws-sdk-rust. There is a github issue to track this feature, and there are other rust s3 clients that support it.

That's it - Thank you for reading, I am learning rust right now and would appreciate your feedback!

DEV Community: Avi Illouz