How I Discovered That the AWS Lambda Runtime Might Be Written in Go

Have you ever wondered how your Lambda function code is triggered?

Each runtime has to conform to the AWS Lambda runtime API.

In managed runtimes, you don't need to take care of that; that's the whole point of using a managed runtime! AWS takes care of that for you.

But how?

It's hard to say because it's not clearly documented. However, while I was conducting some unrelated experiments, I made a very cool discovery!
I think you'll find it interesting too!

If you're working with Go and care about performance, you might want to look at some important metrics, such as:

• the number of allocations
• the time taken for each init() functions

To do so, you need to setup a environment variable:

GODEBUG to inittrace=1.

And tada 🎉!

In addition to information about the binary I wanted to test, I'm now able to see traces from another binary, which seems to be the one from AWS that conforms to the runtime API.

Here is an example of the output 👀👀👀

init internal/bytealg @0.008 ms, 0 ms clock, 0 bytes, 0 allocs
...
init go.amzn.com/lambda/fatalerror @3.7 ms, 0.031 ms clock, 336 bytes, 2 allocs
...
init go.amzn.com/lambda/interop @6.5 ms, 0.023 ms clock, 1128 bytes, 32 allocs
init go.amzn.com/lambda/telemetry @6.5 ms, 0.067 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/core @6.6 ms, 0.007 ms clock, 224 bytes, 12 allocs
init go.amzn.com/lambda/rapi/rendering @6.7 ms, 0.086 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/rapi/handler @6.8 ms, 0 ms clock, 256 bytes, 2 allocs
init go.amzn.com/lambda/rapid @6.8 ms, 0.099 ms clock, 16 bytes, 1 allocs
...
INIT_START Runtime Version: nodejs:18.v9 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:7d5f06b69c951da8a48b926ce280a9daf2e8bb1a74fc4a2672580c787d608206

Here, we can clearly see some explicit package names coming from AWS.

Another important thing to note is that those logs are output BEFORE the INIT_START log line, which provides further evidence that it might be how AWS Lambda is managing your code!

Voila! 🎉
What do you think?

You can find me on LinkedIn and Twitter!

Comments

[Gernot Glawe]

That is a very remarkable finding. I had to test it right away. You're right.
So all Lambda functions rely on Rust as a base for the micro-VMS and on top of that, go as a Lambda starter.
I love that this shows optimal use cases for Rust and GO, IMHO

[Maxime David]

Thanks for your kind words!

[Maxime David]

wow this is amazing! Thanks a lot for sharing!
however I cannot reproduce it, you're selecting python3.11 and deploy a hello world function in eu-central-1 + set the RUST_BACKTRACE env variable to 1 right?

[Doug]

I need to push them to use Rust.

[Gernot Glawe]

Why? The current approach seem to work quite well!

[Maxime David]

It may be possible to go even faster with a runtime api implementation written in Rust (smaller binary, faster to start)

[Maxime David]

<3 please do that!

[Manuel]

Good read, but are you sure of these?

It's worthy to note that AWS Lambda uses Firecracker for provisioning and running secure sandboxes to execute functions. aws.amazon.com/about-aws/whats-new....

Firecracker, in turn, is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services. firecracker-microvm.github.io/, built by AWS and written in Rust.

[Maxime David]

Firecraker is indeed the underlying VM used by Lambda Functions, but we still need code to conform to the Lambda Runtime API (docs.aws.amazon.com/lambda/latest/...)
I think we're talking about two different topics, WDYT?

[reliability-architect]

If it is easy enough, can we prove that runtime for other languages is also Go? for python or Java?

[Maxime David]

I've just done the test with java17 go1.x and python3.11, all are printing those init info as well

[Neel]

Hi Maxime,

First of all, it's a good and brief read

Just for clarity, the logs you attached above, are those from a Node JS based lambda function?

[Maxime David]

Yes!

[Adrian B.G.]

Congrats, that's a nice finding. But I hope you realized before making it public it also raises a security risk for AWS as by identifying what libraries are used attackers can identify to which vulnerabilities are exposed. Basically this gives access to a large part of source code.
IMO you should have report this to AWS first, get a bounty reward and then make it public.

[Maxime David]

Thanks, I'm not sure since the package name seems to align with what is already open source here: github.com/aws/aws-lambda-runtime-...

[raddevus]

Hey, that’s a very cool catch. Way to stay alert and think outside the box and figure things out. That’s a very cool mindset.

Thanks for writing this one up. Very interesting.
If you get a chance, would you mind taking a look at my latest article here on dev.to? Software Developer, Are You Just A Hammer?

I think the kind of mindset it took you to discover this about AWS is a similar mindset to what I’m talking about in my article.

[Maxime David]

Thanks @raddevus! Glad you liked it!