How I Write Meaningful Tests for AWS Lambda Functions

A Harsh Truth

If you are going to write meaningless unit tests that are more likely to mask errors than expose them, you are better off skipping the exercise altogether.

There, I said it.

Your time is precious and could be spent on better things than achieving a hollow coverage percentage.

Effective testing of code has long been a challenging problem in programming, and newer tools like AWS Lambda seem to bring out the worst in developers when it comes to writing tests.

I think the main reason for this is that it’s more difficult (or at least less intuitive) to mirror the Lambda production environment locally. And as a result, some developers abstain from testing entirely.

I know because I’ve done it myself, even for projects in production. Instead, testing was done integration-style only after code was already deployed to the cloud.

This is extremely manual and wastes time in the long run. Another approach I’ve seen results in tests that look something like this:

This is the unmistakeable sign of an engineering team with a test coverage requirement but a lack of accountability. And no explanation is needed that the above is a no-no.

So, how do we go about transforming the sad test_lambda_function.py file above into something meaningful?

Before we can dive right into testing our Lambda code, there are a couple hurdles in the way. We’ll cover each of these individually and determine how to best handle them. Once dealt with, we are then free to test Lambdas to our heart’s content!

Note: I’ll be including small snippets of code throughout the article for clarity. But at the end there will be a full working code example to reference.

Hurdle #1: The Lambda Trigger Event

Every Lambda function gets invoked in response to a pre-defined trigger that passes specific event data into the default lambda_handler() method. And your first task for effectively testing a Lambda function is to create a realistic input event to test with.

The format of this event depends on the type of trigger. As of the time of writing there are 16 distinct AWS services that can act as the invocation trigger for Lambda.

Below is a code snippet with several examples of inputs that I most commonly use:

The full list of sample input events can be found in the AWS documentation. Alternatively, you can also print the event variable in your lambda_handler code after deploying and view the payload in CloudWatch Logs:

Once you have that example, simply hardcode it in your test file as shown above and we’re off to a fantastic start!

Next up…

Hurdle #2: AWS Service Interactions

Almost inevitably, a Lambda function interacts with other AWS services. Maybe you are writing data to a DynamoDB table. Or posting a message to an SNS topic. Or simply sending a metric to CloudWatch. Or a combination of all three!

When testing it is not a good idea to send data or alter actual AWS resources used in production. To get around this problem, one approach is to set up and later tear down separate test resources.

A cleaner approach though, is to mock interactions with AWS services. And since this is a common problem, a package has been developed to solve this specific problem. And what’s better is it does so in a super elegant way.

It’s name is moto (a portmanteau of mock & boto) and its elegance is derived from two main features:

1. It patches and mocks boto clients in tests automatically.
2. It maintains the state of pseudo AWS resources.

What does this look like? All that’s needed is some decorator magic and a little set up!

Say we read data from S3 in our Lambda. Instead of creating and populating a test bucket in S3, we can use moto to create a fake S3 bucket — one that looks and behaves exactly like a real one — without actually touching AWS.

And the best part is we can do this using standard boto3 syntax, as seen in the example below when calling the create_bucket and put_object methods:

Similarly, if we write data to DynamoDB, we could set up our test by creating a fake Dynamo table first:

It requires a bit of trust, but if the test passes, you can be confident your code will work in production, too.

Okay, but not everything is covered by moto…

Yes, it is true that moto doesn’t maintain parity with every AWS API. For example, if your Lambda function interacts with AWS Glue, odds are moto will leave you high and dry since it is only 5% implemented for the Glue service.

This is where we need to roll up our sleeves and do the dirty work of mocking calls ourselves by monkeypatching. This is true whether we’re talking about AWS-related calls or any external service your Lambda may touch, like when posting a message to Slack, for example.

Admittedly the terminology and concepts around this get dense, so it is best explained via an example. Let’s stick with AWS Glue and say we have a burning desire to list our account’s Glue crawlers with the following code:

session = boto3.session.Session()
glue_client = session.client("glue", region_name='us-east-1')
glue_client.list_crawlers()['CrawlerNames']
# {“CrawlerNames”: [“crawler1”, "crawler2",...]}

If we don’t want the success or failure of our illustrious test to depend on the list_crawlers() response, we can hardcode a return value like so:

By leveraging the setattr method of the pytest monkeypatch fixture, we allow glue client code in a lambda_handler to dynamically at runtime access the hardcoded list_clusters response from the MockBotoSession class.

What’s nice about this solution is that is it flexible enough to work for any boto client.

Tidying Up with Fixtures

We’ve already covered how to deal with event inputs and external dependencies in Lambda tests. Another tip I’d like to share involves the use of pytest fixtures to maintain an organized testing file.

The code examples thus far have shown set up code directly in the test_lambda_handler method itself. A better pattern, however, is to create a separate set_up function as a pytest fixture that gets passed into any test method that needs to use it.

For the final code snippet, let’s show an example of this fixture structure using the @pytest.fixture decorator and combine everything covered:

We’ve come a long way from the empty test file at the beginning of the article, no?

As a reminder, this code tests a Lambda function that:

1. Triggers off an sqs message event
2. Writes the message to a Dynamo Table
3. Lists available Glue Crawlers

By employing these strategies though, you should feel confident testing a Lambda triggered from any event type, and one that interacts with any AWS service.

Final Thoughts

If you’ve been struggling to test your Lambda functions, my hope is this article showed you a few tips to help you do so in a useful way.

While we spent a lot of time on common issues and how you shouldn’t test a Lambda, we didn’t get a chance to cover the opposite, yet equally important aspect of this topic--namely what should you test, and how can you structure your Lambda function’s code to make it easier to do so.

I look forward to hearing from you and discussing how you test Lambda functions!

Thank you to Vamshi Rapolu for inspiration and feedback on this article.