Find out how I used popular open-source tools to build a serverless microservice template, which significantly streamlines testing.
Using practical examples, learn how to apply hexagonal architecture principles to increase the testability and maturity of the code. See how you can utilize design patterns and other techniques that you've been using for years. You don't need to forget about them when switching to serverless!
On the contrary, based on my experience, they still have a place in modern solutions as they increase readability, maintainability, and testability of the source code.
If you think they are better ways to program than hackish all-in-one-single-file-lambda-functions then you will love ❤️ what I've prepared for you.
Sounds too good to be true?
Wait, there is more! 😎
Automated integration and end-to-end (e2e) tests significantly streamline developer's workflow. Finally, you can break out from: code -> deploy -> manually invoke Lambda -> check logs -> fix bugs -> repeat cycle!
But that's not all!
THE BEST PART: this template is available on GitHub for free 😃
You can use it right now!
Before I'll explain the solution, allow me to summarize common problems that lead me to the creation of this template.
Fallacious simplicity of the Lambda function
Every developer taking his first steps in serverless already has a lot of experience gained on previous projects. Most often this means that he created monolithic applications in heavy languages (such as Java or C#). Of course, some people already have a lot of experience in microservices, but these are still bigger components than Lambda functions.
When switching to serverless and scripting languages such as JavaScript or Python people tend to explore the freedom offered by these technologies. There is nothing wrong with experimenting and playing around. Alas, all too often I speak with people who have used the hackish (all code in single file Lambda) approach on production services, and now they suffer from poor maintainability and lack of tests.
It is very tempting to implement the Lambda function in just several lines. Unfortunately, in a long run, it doesn't pay off.
Lack of tests
The direct effect of hackish implementation is poor testability. Monolithic code is really hard to test, so people don't write any tests. Simple as that. The repercussions of not having tests are rather obvious for seasoned developers, so I will not explore that topic here.
However, some people do test their serverless applications. They write automated unit tests for business logic or boilerplate parts that operate on AWS services using mocks.
While mocks are not bad (I use them myself) you need to know when you should apply that technique. And more importantly, when not 😉
Mocking all AWS services is not going to give you any warranties that your code will work when deployed into the cloud. Mocks give you a false sense of trust. That also applies to the localstack and similar tools that emulate AWS in Docker.
Think about why we test?
In my opinion because of two reasons:
- to have a trust that our code behaves as we think it does
- to protect ourselves from regression bugs after new changes are introduced
An automated test suite will give us immediate feedback that something is wrong with our code.
Lack of it will force us to do manual testing after every change, or you can be brave and just deploy it to the prod. I was just kidding, please don't do it 🤣
I don't have anything against manual testing, but it doesn't scale, requires knowledge of the system (ie. new joiner will not know how/what to test), and is slow. Also, you can't enforce it. I mean you cannot run manual tests in CI/CD pipeline.
There is also one more annoying thing. All too often in projects without tests or with bad tests, I hear my colleagues say "...but it worked locally on my machine". I really don't care! 😤
As a programmer and a person who takes responsibility for providing a working solution, which is free of bugs, I need to write code easy to test and maintain. And to have certainty that it works on the prod in the cloud, not on someone's else laptop.
Solution: How to test serverless?
To address the problems stated above I have prepared a highly opinionated project template for Serverless Framework that applies hexagonal architecture principles to the serverless world.
The template project was created with two goals in mind: streamlined developer's workflow and easy testing because, sadly, both are not common in serverless development yet.
The template is available on GitHub under serverless-hexagonal-template name.
How to use it?
To start using it, you need to create your new project from that template:
sls create --template-url https://github.com/serverlesspolska/serverless-hexagonal-template --name <your-project-name>
This command will create your new project. Given that you have Serverless Framework, if not install it: npm i -g serverless. More information in docs on GitHub.
Testing approach
All boilerplate configuration of the jest testing framework, plugins and other open-source tools is included. New project if fully configured and ready to be deployed.
The template contains two sample Lambda functions and a set of:
- unit tests
- integration tests
- end-to-end (e2e) tests.
This division was introduced because different types of tests fulfill different needs.
Unit tests are executed locally (on the developer's computer or CI/CD server) and don't require access to any resources in the AWS cloud or on the Internet.
However, integration and e2e tests require real AWS services deployed in the cloud. Therefore, before starting those you need to deploy the project by executing sls deploy.
Integration tests
Afterward, when triggering npm run integration to start integration tests special Serverless Framework plugin (serverless-export-env) is connecting to the AWS account and saves locally in .awsenv file all Lambda's environment variables.
stage=dev
region=eu-central-1
service=serverless-hexagonal-template
httpApiGatewayEndpointId=qjc1234u19
tableName=serverless-hexagonal-template-dev
message=Hello World!
Sample contents of .awsenv file.
Next, values from that file are injected into jest test context. That means that whenever your code refers to, let's say, process.env.MY_ENV_VAR during tests it will be resolved with the same value as it was running inside the Lambda function in the cloud.
In this way, the code of the application (microservice) can be tested locally while using real resources in the cloud. The best thing is that when writing clean code in accordance with a hexagonal architecture, the implementation code is not aware of the test context. You don't have to add any special things to it to make it testable (That would be ugly, wouldn't it?)
Automated jest test suites are executed locally. They test your local files against resources in the cloud. For example, in serverless-hexagonal-template, I implemented tests that use the DynamoDB table. Source code available here & here.
The other test (source code) focuses on AWS API Gateway and Lambda function integration. This is a huge deal, as serverless solutions heavily depend on multiple resources in the cloud. Many errors origins from the wrong configuration. Having such integration tests allows us to test this area.
const { default: axios } = require('axios')
axios.defaults.baseURL =
`https://${process.env.httpApiGatewayEndpointId}.execute-api.${process.env.region}.amazonaws.com`
describe('createItem function', () => {
it('should respond with statusCode 200 to correct request', async () => {
// GIVEN
const payload = {
a: 10,
b: 5,
method: 'add'
}
// WHEN
const actual = await axios.post('/item', payload)
// THEN
expect(actual.status).toBe(200)
})
...
Snippet of an integration test.
Integration and service configuration problems are the main drivers behind changes regarding how the industry looks at testing practices.
On the left classic test pyramid. On the right honeycomb proposed by Spotify.
Therefore, I put so much emphasis on integration testing as it is simply more important in serverless applications.
To be honest, it's not just serverless. In every distributed system, unit testing is just not enough.
End-to-end tests (e2e)
Sometimes integration tests are not enough, as we need to test the whole chain of communication between a set of components.
An example of such a test would be a POST request sent to API Gateway /item endpoint and a check if processItem Lambda function was triggered by DynamoDB Streams as a result of saving new item by createItem Lambda function invoked by the request. Such an approach tests the chain of events that happen in the cloud and gives confidence that integration between multiple services is well configured.
These chains of events are of course nothing more than Event Driven Architecture in practice. These are what make the cloud-native approach powerful. This also explains why use of localstack and similar solutions it risky. There is no guarantee that these integrations work locally as they do in AWS.
By the way, e2e tests work great when you want to test code that connects to RDS databases that cannot be accessed from outside of the VPC.
Hexagonal architecture
It naturally introduces order into our code, as the division into independent modules becomes intuitive. It allows for better separation of problems and makes it easier to write code compliant with the Single Responsibility Principle (SRP). These are key features of an architecture that is easy to maintain, extend and test.
The selection of this particular architecture style is coupled with the proposed project directory structure and naming conventions. You can read more on those in the documentation.
Suffice to say, it defines where what should be placed (i.e. source code in src/ folder, tests in __tests__/ etc.) so you don't need to waste time thinking about it every time you start a new project and creates a common language for your team members. Thus, decreasing cognitive overload when switching between projects started from this template.
How did I create the template?
The template has been worked out as a result of years of development in the Lambda environment using Serverless Framework. It also takes from the collective experience of the community (to whom I am grateful) embodied in books, talks, videos, and articles.
I was fed up with the poor developer's workflow in serverless:
- write code
- deploy
- manually invoke Lambda
- check logs
- fix bugs
- repeat
It's very unproductive!
I decided that I want to fix this problem. I focused on testing because I knew that solving it, will allow me to work in a much more mature way. Many years ago I was a Java developer and I knew that a developer's flow can be much better.
I spent many evenings reading about testing serverless and experimenting. Fortunately, I was using hexagonal architecture for a while now, so it was easy for me to think about testing in the context of singular code components, and not whole Lambda functions. Eventually, I found some articles about the serverless-export-env plugin which was the missing link that allowed me to tight everything together in an easy, automated way. That was of paramount importance for me. I knew that process must be simple and fully generic, so I could use it on any project.
When I started using this approach I immediately noticed how much my development workflow was improved. Finally, I could make changes on the fly!
I was able to write 70 to 90 percent of a code without constant re-deployments. That was a HUGE improvement! In some cases, I used TDD (Test Driven Development) which is simple to do in this setup.
After implementing several microservices using this approach, I was confident that this method worked. I decided that I want to share this approach with the community. I love to help people around the world build and ship projects using awesome serverless technology and help them learn and become better developers. It was a logical thing to do. 😎
However, instead of writing a plain article, I decided to create a Serverless Framework template that embodied all the things and practices that I knew, so everyone could start using it immediately and gain all described benefits without the hassle.
Why should you use it?
In a nutshell, using serverless-hexagonal-template will give you:
- Production-ready scaffolding of a serverless microservice
- Greater confidence in your solution (tests!)
- Efficient and repeatable developer's workflow
- Well-thought-out project structure
- Increased code reusability
- Clean code and mature design - use of patterns and good practices that you learned over the years
- Possibility to run tests in CI/CD pipelines.
Also:
- No more constant re-deployments to test code
- No more manual testing
- No more hackish single-file implementations
- No more regression bugs
- No more it worked on my computer excuses 😉
I experienced this firsthand
My journey with serverless has been going on since 2016. Before I started using this approach I had many projects with unit tests or without tests at all. It was hard to add new functionalities to them without breaking existing things or at least being afraid of such eventuality. Unit tests simply were not enough. Every change had to be deployed and manually tested.
Nowadays, implementing and modifying projects is a completely different story. The addition of integration and e2e tests allowed me to gain confidence whenever I introduce changes. My workflow is not interrupted anymore by project deployments to the cloud. Of course, they are still needed but almost everything can be tested as soon as you provision resources for the first time and define environment variables.
In summary, this saves a lot of time and makes the developer's life easier.
Try it out!
So, if you want to have an awesome developer's flow and well-tested solution give it a try. It will take you no more than 5 minutes.
- Create your project from a template
sls create --template-url https://github.com/serverlesspolska/serverless-hexagonal-template --name <your-project-name> - Install dependencies
npm i - Unit test
npm run test - Deploy to the cloud
sls deploy - Run integration tests
npm run integration - Run end-to-end tests
npm run e2e
Then analyze my code and learn how to test serverless applications. Start using this template in your projects, and give me a star ⭐️ on GitHub: serverless-hexagonal-template. Good luck and happy testing!
Top comments (4)
Do you have a youtube video that shows this implementation in action?
Hi @redeemefy I only have recordings of my presentations in Polish.
Can I help you in other way? What would you like to know?
I see that the deploy command is
sls deploy .and you are not specific for what stage you are deploying. Are you using an.envfile to do that?Thank you for your patience. In this article,
sls deploydeploys service (your project) to thedevstage, which is defined in thedeployment.ymlfile. The.envfile is not used at all here.However, since writing this article, I have updated the project on GitHub and right now
sls deploydeploys the service to adevstage which is dedicated to a particular developer. It uses your username taken from your computer and createsservice-dev-usernamedeployment. In that way, many developers can work in parallel on a single project.More on that can be found in the documentation, section Deployment
Though, you can still deploy to other stages by explicitly providing stage value
sls deploy -s <stage> # stage = dev | test | prod