TLDR; A story about AWS app development frustrations and how I'd implement better testing practices in my SDLC for faster feedback (and a better experience).
A few years ago, I was briefly introduced to LocalStack while working as a DevOps Engineer on a platform built with several AWS-native microservices. Before my time, during the initial build, the team had developed a testing suite using LocalStack. It was out of date, and we never updated it, but in hindsight, I wish we had.
Development Challenges
The platform, built on AWS, used a mixture of serverless and managed services orchestrated through events and queues.
Naturally, this meant when making functional changes to the platform, we had to modify the payloads being exchanged between components. As a severely fat-fingered individual, every change of this nature required lots of testing.
Fortunately, we had sandbox environments that we could use to test our application during development.
Unfortunately, we only had two sandbox environments, which caused several issues:
- Only two engineers could actively develop and test their branches at once.
- A "dirty engineer" who didn't clean up resources after testing would cause problems for whoever came next.
- We had to wait for requests to pass through the system and dig through logs and traces to uncover behavioural events.
Essentially, our feedback loop was very slow. If our development process was an animal, it would be a snail.
As a result, much of our development time was spent submitting payloads and waiting for requests to process, only to discover the issue was a simple typo or an incorrect IAM policy.
This ad hoc approach to testing resulted in extended timelines and quite frankly, could just be annoying to deal with.
To mitigate bugs appearing in releases, we used what I would imagine is a pretty standard CI/CD pipeline.
Our CI/CD pipeline at least validated our deployments, but even with our pipeline, we were slow to release features because fundamentally, we relied too much on our pipeline - which had to build and deploy our end-to-end application into a live environment so we could test it.
The importance of fast feedback
I used to (and sometimes still do) fall into the trap of jankily getting a new feature over the line, which often resulted in more debugging time overall.
The further we get from our own machine, the longer it will take us to get feedback, essentially:
- On our own machine, it's fast to debug and uncover issues we can quickly action.
- In our CI pipeline, the runner executing our pipeline needs to load, dependencies need to be installed, and we need to wait for the results to become available for us.
- Lastly, in a live environment, we need to actively view our observability systems, looking at logs and traces to identify what the issue might be.
The more we can do on our local device, the faster the feedback and quicker (and less annoying) our development experience will be.
Where LocalStack fits in
LocalStack is an AWS, Snowflake (and soon Azure) service emulator that runs locally in a container, in a CI/CD pipeline, and now, also in LocalStack's own cloud. It's a commercial product, but the free edition works great in most cases.
Rather than creating a LocalStack tutorial, if you haven't used it before, you can reference these resources to learn more:
TLDR; When you start LocalStack, it exposes native cloud APIs, so any tool that works with native APIs should work with LocalStack. It doesn't just emulate the API, but also the service behaviour, allowing you to deploy emulated cloud-native resources (such as SQS Queues or Event Buses) that you can use with your application in the same way you'd use the live service, all locally.
Environments can be ephemeral, allowing you to deploy completely clean each time and validate 100% that your application can deploy end-to-end, which is useful for developing multi-tenant applications you may have to re-deploy.
However, it's important to differentiate LocalStack from AWS services themselves. I've read stories from those who have developed their whole application end-to-end using LocalStack before proceeding to deploy directly to AWS.
LocalStack is ultimately an emulator, so use LocalStack for initial testing but not end-to-end testing. You still need to validate how your application performs in a live environment.
How I would have built LocalStack into the software development lifecycle (SDLC)
We can make the best use of LocalStack in various areas of our SDLC, but not everywhere - we still need to involve AWS.
LocalStack works best on the left (earlier in development). While writing code, it's incredibly useful to spin up my LocalStack instance and validate that my Lambda Function not only deploys correctly, but also properly integrates with other AWS services, that it sends and receives the proper data to the SQS Queue, and that it can interact with my Database correctly.
However, we should be writing structured tests along with our features that validate that our code works as expected. You can use any test framework you'd like, since we're essentially interacting with native AWS services (just locally).
Building guardrails with our tests is just as important and from an SDLC perspective, there are a few ways we can do that:
- Pre-Commit
- CI/CD Pipelines
- Human feedback (via observability data collected)
Pre-Commit: Testing locally
Pre-Commit is a python package we can install. It hooks into our git commit command, so that before our staged files are committed, the hooks we've configured execute on them.
It’s an important step in the SDLC because it runs tests locally on our machine, ensuring fast feedback.
We can build hooks in many languages, including plain scripts that can run a series of our tests and return successfully to pass, alternatively, we can build more complex and reusable hooks using a fully-fledged language.
That means we can create a Pre-Commit hook that does the following:
- Starts our LocalStack container
- Deploys our service's AWS resources to LocalStack using IaC
- Executes our AWS integration tests for our service
The .pre-commit-config.yaml file could look like:
# .pre-commit-config.yaml
- repo: local
# defining our hooks
hooks:
- id: cdk-deploy
name: Deploy AWS Resources
entry: bash # our hook starts with `bash`
language: system
args:
- -c
- |
set -e
export CDK_DISABLE_CONTEXT_SAVE=1
export AWS_CDK_DISABLE_VERSION_CHECK=1
cdk deploy
pass_filenames: false # don't pass the staged files as arguments
always_run: true # always run this step
- repo: local
hooks:
- id: integration-tests
name: Run Python Tests
entry: python # our hook starts with `python`
language: system
args:
- -m
- pytest
- tests/test_lambda_unit.py
- -v
files: ^(lambda/|tests/).*\.py$ # only run if these files are staged
pass_filenames: false
require_serial: true #only run after cdk-synth has finished
You should also add other common checks to your pre-commit setup, such as security scans, linting, formatting checks, and the like.
CI/CD: Testing in-pipeline
Continuous integration and delivery are pretty standard these days. To make matters easier, you can execute pre-commit hooks in your CI/CD pipeline using the pre-commit run --run-all-files command.
I've had the best developer experience when having a pipeline I can run from my development branch to deploy to a sandbox environment, so I'd still recommend incorporating that functionality into your development pipelines. The perk of this approach is that we can enable additional observability services such as CloudWatch logging, application tracing, and OpenSearch, all of which are incredibly useful when debugging. Of course, sometimes nothing beats debugging in a live environment, when you can just search through logs in your OpenSearch instance.
For those who use GitHub Actions, you can make use of the LocalStack Action. If you don't use GitHub, there are also examples for other CI platforms such as GitLab, CodeBuild and BitBucket.
# .github/workflows/tests.yml
name: Tests
on:
workflow_dispatch: # I can manually run the pipeline
pull_request:
branches: [ main]
env:
AWS_DEFAULT_REGION: us-east-1 # We can use any region, as it's just emulated
AWS_ACCESS_KEY_ID: test # We can use any credentials
AWS_SECRET_ACCESS_KEY: test
jobs:
localstack-tests:
name: Deployment & Integration Tests
runs-on: ubuntu-latest
needs: [ linting, sast-scan ] # run your linting and SAST stages first.
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start LocalStack
uses: LocalStack/setup-localstack@v0.2.2
with:
image-tag: 'latest'
install-awslocal: 'true'
configuration: |
DEBUG=1
- name: Set up Python
# ...text omitted
- name: Set up Node.js
# ...text omitted
- name: Install runtime and test dependencies
# ...text omitted
- name: Bootstrap & Deploy CDK to LocalStack
# ...text omitted
- name: Integration Tests
# ...text omitted
- name: Destroy stack (cleanup)
# ...text omitted
Enabling debug in LocalStack is important so you can review any issues your application might have with the emulated AWS services.
Some final advice
Building your own SDLC requires understanding how you and your team prefer to develop, the nature of the application itself, and the budget and time you have available.
My recommendations for testing with LocalStack are:
- Before trying to test with LocalStack, use it. Read the Quickstart Guide and deploy some resources.
- Use LocalStack to test integrations between your application and AWS services. For example, testing how your application interacts with SQS or DynamoDB.
- Make your tests simple but meaningful - overcomplexity can result in avoiding testing altogether! Focus on validating practical behaviour.
- Make an effort to keep your tests and LocalStack up-to-date, as new features are released in LocalStack and your application, it's important to ensure your tests continue to work.
Finally, LocalStack is a commercial product. The free version covers practically every integration service, but if you use EKS, or want to test out your full end-to-end application entirely, then LocalStack Ultimate might be the way forward.
These days, whenever I’m developing on AWS, I validate my logic locally using LocalStack. It lets me spin up a PoC in minutes and, more importantly, build a robust testing process for myself and others - catching issues early and saving hours of development time (and probably pain).
If you've used LocalStack before and have any advice, or if you've learned any interesting lessons yourself, drop them in the comments!
Top comments (0)