Scheduling automated deletion of AWS Cloudformation stacks

Sebastian Schlecht — Sat, 23 Oct 2021 08:30:42 +0000

Article preview: the full article can be found here.

When developers within a team are working on a feature, they are often faced with the situation that each developer is trying to work on a certain aspect of the feature that requires a different version of the infrastructure than we’ve deployed via a Cloudformation(CF) stack, be it an ECS(Elastic Container Service) cluster with multiple task definitions, or when using any other service provided by AWS.

We at Capmo also ran into a similar situation within our teams. The challenge was not the multiple versions of our infrastructure being deployed: It was how were we going to manage the clean up of the Cloudformation stacks once their work is done. That’s when the idea of auto-deleting the CF stacks came up. After doing some research on our side, we found out that the benefit of owning a piece of infrastructure stack for testing a feature makes the process more agile, flexible and reduces any dependency that might arise when subsequent infrastructure deployments take place, overwriting the previous version of the deployment.

Since our infrastructure deployments are taken care of by CDK, we wanted to have a solution in place which we can integrate with our existing Cloudformation stacks without any hassle. So we decided to package our solution that can be used in all our microservices if needed. And also with the possibility of making it open-source at some point in the future.

Automating deletion of Cloudformation stacks

Now that we knew what we wanted, the question was how to build the solution. We would need a trigger that would initiate the auto-deletion of the Cloudformation stack. This can be achieved with the help of an event, for which we used an EventBridge rule. EventBridge rules have the ability to match an event pattern and trigger its targets. The target of choice was a Lambda function that would initiate the auto-deletion sequence. A serverless solution was a clear answer to this as it makes write and deploy easier, with no effort of managing the deployed resources. All of this was deployed within the same Cloudformation stack that we wanted to delete. Now let’s dig deeper into how we implemented this.

To read on, the full article is published on our engineering blog.

Serverless Nest.js micro-services & integrations without HTTP

Sebastian Schlecht — Sat, 09 Jan 2021 12:19:19 +0000

Originally posted on our engineering blog.

Serverless Nest.js micro-services & integrations without HTTP

At Capmo, we are currently evaluating to move our entire backend stack to Nest.js because of its many benefits (which I won’t mention in this blog post as there are plenty of others on that matter).

Since we already have a legacy application running in production, we unfortunately are not looking at a green field here but need to move building blocks piece by piece — among others we have some serverless micro-services that are built using AWS Lambda which we also want to migrate and extend to keep a consistent structure and experience across our backend stack. Those services mainly use native AWS integrations via API Gateway, S3 events or SQS triggers. While there is material on the Internet on how to integrate Nest.js with API Gateway (e.g. here), most approaches rely on a proxy integration between API Gateway and a (cached) express.js or fastify instance. There are ways to interface with other cloud-native integrations using Nest.js micro-services, but many transports don’t come out of the box and we found that wiring up new ones is quite cumbersome and thus were looking for a more straightforward approach that might play nicely with AWS native integrations for Lambda.

Standalone applications to the rescue

Nest.js already provides a guide on how to use the framework in non-http contexts like cron-jobs. In such a scenario, one can benefit from its structure, modularity and other goodies without the overhead of an HTTP server.

When looking at native Lambda integrations however, one might still want to preserve the concept of a request scope. This can be useful for use-cases like preserving native AWS requestIds throughout the lifecycle of an invocation for logging, or pass authorisation headers down to app modules. With standalone apps as described in the guide however, one has only access to singleton scoped providers. What if, for example, we want to share a requestId or userId among all providers within a single invocation / execution context to make logging more transparent, handle authorisation/tenancy, etc.?

In such a case, we can leverage Nest.js’ dependency injection system to create a request-scoped sub-tree for us, which we can use during the life-cycle of a Lambda function invocation. We first instantiate and cache a standalone Nest.js application context, and then create a scoped sub-tree for each invocation:

Subsequently, we can now use request scoped providers throughout our application and inject the { context } object from line 44 through the @Inject decorator.

    @Injectable({ scope: Scope.REQUEST })
    export class CatsService {
      constructor(@Inject(REQUEST) private requestContext: Context) {}
    }

whereas requestContext in this example is the injected Lambda execution context, but can be arbitrary — for example we can extract headers from the invoking request or event-payload like a userId or tenantId which we can make available to all services through DI.

A note on memory-leaks

The attentive reader now might think — what about memory-leaks? We are not cleaning up the instantiated sub-tree in the example above and cache the application context over multiple function invocations?

Luckily, Nest.js uses WeakMap internally, hence as soon as the reference to contextId (which is an object!) is being garbage-collected, the request-scoped sub-tree will also be garbage-collected subsequently. Details can be found here.

The example above illustrates how to wire up a Nest.js app without HTTP overhead directly to Lambda integrations and cache the underlying dependency injection contexts. We plan to use this internally through AWS CDK until Nest.js’ serverless core matures in order to progress our application landscape.

And what’s the downside here?

The downside here is that we are losing quite some framework mechanics when invoking services directly, like for example pipes, guards or exception filters. This is not a problem when going the HTTP proxy way but there is definitely room for improvement when going the serverless way without using HTTP or the Nest.js style micro-services pattern.

PS: We are looking for kick-ass software engineers to join our team to shake up the construction industry. If we caught your interest, feel free to get in touch here or directly via Github/LinkedIn.

DEV Community: Sebastian Schlecht

Scheduling automated deletion of AWS Cloudformation stacks

Automating deletion of Cloudformation stacks

Serverless Nest.js micro-services & integrations without HTTP

Serverless Nest.js micro-services & integrations without HTTP

Standalone applications to the rescue

A note on memory-leaks

And what’s the downside here?