This is a story and guide on how i reduced the pipeline execution times from a monorepo-bitbucket based serverless project from 20min45sec down to 3min24sec.
Introduction and brief description of the deployment configuration.
While I was working on the company's backend monorepo the other day, I noticed how slow the pipelines had become after moving to serverless v3.
We now have a deployment model that uses bitbucket pipelines for development, staging and multi-tenant production environments; incorporating technologies such as Serverless, Nx, NestJS and AWS.
In the past, the deployment of cloudformation stacks was done separately; as we used the multi-repository model and each service could independently run its deployment to AWS; being able to run pipelines in separate repositories at the same time.
Once we moved to the monorepo model, in order to enforce code reusability and reduce the amount of boilerplate code spread across microservices; it started to increase the deployment time of the microservices, using bitbucket pipelines.
Our configuration for the pipelines consisted of a deployment model based on the changes of the folders involved in the commits; it was the approach that the SRE team worked on to avoid running unnecessary cloudformation stack redeployments; this configuration looks like this:
bitbucket-pipelines.yml
image: sundevs/node-serverless:0.2.3
definitions:
steps:
# CRM ENGAGEMENT #
- step: &deploy-to-dev-crm-engagement
name: Deploy crm service to development
caches:
- node
deployment: test-crm
condition:
changesets:
includePaths:
# only files directly under apps/crm-engagement/ directory
- "apps/crm-engagement/**"
script:
…
The only conditions under which a deployment of several services at once can occur are:
- That you have made changes to more than one microservice at a time on the same branch.
- You have made changes to a library; this will cause those services that are using that library to be redeployed.
- You choose to redeploy several or all of the microservices through a manual action pipeline.
The problem at a high level
A few days ago, in order to raise the bar on the quality of our team's code base; I took the task to start documenting the microservices under the same branch; obviously this change has implications on the deployment of affected microservices in that branch. When documenting the whole project and opening PR; according to the pipeline configuration we have; the 3 services were going to be redeployed and the pipelines looked like this:
Where we can see an abysmal deployment time (20min45sec), especially for the last microservice; which has several lambdas to pack (about 8).
For this reason I decided to reduce those deployment times; and here I tell you how:
Analyzing the pipeline
Among the script command lines, we can see that the command serverless deploy - stage ${STAGE_NAME} - verbose - force was the one that consumed most of the pipeline execution time; with about 10min!
Improving dependency resolution with serverless-jetpack.
Analyzing that particular step of the pipeline; I could notice that what took the longest time was the dependency resolution and exclusion.
For this reason, the first experiment to reduce deployment times I did with this microservice; I installed the serverless plugin Serverless Jetpack, the details of this plugin and how it can improve the dependency resolution of your serverless project can be found here.
Immediately I could see improvements in deployment time:
Reducing the overall deployment time by 3.32 min! For a total of 17m13sec.
While the implication of this change for the code base is minimal:
+1 line of code.
Reducing the packaging time by removing the individual packaging.
The packaging of the lambda functions was also one of the steps in the execution of the serverless deployment command that took a long time; investigating I found out that we were using the wrong packaging model. It was configured for individual packaging, when it was really indispensable for our microservices to package the functions separately. You can read more about it and see if this change applies to your serverless project here.
Reducing the total deployment time by 6 Minutes, 8 Seconds! For a total of 11m5sec.
While the implication of this change for the code base is minimal:
+1 line of code.
Changing the cloudformation deployment method.
Reading the serverless documentation, I came across this article about deployments, which states the following:
Since Serverless Framework v3, deployments are done using CloudFormation change sets. It is possible to use CloudFormation direct deployments instead.
Direct deployments are faster and have no downsides (unless you specifically use the generated change sets). They will become the default in Serverless Framework 4.
You are encouraged to enable direct deployments via the deploymentMethod option:name: aws deploymentMethod: direct
So I decided to give it a try and make the change in the serverless configuration files of the microservices:
Reducing the total deployment time by 1 Minute, 53 Seconds! For a total of 9m12sec.
While the implication of this change for the code base is minimal:
+1 line of code.
Changing the execution model of bitbucket pipelines steps.
Not everything is rosy, but after a failed attempt trying to use the serverless-esbuild plugin (as you can see in the commit name). I managed to reach the 3 min! mark; making use of a bitbucket pipelines feature called Paralell Steps, which allows to execute several steps of the pipeline at the same time.
The result looks like this:
Reducing the total deployment time by 5 Minutes, 48 Seconds! For a total of 3m24sec.
While the implication of this change for the code base is minimal:
+1 line of code.
While I recommend it to improve pipeline execution times for this deployment model, I caution that its use may have implications on your workspace billing; bitbucket gives us a very subtle warning about this:
Summarizing the changes:
- Install serverless-jetpack to improve dependency resolution.
- Change the packaging model of the lambda functions, removing the individual packaging.
- Change cloudformation deployment method.
- Change the execution model of bitbucket pipelines steps.
We went from deployment times of up to 20min to as little as 3min, only changing 4 lines of code per service.
Top comments (1)
Very interesting 👏