DEV Community: Cesar Pérez

How to improve serverless deployment times with just 4 lines of code.

Cesar Pérez — Mon, 22 Aug 2022 16:35:07 +0000

This is a story and guide on how i reduced the pipeline execution times from a monorepo-bitbucket based serverless project from 20min45sec down to 3min24sec.

Introduction and brief description of the deployment configuration.

While I was working on the company's backend monorepo the other day, I noticed how slow the pipelines had become after moving to serverless v3.

We now have a deployment model that uses bitbucket pipelines for development, staging and multi-tenant production environments; incorporating technologies such as Serverless, Nx, NestJS and AWS.

In the past, the deployment of cloudformation stacks was done separately; as we used the multi-repository model and each service could independently run its deployment to AWS; being able to run pipelines in separate repositories at the same time.

Once we moved to the monorepo model, in order to enforce code reusability and reduce the amount of boilerplate code spread across microservices; it started to increase the deployment time of the microservices, using bitbucket pipelines.

Our configuration for the pipelines consisted of a deployment model based on the changes of the folders involved in the commits; it was the approach that the SRE team worked on to avoid running unnecessary cloudformation stack redeployments; this configuration looks like this:

bitbucket-pipelines.yml

image: sundevs/node-serverless:0.2.3
definitions:
 steps:
 # CRM ENGAGEMENT #
 - step: &deploy-to-dev-crm-engagement
 name: Deploy crm service to development
 caches:
 - node
 deployment: test-crm
 condition:
 changesets:
 includePaths:
 # only files directly under apps/crm-engagement/ directory
 - "apps/crm-engagement/**" 
 script:
 …

The only conditions under which a deployment of several services at once can occur are:

That you have made changes to more than one microservice at a time on the same branch.
You have made changes to a library; this will cause those services that are using that library to be redeployed.
You choose to redeploy several or all of the microservices through a manual action pipeline.

The problem at a high level

A few days ago, in order to raise the bar on the quality of our team's code base; I took the task to start documenting the microservices under the same branch; obviously this change has implications on the deployment of affected microservices in that branch. When documenting the whole project and opening PR; according to the pipeline configuration we have; the 3 services were going to be redeployed and the pipelines looked like this:

Where we can see an abysmal deployment time (20min45sec), especially for the last microservice; which has several lambdas to pack (about 8).
For this reason I decided to reduce those deployment times; and here I tell you how:

Analyzing the pipeline

Among the script command lines, we can see that the command serverless deploy - stage ${STAGE_NAME} - verbose - force was the one that consumed most of the pipeline execution time; with about 10min!

Improving dependency resolution with serverless-jetpack.

Analyzing that particular step of the pipeline; I could notice that what took the longest time was the dependency resolution and exclusion.
For this reason, the first experiment to reduce deployment times I did with this microservice; I installed the serverless plugin Serverless Jetpack, the details of this plugin and how it can improve the dependency resolution of your serverless project can be found here.

Immediately I could see improvements in deployment time:

Reducing the overall deployment time by 3.32 min! For a total of 17m13sec.
While the implication of this change for the code base is minimal:
+1 line of code.

Reducing the packaging time by removing the individual packaging.

The packaging of the lambda functions was also one of the steps in the execution of the serverless deployment command that took a long time; investigating I found out that we were using the wrong packaging model. It was configured for individual packaging, when it was really indispensable for our microservices to package the functions separately. You can read more about it and see if this change applies to your serverless project here.

Reducing the total deployment time by 6 Minutes, 8 Seconds! For a total of 11m5sec.

While the implication of this change for the code base is minimal:

+1 line of code.

Changing the cloudformation deployment method.

Reading the serverless documentation, I came across this article about deployments, which states the following:

Since Serverless Framework v3, deployments are done using CloudFormation change sets. It is possible to use CloudFormation direct deployments instead.
Direct deployments are faster and have no downsides (unless you specifically use the generated change sets). They will become the default in Serverless Framework 4.
You are encouraged to enable direct deployments via the deploymentMethod option:
name: aws
deploymentMethod: direct

So I decided to give it a try and make the change in the serverless configuration files of the microservices:

Reducing the total deployment time by 1 Minute, 53 Seconds! For a total of 9m12sec.

While the implication of this change for the code base is minimal:
+1 line of code.

Changing the execution model of bitbucket pipelines steps.

Not everything is rosy, but after a failed attempt trying to use the serverless-esbuild plugin (as you can see in the commit name). I managed to reach the 3 min! mark; making use of a bitbucket pipelines feature called Paralell Steps, which allows to execute several steps of the pipeline at the same time.

The result looks like this:

Reducing the total deployment time by 5 Minutes, 48 Seconds! For a total of 3m24sec.

While the implication of this change for the code base is minimal:

+1 line of code.

While I recommend it to improve pipeline execution times for this deployment model, I caution that its use may have implications on your workspace billing; bitbucket gives us a very subtle warning about this:

Summarizing the changes:

Install serverless-jetpack to improve dependency resolution.
Change the packaging model of the lambda functions, removing the individual packaging.
Change cloudformation deployment method.
Change the execution model of bitbucket pipelines steps.

We went from deployment times of up to 20min to as little as 3min, only changing 4 lines of code per service.

What i've learned after 8 months as a remote Web Developer (Part II) 🇻🇪

Cesar Pérez — Thu, 22 Jul 2021 10:38:08 +0000

Well, it wasn't actually a job. It was a trainee position at a local company, i was expected to meet certain requiremets as per say:

* 4h availability
* Web development basic knowledge

In order to continue the application process, i had to complete some forms, the problem was... They sent those to me 20 DAYS AGO!. When i got into this "unplugged" thinh i wasn't expecting to be called accepted from a company due to my lack of experience.

Having nothing to lose, i completed the forms, contacted the employer, and got lucky!. The entry position was still available and the team were really glad to had more people into the trainee program.

A few days later i got my first meeting with the team and the rest of the trainees, there i had the opportunity to talk to the CEO and the developer in charge, they were actually really approachable (?), it was a small company willing to grow form developers. The job tasks were mainly related to the maintantence and development of websites from clients that were developed in Wordpress.

First weeks as a trainee 👨🏼‍💻

As a trainee, my first assignments were mainly related to the development simple code snippets that extended the functionality of those sites through hooks with php, some javascript, html and css and when there wasn't any hard task, i'll had to write some tech-related articles in order to contribute to the SEO of the company website.

After two weeks, the CEO noticed my enthusiasm and will to learn, so the 9th of November he came with a salary proposal and i got my first Job as a Junior Web Developer!.

I gotta admit that the next months were kinda rough, i hadn't used wordpress before and never programmed in PHP, so i had to learn in practice: reading docs, blogs, forums, and deep diving into the code of some wordpress plugins were some of my daily tasks as a junior.

Eventually i started to feel comfortable with most of those assignments and got confident with my problem-solving skills. Then, after 3 months i was promoted to Web Developer!

Projects 🖥

One of the main responsabilities that came with thte promotion was being in charge of talent acquisition and (capacitacion?), i had the opportunity to be both interviewer and teacher of web developer entring trainees!. Besides of this, as a Web developer from the company, i worked in some projects like:

Cautive Portal and backend for a Hotspot Wifi.
E-commerce's.
Wordpress Plugin Development.
Development of python trading Bot.
Development of a python productivity monitoring tool.

After a few months I managed to create a small development group in the company, with them I had the opportunity to learn how to work as a team, management, leadership, code documentation and folder structure best practices and repository branching models in order to keep the projects organized.

But things weren't looking all that promising, the salary was ok, considering that the company was established here in Venezuela and i was earning quite good. What i didn't actually liked was the lack of a clear company vision, i felt the need of being in a more well-stablished company where i had the opportunity to keep growing as a developer, doing more advanced projects and learning along the way, so in may of 2021 i started to apply to other companies outside of the country and after 2 weeks i got another offer!

What i've learned after 8 months as a remote Web Developer (Part I) 🇻🇪

Cesar Pérez — Fri, 09 Jul 2021 02:21:48 +0000

Well, before i start telling you my story, i've gotta say it is one of a kind… Let me introduce myself first. I'm Cesar Pérez, and i'm a 20 years old Venezuelan Web Developer, Electronic Engineering Student and self named HotCakes 🥞Master.

Agus, the traversy 🎮

My journey as a developer started on April of 2020, a couple of weeks after the pandemic started i finished my ongoing semester as an Electronic Engineer and got into a Game Jam competition that was being organized by the students of one of the university clubs. I teamed up with a couple of friends to make a game in two weeks, all of us without any previous experience but willing to win, driven by the pure of excitement and experience, and a couple of bucks of reward. My teammates were in charge of the history concept development, character/level design and animations, while i was responsible for programming everything, from character movement, enemy behaviors, bosses machine states to health system, camera, an all of that stuff.
But I didn't knew the language neither how to use the game engine, so i had to learn C#, test the game and get used to Unity game Engine ASAP!. I decided to follow a huge reproduction list from Youtube that covered everything and after long everyday programming journeys and with the help an support of my team we managed to finish up the game just a couple of hours before the deadline. I'll never forget that last day; my eyes burned so much that i had to wear sunglasses in order to keep programming the game. We named it Agus, the traversy. And no, it wasn't my idea.

Did we won?, no, definitely no. We were so tired and excited because we finished that we were too lazy to test the game one last time, one pixel bugged kept the judge from finish up the game. So we got disqualified i guess.
Was i happy?, YES. Until then i hadn't been so excited and disciplined about something a long time ago, that's what got me hooked into programming, a buggy game. But, i decided that game development wasn't my thing, setting up the scenes, the sprites, animations and all that stuff were dependent of someone making the assets for me because i didn't have that skill.

The Mobile App 📱

After that journey i came with what i thought was the next 1 million dollar idea. A delivery app… What a non trivial and original idea, isn't it?. Got teamed up with two friends from the university and one of my cousin and we spent the next couple of months making this idea into a reality,i was in charge of the mobile app development, it was programmed on Scratch programming language on top of Kodular, a platform for making Android Apps, the web app was made on React by a friend of mine who had more experience on web development.
Six months later we realized that it wasn't that much of good idea and there were already a lot of well established delivery app's companies in the country with more people and resources involved, so i decided to take a break and rethink about my life.

The routine 💻

At that time i started to feel curiosity for web development, it was both the most viable and the only programming option for me at the moment because i neither had the system resources to build native mobile apps or the patience to get into complicated syntax and environment setups, plus, i knew a little about html, css and javascript from some courses i took a couple of years ago.
And there i was again, hooked into programming for more than 8 hours a day. I used a lot of free resources to learn: Youtube videos, Udemy courses, free bootcamps ( cough thanks freecodecamp cough), removed any distraction and got into this routine for approximately 2 months:

6AM: Watch 1 or 2 Youtube tutorials about javascript and take notes.
7AM: Breakfast.
8AM: Freecodecamp exercises.
9AM: Watch 1 or 2 youtube tutorials about javascript.
10:30AM: Practicing what i've learned.
11:30AM: Break, helping with Lunch and making chores at home
1PM-3:30PM: Watching videos/courses about technologies i was interested like SASS or Refreshing my knowledge about HTML and CSS.
3:30PM-5:00PM: Make a cake, exercise, or whatever i felt like doing.
7:00PM: Dinner.
8:00PM : Watching short videos about web development.

I've gotta say that it wasn't the most efficient one but it surprisingly worked, because in 30th of September of 2020 i got my first job offer!…

Suscribe for part II!