DEV Community: Farmijo

Building BIG systems based on serverless

Farmijo — Wed, 10 Mar 2021 11:42:00 +0000

This is going to be a short post, with the idea of being an intro for a future serie of more extended ones if seems interesting and if I get some time.

I wrote a post weeks ago sharing some thoughts about serverless architectures and their tradeoffs.
I had a previous bias on the use of Serverless solutions (specifically on serverless functions). I was thinking of them just as a way to trigger simple tasks without caring too much of managing resources, but I've never thought of build an entire system based on them. Things and thoughts change and now I see a big advantage on them.
But serverless architectures and the frameworks that allow to work with them don't tell anything about how to build stuff, they just provide a way to build and deploy easily the systems.

Currently, we're having some funny time refactoring properly part of our core codebase. With several years of tech debt accumulated, some of the services developed have an (ironic) interesting coupling to the serverless way of thinking (like invoke lambdas inside other lambdas, as if it was an http request). Yep, this is giving us some hard time while refactoring things.

But, the solution to our problem was quick to find and is quite simple to understand (not as simple to implement, but we're doing it little by little).
With inspiration good old classic hexagonal architecture. , we're starting to rebuild things without caring on how the entrypoints are. Some conclusions that we arrived were:

Treat serverless solutions as another framework and just avoid coupling to them.

Thinking on each lambda as a controller or as an event handler that will call to internal layers, just as another part of the infrastructure layer. Not caring about which type of trigger will call our services, just defining contracts to use them.
Build the app from the domain to the external layers even if it's a simple solution . Once done that, you won't care if the entrypoints are done via Lambda, Azure function or via expressJS routes and they will scale easily.

About the internal calls to other lambdas... we're refactor them to be more event-driven. Or, for tricky cases, wrapping adapters over them (we're having in parallel a lot of conversations about domain modeling lately)

And, for the major architecture... Abstract also to the idea that services are run serverless (keeping an eye to performance/cold starts and similars, though). Build services as if they're another microservice more.

Thoughts on Serverless Architectures

Farmijo — Sat, 30 Jan 2021 02:46:33 +0000

Several months ago, I started working in a small startup. Given their growing volume of clients and needs and due to the small team, they decided to build most of their core infrastructure through AWS lambda functions, using the serverless framework for it. So, we're running our whole business through lambda and it works! It's amazing to see that a whole working business is built over a such simple architecture.Before working here, I just saw lambdas as a way of build crons or small tasks, but it could be bigger than that.

After working with it for some time, I'm starting to see a lot of of advantages but some cons too. Summarized, is a really good way to make things scalable and provide a unified framework to build things without worry too much about infrastructure issues.
But, it have some trade-offs to be known too.

Pros:

Serverless, its name says it, don't worry about servers. Just code, deploy and it will work. On this days of overwhelming complexity through containers and kubernetes, is fine to avoid this kind of worries.
Prod and Dev environments can be easily replicable at 100%! No need to worry about environments too!
Easy deploys, easy rollbacks. And, through Serverless framework, be platform agnostic is quite easy (not 100% transparent, though)
All of this allows to build and ship things really fast!

But, there are things to take care off. Some of them are tradeOffs, some of them just points to take into consideration:

Serverless framework does not provide any information about how to build your stuff. It just provides an easy way to build and deploy a Serverless infrastructure on a lot of cloud platforms. The way that it avoids the server problems and configs does not mean that it helps you to properly develop things. You can develop HORRORS (coupled lambdas, twisted Code,..) and deploy it easily. Its advantage is on avoid to have dedicated staff into infrastructure, not into development.
You're slave of a cloud provider. Someone will manage your servers, responses time will depend on them, scalability too... if you Work with Serverless, Make sure that you're able to migrate to other alternatives easily.
Infernal YAMLs. You will be defining too much stuff through them. Be sure that you're writing them properly too.
Local development can be problematic if you start to couple lambdas. One lambda development works fine, two at the same time, don't. They don't react on local quite fine to external events or invocations, so you will need to take care of how you structure your Serverless architecture

Having some cons, I think that it can boost the productivity of a team and fits perfectly for MVP constructions. But, think about it more as a sinergy than as a Swiss knife. A poor codebase built through a Serverless architecture won't be better than a poor codebase on premise. But, if things are built properly, the boost of productivity is gigantic.

About the past, present and future of QA positions.

Farmijo — Sun, 25 Oct 2020 21:19:22 +0000

So, after several years of work in QA related positions, some months ago I decided to make a side step on my career and move to a development position. In part, this change was due to personal interests (at the end, I’m a quite curious guy in everything related to techie stuff and QA positions were not fulfilling my needs anymore), but also because I realized that everything that involved the work done on a QA positions made more sense as a set of skills and tasks that a team should perform and possess than a list of tasks that should be done by a specific department.

Previously, on QA teams...

Let’s say that the specific idea of a QA team in origin was to “assure the Quality of the delivered product”.
On a generic way, Quality can be defined as the possession of properties on an object that are able to fulfill or overpass the expectations. Basically, that the product that we’re building works on the way that was originally designed (does what is expected to do and performs properly).
Therefore, the QA team should be responsible of the processes and tools to be able to assure that the product built fulfills its expectations.

This is the theory, but,...

Let's say it: in practice, the QA team tasks usually are limited to test, manually, at the end of the development cycle, the features that are going to be delivered, to assure that anything else was broken as a side effect and report any failure.
So, we end with a lot of developers writing features and not so many testers testing them out. The testing could be more or less planned, could be done at different ways, but at the end, it is perceived the same way: Clear separation between the two sides. Dev team is the owner of the development tasks, QA of the QA related tasks, aka testing.

This is where conflicts could start to appear:
QA team, or the QA Engineer present on the squad on Agile oriented environments, that should be responsible of the processes to assure that everything worked properly, is seen as the one that just tests stuff. Quality assurance becomes just testing.
And, Inconvenient Truth incoming: Development requires specific technical knowledge while usually to testing is more a question of soft skills, often undervalued. Usually, QA Engineers are people that do not even come from a technical background.
This could be seen as: developers are working on hard, interesting stuff and QA are, well, breaking it and taking notes on the failures.
So, the thing ends into undermining and undervaluing the QA tester labour.

Thinking on it a bit deeper

Quality should be not even a goal, but a mindset. Is not something limited to a serie of tasks that should be performed, but a way to perform on your daily basis. And, of course, should not be a single team member responsibility but a whole team involvement.

Switch the mindset of “I should develop this feature” to “I should make this feature work properly”. Turn off the mindset of “QA will find bugs if there are” to “Lets find a way to know if this is working as expected”.
Let’s be clear ( another inconvenient truth incoming): Testing is more a question of attitude than of aptitude.

And, as a QA Engineer, switch your mindset too. Explain your way of thinking, your way of work. Learn new things. Teach your team about tools that you use, spread your knowledge. Your work should not be only based on testing, demonstrate that it is like that. Get involved on other tasks away from testing, propose processes modifications, propose ways of automate testing, anything that could lead to help the team to, cof cof, assure the quality on what you’re doing. You’re in a field that requires constant adaptation, it works that way for you too.