This is a post that I have had brewing for maybe a year but I never took the time to write my thoughts down. By the end of this you should hopefully understand how I see the modern architect's role in a serverless world of empowered engineers rapidly delivering software. I will also include a tiny piece of how a DevOps/Platform team can be integral without being a blocker.
You might have seen my re:Invent talk below before which was super short at 8 minutes so I had to leave out key details, this article will add some more flavour.
The Situation Today
Today, technology does not stick around for long. There was a period of time where you could be the expert of something like IBM Websphere v7 and that knowledge would serve you for 3 or 4 years before you needed to reskill. That is not the case today, it does not matter where your favourite tool is on the technology spectrum it will be different 12 months from now than it is today. You can choose to stay behind the latest version for stability but you forever run the risk of falling out of support and exposing all of your customers to malicious parties.
You may cite some of the long serving frameworks like SpringBoot, React or Angular as showing longevity but look back at the initial version of all of those frameworks and tell me it looks like it does today. Also look at the recent incidents like Log4j and spring4shell to show how much you need to keep those services up to date. Ruby on Rails does seem like the exception to this looking in from the outside but I don't have enough experience in it to comment.
I was personally bitten by SpringBoot, I used to be the Architect over a fleet of microservices and it felt like all we did was refactor to the latest versions of Spring and apply security patches which meant we couldn't deliver as many new business facing features as we wanted. I spent way too much time talking about Spring innards which I really didn't want to care about. This is not a criticism of SpringBoot specifically, any framework is the same.
Serverless To The Rescue
Serverless was originally pitched as the antidote to this - just write your business logic, we will keep your runtime patched / upgraded oh and just pay for what you use. The idea in its purest form is that when something like log4j happens, if you don't package your own dependency then AWS will handle it for you while you focus on your customers.
The Reality
Pretty much everyone's first adventure into the world of serverless is to use AWS Lambda, usually through a monolith (nothing wrong with an appropriate monolith) which usually means they package something along the lines of SpringBoot inside the function along with all of their usual dependencies, treating Lambda like a regular container only pay per invocation. Usually but not always they have a bad experience with serverless here whether it be cold starts, costs or timeouts. They also need to keep updating the function as often as they would have a standalone service.
Enter their aligned Architect to help. Let's have some fun with different Architect personas to see how this turns out
Traditional
The traditional time to engage Architecture was several months before you wrote a single line of code, you should bring detailed technical alternatives with all of the pros/cons clearly documented with a recommendation. The Architecture group can then tell you if what you are planning aligns with strategy.
If you phone up this group and say "hey, I built a thing with Lambda but I am not seeing it perform how I would expect", the answer you will get will be somewhere along the lines of "we have a container strategy where you should place that workload here for the performance you expect", "what are you using for logging? We have a logging strategy... oh and while you are here, we have an eventing strategy since Lambda is event based". So ends your experience with serverless and you spread the misinformation that it doesn't work for your architectures
Modern
I love to say that modern Architects aspire for evolutionary architecture, making the best decisions we can in the moment with the acknowledgement that we may have to refactor should some new information come to light or should AWS enhance a different service which changes underlying assumptions. This requires very different thinking as you need to be very fast at decision making and very good at keeping your documentation in check.
The first thing I would expect a modern Architect to ask is "why Lambda?", "did you try to implement this system functionless?". They will ask this because they are following the serverless spectrum.
The spectrum is explained in this re:Invent talk by Linda Lian, the only addition I make is that writing no code using direct integrations is more serverless than Lambda so this spectrum from 2018 is incomplete
Serverless-first means that you should start with the most serverless implementation and keep falling back as you find it to be inappropriate until you find the goldilocks solution. This is fantastic for a couple of reasons:
- You now know that your workload is running in the best place
- You know exactly what it would take it move it up the spectrum
We now have a new problem - If your Architect says "hey, you don't need that Lambda Function, you can have a Step Function directly integrated into an API Gateway which will be much more performant". There is no gentle on ramp, you now need to learn Amazon API Gateway fully and AWS Step Functions. None of your previous skillsets now apply to your serverless workload. This can leave you feeling very frustrated.
Enabling Architecture Is Needed
It is completely unreasonable to expect every engineer in your company to be experts at all of AWS (unless your company is 5 people). It is also completely unreasonable in a serverless context to expect all serverless infrastructure to be provisioned by a central team since with direct integrations, your infrastructure is your business logic.
You can however use the strengths of both groups together to be stronger than ever
The north star is that you want your development teams delivering solutions to production as fast as humanly possible without being constrained to a single solution. This means development teams need a safety net that if they try anything dangerous will protect their reputation and the company.
There are various industry shared ideas for solving this problem like Spotify sharing their Golden Path methodology. They have one "fully paved" golden path for creating software and you are allowed to deviate from the path but under the expectation that you will have no support and you have to report back on if the golden path should change after you learn the pros/cons.
Creating A Safety Net
I think we have improved on the golden path concept. Liberty Mutual shared back in 2019 a tool we created internally for our developer safety net that we called Radar. The TLDR is that if anyone tries to deploy anything deemed dangerous, it is instantly deleted.
Central teams own the creation of the rules for the safety net and each rule comes with documentation of why it needs to exist. Now instead of a slim golden path that leaving brings you no support we have a safe space to experiment.
Driving reuse through inner source
The next thing we did was create a place where engineers could share working solutions with others called the Software Accelerator. Rajesh and team talked about it in detail at CDK Day which you can watch below
The TLDR is that we created a place where in a couple of clicks anyone could have a working codebase, build and deploy pipeline. We also allowed anyone to contribute to it, this created a flywheel of innovation within the development community
Using AWS Well Architected For Advice
The final piece is - how does anyone know that the patterns are high quality? Sure, they aren't dangerous because they aren't immediately deleted by our safety net but there is a large margin between not dangerous and well architected.
This is where we introduced regular Well Architected Reviews, we do this at a pattern level in the accelerator but also on all deployed solutions. This way architects can regularly pass knowledge down to the teams who can gain confidence that what they have built is appropriate.
I shared more about how we combine Serverless-First, Well Architected and AWS CDK recently at the Pretoria AWS UG meetup that you can watch below
Revisiting Our Team From Earlier
If you create an environment like above then the conversation with the architect isn't as scary. Instead of saying "hey, I need you to learn everything about API Gateway and Step Functions" what I am saying is "here is a well architected pattern you can deploy right now to solve this problem". The conversation starts with working, well-architected software.
It also allows Architects to see common problems across teams during well architected reviews and work with DevOps to create the solution through the accelerator before they have even been asked. They can then work out any kinks which would block wider adoption before it is make or break on a real project.
Conclusion
In a serverless-first world, Architecture, DevOps and Delivery teams need to work together to solve different pieces of the same problem. Rather than thinking about one single golden path, think about a spectrum of capabilities that when surfaced in a well-architected way can supercharge your delivery teams and help your business thrive.
Inside this enabling environment teams can create their own Golden Paths if needed, nothing is off the table which is the beauty of the setup.
Top comments (4)
Thanks for sharing this @nideveloper, it was very informative.
I'd love to see something like what you guys have done at Liberty be made available to the greater public. Startups, for example, may not have time (nor desire) to put together a set of pre-made templates for building applications, never mind all the needed guardrails in place like the Radar system (which sounds awesome, by the way).
I imagine things like Proton, or the CDK L3 Constructs, are good places to start. More work is needed to establish good patterns and make following them as painless as possible!
You can also look at things like Spotify Backstage (backstage.io/) which is open source as an alternative to our Software Accelerator and SaaS products do now exist that do some of what we get from radar. I do agree though that instead of just basic examples listed everywhere we need to agree upon a set of well architected patterns that everyone can just pick up and use safely. AWS has been trying with the AWS Solutions Constructs (docs.aws.amazon.com/solutions/late...) but I think we need a few of the biggest AWS customers to come together and agree what works well for them then trickle it down that way
Thanks Matt, I'm definitely going to look into Backstage.
Thanks for writing this Matt- I completely agree with your approach. For those looking for validated serverless patterns check out: serverlessland.com/patterns