Cover Image Credit: Thanks James Bowman!
I've spent a lot of time recently writing about Cloud Engineering. The more I learn about Cloud, the more I'm humbled (and quite frankly, scared) about how much there is to know...
Wanna learn Serverless? Sure, you need an AWS account, oh and it needs to be setup securely, oh and your Lambda needs an IAM policy. Oh and you'll also need to know a programming language too, right? Oh! And you'll need to understand events... and logging, you'll want some of that too... What's the difference between logs and metrics? Who knows! Just remember to structure those logs! (What the hells a structured log?). Oh and you wanna make sure your error handling is setup right, too. Oh and don't forget to access your Lambda you'll need to hook up to API Gateway (maybe you should use Serverless framework?). You'll probably wanna learn infrastructure as code, too otherwise things will get messy. Oh and since we're building an API we better make that RESTful compliant, too. And we'll want an API standard, how about JSON:API? Or GraphQL? How should we test that? Integration testing? On Local?
You get the point (ha! And I really, really could keep that example going on!). All that for potentially just a single API...
So my question is: What Cloud technologies are you trying to learn at the moment, and what are some of the hurdles you're facing? What techniques and concepts are you looking into, and what things are you ignoring?
Lou is the editor of The Cloud Native Software Engineering Newsletter a Newsletter dedicated to making Cloud Software Engineering more accessible and easy to understand, every 2 weeks you’ll get a digest of the best content for Cloud Native Software Engineers right in your inbox.
Oldest comments (7)
Looking at serverless on Azure at the moment. Trying to tune Logging and metrics. Most of the cloud providers have a free quota of logging limits and you run a couple.of perf runs most likely we may exceed that. Trying to find the sweet spot of sampling there.
Next is trying to control the scaling of functions so that I dont introduce back pressure on downstream systems
Hey Hari! Ah yeah — I guess if you've got a dynamic or unpredictable workload you'd want to sample based on invocations, rather than a static percentage, etc... 🤔
Oh, you're throttling your own service to accommodate downstream services? Is it something you could do with a queue? Or the downstream have a specific integration pattern?
‘What's the difference between logs and metrics? Who knows!’
Well, question like that is likely due to lack of general software development knowledge, regardless if the engineer is working on server or serverless ;)
I think it’s not fair to wrap all these questions under ‘cloud engineering’. Actually, very unfair.
Rather, the whole spaghetti in the post is true for any type of software engineering, if the engineer is new to the domain.
Specially for serverless: one of the core advantage is the abstraction from all the infrastructure setup of a server. When comparing efficiency, it’s worth comparing the lines of code (service setup, network, hosting....etc) required for a traditional service and, for example, for a lambda+API Gateway.
Serverless definitely nailed it in many aspects ;)
Let me re-clarify the purpose of the post, it seems my humour was lost on you: I'm curious what people are focusing on with regards to cloud technology — specifically I want to know what criteria they're using to prioritise their learning, what they're learning, and what stands in the way.
Moving on: I'd be careful with opinions like: "likely due to lack of general software development knowledge" — you are assuming everyone has your level of understanding. Such a line of thought is fraught with difficulty. It's also not going to lead to a very fruitful starting point for a #discuss...
Specifically on the topic of logs vs. metrics that you call out, consider: We can mine metrics from logs. Therefore it could be reasonably debated that metrics are redundant — and instead we could simply use logs (or at least, in theory). Topics have nuance. It makes for interesting discussions.
I'm not bashing Cloud Engineering, it's not really a question of "fairness". I am (hopefully, lightheartedly) addressing the fact that Cloud can be overwhelming. Which I think is something other engineers will relate to. Just because Cloud can be confusing doesn't mean that other areas of software are not equally as confusing, that's hardly the point I'm trying to make.
If you wanna take a stab at sharing what you're working on and maybe some of the difficulties that you're overcoming it might lead to some cool discussion, but you've got to be willing to concede that you don't have all the answers first.
Oh no no, the humor was absorbed silently!
Absolutely, I do not have answers for most questions (perhaps just having a bit of clue on where to find answers).
Didn't mean to bash on anything. I'll drop the terms
fair/unfair:)Though, I hold the opinion on 'general software development knowledge'. I believe a sufficient software engineer should know what are the necessary components of a service (logs, metrics, data storage, server/host, endpoint ... etc), which is the minimum bar.
yeah, I made an assumption of engineer audience. that's on me.
My apologies that 'lacking of general software development knowledge' does sound a bit negative, but that's perhaps true for all junior engineers. It's not necessarily a bad thing at all. It just means room to learn.
I guess I was just nit-picking that some of the doubts mentioned in the post are true for any type of software development.
Some of my learning curves when building up a serverless service:
Thanks Dom, I've updated the article with credit for the cover!
I'm trying to find out wich one of three AWS orchestrators I really need to use: ECS, EKS or Fargate :D