DEV Community: Marceli-Wac

What technology would be suitable for region-based, serverless backend solution on AWS?

Marceli-Wac — Sun, 29 Mar 2020 18:36:59 +0000

I am currently planning the development of the backend for my personal website. In short, the front-end is written in React and at the current stage is essentially ready to consume backend. It is hosted on AWS using the S3 bucket set up for website hosting and exposed using CloudFront distribution. This means that the access to the website itself is cheap in terms of latency, regardless of the location, as the results of querying that website are stored in CloudFront's cache.

Now, I'm planning on providing the content for that website using some sort of serverless-enabled document database (considering MongoDB with Atlas and DynamoDB with Global Tables). My only worry is that at the moment, not many of these solutions provide an out-of-the-box region-based routing that optimises the location from which resources are served. MongoDB Atlas has some nice VPC-peering options which integrate really well with AWS, but seem to lack any region replication; DynamoDB on the other hand has global tables which does exactly that, but does not expose a single endpoint which would route requests to the closest database (or at least I could not find a way to set it up like so). One thing that I could think of is to setup the manual routing to the databases via Lambda@Edge, which would be both serverless and region-based, but that sounds like a lot of hassle and manual setup to find the best database server and just seems flaky.

Another way is to use a static site generator, or perhaps even a script that would compile a JSON from the database with website. Together with DynamoDB streams (or any other db hooks) this could allow me to deploy new version of my website with updated data. Considering features like comments etc. this obviously seems inefficient as the website would have to be re-deployed with every change to the database and that takes few minutes.

Finally, I'm wondering whether exposing a single database endpoint in just one region via Lambda@Edge and using it as part of the CloudFront distribution viewer request would be a viable solution for my use case.

Overall, the setup is a bit overkill for the traffic I am expecting and I don't suppose that a second of delay for someone on the other half of the world will make a difference. Then again, we live in the world where all of that server(less) infrastructure is essentially free, so why not go for it and learn something in the process.

Any answers, opinions or questions are very welcome!

How I solved the problem I didn't know I've had - answer to the question of what programming is all about

Marceli-Wac — Mon, 18 Nov 2019 16:09:54 +0000

Some context for this post

I'm a software engineer by trade. Over the past five years I've been working on both personal and commercial software projects spanning a variety of stacks and technologies from web development to embedded devices. I'm far from the person who knows everything about computer science, but I'd like to think that I can explain a lot of the concepts behind it pretty well.

In one of my recent conversations with my girlfriend she asked me what's the point of programming. After a short pause, I replied: It's mostly just problem-solving.

Seeing as this didn't convince her, we've both went through the effort of teaching her the basics of JavaScript and writing a piece of code that solves some of the maths problem she has to deal with at work. While this probably helped her understand what programming is all about, I knew that neither of us was completely satisfied with that as an answer.

This post is about giving her, as well as everyone else a better example of what problems you can solve with programming, and how some of these problems we might not even be aware of.

Where it all started

About two years ago I applied for (and got offered) a position at summer internship program. It was a way to gain some professional work experience and earn some money, but also kill the time between the university terms. Few weeks into my new job, I realised that a nine-to-five job does not actually take all of my time. After all, there's 24 hours in a day! I had to find something to work on in my spare time.

Around that period, one of the things that has also started to gain some real traction was Amazons Alexa - a voice-first virtual assistant and new big thing by Amazon. The popularity of NodeJS was also sky-rocketing and when I learn that Alexa uses Lambda which can run Node... I knew that this is definitely something I want to get my hands on. Another thing that convinced me to do it was Alexa Perks Program offering the developers a chance to win an Alexa-enabled device (among other promotional items) when publishing a Skill (that's the Amazon's nomenclature for Apps running on Alexa). Without hesitation, I bought a book to teach myself Node.JS and get started with building Alexa Skills. After a week or so of playing with Node I looked up sample Alexa projects on GitHub. I was ready to write my first skill!

There was only one problem... I didn't have a project to work on.

Open APIs and TasteDive

I looked around for some open APIs that I could use for my project and quickly stumbled upon TasteDive. TasteDive is a "recommendation engine that lets you explore your taste", or in other words - a service that lets you find the movies, books, games etc. that you might like, based on a given title.

Great! That will do! - I thought and went ahead to explore what the API had to offer. The interface turned out to be relatively simple yet very powerful. I started working on the project, spending few hours every day after work and over the weekends. After a week or so, the project was ready. I submitted my skill, completed the certification process (after two failed attempts) and just like that, my skill was available on Alexa.

Not much later, an Amazon parcel has arrived at my doorstep and the echo was set up in my room. Mission complete!

The problem I didn't know I had

After submitting my app to the Skill marketplace in a fire-and-forget manner, I've had more time in the evenings to relax. It didn't take more than a week for me to completely clear my IMDB watchlist and I quickly found myself looking for the next movie or TV show to watch. For two weeks straight I was scouring IMDB and Rotten Tomatoes for relevant movies that would be worth my time (although I have to admit, in the end I had also settled for the ones that weren't worth anyone's time). And then it hit me...

I HAVE LITERALLY JUST SOLVED THAT PROBLEM

Since then, I have started using Delphic Oracle - my Alexa Skill that consumes TasteDive API (see The Crumbles if you are from UK) and it has proven to be significantly easier and more accurate compared to what IMDB recommends. It felt great!

As all good things in life, summer too came to an end. Few weeks later I forgot all about my Alexa Skill and got busy working on my degree. It was not until about a year or so later that I went to the Alexa dashboard and (to my surprise) saw that people were actually using it! It's not any kind of a promotion and I don't expect anyone to go and use my app when they're done reading this post. What I am however getting at, is the moral of the story here.

Computer science - in its all shapes and sizes, aims to solve some sort of a problem. It does not matter whether it's a problem that millions of people face on a daily basis, a niche task within a team of people or even something that you struggle with personally. So long as it's a problem (even if it's a problem that you don't know you have) - software is there to solve it.

Kubernetes - Explained Like You're Five

Marceli-Wac — Sat, 21 Sep 2019 23:18:44 +0000

Kubernetes - by definition is a container orchestration tool.

That's really all there is to it, so let's break it down! Before that however I'm going to give a Docker quick overview, mainly because it's a related topic and will be beneficial for those who wanted to get into Kubernetes but never really learnt docker.

As a disclaimer, this post is written as a reply to the question by Hassan Sani. It is also not the most comprehensive explanation, but rather a quick description for those who wish a 5-year-old-explanation. The original question asked:

Explain Kubernetes to me like I'm Five

Hassan Sani ・ Sep 21 '19

#discuss #kubernetes #devops #explainlikeimfive

What is Docker?

Docker is a piece of software that fundamentally "packages" applications into runnable programs. Of course there's a little bit more to it than that, but let's assume that Docker does just this. As an example, think of an application you can write in your favourite language. That application serves some purpose, and you as a developer probably have the knowledge on how to get from having that source code to running it as a program. This can be done by the means of compilation (i.e. C++, Java) or interpretation (i.e. JavaScript, PHP) etc.. Regardless of the language specifics, there is a connection between the raw code and its product in action.

Now that you have your program, you would probably like to spread the love and allow others to benefit from the long hours you've put into its development. So let's say you want to share that great To-do app you've just written with your friend. Maybe he has the experience to build it from code, maybe not. Either way, this might not even matter because if he uses different operating system (OS), chances are you are using libraries that are installed locally on your system so the code won't work on his system anyway.

That's where Docker comes in. Remember when I said it takes your program in its source-code form and transforms it into a runnable?

But my friend can't run my todo.exe on his Mac!

I know, I know! With Docker, running any application in any OS is possible because "packages" created by Docker are not just executables; instead they are images of an entire OS with a pre-loaded application.

That seems like a lot of unnecessary stuff coming with my application!

It might, but it's not the whole story! Nowadays packages like alpine are widely available linux distributions that only take 5MB.

To use it, all you have to do is follow few simple steps (see a decent guide below) and transform your application code into a docker image. Share it with your friend and now all he has to do is find himself a docker runtime to run it! Easy peasy!

5 part Docker series, beginner to master

Chris Noring ・ Mar 10 '19

#docker #beginners #devops #showdev

but...

Alright, so you're saying that your friend does not want to download extra packages just to try out your To-do app. Fair enough, I've got you covered. It turns out, that Kubernetes is exactly what can help you achieve that!

The power of Kubernetes

Let's recap quickly where we are. You have a To-do application that you've graciously written to help your friends get through their day and because they use some fancy OS that you don't have access to, you've packaged your application into a docker image. Now all they have to do is run the docker image and profit. Except they can't do that, because all that docker-engine mumbo jumbo seems like a lot of effort and they don't want to clutter their OS with unnecessary packages.

Okay. As I've said, Kubernetes has got you covered. Imagine, that the To-do application you have written can run as a standalone web application. All this means is that you can start it somewhere on a server and make it available at some endpoint (I'm looking at you, localhost:3000). Maybe you are a CLI-ninja who enjoys setting up AWS EC2 servers, security groups, load balancers etc., or maybe all you want to do is expose that docker image you've just built without all that faff surrounding setup.

I'll quote again - "Kubernetes is a container orchestration software"; and we're finally at the stage where we get to demystify that orchestration part of it. Running a publicly available software of any kind, size and shape, always comes with several considerations. These might include scaling (responding to the increased use, or load of your application), delivering (exposing your service to other users) or securing (restricting access to or otherwise protecting) your application, but also covering more pragmatic scenarios. What happens when your application crashes? Would it restart itself? What about existing session? And what if you've made a mistake and there's a gigantic memory leak that now prevents you from even accessing the server?

Kubernetes deals with all of these questions and considerations in an elegant manner. It orchestrates your containers, and it does that by tying your packaged software to the infrastructure in the same way Docker ties your code into its actionable form - the runnable software.

I'm not going into the great depth in explaining the concepts and structure of Kubernetes, because there are loads and loads of great resources that will do it better than I can, but the bottom-line is, you should RTFM (which you will find here). I'll also attach one of the Dev.to tutorial series I've found to be both helpful and reasonably in-depth.

Learn Kubernetes, Part I, Basics, Deployment and Minikube

Chris Noring for Microsoft Azure ・ May 18 '19

#devops #kubernetes #tutorial #docker

The things you should know

Kubernetes operates on a cluster. This means that there is usually more than a single (virtual) machine; this provides redundancy and scalability options but supports the core kubernetes services ( kube-apiserver, kube-controller-manager and kube-scheduler). There is of course more services you can use to tweak your cluster to do what you like, such as for example the KubeDNS - the DNS service responsible for routing within the cluster.
Cluster comprises of nodes. The least you can have is a single master node. Master here is the keyword since Kubernetes distinguishes between the two types of nodes: masters and non-masters. The bottomline of the difference between the two is that master-nodes govern the behaviour of "what goes where" within the cluster. This means that if you deploy (I will get to this) you application on the cluster, Kubernetes master will decide what actual resources should be provisioned and what objects and structures should be created and where. The non-master nodes serve as an expansion to the architecture but do not include the services such as kube-apiserver (instead they run kubelet and kube-proxy that allow them to "listen" to the master nodes).
kube-apiserver is a service that provides an interface to the cluster. This means that to deploy your application, you provide the cluster with a configuration file that describes exactly what you want. For example, you might want to specify what docker image you would like to run, which ports should be open for communication, under what domain will it be available and how many replicas (copies of your application running in parallel) you would like. The complete list of possible configuration options is extensive and there are multiple resource types you can use to make the service available. Each with their benefits, drawbacks and quirks.
Most commonly, the kubectl CLI utility is used to communicate with the cluster. Yes, it turns out you will need some of these CLI-ninja skills after all, but on a positive note - it's not complicated at all and the configuration files have multiple supported formats including JSON and YAML, so the process can be really painless.

There is a more complete list of concepts available on the Kubernetes website. I suggest you give them a quick look if you want to get a better grasp of who does what and what goes where.

Should I install it on my machine or go for the cloud?

Finally, before you dive down into installing the Kubernetes clusters on your bare-metal servers, look for the tools that will help you with this task. In my days, I have found kops (kuberentes operations) to be an extremely useful software. By far, I can credit the best introduction to Kubernetes on AWS I have seen to the following YouTube series by Jeffrey Taylor.