Completing a side project. Presenting AVCUD.

George Stefanis — Sat, 08 Aug 2020 13:24:36 +0000

During the lockdown I needed a way to distract myself. I decided to watch a movie but the problem I'm having is that I get paralysis from all the options. I normally end up scrolling through endless amounts of movies and in the end I don't pick anything. I decided this time to watch the best movie for each year from the year I was born until 2019.

There is one problem though. How do you define "the best" movie? Is it the box office income? Is it the IMDB score? Is it the most Oscars? I decided to do what every normal person would. I did create a weighted function that takes all three into account, creates a score and ranks them by it. On the back of that I did create https://avcud.com

The idea is simple, you put a year in the search box and the app goes through all the data I mentioned above, crunches the numbers and gives you a visual list of the movies starting from the highest ranked. Under each movie it gives you some basic info (box office income, IMDB score, Oscars won/Oscars nominated). Clicking on the movie title gets you to the movie page where you can see more info and even get suggestions on other similar movies.

I am really proud of this. It's the first time I can safely say i did deliver on a product idea I had. It does solve my problem exactly and I ended up using it all the time. I hope you like it. Next steps include creating a page to see individual actor profiles (I like searching movies by actors/directors) and curated lists of movies. I also have some technical issues I need to solve.

As far as technologies are concerned the backend is written in Python3 and dockerized. The frontend is written in Svelte and server side rendered with Sapper (really enjoyed using Svelte). The site is hosted in Google Cloud and Firebase. My technical problems are concerned with Google Cloud. That will be a future blog post.

If you have any feedback please let me know. Check it here: https://avcud.com/

How do you debug?

George Stefanis — Wed, 05 Sep 2018 12:21:15 +0000

I have observed 2 broad categories of debugging issues in code bases.

The physicist. This type of person really carefully reads the code. Doesn't touch anything, doesn't log much. Just reads and tries to make a mental model of what's going on. They will only change code, or step through the code/add breakpoints when they are pretty sure what's wrong.
The electrician. This type of person likes to probe things. Like an electrician with an electrical tester they comment things, log results and build understanding through experimentation.

The first group seems to me more analytical and organized in a way. The second group is more intuitive and adventurous in their testing.

Have you observed similar things? How do you approach debugging?

Understanding microservices 101. A frontend developers view.

George Stefanis — Thu, 16 Aug 2018 10:49:23 +0000

I've been working predominately with the frontend for more than 10 years now. While i've been contributing to different services and backends, I never spend the time to see how microservices tie together. How does it all doesn't fall apart?

To me this is what a backend looks like. An app that has some API endpoints defined and returns a JSON object. But the problem is that this is how the world was 10 years ago.

I had to look closer and find out how that old image i have in my head translates to todays golden standard. Microservices. I broke it down to 4 sections.

Section 1

This bit is easy. This is the micro service itself. You take all that code and you move it to its own repo. Easy. Use the framework of your choice.

So doing that we should end up with two services, User service and Transfer service.

Section 2

OK. This is now a bit complicated. Where should this annotation live? Inside the micro service? Outside? If outside where? How would the system know where is what?

We need something that knows how to translate a url to a meaningful route and talk with the right service. Kinda like a traffic controller.

We need a gateway. This is one server that works as a single entry point to the system. This is basically the facade design pattern. Netflix has an open source project for that called Zuul.

Cool, solved it! Nope. How does the gateway know where the server is? It needs an IP! Yeah but what happens if the IP changes? Someone needs to let it know.

We need a discovery service. Blame the cloud, not me. If this was 10 years ago we would have hardwired locations. Now we need a service registry though. A service registry is basically a database of services, their instances and their locations. You call it, you ask where service A is, it tells you. Simples!

There are various tools for that:

Netflix Eureka (TW uses that)
Apache zookeeper
Consul (We use that too)
Etcd

OK! We are done. Almost…

How do you deal with failures? What do you return when one service fails? Worst, what happens when a service doesn’t fail but it keeps running?

In the old days of monoliths you should have implemented a timeout. You still need to, but that alone isn’t enough. A useful pattern is that of a circuit breaker. When things go wrong for a long time the circuit breaks and you know in a sensible time that you need to take some action or try again. Netflix Hystrix is a useful tool that deals with that. Alternatives to that is Envoy (that we use), Akka and maybe more that I can’t possibly know. Note that most of these tools are a lot more powerful that just circuit breaking.

That should be good for now.

Section 3

In an old monolithic system having shared resources was easy. Put them in a folder and share them wherever you want. No problem. Now sharing things is not straightforward. How do you share configuration across services?

We need a configuration server. You can think of this as yet another service that just stores configuration we should share between all our other services. There are various tools that do that:

Netflix archaius
Spring cloud offers a solution (we use that currently)
AWS offers its own config service
Kubernetes also offers a config service too. (we are heading there at TW)

Yeah cool, but why share configuration?

“One of the key ways to identify what should be constant from service to service is to define what a well-behaved, good service looks like. What is a “good citizen” service in your system? What capabilities does it need to have to ensure that your system is manageable and that one bad service doesn’t bring down the whole system? “ - Sam Newman, Building Microservices

Configuration is only one aspect of the answer to the above. It’s just one thing that makes our services consistent and well behaved (hopefully). There is a lot more obviously.

Section 4

This to me was the trickiest concept to get in terms with. How do I share functions between micro services? Also, how do micro services talk with each other? How does the Transfer service talk with the User service now?

OK, regarding sharing functions this is a matter of principle. Here is another quote from Sam Newman and his book Building Microservices:

My general rule of thumb: don’t violate DRY within a microservice, but be relaxed about violating DRY across all services

So the answer to my first question is, copy functions across if you need to, but don’t copy them within your micro service. Keep it DRY. Unfortunately school made us feel guilty for copy/pasting things around. It’s not always bad.

The second part of my questions is more interesting though. How do services talk with each other? To answer that we need to go back to service discovery. When Transfer service wants to talk to User service the following need to happen:

Transfer service goes to the discovery service.
Asks, give me the address of the User service
Discovery replies with an IP or an address.
Transfer service does an API call to that address.

This is called client-side discovery.

Client-side discovery is good because there are less moving parts. You are just utilising the existing parts of your system. It’s also bad. The bad thing is that you are tightly coupled with that discovery service. What happens if discovery dies? Also, you need to implement that discovery logic I described above to all of your services. That’s a lot of boilerplate.

We have an alternative. Server-side discovery. In server-side discovery the client makes a request via a router (a.k.a load balancer) that runs at a well known location. The router then calls the service registry (no you can’t avoid that completely) and you get the address you were looking for. Solution for that method are the following:
* AWS Elastic Load Balancer (ELB)
* Kubernetes proxy
* Marathon proxy

This is simpler by comparison with the client-side discovery because the service now has to only do a call to the router. No implementation of the discovery logic. Also, if you are on AWS it makes sense to just use ELB. The negative is that you add yet another moving part, yet another network hop in your system. Also whatever you choose to use has to support all the protocols you use (gRPC, Thrift, HTTP etc).

Monitoring

If monitoring is important on a monolith it’s twice as important on a distributed system with many micro services lying around. The challenge of finding why things failed can be a nerve breaking problem. To be able to get answers and avoid failure you need to have solid monitoring in place. There are a lot of options here.

The most important thing is visualisation of those metrics that we collect. On that front Graphite is a monitoring tool a lot of companies (including TW) use. You can use for example a tool like logstash to push metrics into Graphite and then get interesting graphs. That can help teams know when their services perform bad. Collecting metrics like inbound response, error rates and application level metrics are only a start. On top of that you can add alarms that can let your team know when things went bad.

While this is a great solution it’s not without problems. The biggest problem with Graphite is that services need to push to it, so it can be aware of what’s happening. If there are networking issues or the service is dead then your team won’t even know what’s wrong. Plus it adds another burden to the service.

Another alternative to that is Prometheus. With Prometheus instead of services pushing to it, Prometheus pulls the metrics. That immediately alleviates the burden from the services. Also Prometheus offers alarms without the need of extra software. So one less thing to worry about.

In my eyes there are no clear winners on this. The only clear thing is that if you are not monitoring things you will have a horrible time.

Basic concepts that you need in a micro service architecture

Automate everything, and if the technology you have doesn’t allow this, get some new technology!
If possible, move to a single-service per host/container. you need one CI build per microservice
Focus on maintaining the ability to release one service independently from another, and make sure that whatever technology you select supports this.
At scale, “partially broken” is the normal state of operation.
CAP theorem (Consistency, Availability, Partition Tolerance). At its heart it tells us that in a distributed system, we have three things we can trade off against each other: consistency, availability, and partition tolerance. Specifically, the theorem tells us that we get to keep two in a failure mode.
- Consistency is the system characteristic by which I will get the same answer if I go to multiple nodes
- Availability means that every request receives a response
- Partition tolerance is the system’s ability to handle the fact that communication between its parts is sometimes impossible.

Recaping

So here’s the ingredients you need for a micro service architecture:

Gateway + Discovery service. Software that might be of interest:
- Zuul
- Eureka
- Zookeper
- Consul
- Etcd
Some circuit breaker to deal with failures.
- Hystrix
- Envoy
- Akka
Some configuration sharing service/server
- Archaius
- Kubernetes
- AWS config
- Spring cloud
Monitoring
- Graphite
- Grafana
- Prometheus
- Zabbix
- Nagios

There is clearly a lot more to this. This is just one little piece of the puzzle. There are a lot of tools to deal with provisioning of servers, availability based on traffic, orchestration of containers etc.

How do you learn from online material? Notes? Live coding? Just listen?

George Stefanis — Wed, 06 Jun 2018 14:01:38 +0000

I was wondering what's the prefer method that people follow whilst doing an online course?

Listen and take notes?
Listen and code as you go?
Just listen and do something later?

DEV Community: George Stefanis