DEV Community: Sezgin Ege

Transient Errors: Retry Wisely

Sezgin Ege — Sun, 31 May 2020 00:00:00 +0000

Each remote service that we call eventually going to fail. No matter how reliable they are, it is inevitable.

“Everything fails all the time” — Werner Vogels

These failures can come from a variety of factors; network issues, hardware problems, temporarily unavailable services, exceeded response times, etc.

Some of these failures might have been resolved automatically and in a short period, if the remote service is invoked again, it immediately responds successfully. We call these kinds of errors as transient errors.

When we encounter a transient error, there are a few things we can do. The simplest option would be to log an error and give up. Since transient errors most likely to be resolved when you retry, you may guess that this is not the wisest option. So, the correct strategy would be the retry the failed operation.

Retry the failed operation without having a clear strategy most likely will create extra load on remote service and therefore it probably will make the situation worst.

Some questions that should be answered before applying retry strategy are:

How would client determine if error transient or not?
How often client should retry?
How long client should keep retrying?
When client should give up?

So, when to retry?

You can identify that it is a transient error

If the remote service returns TransientErrorException, it is great. However, this is not always likely. In those situations, we need to be smart while interpreting the errors.

Client Errors: These are the errors caused by client itself. Examples are badly formed requests, causing conflict, or doing too many requests. In those cases, remote service returns 4xx error. The only way to handle client errors is to fix either client or request itself by human intervention. _ There is no point to retry these requests. _

Server Errors: These are the 5xx errors that indicate something went wrong on the server side. In those cases, it is usually safe to retry since every 5xx error is not as transient error.

Network Errors: These are the errors due to network issues. Examples are package loss, router/switches etc hardware issue, etc. Safe to retry if you can identify those.

The service is idempotent

Idempotence means, when making multiple identical requests has the same effect as making a single request ¹

According to the definition of the spec of HTTP², GET, HEAD, PUT, and DELETE are idempotent operations. So, you are fine to retry those requests unless advised by a remote service owner. On the other hand, POST & PATCH are not idempotent and if idempotency not applied, it is not safe to retry since it might cause side effects such as charging a customer multiple times.

Retry strategies

Several strategies that can be applied as retry mechanism. Choosing the right strategy depends on the use case.

We can retry immediately right after the operation failed. This is the simplest retry strategy that we can implement. It is a good idea to give up or fallback a better strategy after the first failed retry operation since continuously retrying will create too much load on remote service.

We can retry at fixed intervals right after the operation failed. This strategy gives more time to remote service for recovery.

Both of these strategies are useful for applications that user interact with it since these strategies retry the failed operation and if operation not successful, most likely to give up. So that, the user does not have to wait for a long time.

If your service/application does not directly interact with a user and/or you have a luxury to wait more (e.g. background operations), exponential backoff is a strategy that you should try. This strategy is based on increasing the wait time exponentially between subsequent retries. This is a pretty useful technique since it gives remote service more time to recover and create less load than previous both strategy in a given period.

So, what exponential backoff strategy looks like?

Here is a simplified pseudo code for exponential backoff strategy algorithm in a simple way:

retries = 0;
retry = true;
do {
    wait for Math.MIN((2^retries * 1000) milliseconds, MAX_WAIT_INTERVAL)
    status = fn(); // retry function
    set retry true if operation status failed
    retries++;
} while(retry && retries < MAX_RETRY_COUNT)

Distribute load by adding some jitter

Most likely that there will be multiple instance of client and therefore, if all the requests from those clients fail at exactly the same time, we don’t want these retries to be overlapped. Adding jitter will distribute load more evenly. With jiter, our algorithm would be;

retries = 0;
retry = true;
do {
    waitTime = Math.MIN((2^retries * 100) milliseconds, MAX_WAIT_INTERVAL)
    waitTime += random(0...3000)
    status = fn(); // retry function
    set retry true if operation status failed
    retries++;
} while(retry && retries < MAX_RETRY_COUNT)

Summary

Retry is a powerful technique that allows the client to offer higher availability than its dependencies if applied correctly. Retrying the failed operation without having a clear retry strategy most likely will create extra load on service and will make the situation worst.

So, use wisely!

Idempotent rest APIs ↩︎
Idempotency on HTTP methods ↩︎

The Analysis of Turkey’s Big Plan: Train 1 Million Software Developers

Sezgin Ege — Fri, 01 May 2020 22:00:00 +0000

The Treasury and Finance Ministry of Turkey has kick-started a campaign on 21st April which aims to train at least 1 million software developers.

It has been announced on Twitter:

We put into practice today the revolutionary ‘1 million software developer’ program, which will make our country one of the leading countries in the world in software development. A historic step is being taken for Turkey today.

Turkey has scarce human resources in software development with about 140,000 employees. Since their number is limited in Turkey, software developers have to take responsibilities of a system analyst, system testing expert, project supervisor, database expert, designer, and security expert at the same time. Firms always complain about having difficulty in hiring in these positions.

We aim to solve this problem with the “1 million software developer” program. Software developers will work much more efficiently since they will only work on the areas they are experts.

31 courses of nearly 47.000 minutes are already uploaded on the training portal. The number of courses will increase to 100 until the end of the year.

Each exam score of completion of these courses will automatically be added to their CVs. Companies will be given access to the CV pool.

Courses will be free of charge and everyone can access it.

After the “1 million software developer” program has been announced, of course, people started the criticize it. While veteran software developers support the idea that the “1 million software developer” program will affect the businesses in a bad way ( Quality vs Quantity ), students/hobbyists, etc. believe that this program will help them to learn how to code and get a job way easier.

Why this program is promising?

We are living in a world which constantly transforming. It is getting more connected, more digital every day. We need to understand today’s world, use the tools that exist and leverage them to improve our productivity. These free courses are giving people a chance to do it exactly.

Let me tell you a story. Once I have met with someone. Their team was responsible to download product images everyday from several websites and upload their internal systems. Can you believe that how much time they could save if any of their team members would now basic scripting? They could automate all the processes.

It does not have to be all examples about coding. Imagine someone, who has a local shop, manually keep track of inventory. This is an error-prone process. It takes a lot of time. Knowing how to use office programs would help to save time and could make their life more productive.

Some Early Concerns

There is limited soft skill trainings

Software development is not just about coding from morning to evening. It would be really easy if it was.

As a software developer, you need to be part of several meetings -planning, grooming, design, mentorship- with different stakeholders -managers, UI/UX designers-. During each meeting, you might have to mimic a different role. You cannot build the right product without understand customer problem. You cannot deliver a product on time if you don’t estimate well or have a good judgment about tradeoffs between decisions. Most importantly, different stakeholders use different mediums to share information. While some of them use planning boards, others might use design documents. Having success as a software developer not just depends on coding skills, but most importantly it requires to have good soft skills.

It is all about online courses

Nowadays, it is possible to find a course anything you might be curious about. Are you curious about machine learning? Sure. Tons of ML course available today. Wanna be a web developer? Here is the React training. However we still have too many open positions today and companies are having a hard time hiring highly qualified people. Making online courses definitely will increase the number of people work in this field. However this will not improve the “quality”. Software development is not just about doing working software but also well-crafted software. I have been professionally doing software development for the last 6 years, following several programming blogs, Youtube channels, people on twitter. Even though I learn a lot from exposing myself to all the information what others share, I must say that doing a bad decision on real product teaches you way more than anything. I hope, they somehow find a way to support these courses with real-life practice.

Trainings needs to be more focused

I enrolled in the program to give it a try -requires Turkish citizenship id-. I must say that I am shocked when I see there are 695 trainings on Javascript course. It is all about pure, vanilla Javascript. This might sound amazing but I can assure you that it is not. There are hundreds of videos that explain each DOM event, each function that Date object has. Who needs to know each DOM event? All the efforts that spend to create these videos could have been used to prepare more structured, detailed videos.

There is no “guidance”

This platform mostly will be used by people who have little or no knowledge about programming. They need guidance to achieve their goals. Today, courses have been grouped such as software development, cybersecurity… It is easy to get distracted and jump one course to another or struggle to find out what to learn next if you have little experience. It would be great if they can add roadmap functionality (e.g. roadmap.sh)

Conclusion

There is nothing wrong with encouraging people to learn to code. This program may help people to do headstart. Also I believe that it will get better over time.

On the other hand, this program should not be announced as a “hope” to find a job (complete course, get the job). Companies are looking for people who do more than complete courses. People must be aware of that.

Weekend Project: Building Augmented Reality App With Javascript/HTML

Sezgin Ege — Sat, 04 May 2019 00:00:00 +0000

Augmented Reality (AR) got my attention after I have seen few interesting projects on my twitter feed and I wanted to do small AR demo with using basic web development skills, just to experiment AR stuff.

Here is the result:

Doing some AR experiment 🕺 I will publish details on my blog later #AugmentedReality #groot pic.twitter.com/WPTALpZGj9

— Sezgin Ege (@sezginege ) 27 Nisan 2019

I have only used a-frame framework and AR.js during development.

a-frame: is a web framework for building virtual reality experiences.
AR.js: is a javascript framework that enabling AR features directly on web browsers without sacrificing performance. It works on any phone that supports webgl and webrtc.

Let`s see the code

I have added comments to the code. It is pretty straightforward. Basically, we define our scene, initialize arjs, add camera and marker for origin point. Also we define the entity that we would like to show on origin point. I have used 3D groot model but you may want to show static image/box etc. In that case, you need to use a-box or a-image components.

How we can test this?

If you open index.html file directly on your browser and check developer console, you may see CORS issue:

Access to XMLHttpRequest at 'file:///<workplace>/AR/test/scene.gltf' from origin 'null' has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https

Please have a look CORS wiki page to understand what CORS means: https://en.wikipedia.org/wiki/Cross-origin_resource_sharing

In order to get rid of this issue, we need to run http-server. Node.js has a simple HTTP server package. To install, you need to run:

npm install http-server -g

After successfully command executed, we need to go to the folder that index.html exists and then start http server;

http-server

We should see an output like this on our terminal:

Starting up http-server, serving ./ Available on: http://127.0.0.1:8080 http://192.168.1.36:8080

So, server is up and running. We can click any of those addresses and open in our browser. However we will face with two issues.

Testing AR with our browser´s webcam is not comfortable at all.
We need to create SSL certificate for our server. Otherwise, we will see “Webcam Error NotSupportedError Only secure origins are allowed” error: https://github.com/jeromeetienne/AR.js/issues/399

To simplify this process, we can use ngrok which is reverse proxy software that establishes secure tunnels from a public endpoint such as internet to a locally running network service. We need to download and run ngrok.

./ngrok http <port that your local server listening>

We should see an output like this on our terminal:

Forwarding http://e2db416e.ngrok.io -> localhost:8080

Forwarding https://e2db416e.ngrok.io -> localhost:808

Now, We should be able to reach our demo with using https endpoint with any device that has internet connection.

To see our entity on screen, we should show the hiro marker to the camera. Marker can be found here: https://jeromeetienne.github.io/AR.js/data/images/HIRO.jpg

Conclusion

The demo is pretty simple. However if we consider that we did this demo with few lines of html code and few JS libraries, there is a huge potential here to build greater things.

I am finishing the article with one of my favorite AR application. Enjoy!

📣 Career Talk at Yildiz Technical University

Sezgin Ege — Tue, 01 Jan 2019 00:00:00 +0000

Few months ago, I had the chance to talk as a former student at Yildiz Technical University about how I built my career after graduation and a few things I wish to learn a few years ago. Huge thanks to Prof.Dr.Banu Diri and Skylab team to organize this fantastic event!

I really enjoyed to be there and try to answer people´s questions as much as I can ✌️ I hope, we can do these kinds of organizations more often. It seems that we should, after all those great feedbacks.

Few things to emphasize from talk

It would be great to share a record of the talk instead of giving a really short summary but we could not arrange a camera.

Experience is what we get from life, not the what we expect

There is no guarantee that your dream will be always come true. You need to work hard to make them real but sometimes, it is not enough. Also, you should not forget that decisions are not good or bad on its own. We, as decision makers, make them good or bad, depending on our actions after the decision. Whatever you do or you decide, do it wholeheartedly, and it will be a good decision.

Have a plan

Having a plan will keep you oriented and focused to reach your end goals. Do not let having a plan to create pressure on you. You cannot have a plan for everything but thinking on your career or your health is a good thing to worry about.

Work on things that you enjoy

I know It sounds like a cliche and I agree that. It is nearly impossible to enjoy all the time from the things that we are working on. Everything that we do has positive or negative sides. The important thing here is that how do you feel the end of the day. Are you happy? If yes, go ahead and if not, it might be a good idea to think about it.

FAQ

Which blogs do you follow?
- Most of the time, discussions happening on Twitter. However, I believe that blog is still a thing and there are good authors/companies to follow. I cannot list all of them but those are at least my favourite ones.
- http://blog.codinghorror.com/ by Jeff Atwood
- https://martinfowler.com/ by Martin Fowler
- https://medium.com/netflix-techblog by Netflix
- https://medium.com/airbnb-engineering by Airbnb
- https://www.tedinski.com/ by Ted Kaminski
- https://www.hanselman.com/ by Scott Hanselman
- https://ahmet.im/blog/ by Ahmet Alp Balkan
Which talks do you suggest?
- Please have a look at this repo: https://github.com/hellerve/programming-talks
Which resources we should look to improve our technical skills?
- There are different ways to solve the same issues. It is good to look for If someone already invented the wheel. So, I strongly suggest looking for design patterns which are "simply a generalized, reusable solution to a commonly occurring problem."
- Design patterns for humans
- Try to understand how large scale systems work:
- The System Design Primer
- What I cannot create, I do not understand - Richard Feynman
- Build your own x
What was your English level when you join Amazon and what is now?
- When I joined Amazon, I believe that it was B1. I was able to understand people easily. But my problem was to talk fluently. So, I was too afraid to go into the conversation with people. I still make quite a lot of grammar mistakes but at least now, I can communicate without much difficulty.
Do you suggest to live abroad?
- Definitely. I am so happy to meet and work with the people all around the world, listen their stories and learn from them.

Conclusion

I am really grateful for the invite and I really enjoyed to be there! Thank you so much everyone!

Let´s Try Again: New Blog

Sezgin Ege — Sat, 29 Dec 2018 00:00:00 +0000

I really wanted to stop giving status updates like this on my blog. I believe that Twitter is a good platform to share those updates but let´s make an exception and I hope, this will be the last.

What happened to the old blog(s)?

I had a few blogging experiences with different platforms (Blogger/Wordpress). I was blogging about technical stuff and I was happy with the traffic I have. However at some point, I just lost my motivation and I thought, blog posts I shared were not good enough from a technical perspective. So, I stopped blogging.

What has changed?

I believe that I have more interesting stuff to share now. Years past after my last blogging experience. During those years, I found a chance to work with different teams on different interesting projects and had to solve too many issues. I started to live abroad far away from my home country. I travelled a lot and met interesting people. So, I have changed 😋

The Motivation behind this blog

We all face different problems in our daily routine and the first thing we do is that check other people´s blog posts/wiki pages/news etc. It is great to see that there are too many people outside ready to help other´s but I believe that this process should be mutual.

What kind of blog posts you should expect?

This blog will be a place for me to share my experiences/ideas on a variety of topic. But probably, the main content will be about technical stuff.

Conclusion

I am really happy to blog again and I hope, I can keep this blog alive for years!