DEV Community: Andrey Goncharov

How I got multiple FAANG offers in Europe

Andrey Goncharov — Tue, 31 Aug 2021 09:20:17 +0000

Yet another story of how an ordinary guy from the middle of nowhere in Russia managed to get multiple FAANG offers. This time, in Europe.
I'll tell you a little about my background, share several tips and tricks that helped me to land the interviews and prepare for them, talk about the interviewing process, reveal my final choice.

TL;DR. Sent hundreds of CVs. Solved 100 challenges on Leetcode. Read a book on system design. Went through 9 interviews. Got 4 offers. Accepted 1. Lived happily ever after.

Background

I guess I shed some light on this already, didn't I? We have a guy in his late twenties from the middle of nowhere in Russia. In fact, it is Voronezh, Russia, but have you ever heard about it? My point exactly. 6-ish years of professional commercial experience. Some degree in CS. For simplicity, let's call it a Master's. It is not a true Master's, but this article is not about the system of education in Russia, right? 4 out of these 6 years I spent working at IT outsourcing companies. For the rest of 2 years, I worked remotely for US-based startups. I have a couple of small libraries on GH. Try to do conference talks from time to time. One last thing I tried to leverage to make my CV stand out is this blog. It's, let's face it, tiny, but better than nothing. Hopefully, it shows the world my love for tech. Oh, yeah, I am also pursuing my proper Master's from Georgia Tech online. I think this nicely wraps up my life story. By the way, my name is Andrey G. Nice to meet you!

G. stands for Goncharov. I wanted to spare you the pain of trying to read it.

I like thinking of myself as a bit of an adventurer. One adventure I nourished in my mind for a long time was to move to another country. A year ago, this desire matured into concrete actions. I did a research and outlined a list of countries for the relocation. Once I had the list of countries identified, I decided to give myself a break until the end of the year. At the end of January, after a week of snowboarding, I felt energized enough to get back to the relocation mission.

Sending out applications

Timeline: January - February 2021.

Time to send the actual applications. At first, I thought about applying to only well-known companies. Well-known to me. As I started going through the list of companies in my head, the "to me" part of the previous sentence raised more and more doubts. First, there are great companies out there I haven't even heard of, or, at least, can't remember right away. Second, surprise-surprise, human memory is not perfect. I wrote down the companies I know by heart, and went on with the search. I tried googling various lists of great companies to work at. I browsed dozens of vacancies on LinkedIn. While it all helped my list to grow, the ultimate solution I came up with was to go through the list of S&P 500 companies. If that's not enough for you, you can add NASDAQ on top of it.
A natural question here is why only publicly trading companies? First, I had already tried working for small and medium-sized businesses, wanted to try what it feels like to work for a proper corporation at a proper corporation-sized scale. Second, I figured that a FAANG entry on my resume would act as a quality stamp, opening many doors in the future. Third, I hoped that a rigorous hiring procedure would help to concentrate true talent inside of the company. When you are surrounded by people willing to methodically take a VM apart piece by piece just to figure out why one operation works 5% faster than the other one, it, kind of, becomes a new norm for you, don't you think? At least, it is a heck of a story to tell at a conference. Forth, I figured that larger companies would be less afraid to relocate candidates. Last, my wife is about to start her Master's, and FAANG pays pretty darn well.

Tip #1. The end of January is too late to start sending out the applications if you consider relocation to the US. Most of the US-based jobs with relocation assume you getting an H1B visa. This type of visa is issued only once a year, and your sponsoring company has to apply all necessary documents by mid-March. FAANG giants rarely operate at this pace.

As many other articles on the web suggest, I tried to find referrals for some of the biggest companies. I reached out to folks I studied with and met at conferences. I even tried reaching out to people I do not know at all asking for a reference. As one of the Googlers told me, when they refer somebody, they put in how well they know a candidate. If it is a referral for an anonymous on the internet, it is worthless. Apart from referrals, I mostly used companies' own websites to apply for the jobs.

The first week of February went by. I sent, probably, several hundred applications. My rough estimate is 400. And... nothing. Some of the companies froze their relocation programs due to COVID, some of them silently ignored me. Cosmic powers, my spidey senses, and just common sense came together and hinted that I need either patience or a new strategy. Further thinking transformed it into "patience AND a new strategy". I thought that even I still might see the fruits of my previous work, a new shiny way of getting a one-on-one with a recruiter wouldn't hurt.

Someone smart once said, "A wise man's question contains half the answer". I wouldn't dare calling myself wise eve remotely, yet when asked myself how I was going to make recruiters notice me, the answer was obvious - send messages to the recruiters, dummy! Without any hesitation, I opened my LinkedIn, found that magical search bar, and started looking for recruiters from the companies I wanted to get in the most.

Website applications got me interviews with Booking in the Netherlands, JP Morgan in the UK, Amazon in Germany, Microsoft in the US, Lyft in the US. Spamming recruiters worked with Uber in Netherlands and Databricks in the Netherlands. It also kind of worked with Microsoft, but Microsoft in Poland, that was not my area of interest. It was a bit complicated with my referral for Facebook. I reached out to Michel Weststrate, whom I knew as a creator of immer and mobx. He was a super-nice guy, and he referred me for a frontend role. I was immediately rejected. I decided not to give up, and asked my ex-colleague from Hazelcast, master of everything distributed, Ensar Basri Kahveci, for help. I sent him 3 different roles at Facebook I wanted to try - frontend in the UK, generic software engineer in the UK, and something else I don't even remember now. My thinking was that even if my first application for the frontend role was rejected, on a different day a different recruiter might be looking at a new pile of applications, and he might make a different decision about mine. As to the generic software engineer role, I thought that a totally different recruiter might deal with that role, so it was worth a try. On the next day, I connected on LinkedIn to a recruiter from Facebook in London. To my surprise and joy, he read my message! He told me that he saw the referral in the system, and initiated a call.
My last referral that eventually worked came from Alex Salo - a god of math I met back in my uni days, who now works at Google. I say eventually because I did not have a reliable way to contact Alex, so by the time he referred me, I already maxed out the number of applications one could send to Google in a month. My website applications to Google, US, did not work. So after a month, in March, I tried to apply to Google, Germany, and Google, Switzerland, via Alex's link, and I got the interview!

Tip #2. Big corporations are gigantic monsters. Every limb of that monster usually functions independently. Apply to different positions in the same region, apply to different regions, apply at different dates. You may fail to pass the filter for 99 job ads, but you will be selected for the 100th.

In the end, I interviewed with Booking, JP Morgan, Amazon (twice!), Microsoft, Lyft, Databricks, Uber, Google, and Facebook.

Preparation

Most of the big tech companies (if not all) are going to assess your skills based on 3 criteria:

Domain-specific knowledge
Data structures and algorithms
System design
Soft skills / cultural fit

Domain-specific knowledge

You probably won't see it if you apply for a generic software engineer role. These questions will arise if you apply for more specific roles like fullstack engineer, frontend engineer, and etc. I saw them when I tried myself for a fullstack role at Uber and Booking, a frontend role at JP Morgan.
I did not invest any time into preparation for domain-specific questions hoping that my entire career prepared me for them.

Data structures and algorithms

Leetcode. Lots of leetcode.

Tip #3. I would advise choosing a language you are most comfortable with, even if it does not have all the necessary data structures out-of-the-box. I am most fluent in JavaScript/TypeScript. It lacks many popular data structures like heaps. Every time I needed to use a heap in my solution, my interviewer and I just imagined that I had a library with a reasonable API. From my experience, it is never going to be a problem. Worst case scenario, they ask you to write a heap from scratch. Practice it a couple of times, and it should not present a problem for you.

If you did not touch algorithmic challenges for a while, I would suggest starting with the easy collection of top interview questions. Do not do all of them. Just the recommended ones in each section. After that, I would suggest moving to the medium collection of top interview questions. Like before, go only through the recommended challenges in each section. At this point, you should be able to tell which sections are easier for you, and which are the most problematic. Spend extra time on the sections you find the hardest. Now would be a good time for your first coding interview.

Tip #4. It is ok to be overwhelmed with the complexity. I spent days and weeks feeling utterly stupid. It gets easier once accumulate the critical amount of knowledge. Set for yourself a strict schedule and follow it. I committed to spending up to 2 hours a day going through one section at a time. Leetcode has 6 coding-related sections in each group. This routine gave me an opportunity to more-or-less evenly progress through each section with one extra day to focus on my weak points.

Once you go through the first coding interviews in several companies, and you identify which companies are still in the game, it might be a good idea to start exploring company-specialized sets of challenges. It is no longer free on Leetcode, but it is worth it. I did not have time to solve the specialized sets for each company I interviewed with, so I decided to focus on Google and Facebook. After all, how different these coding tasks could be in other companies? One week I went through the Facebook set, another week through the Google one. And so on.

System design

You will be given a high-level description of a system. You will need to design a high-level architecture for the system by drawing boxes and explaining how they are interconnected and why you need them.

At some interviews you will be asked to focus more on the API design, at others, you will need to provide more insight on how you plan to make the system reliable, going into the depth of availability and performance.

One book that helped me the most was "System Design Interview – An insider's guide" by Alex Xu. It is concise, yet covers a dozen typical systems. As I had a chance to do some system design on my daily job, it was the only book I read. I wanted to spend more time on the algorithms. If you want to dig a bit deeper, check out this great blog post by Gergely Orosz.

Soft skills / cultural fit

It was probably the easiest part for me. I went through "Rock the behavioral interview" card on Leetcode, and wrote down the answers for their questions from my own life and work experience. Do not rely on your memory! Human memory sucks. Write it down.

Preparation recap

I started sending out applications at the end of January. My last interview was in mid-April just like I planned. It gave me 10 weeks of preparation. The first 2 weeks I spent doing only leetcode for 2 hours a day on workdays, 4 hours on Saturdays, leaving Sunday off. Around that time I had my first technical interview. I kept on going through leetcode at the same pace, but without a day off, for 2 more weeks. Next, I started reading one chapter from "System Design Interview – An insider's guide" every workday in addition to leetcode. Once I finished it, I returned to the old leetcode-only routine. It was not easy. I had a full-time job. I needed to work on my Master's from Georgia Tech. I had to ruthlessly prioritize and make sacrifices. I fell in love with to-do lists. I think what helped me the most is starting my day with the preparation and then gradually going through my to-do list starting with the most important tasks.

Interviews

Tip #5. If your interviewing skills are a bit rusty, most likely, you are going to bomb your first couple of interviews. Engage with all the companies from your list as soon as possible, but try to schedule the technical interview for your dream companies at a later point.

I would be thrilled to work at any of the companies from my list above, so I had a really hard time following the advice from "Tip #4". I slightly preferred some of them over the others, but to be respectful to all of them, I am going to tell you about my interview experience at each one of the companies in no particular order.

Lyft

I managed to get a call with a recruiter only. It was the end of February, and we just had no time to go through all tech interviews by the time they would have to file for my H1B visa. See "Tip #1". End of story.

Score: 0 / 9.

Microsoft

I passed the phone call, where I had a leetcode-medium coding challenge. Then after a week of silence, they told me they thought I had a work permit in the US, and they were not willing to sponsor an H1B at that moment. I guess I forgot to click on a non-resident checkbox in my application. Sorry, Microsoft!

Score: 0 / 9.

Databaricks

I failed on the first coding interview. They asked me a medium-complexity leetcode-like question. I did not find the optimal solution. Well, you live, you learn. That's why it is crucial to have several interviews lined up.

Score: 0 / 9.

Booking

I got two domain-specific interviews. Another interview was split in half. First, it was an easy leetcode-like question, then they extended it to a system design question. At the moment, I was working on an offline-first application with multi-user collaboration. I guess it made me over-engineer my final design. After that, I had a behavioral interview with a manager. In the end, I was told that all went well, but the system design. My recruiter offered me to try for a different team, which I did. It was just another behavioral interview. As result, they told me I did not show enough passion.

Score: 0 / 9.

Uber

After a quick call with a recruiter, I got my first domain-specific coding interview. At that moment Uber hired for middle-level and senior-level roles. That interview was supposed to put me either on a middle-level track or on a senior one. After the first round, I was evaluated as a middle-level engineer. Subsequently, I had two more leetcode-like algorithmic challenges, a classical system-design interview, and a behavioral interview with a manager. The results were inconclusive. Uber was not ready to offer me a senior role as they thought my coding was weaker than necessary. They were not ready to offer me a middle-level role as they liked my soft and system design skills. They figured I would be bored without constant exposure to architecture-related tasks, and that's something they provide for their seniors.

Score: 0 / 9.

JP Morgan

After a call with a recruiter, I was up for a full day of interviews. This time no first screening with a coding challenge. I got two domain-specific sessions with more-or-less standard questions about JS and React. After that, I had a behavioral interview with a manager. They really liked that I managed my own portfolio at that time. It was one of the least stressful interview loops. As result, I got an offer for a senior position with relocation to London!

Tip #6. If you have experience in the company domain, do not be shy and talk about it! Surprise-surprise, it matters.

Score: 1 / 9.

Amazon

It is a twisted story that proves tip #2. FAANG corporations do not function as a holistic unit with regard to hiring. You can be rejected in one region, but accepted in a different one!
My first interview was with Amazon in Munich. They sent an online screening test with a couple of easy leetcode-like tasks. Shortly after, I was on a call with an engineer solving a medium leetcode challenge. It took about 30% of the interview, while the majority of the time went into the behavioral part of the interview. I thought I did well... and I was wrong. A few days later, I got a rejection letter.

Be prepared! Amazon asks behavioral questions on every call. Interviews are no longer exclusive to checking one area of knowledge like algorithms or system design. They always check two - your soft skills and either algorithms or system design.

Tip #7. It is OK to use the same story for all your interviews. Usually, a good life story covers several leadership principles they want to hear.

Luckily, it was not the end of the story. I did not stop applying for various vacancies at Amazon. About a week later I got a call from Amazon Cambridge. They did not seem to know or care about my previous failure with Amazon Munich. They did not even mention it. I got the same online screening test. It had different tasks, but the structure was the same - a couple of easy leetcode challenges and forms to explain why I solved them the way I did. A few days later, I jumped on a call with an engineer from the team. This time it was a bit different - same behavioral questions, but two medium leetcodes. I found the optimal solution for the first but did not manage to do the same for the second one. Nevertheless, I passed.

After a month of preparation, I met the team over 4 interviews. 2 medium leetcodes, one system design, and one coding task that I would describe as application/API design. The last one presented me with a vague list of business requirements. I had to clarify them, set meaningful constraints, and then code a class implementing the logic. In the end, they loved my algorithms, my application/API design, but did not meet expectations with my system design skills. If only a guy from Uber could have called and told them about my mad system design skills. It could have been the end of the story, but it was far from over. I auditioned for the backend role in Cambridge, and the guys thought that even though I lacked the necessary system design skills for their team, I still could be a good match for a frontend role. The thing was that they did not have any frontend roles in Cambridge. I had another call with a recruiter, and she redirected me to... surprise-surprise, Germany! There and back again.

They set another loop on interviews. This time much shorter. As far as I remember, it was only a single medium leetcode and another system design. This time they loved it and I got an offer. Offer for a middle-level position. Still counts as an offer though, doesn't it?

Tip #8. Be proactive and talk to your recruiter about the level you are interviewing for. Make convincing arguments, share that you have interviews with other companies for a higher level. The bottom line, do anything to be put in a loop for a higher position.

Score: 2 / 9.

Google

I received a standard FAANG interview package. One phone screener with a medium leetcode. After 2 weeks of prep, an exhausting day with 5 interviews - 1 behavioral, 1 system design, 3 algorithms. Each algorithmic interview was an easy leetcode followed up by a medium leetcode. System design and behavioral were canonical. I got positive feedback. Positive feedback and approval of the committee! Yes, before you get an offer you need to go through a team matching process. To get there, first, some mysterious committee needs to approve your level. In my case, it was L4 - yet another middle level. Remember, tip #8? I wish I knew it back then.

Score: 3 / 9.

Facebook

At first, it seemed like another standard FAANG interview package. However, there was a small catch. They were on a hiring freeze for E4 (middle level). So I had to pass the bar for E5 (senior) without a possible downgrade. A first screening interview with one easy and one medium leetcode. A month of preparation. Finally, 2 days of interviews. 2 algorithmic interviews on day 1. 1 system design and 1 behavioral on day 2. As Mr. Eminem sang, his palms are sweaty, knees weak, arms are heavy... Both algorithmic interviews were "easy leetcode with a medium leetcode followup". I did well on the first one, and absolutely horribly on the second. System design was focused mostly on the API and the DB schema. The behavioral one was more or less standard. They went without a hitch. I heard multiple times about rigorous requirements at FAANG. I was almost ready to say goodbye to this opportunity. Luckily, Facebook was super nice and gave me another chance. They set up another coding interview. If you recollect what Mr. Eminem sang in that song later, it is "There's vomit on his sweater already, mom's spaghetti. He's nervous, but on the surface, he looks calm and ready". Well, I was as far from calm as possible. As result, on the call, I calculated big O for the easy leetcode wrong, but I realized it only after the interview. The interviewer did not seem to notice it or decided not to mention it. However, he thought my solution had a bug. After running various test cases out loud without any luck of finding that bug, I switched to the second part of the interview with the medium leetcode. For that one, I found the best solution and did my math well. I knew that it was my last chance of getting in, and I like thinking of myself as of someone who does not give up that easily. I asked the interviewer to copy-paste my solution for the first task to debug it at home. I created an online test suite with that very implementation and a variety of edge cases. There wasn't a bug! I desperately needed to somehow communicate that sacred knowledge to my interviewer! But how? Facebook does not reveal emails of your interviewers. What it does is reveals their names. It was my lucky day. The guy was an active open source contributor. I found his email in his commits and emailed him the test suite and the correct big O calculation. I do not know if that email had any effect. I never got a reply. At least, I knew I did everything I could. And it worked! A few days later, I received a call from the recruiter with an offer for a senior role in London!

Score: 4 / 9.

Grand finale

After interviews with 9 different well-known prestigious companies, I got 4 offers. After a week of negotiations (here is a great guide on how to do it) and hesitations, I happily decided to move to London with Facebook. I now hope for two things. First, that my story helps someone else in the middle of nowhere in a non-English-speaking country realize that they do not need to be a super extraordinary genius to get to FAANG in a major city. All they need is hard work and dedication. Second, that this is the end of a chapter, not the end of a story :)
Stay tuned for new articles by following me on Twitter or LinkedIn! Subscribe to my newsletter or RSS. Drop me an email if you have any questions.

Immigration in IT: Choosing your new home objectively

Andrey Goncharov — Mon, 26 Oct 2020 13:59:42 +0000

I now have a new shiny blog. Read this article with the latest updates there https://blog.goncharov.page/immigration-with-numbers

So, you have decided to move your ass someplace else, seeking a better life. The whole world is at your feet, but "the whole world" is not a specific place where you can be. You have to choose, if not a city, at least a country. Nowadays, there are plenty of places where your career can skyrocket, and you can have a great quality of life. Of course, it depends on your definition of the "quality of life". In this article, we will try to set subjective factors that drive your decision aside. We will look at the numbers in different areas to rate and, later on, filter our potential relocation prospects for the next X years.

If you are more comfortable reading in Russian check out this article on habr.com.

Criteria

Before we do the measurement itself, we need to know what to measure. So what do we expect from our new home?

Freedom! Freedom of speech, freedom of political and religious views, functional democratic institutes.
No corruption... Ok, let's leave ponies and unicorns in our hopefully happy childhood memories. As little corruption as possible. I would like to see where my taxes go.
Powerful economy. Free market, relaxed regulations, highly diversified high-tech economy. Living off oil and gas is not a good long-term vision.
Wealth, comfort, safety. As software engineers, we can make decent money at any point of the globe. However, our well-being is not everything. Well-being of our neighbors, who are not software engineers, matters as well. Let's just suppose that when the basic needs are satisfied, human beings tend to get creative, self-improve, create new breakthrough companies, and, therefore, lead the nation to new highs. I am not a socialist in any way. I am just saying that your new Tesla might not make you half as happy if dozens of people next to you barely make enough money for food and clothes.
Soft immigration policy. Permanent residence in X years. Passport in X + Y years. Work permit for your spouse.
Moderate taxes. Who wants to pay more, right?
Hot IT job market. Yes, there are plenty of remote opportunities, but, most probably, you will need a local job to get that work visa.
Science. Well-developed and well-sponsored science. Let's just suppose it will lead to more innovation in the technological sector of the economy. Moreover, I, personally, would love to believe that at some point in my life I will do research. After all, I might get tired of making my React application draw just another form to save just another JSON with user data to the database.
Strong army. "People who do not want to feed their own army will soon be forced to feed someone else's." That is, allegedly, Mr. Napoleon Bonaparte, not me.
Immigrant integration and language. Ideally, people should care not where you come from, but what human being are you. Ideally, they should speak English. I can barely come up with a 10-word sentence without 10 mistakes in it, please, do not make me learn another language!
Strong passport that allows as many visa-free entries to different countries as possible.

Approach

To be at least relatively object we will not base our judgment on beautiful or not so beautiful stories and videos of various bloggers, vloggers, and who-knows-what-loggers. Instead, I suggest using indices by, in most cases, non-profit independent organizations.

Freedom
1. Freedom House. Locates in the USA. They promise a rating of political and civil liberties in 210 countries. Founded in 1941. Funded, mostly, with US federal grants.
Corruption
1. Transparency International Corruption Perceptions Index. Located in Berlin, Germany. Transparent, publish annual budget reports. There is a huge list of those, who chip in for the research.
Economy
1. Index of Economic Freedom by American Heritage Foundation. Mostly funded by individuals. Who exactly? I wish I knew.
2. The Global Competitiveness Report by World Economic Forum. Powerful community with 50-year history. They claim a remarkable list of partners. They publish an annual report on how they spend the money. However, it is unclear how much they get from each partner.
3. Atlas of Economic Complexity by Growth Lab in Harvard.
Wealth, comfort, and safety
1. Human Development Index by United Nations Development Programme. Funded mostly by European Union and North America. 100% transparent.
2. Gross National Income per Capita and Average Annual Wages by OECD. Funded by 37 countries. Transparent structure and budget.
3. Victims of Intentional Homicide by UNODC. Yep. United Nations itself. Lots of docs on where the money comes from.
Army
1. SIPRI Military Expenditure by Stockholm International Peace Research Institute. It is unclear how exactly they spend their budget, but they have a detailed list of those, who provide it.
IT job market
1. Let's take a number of job openings on LinkedIn and divide it by the population in millions. For Russia, we will get the vacancies from the local resource - hh.ru.
Immigrant integration and language
1. Migrant IntegrationPolicy Index. There is a list of partners and experts. I could not find the detailed info on the budget.
2. EF English Proficiency Index. Since there is no huge "Donate" button on the website, I assume the organization is commercial. Private commercial organization. So no reports. Sorry, I could not find anything better. At least they sponsored the Olympics several times.
Passport strength
1. Passport Index by Arton Capital. "Capital", kind of, gives us a hint about the commercial nature of the organization. Unfortunately, nothing better.
Science
1. Let's take "Share" value from Nature Index and divide it by the population in millions. Published by Nature. Commercial. If we trust Wikipedia it is extremely respectable.
Taxes
1. International Tax Competitiveness Index by American Tax Foundation. Finally, another non-profit! They publish a financial report, but it is still unclear how exactly it is funded.
2. All-in average personal income tax rates at average wage by family type by good old OECD.
Immigration policy
1. There are some indices trying to somehow rate the migration by people inflow and outflow, but it is not what we are after. We need to rate based on the following criteria:
  1. Requirements for immigration visa.
  2. Requirements for permanent residence.
  3. Work permit for spouse.
  4. Requirements for naturalization.
  5. Multiple citizenship, i.e. if we need to give up an existing passport to be naturalized. Since there is no easy way to assess these factors, we will, first, filter the countries by other indices, and then have a detailed look at the immigration policy of the finalists.

If you want to play with the data right away here is the link to the Google Spreadsheet. I am currently located in Russia. So do not get surprised with a separate bottom row with the numbers for Russia. Feel free to clone the table and replace it with the country you want.

Filter by freedom

We will use the Freedom House Index. They rate political and civil liberties. We could have simply taken all the countries marked as Free (rating above 70 with a few exceptions). Immigration is a big deal. Let's shoot for the stars and raise the bar to 80. Results:

Netherlands
Germany
Luxembourg
Switzerland
Singapore
France
Ireland
United States
Belgium
United Kingdom
Austria
Canada
Sweden
Australia
Czech Republic
Denmark
Spain
Italy
Finland
New Zealand
Japan
Norway
South Korea
Iceland
Slovenia
Cyprus
Poland
Greece
Lithuania
Slovakia
Latvia
Portugal
Chile
Croatia
Argentina
Estonia
Romania
Bulgaria
Uruguay
Trinidad and Tobago
Panama
Costa Rica
Mongolia
Ghana
Taiwan

Here we face the first evidence that humans can never stay 100% objective. I've heard so much nice stuff about Singapore, so I decided to let it pass trough to see how it performs by other criteria.

Filter by Human Development Index

Truly remarkable rating built on 3 pillars: life expectancy, education, and wealth.

If I had to use a single index, I would choose this one. Multi-factor ratings rule!
Let's sort the countries in descending order by Human Development Index.

Country	Human Development Index
Norway	0.954
Switzerland	0.946
Ireland	0.942
Germany	0.939
Iceland	0.938
Sweden	0.937
Singapore	0.935
Netherlands	0.933
Denmark	0.93
Finland	0.925
Canada	0.922
New Zealand	0.921
United States	0.92
United Kingdom	0.92
Belgium	0.919
Japan	0.915
Austria	0.914
Australia	0.914
Luxembourg	0.909
South Korea	0.906
Slovenia	0.902
Spain	0.893
France	0.891
Czech Republic	0.891
Italy	0.883
Cyprus	0.873
Poland	0.872
Greece	0.872
Lithuania	0.869
Slovakia	0.857
Latvia	0.854
Portugal	0.85
Chile	0.847
Croatia	0.837
Argentina	0.83
Estonia	0.822
Romania	0.816
Bulgaria	0.816
Uruguay	0.808
Trinidad and Tobago	0.799
Panama	0.795
Costa Rica	0.794
Mongolia	0.735
Ghana	0.596

Look at that! My bet on Singapore paid off!
I love simplicity, so I would like to take 0.9 as the bottom line. One thing stopping me is that Microsoft has an office in Prague. I do not want to discard a chance to work there just because the number was slightly lower than I wanted. So let's take 0.891 as the bottom line to let Czech Republic pass to the next stage.

We are now left with:

Norway
Switzerland
Ireland
Germany
Iceland
Sweden
Singapore
Netherlands
Denmark
Finland
Canada
New Zealand
United States
United Kingdom
Belgium
Japan
Austria
Australia
Luxembourg
South Korea
Slovenia
Spain
France
Czech Republic

Filter by population

Luxembourg and Iceland show great results so far. Population there is 0.62 and 0.35 millions of people respectfully. I reckon these countries might easily be by-invitation-only clubs that do not welcome new members. To make filtering a bit easier, let's assume that we will only consider countries with 5 million of people and greater. Time to say goodbye to Luxembourg, Iceland, Slovenia, and New Zealand.

Here is what is left:

Norway
Switzerland
Ireland
Germany
Sweden
Singapore
Netherlands
Denmark
Finland
Canada
United States
United Kingdom
Belgium
Japan
Austria
Australia
South Korea
Spain
France
Czech Republic

Filter by corruption

We will rely on the data by Transparency International Corruption Perceptions Index.

Country	Corruption Perceptions Index
Denmark	87
Finland	86
Switzerland	85
Singapore	85
Sweden	85
Australia	85
Norway	84
Netherlands	82
Germany	80
United Kingdom	77
Austria	77
Canada	77
Belgium	75
Ireland	74
Japan	73
France	69
United States	69
Spain	62
South Korea	59
Czech Republic	56

I wish I could take 75 as the bottom line, but it would effectively filter out the USA with tons of amazing work opportunities. Let's adapt our wishes to reality and say that 69 also sounds not that bad.

We now have:

Denmark
Finland
Switzerland
Singapore
Sweden
Australia
Norway
Netherlands
Germany
United Kingdom
Austria
Canada
Belgium
Ireland
Japan
France
United States

Economy

As the primary economic filter, we will use Index of Economic Freedom. It rates inviolability of private property, the effectiveness of the jurisdictional system, the amount of national debt, taxes, business opportunities, and my beloved market freedom.
The creators of the index mark countries as Free if they score over 80. They say the countries are Mostly Free if they score over 70. Let's leave only Free and Mostly Free ones.

Country	Economic Freedom
Singapore	89.4
Australia	82.6
Switzerland	82
Ireland	80.9
United Kingdom	79.3
Denmark	78.3
Canada	78.2
Netherlands	77
United States	76.6
Finland	75.7
Sweden	74.9
Germany	73.5
Norway	73.4
Austria	73.3
Japan	73.3
Belgium	68.9
France	66

Farewell, Belgium, and France.

Nd to the next stage pass:

Singapore
Australia
Switzerland
Ireland
United Kingdom
Denmark
Canada
Netherlands
United States
Finland
Sweden
Germany
Norway
Austria
Japan

You will be surprised, but even at this stage no poor countries left. In all of the above gross national income per capita (smart word for "person") is over $40000. Average salary is also above $40000.

Country	Gross National Income	Average Salary
Singapore	91557	53244
Norway	70525	50956
Switzerland	69545	64109
United States	66637	63093
Ireland	66342	47952
Denmark	58700	55253
Netherlands	58135	54262
Austria	56733	50868
Germany	55979	49813
Sweden	54742	44196
Australia	51924	53349
Finland	49540	44111
Canada	49431	48849
United Kingdom	46253	44770
Japan	42872	40573

As you might remember, Japan scored poorly on corruption (73). Let's leave it out by the confluence of factors.

IT job market

Let's take a number of job openings on LinkedIn and divide it by the population in millions.

Country	Total number of IT job openings	Number of IT job openings per one million people
Netherlands	128194	7410
Germany	480737	5994
Switzerland	37043	4410
Singapore	22105	3565
Ireland	12972	2495
United States	804444	2439
United Kingdom	159013	2417
Austria	16237	1824
Canada	49320	1308
Sweden	11453	1123
Australia	20313	797
Denmark	3695	626
Finland	2244	401
Norway	1334	243

Based on the numbers, I feel the urge to discard Norway, Finland, Denmark, and Australia. However, Australia has offices of several major IT corporations at once - Amazon, Google, Microsoft, Atlassian. So we will let Australia keep its place.

Results:

Netherlands
Germany
Switzerland
Singapore
Ireland
United States
United Kingdom
Austria
Canada
Sweden
Australia

Science

Let's take the adjusted number of articles (Share) from, Nature Index and divide it by the population in millions.

Country	Total number of articles	NUmber of articles per one million people
Switzerland	1467.12	175
Singapore	632.26	102
United States	20462.07	62
Sweden	637.11	62
Germany	4602.86	57
United Kingdom	3762	57
Netherlands	963.86	56
Australia	1275.34	50
Canada	1635.2	43
Austria	363.05	41
Ireland	122.23	24

Ireland - you are the weakest link. Bye!

Our finalists:

Switzerland
Singapore
United States
Sweden
Germany
United Kingdom
Netherlands
Australia
Canada
Austria

Apart from data we have used for filtering, let's add a few other sources::

Finalists as a filtered view in Google Docs

Country	Freedom House	Corruption Perceptions	Economic Freedom	Economic Competitiveness	Economic Complexity	Human Development	Gross National Income per capita ($)	Average wage	Homicide rate	Military budget ($m)	Population (millions)	Total number of IT jobs	Number of IT jobs per 1 million people	English proficiency	Migrant Integration	Passport Power	Total number of scientific articles	Number of scientific jobs per 1 million people	Tax Competitiveness	Tax (%) for average income	Tax (%) for 133% of average income	Tax (%) for 166% of average income
Netherlands	99	82	77	82.4	0.98	0.933	58135	54262	0.58	12059	17.3	128194	7410	70.27	61	115	963.86	56	72.5	37.3	39.8	42.5
Germany	94	80	73.5	81.8	2.09	0.939	55979	49813	0.95	49276	80.2	480737	5994	63.77	63	116	4602.86	57	66.9	49.4	50.8	51
Switzerland	96	85	82	82.3	2.17	0.946	69545	64109	0.59	5179	8.4	37043	4410	60.23	46	116	1467.12	175	79.5	22.3	24.7	26.9
Singapore	50	85	89.4	84.8	1.85	0.935	91557	53244	0.16	11211	6.2	22105	3565	66.82		81	632.26	102
United States	86	69	76.6	83.7	1.55	0.92	66637	63093	4.96	731751	329.8	804444	2439	100		83	20462.07	62	63.7	29.8	32.5	34.2
United Kingdom	94	77	79.3	81.2	1.51	0.92	46253	44770	1.2	48650	65.8	159013	2417	100	56	116	3762	57	60.1	30.9	33.9	37.1
Austria	93	77	73.3	76.6	1.81	0.914	56733	50868	0.97	3237	8.9	16237	1824	64.11	48	116	363.05	41	71.4	47.9	50.8	51
Canada	98	77	78.2	79.6	0.65	0.922	49431	48849	1.76	22197	37.7	49320	1308	100	70	115	1635.2	43	67	30.5	31.2	31.7
Sweden	100	85	74.9	81.2	1.7	0.937	54742	44196	1.08	5290	10.2	11453	1123	68.74	80	115	637.11	62	75.5	42.7	46.8	51
Australia	97	85	82.6	78.7	1.81	0.914	51924	53349	0.89	25912	25.5	20313	797	100	66	116	1275.34	50	76.4	27.9	32.1	34.4

Immigration policy

Information below applies to Russian citizens only! Please, double-check it for your country!

Europe

They have the coolest thing ever - Blue Card. It is a work permit that works across whole European Union except Denmark and Ireland. You need to have a bachelor's degree. There is a solid salary threshold. Spouse CAN work or study.
Apart from the Blue card, each country has its own program for skilled immigrants with lower thresholds. However, the primary benefit of the Blue Card is that you will be able to relocate inside of the European Union and still meet the requirements of EC-long term resident.

Blu Card works for the following countries out of our list of finalists:

Netherlands
Germany
Austria
Sweden

Note that Switzerland does not want your Blue Card! To work there you need to get the local Swiss Work Permit. Your spouse sill CAN work or study.

Singapore

They have Employment Pass. You need to have a job with the local company. Spouse CAN work or study.

UK

The most common way to enter the country is via General work visa (Tier 2). You need a contract with an employer. Your spouse CAN work or study.
There is another option - Global talent visa. If you are a conference star, a book author, regularly publish research papers, you have a shot if you find a good lawyer. You do not need to win a Nobel, but have to have something that makes you stand out. With Global talent visa you will no longer depend on your employer.

Australia

If you get an offer from a major corporation, you will, most probably, wind up with TSS 482 visa. It is a work permit linked to your employer. You can try to work around it by getting Skilled independent visa, but it has an extensive limited queue. With Skilled independent visa you will no longer be linked to your employer. It is no uncommon to enter the country with TSS 482 and get Skilled independent visa later. This way you will have a higher chance of getting the one because the previous experience of working for an Australian employer moves you up the queue.
For both visas, your spouse CAN work or study.

USA

Most of the alien talent comes via H1-B visa. It has a yearly limit. If the number of applications exceeds it, almighty random decides who gets a visa. There are plenty of stories where people tell they got an offer, but could not get in because of this limit.
You could get in via O1, but it has more or less the same requirements as Global talent visa in the UK.
With both of these options, your spouse CANNOT work or study.
There is the only way not to make your spouse give up their career when you move - L1 visa, but there is a catch. The visa is meant for corporations to move their existing employees from one location to another within the US. You have to work for at least 1 year in a different office of the company outside the US to be eligible for L1 visa. The visa is linked to your employer.
This is not the end of the story. The USA is the only country from the list, where you will need your employer to sponsor your permanent residence permit (Green card). It is not true for all cases, but for over 70% of them.
To get your Green card, with a few exceptions, you, as a skilled professional, have 3 options:

EB1 (Priority Workers)
EB2 (Professionals Holding Advanced Degrees or Persons of Exceptional Ability)
EB3 (Skilled Workers, Professionals, and Other Workers)

You can apply for EB1 and EB2 yourself. At the same time, your employer can sponsor them for you. EB3 can only be sponsored.
Let's take a look at the numbers of issued visas in 2019.

EB1 - 2,223
EB2 - 3,497
EB3 - 13,028

Even if all EB1 and EB2 applications were self-petitioned, it means that the percentage of self-petitions is 30.5%. In real world, it should be even less.

Immigration policy cheatsheet

Country	Program	Work permit for spouse	Years to citizenship	Multiple citizenship allowed
Netherlands	Blue Card	+	5	-
Germany	Blue Card	+	8	-
Switzerland	Swiss Work Permit	+	10	+
Singapore	Employment Pass	+	~3	-
United States	H1-B or O-1 or L1	--+	~8	+
United Kingdom	General work visa (Tier 2) or Global Talent visa	+	6	+
Austria	Blue Card	+	10	-
Canada	Express Entry	+	3	+
Sweden	Blue Card	+	5	+
Australia	TSS 482 or Skilled independent visa	+	4	+

For freelancers and contractors

Other countries have some programs for startups, but, as far as I understand, they are meant for bigger companies that create new jobs, not for lonely contractors and freelancers.

Results

A few years ago, after several business trips, I was under the spell of the USA. At this moment, after visiting some other European countries, having collected objective numbers, having studied their immigration policies, I am doubtful I will try moving to the USA over the next years.

Another important factor for me, personally, is the language you can get your Master's in. Next year my wife is going to graduate from the local Russian university with her Bachelor's in Architecture. Unfortunately, Switzerland does not provide a whole lot of Master's in Architecture in English. So we decide to rule it out.

Here is my personal list of finalists:

Netherlands
United Kingdom
Canada
Australia
Sweden
Singapore
Germany
Austria

Here is the same list as a filtered view in Google Spreadsheet

Title image is based on https://www.deviantart.com/gio0989/art/World-map-wallpaper-127690139

balalaika IT newsletter #12

Andrey Goncharov — Sat, 17 Oct 2020 12:49:00 +0000

This is our 12th newsletter. It means we are exploring the world of web development together for half a year. Sad to say, but there will not be any new issues of the newsletter for a long while. This is not a goodbye, rather a see you later. In the meantime, feel free to subscribe to the personal blogs of our curators: https://medium.com/@apechkurov and https://blog.goncharov.page/. Let's stay in touch!

General

In this post Brandon Harris shows us how to synthesize billions of rows of true time series data with an autoregressive component, and then explore it with ClickHouse, a big data scale OLAP RDBMS, all on AWS.
DuckDB - an embeddable database for analytical workloads.
Do you think mmap is faster than syscalls? Well, mmap could make your app run faster, but with strings attached. Sasha Fedorova gives us a great in-depth explanation of how it works.
Distributed transactions are hard to implement right. In fact, many modern databases do not even bother with supporting them. FaunaDB is not one of those, as it builds transactions on top of Calvin algorithm.

Java

Tiered JIT compilation can be found in multiple modern runtimes and JVM is no difference. Some time ago Rafael Winterhalter gave a nice talk on basic principles behind C2 compiler in HotSpot.
ForkJoinPool is available since Java 7 and used in several places within standard library, like parallel streams. It embeds many good ideas, like queue per worker and work stealing. If you are not familiar with these details, you may find them in the original paper by Doug Lea.

JavaScript

Monomorphism is a very important definition for JIT compilation in JavaScript (and not only - see the above Rafael Winterhalter's talk). Several years ago, Vyacheslav Egorov wrote a brilliant post to explain how V8's JIT compiler makes use of monomorphism.
esbuild - a new kid in the family of bundlers and minifiers. Written in Go. Claims to be extremely fast.
NPM 7 is out. Lots of new goodies. Read more about them here.

balalaika IT newsletter #11

Andrey Goncharov — Sat, 03 Oct 2020 09:23:14 +0000

If you like this newsletter subscribe to our new issues at https://balalaikait.com/

General

Joseph Gentle tells us about the rise of CRDTs and why he thinks in could beat Operational Transform in many aspects.
One who wants to write fast code has to understand weak and strong points of modern HW. Cliff Click gave a brilliant talk where he explains those points.
Thread-Per-Core (TPC) is a valid approach for building high-performance server-side software. Yet, it is not trivial to build it right and has a lot of quirks. Some time ago, Redpanda team managed to use TCP architecture on top of Seastar framework and shared their experience.

Java

James Gosling recently gave a lengthy interview on Java, JVM, Emacs, and the early days of software engineering.
Kotlin gets more and more popular among the JVM community. In this post Andrey Breslav sheds some light on the secrets of fast compilation in Kotlin.

JavaScript

Atomics module, which holds low-level synchronization primitives useful for Web Workers, keeps getting new features, like Atomics.waitAsync method, which is currently at TC39's stage 3. V8 team wrote this blog post to explain how Atomics can be used to write sync and async lock primitives.
A great reminder that even high-level languages like JavaScript still suffer from memory leaks. This time the case is kind of unusual - detached windows. Be a good engineer and clean up after yourself!

React

Kent C. Dodds tells us about common pitfalls and myths related to useEffect.

balalaika IT newsletter #10

Andrey Goncharov — Sat, 19 Sep 2020 14:26:41 +0000

If you like this newsletter subscribe to our new issues at http://balalaikait.com/

General

Back to basics. Do you remember how NAT traversal works? David Anderson wrote a lengthy post with great visuals to remind you.
Steve Yegge explains why vendor locks mixed with poor backward compatibility could make anyone question the benefits of a cloud.
CockroachDB team recently introduced Pebble, an open-source K/V store inspired by RocksDB. Read this blog post to learn why they decided to do write it.
Here is a great series of blog posts on implementation of Linux epoll for those of you who enjoy learning about kernel.
If you're not familiar with Log-structured merge-tree data structure (aka LSM tree), but want to learn it, Tarantool DB team did a great job on documenting their implementation.

Java

Java 15 is GA. It brings internal JVM changes (say, it disables biased locking), stable releases of low-latency garbage collectors (ZGC and Shenandoah), experimental APIs (such as Foreign-Memory Access API), and some more goodies.
JNI has a fame of being "slow". But where the JNI overhead comes from and how critical it is? If you want to hear an explanation with all low-level details, here is a great talk by Cliff Click.

JavaScript

Moment.js team now considers the library to be a legacy project in maintenance mode. It is not dead, but it is indeed done. Read more about the reasoning here.

React

Animations can make a site stand out. Or, they can just as easily kill the experience. Prasanjit Singh presents us with the ground rules to master web animations.
AVIF is a fairly new kid on the block and it promises us a new future of the leaner web with smaller images. JPEG is not dead yet, but we keep our fingers crossed. Here is a concise read from Jake Archibald on the new image format.
Wondering how Facebook's Recoil works? Bennett Hardwick explained it for you in 100 lines of code.

balalaika IT newsletter #9

Andrey Goncharov — Sat, 05 Sep 2020 10:58:01 +0000

If you like this newsletter subscribe to our new issues at http://balalaikait.com/

General

CPU caching 101 from Nick Evanson. A concise read about the fundamentals to get you started on how it works and why we need it.
Over the last two years, Uber has attempted to reduce microservice complexity while still maintaining the benefits of a microservice architecture. With this blog post we hope to introduce our generalized approach to microservice architectures, which we refer to as “Domain-Oriented Microservice Architecture” (DOMA).
Distributed locks are infamous for being hard to use right. Some time ago Hazelcast team introduced locks as a part of CP Subsystem based on Raft, a well-known consensus algorithm. This blog post describes potential pitfalls of distributed lock and how FencedLock solves them.
Great engineers of the past have a lot to learn from. In this talk explains how Erlang was designed to scale well on multicore machines.

JavaScript

ES6 introduced many nice built-in collections, such as Map, Set, WeakMap, and WeakSet. Unfortunately, the spec does not put many requirements for concrete implementations. If you want to learn practical details of Maps and Sets implementation in V8, read this blog post by Andrey P.
JIT compiler is what makes V8 and other JS engines blazing fast. But exactly does it do? Watch this talk by Franziska Hinkelmann to learn the answer.

React

Is it possible to build SPAs purely in Rust and without writing a single line of JavaScript? The short answer is YES! Read about the experiment here.

Node.js

A concise guide on how to debug segmentation faults in Node.js.

Java

Did you know that java.security.SecureRandom may be more or less secure depending on the configuration? This blog post explains how to use it in the right way.

How to display a gazillion of metrics and keep your sanity

Andrey Goncharov — Wed, 02 Sep 2020 12:29:06 +0000

I now have a new shiny blog. Read this article with the latest updates there https://blog.goncharov.page/how-to-display-a-gazillion-of-metrics-and-keep-your-sanity

Large scale equals distributed. Distributed equals inevitable complexity. Complexity at runtime equals extensive monitoring. At Hazelcast, doing distributed systems well is our bread and butter. It means we have no choice but to be huge fans of collecting all kinds of metrics to stay on guard of the data our users trust us with.

In Management Center 4.2020.08, we drastically changed the model of how we transfer the metric data from the cluster members to the Management Center, how we store it, and how we display it. In this post, we are going to talk about the latter bit of the triad.

We will discuss what to do when you want to display all the data at once, but your users have a limited number of monitors and only one pair of eyes. We will speculate about what users actually want to see when they look at a chart of a monitoring web app. We will go over different approaches to filter the data, and how an average, a median, and a definite integral play their key roles.

4.2020.08! 4.2020.08! 4.2020.08! Why are we so excited?

If you are not interested in the new features of 4.2020.08 you can jump straight to what users want to see and why.

This release stands on 3 pillars:

Enhanced UX for developers
Unification with the Jet Management Center
New metric engine

Pillar 1. Enhanced UX for developers.

We love developers and loathe unnecessarily complex UIs. As you might know, Hazelcast Management Center supports numerous security providers. Prior to 4.2020.08, the default choice was to use username/password based authentication, which we called the "Default" authentication. We are happy to present you with the "Dev Mode" security provider!

Internally, we call it "Just click save" security provider. Go ahead and try it yourself. We are keen to hear back from you!

Pillar 2. Unification with the Jet Management Center.

What the heck is Jet? It is a blazing fast distributed computing platform built on top of Hazelcast In-Memory Data Grid. Here you can read more about it. Previously, Jet had its own Management Center, but not anymore. One Hazelcast - one Management Center!

Pillar 3. New metric engine.

Since the dawn of time we used the "push" model to collect the metric data from the cluster. It is not even entirely correct to use the word "collect" here. Collecting is an active process. We received the metric data from the cluster. Every node had to know the address of the management center, and it pushed its metric data to that address.
With 4.2020.08 we are trilled to reverse that paradigm and start polling cluster members for metric data! It is a huge change but we have not stopped there. Here is a list with new major metric-related features:

The "poll" model instead of the "push" one.
A new metrics storage for time-series data implemented on top of RocksDB.
Prometheus exporter for the metric data.
A new customizable UI widget to display the metric data.

Having that said, let's dive deeper in the anatomy of the widget, and what problems we faced while creating it.

The widget

What users want to see and why

What is the purpose of monitoring? One could argue that the whole point is to provide top management with pretty charts and multi-page reports with statistics. Jokes aside. What is the real purpose of monitoring? Why do we stare at all these graphs for ours? The ultimate goal is to detect abnormalities as early as possible. Why? Because any abnormality is a predecessor to potential issues.

Speaking of monitoring in general, we are usually speaking of time series data (a list of data points in time order, in our case they come to the front end as { time: number; value: number }[]). Speaking of monitoring of distributed systems, we are usually speaking of multiple time series. Multiple time series cause multiple problems.

Problem 1. How do we analyze graphs?

We have several data points. We use fancy or not so fancy methods of interpolation to draw continuous lines that resemble the original continuous real-world data standing behind the discrete data. How do we detect abnormality there?

We can analyze graphs:

Individually
As a group

For individual analysis, we already have the data points containing scalar values. Comparing scalars is, if not simple, at least doable.

For group analysis, we have to somehow calculate a single scalar out of a time series for each time series, and compare the resulting scalars. There are endless possibilities of how one could do the calculation, depending on the needs of the business. We will go over a small bit of them that we chose for our application down below.

As we can see, for both methods of analysis we end up with a list of scalars that we need to somehow compare.

Problem 2. How do we find an abnormality in a list of scalars?

Generally speaking, we have 4 ways of describing abnormality:

We can define a range a values that are considered normal. If anything falls out of the range then it is abnormal. For example, we can say that [-42,42] is our normal range. Then 7 would be considered normal, and 43 would be abnormal.
We can define a predicate for normality. If a value satisfies the predicate than it is considered normal. Otherwise, it is abnormal. For example, we can say that all integer odd values are normal range. Then 7 would be considered normal, and 7.5 or 42 would be abnormal.
We can define a range a values that are considered abnormal. If anything falls out of the range then it is normal. Basically, it is a negation of #1.
We can define a predicate for abnormality. If a value satisfies the predicate than it is considered abnormal. Otherwise, it is normal. Basically, it is a negation of #2.

We can also detect abnormality iteratively, using different methods (listed above) at different stages, chaining them. For example, we could consider abnormal only integer odd values that belong to [0, 42).

With a few exceptions, predicates make the most sense for analyzing potentially infinite lists, as they, usually, require limited processing power and memory.

Ranges are not that simple. Ranges can be:

Static
Dynamic

Static range is constant predefined range that does not change when new metrics data comes in. Say, we have a cluster of 30 machines. Every machine has 16 GB of RAM. Knowing the average expected load, we could identify the normal range for the used RAM as [1.6 GB, 14.4 GB] (from 10% and up to 90%). That would be a perfect example of a static range. It does not depend on the number of machines or the phase of the moon.

For some metrics and under certain conditions, we could define a range of normal values beforehand, without analyzing the metric data itself. Say, we have a cluster of 30 machines. Every machine has 16 GB of RAM. Knowing the expected load, we could identify the normal range for the used RAM as [1.6 GB, 14.4 GB] (from 10% and up to 90%).
However, not all metrics have a universal predefined range of possible values that make sense for any system. For those metrics, we have to evaluate them by comparison to their peers, i.e. process the whole list of scalars to define the normal range, and then process the list all over again to identify the abnormal range. We can see that the complexity here scales at least linearly with the size of the list.

As a result, we see that abnormality could be identified in 2 ways:

Absolutely
Relatively

The absolute abnormality detection is simpler to understand and implement. It also, usually, requires less processing power as it does not require a dynamic range calculation. It could be used on a potentially unlimited list of scalars. However, it has a restricted area of application as not all metrics have a constant predefined range of metrics.
The relative abnormality detection is more complex. It requires a dynamic range calculation and, therefore, could be used only on the limited list of scalars. However, it could be applied to any metric in the cluster.

The absolute abnormality detection requires users to have deep knowledge of the system to define the normal/abnormal range. In comparison, the relative abnormality detection allows users to easily identify if any metric on any node significantly deviates from the average cluster value without the prior knowledge of the system, or without the expert knowledge of the normal reference range for the system.

Problem 3. How do we display the abnormality?

Say, we have 100 time series. We identified 20 of them as abnormal. How do we display them?

We could display all 100 graphs, and somehow mark the 20 abnormal ones. For instance, we could color code them, but would you be comfortable browsing through 100 graphs? I certainly would not be. I seriously doubt any living human being could make any sense of that mess.

We could display only the 20 abnormal ones. But then we wouldn't know what is considered normal. What was the benchmark against which we identified the abnormality?

Should we display the 20 abnormal ones and additionally draw a graph with the calculated reference value, considered normal? Yes and no! Yes, because it is the correct principal. We should display the abnormal lines along with the reference normal one. No, because 20 graphs are still too much to comprehend. Various studies on how many items a human being can focus on simultaneously show different results on a range from 3 to 7-9. As the final number varies from study to study, it is still a single-digit number everywhere. It means that 20 is still not good enough. How do we go from 20 to a single-digit number?

Previously, we said we could detect abnormality iteratively. Why don't we apply that idea here? Once we detect the initial list of abnormal scalars, we could sort them in the ascending order by their deviation from the reference normal scalar, and leave only those with the biggest deviation.

After experimenting a bit ourselves, we decided that 4 is a good number. So for a single metric we decided to display up to 3 abnormal graphs along with the reference normal one.

Filters

Hazelcast cluster could have hundreds of members. It means that for every metric we have hundreds of time series. We could not display all of them. Inevitably, we had to somehow help our users to find the abnormality and filter out the result graphs.

As of Management Center 4.2020.08, we provide our users with 6 different filters:

Peak Top 3
Peak Bottom 3
Average Top 3
Average Bottom 3
Outliers
Manual Selection

Manual Selection

Let's start with the most boring one - Manual Selection. It allows users to select several members out of the list of cluster members they want to see the metric data for. Our front end receives the data only for those members. Piece of cake.

Peak Top 3/Peak Bottom 3

Do you remember what we have to do to analyze time series as a group? We have to calculate a scalar value for every time series. When you think of a calculation approach, one of the first things that comes to mind is just to take the absolute maximum value of the function.

Given that every data point is { time: number; value: number }, the implementation is straightforward:

const timeSeriesResScalar = Max.max(timeSeries.map(({ value }) => value))

After that, we sort our multiple time series in the descending order by their calculated scalars, and take the first 3 time series. We call this filter Peak Top 3. For the Peak Bottom 3, we swap Math.max for Math.min, but the idea is the same.

We do not calculate any average cluster value (a reference normal value) with this approach, leaving the final judgment of normality to the admin. However, we might start doing it if there is a strong demand.

Have you noticed any problems with the Peak filter? One data point with an extraordinary value can make the whole time series range higher even though all other data points have rather small values. For some metrics with modest volatility Peak filter is exactly what we need. Those short term surges could indicate a potential abnormality. But what about highly volatile metrics? We need a way to minimize the effect of those short-term spikes on the result scalar. Here is where the story of mean and median begins.

Mean and median

Barry has 4 pots of gold. Amy, John and Douglas have 0. The mean value here is 1. The median, however, is 0. How do we calculate them?

Mean is the average you are used to. Calculate the sum of the items in the list and divide it by the number of the items.
Median is the middle value of the list. Sort the list in the ascending order. If the number of items is odd, the middle item of your sorted list is the median. If the number of items is even, sum two middle numbers and divide them by 2.

In terms of monitoring it means that mean accounts for extraordinary spikes. Several of them can drastically change the resulting scalar. Median, on the contrary, filters out the surges.

Should we stop at the mean and the median though?

Average by integral

Let me rephrase it. Should we stop at the arithmetic mean and the median though? After all, we are not calculating an average of a list of simple numbers, these are data points of a time series. By extracting only values and operating only on the values we simply disregard the time factor.

Consider these two graphs:

The arithmetic means for their values are the same. Do you think they adequately represent the average value for both functions? What scalar value could be used to describe a function?

It is the area under the curve. Ladies and gentlemen, please, give your warmest welcome to the definite integral!

Usually, we are speaking of a definite integral of a function, but with a time series we do not have a function. Instead, we have a list of discrete data points. Luckily, we do not have to reinvent the wheel. There is an area of study in math researching specifically this problem called numerical integration of discrete data or discrete data integration.

There are various methods of discrete data integration. We will consider only one of them. The one we used for the widget - the trapezoidal rule.

Consider the following graph:

You can see how the data points of the time series form 3 trapezoids. To find the area under the interpolated curve we need to find areas of those 3 trapezoids and sum them up.

For trapezoid A it is ((2+4)/2)*(2-1) and equals 3. For trapezoid B it is ((1+4)/2)*(2.5-2) and equals 1.25. For trapezoid C it is ((1+1.5)/2)*(7-2.5) and equals 5.625. So the final square is 3+1.25+5.625 and equals 9.875.

To find the average value of the function we now only need to divide its area by its projection on the x axis. Imagine it as if we are trying to find the height of a rectangle with the same length as the projection of the function and with the same area.

So the TypeScript function to find the average could look like this:

export const averageByTrapezoidalRule = (
  data: { value: number; time: number }[],
): number => {
  let totalArea = 0

  for (let i = 0; i < data.length - 1; i += 1) {
    const point1 = data[i]
    const point2 = data[i + 1]

    totalArea += ((point2.value + point1.value) / 2) * (point2.time - point1.time)
  }

  const average = totalArea / (data[data.length - 1].time - data[0].time)

  return average
}

Average Top 3/Average Bottom 3

With all that knowledge, let's now build an algorithm for the Average Top 3 filter:

For every time series calculate its average by the trapezoidal rule.
Sort the multiple time series in the descending order.
Take the first 3 of them.

For the Average Bottom 3 we just reverse the sorting.

For both of these filters, we still do not calculate any average cluster value (a reference normal value) leaving the final judgment of normality to the admin.

Outliers

Alrighty, so all previous filters require some expert knowledge of the system to identify if the displayed values are normal or abnormal. Could we create a filter that unleashes the whole power of the relative abnormality detection, and helps us find the abnormal graphs based on the peer values?

It is our default filter. We call it the Outliers or simply auto. The core idea is to calculate the reference normal value based on the multiple time series as some sort of average. Calculate averages for every time series using the trapezoidal rule. Compare their averages with the reference value. If it deviates from the reference value by more than 30%, consider it abnormal.

The trick is with finding the reference value. Originally, we did it all on the frontend.

We calculated averages for every time series.
We calculated a median of those averages.
We used the median as the reference value.

Why the median? Say, you have a cluster of 10 nodes. 2 of those nodes deviate from the median by some extreme value, say, 300%. They could shift the reference value, so one of the normally behaving members could start being an outlier. The median mitigates the effect of outliers on the reference value.

Later, our backend provided us with a cool aggregation API for the metric time series data. So now we:

Request the MEDIAN aggregate for all nodes.
1. MEDIAN aggregate is a time series as well. For its every data point it takes values for all nodes for the timestamp, and returns a median of those values.
Calculate the average of the median time series.
Use the calculated average as the reference value.

Now we disassembled our reference value calculation. Only one last question still stands. What happens when there are too many outliers?

If we find more than 3 outliers we sort the outliers in the ascending order by their deviation from the reference value, and take only the top 3 with the greatest deviation.

The final Outliers algorithm looks like this:

Request the MEDIAN aggregate.
Display it as a dashed graph on the chart.
Calculate its average by the trapezoidal rule. Consider it a reference value.
Calculate averages for every time series.
Leave only ones that deviate from the reference value by more than 30%.
Sort them in ascending order by the deviation. Display the top 3 time series with the greatest deviation.

As you can see, with the Outliers we have the reference graph, so it is easy to understand what is considered normal at first glance.

Future plans for the widget

Going further, we would like to:

Make the outliers threshold adjustable.
Allow zooming and out for both X and Y axes.
Make the maximum number of graphs adjustable.

We encourage you try he new Management Center yourself, and get back to us with your feedback! Stay tuned for the new posts with the lessons we learned a truly distributed enterprise-ready in-memory data grid.

balalaika IT newsletter #8

Andrey Goncharov — Sat, 22 Aug 2020 10:24:11 +0000

General

Databases are one of the most complex pieces of software ever made by humans, not only in terms of their internals, but also in terms of the behavior for the end-users. This comprehensive blog post might help you to avoid common pitfalls when dealing with DBs.
Designing a successful language is very hard, no doubts. If you are interested in the topic, watch this talk by Brian Kernighan.
Nowadays application development assumes dealing with distributed systems of various kinds. If you feel that you are at the beginner level in the wild ocean of distributed systems, you should watch this nice overview talk.

Java

Were you capable to catch up with all the new stuff introduced in Java 9, 10, 11, 12, 13, and beyond? If not, here is a busy developers’ guide from Grzegorz Piwowarek.

JavaScript

On August 20, Microsoft announced a new major version of TypeScript - 4.0. It packs lots of goodies. This lengthy post goes over them one by one and it is an absolute must-read for every TS developer.

React

TypeScript is the official language of frontend web development at Airbnb. Yet, the process of adopting TypeScript and migrating a mature codebase containing thousands of JavaScript files didn’t happen in one day. In this article Sergii Rudenko tells us about their TypeScript migration strategy and tools.
On August 10, React published their first release candidate for React v17. Even though it’s almost three years since a major release of React, surprisingly, there are no new features added for the new version. This concise read covers the major updates.

Node.js

Node.js Technical Steering Committee is currently running a survey on unhandled Promise rejections handling. You have a chance to a part and impact the final decision. Here is the blog post which explains the rationale behind the survey.

balalaika IT newsletter #7

Andrey Goncharov — Sat, 08 Aug 2020 07:58:08 +0000

Frontend

A short, yet insightful post about enhanced A11Y at Facebook. You might think it is easy and not a big deal until you have to re-write your whole application to meet the AIRA guidelines.

Node.js

What do you know about the speculative compilation in JavaScript? This lengthy read goes in-depth of how modern JS VMs make your code run faster.
There is a certain learning curve related with contributions to the native side of Node.js core. Luckily this great tips and tricks post from Joyee Cheung is there to help you.

Java

It is not a big secret that GC may introduce latency spikes. Such spikes are especially harmful in case of stream processing engine, like Hazelcast Jet or Apache Spark Streaming. Read this blog post series from Jet team to see results of various experiments with modern Java garbage collectors.
Did you know that there is a Java Engineering Group in Microsoft? Moreover, these guys wrote a nice JEP on Stack Allocation support in HotSpot C2 to eliminate non-escaping heap allocations where applicable.

General

Distributed systems are much more than the AP/CP choice implied by the so called CAP theorem. Conflict-free replicated data types (CRDT) assume eventual consistency, scale really well, and provide math-based way to resolve inconsistencies between replicas. In this talk Martin Kleppmann goes beyond the introductory material on CRDTs.
API seems to be so natural term that we assume it was there since the beginning of computer era. But that is not true. If you want to learn when and how the concept of API was invented, watch this talk from Joshua Bloch.
Dropbox migrated from Nginx to Envoy. In this blogpost Alexey Ivanov and Oleg Guba talk about the old Nginx-based traffic infrastructure, its pain points, and the benefits we gained by migrating to Envoy.

balalaika IT newsletter #6

Andrey Goncharov — Sat, 25 Jul 2020 16:25:44 +0000

Frontend

A comprehensive guide on testing in React from Johannes Kettmann. A single long-read that covers it all.
Did you ever hear of CSS Houdini? Lisi Linhart did not just hear about it, but went above and beyond and wrote a great piece on its performance.
Adobe open-sourced its own design system and state management library for React!

Node.js

Moment.js, Luxon or date-fns? Hopefully, one day there will be a built-in API that if not removes but at least reduces the need for third-party libraries to manipulate date and time. Here is a TC39 proposal for the new Temporal API that is destined to fix JS Date.
Nowadays almost every Node.js developer has a good understanding of the event loop principles. Yet, only some folks understand how node works from the low-level perspective. If you want to learn more about non-blocking I/O, watch this interesting talk by Sam Roberts.

Java

Some of java.util.concurrent.atomic classes expose #lazySet() methods which may be a significant performance win for a single writer scenario in the concurrent mode. If you are not familiar with those methods and want to understand how they differ from ordinary volatile writes take a look at this post and this one as well.
Everyone knows Netty, which is used as a building block in many other frameworks. One of the main reasons behind that popularity is performance. Watch this talk to learn about some optimizations inside Netty, including the "do not try this at home" ones.

General

Linux kernel scheduler has to make a lot of decisions and it has to make them fast. Here is a detailed explanation of how the scheduler works.
Writing correct concurrent code is hard, no doubts. Writing performant concurrent code is even harder. If you aim for both of these goals, read this awesome blog post that compares the costs of different concurrent primitives.

balalaika IT newsletter #5

Andrey Goncharov — Sat, 11 Jul 2020 09:52:29 +0000

Frontend

Does CSS grid seem kind of complicated to you? Do you find yourself constantly fighting it, endlessly browsing web for yet another hack? Here is a remarkable cheatsheet destined to keep your sanity.
An outstanding overview of scrolling technologies and approaches by Zach Saucier.
Security is something frontend engineers are often not thinking about. It seems like a task for the backend team. What possibly could go wrong on the frontend? Here is a lengthy list every React developer should keep in mind.

Node.js

The TSC is going to have "Next 10 years of Node.js" session on July 16. Here you can find the agenda.
What exciting things happened in the JavaScript language and the V8 engine in 2019? Shu and Leszek take a tour of some new features and improvements.

Java

What may be simpler than measuring time, right? In reality, Java has multiple APIs to do the job and there are some important restrictions related to each one of those. Here is a great deep dive on the topic.
Have you ever asked yourself a question of whether the allocation of Java objects on the heap involves any contention? By all means, it should, as multiple threads have to be coordinated during those allocations. Yet, HotSpot JVM has another tricky optimization, called Thread-Local Allocation Buffers which allows avoiding contention in most scenarios. If you want to learn more, here is an extensive read.

General

At first glance, load balancing seems to be simple as a concept. In fact, it is extremely hard to make it right. Guys from DataStax recently implemented the Power of Two Random Choices algorithm for Apache Cassandra. It showed a sizeable improvement in high load scenarios. Learn more here.
The story of the Slack downtime on May 12th, 2020.
ARM is conquering the world. Soon you will find it in servers, supercomputers, and even your MacBook!

balalaika IT newsletter #4

Andrey Goncharov — Sat, 27 Jun 2020 09:00:04 +0000

Frontend

An in-depth guide on building your own 100% reactive renderer from scratch.
A must-read research on CPU cost of JavaScript.
Bootstrap 5 alpha is out! This time no jQuery and no IE11.
A guided tour of what it is like to work on the SpiderMonkey (JavaScript engine) compiler and improve conformance with ECMA-262, the JavaScript Specification. From Mozilla to people with love.

Node.js

Sad to say it, but it looks like hapi.js project is reaching its end. If you happen to use hapi.js, you should consider switching to another web framework in the nearest future.
As of v7 npm support both yarn.lock and package-lock.json. Here is a great read on npm v7 treats them and about the future of npm dependency resolution algorithms.

Java

HotSpot JVM has tons of optimizations all over the place. For instance, even the default hashCode() hides some secrets. Learn more about this optimization here.
As an interesting follow-up, you might want to take a look at the j.l.String#hashCode() implementation which caches the hash code in a non-volatile hash field. This optimization was described in the Java Concurrency in Practice book here and here. Do not do it at home!
If you are not familiar with the history behind Generics, which have a controversial reputation in Java community, you should read this post from Brian Goetz.

General

Jepsen team recently did an analysis of RedisRaft, a new module that provides strict serializability for Redis. As usual, some issues were found. As usual, some promises of fixes were made.