DEV Community: Denat Hoxha

Sharing Data Between Microservices

Denat Hoxha — Tue, 25 Oct 2022 17:20:25 +0000

When I started working with microservices, I took the common rule of "two services must not share a data source" a bit too literally.

I saw it stapled everywhere on the internet: "thou shalt not share a DB between two services", and it definitely made sense. A service must own its data and retain the freedom to change its schema as it pleases, without changing its external-facing API.

But there's an important subtlety here that I didn't understand until much later. To apply this rule properly, we have to distinguish between sharing a data source and sharing data.

Why sharing a data source is bad

An example: the Products service should own the products table, and all the records in it. They expose this data to other teams via an API, a products GraphQL query, and the creation of these records via a createProduct mutation.

The Products service has ownership over the products source of truth, and no other team should reach into this directly, ever. If they want data out of it, they should ask the Products service for it via the contract (API) they adhere to. Under no circumstance should you allow direct access to the database, or you’ll lose the freedom to make changes to your schema. I learned this the hard way..

Sharing data is OK

The fact is, services need data that belong to other services.

Example: A Trip service will need access to passengers (from Passenger service) and drivers (from Driver service) to serve trip overviews.

Trip service asks each respective service for its data, synchronously, to fulfill the original request (getTrips). We can rest assured that the data is fresh, and the requesting client will get a strongly consistent view of the data (some of you might see where I’m heading at this point ;).

This synchronous request/response model to transmit data between microservices is a very natural mental model for teams that start off in microservices, at least in my experience. You need some data, you know where to get it, you ask the owning service for it, it gives the data to you, on demand.

On top of that, serving fresh, strongly consistent data was a no-brainer for teams I was in. Strongly consistent data means up-to-date data, the absolute “freshest” piece of data, straight from the source (of truth). To me, back then, serving anything other than consistent data was unacceptable. How could you serve anything other than up-to-date data? Anything else would be a lie!

We applied these patterns as dogma because we saw no other way, and above all, it felt natural.

Synchronicity and strong consistency don't scale

Architectures that rely heavily on synchronous requests and strong consistency do not scale well. Sometimes it’s just not feasible, or strictly necessary, to always go straight to the source for your data needs.

The Trips service example above looks neat at first, but it’s rarely the case that systems remain that simple. New services are born, and they’ll require data from existing services. Sticking to the synchronous request pattern will, in time, have you end up with a tangled web of requests between services. Here’s a scenario:

A user completed a challenge, runs a completeChallenge mutation to Challenge service
After storing the completion, Challenge service lets the Leaderboard service know, so it can update the leaderboard
The Leaderboard service asks User service for user display names and avatars to build the new leaderboard state
Leaderboard service sees that there’s a new leader in the new leaderboard state, lets Notification service know so it can notify participants that there’s a new leader!
Notification service asks User service for the up-to-date email addresses of the users in that particular leaderboard, so it can send emails out

The User service is clearly a point of contention here: everyone is in one way or another depending on it. Imagine this service being down: it’ll render most of the other services down as well. Not only that, but you’re gonna have to make sure to keep this server buffed up at all times with more replicas and a high-performant DB to keep up with the demand.

On top of that, each hop in this chain of requests adds latency to the entire request. Each hop has potential to add an exponential amount of latency, because every service in the dependency chain could fire off more than one request to its own dependencies. Before you know it, you’ve hit unbearable levels of latency.

Finally, every additional dependency in the request chain increases the likelihood of the entire request chain failing. In a request chain involving five services with an SLA of 99.9% (~9h of yearly downtime), the composite SLA becomes 99.5%. That’s almost 2 days of downtime per year!

We can avoid all of these downsides by asking one question: do services really need up-to-date data?

The Notification service (step 5) arguably does. If a user changed their address and the Notification service doesn’t know about it, it’ll risk sending an email to the wrong address and not getting the notification to the intended user.

The Leaderboard service, on the other hand, probably doesn’t need the up-to-date display names and avatars to build the leaderboard — it’s not that big of a deal if users see stale avatars or display names.

As you can see, services have different data consistency needs. There are trade-offs that we can use as leverage to apply different data sharing methods and build a more robust distributed system.

Enter Eventual Consistency

It’s at this point in my career that I discovered that services can maintain a copy of other services’ data, locally in its own DB tables. It comes with the responsibility to maintain that data, either via events or polling.

Included in this package is the fact that the data may be stale for some time, but that it’ll eventually be updated, meaning the data is eventually consistent. We can’t guarantee that the data isn’t stale, but we can guarantee that we’ll eventually catch up.

The moment it “clicked” for me was when I thought about it from the perspective of a backend service depending on a public weather API for weather data. Instead of retrieving weather data for Prishtina or Berlin every time a user from those respective cities needs weather data, I cache it (maybe multiple times a day) by materializing it in a local table and serve cached data to those users. I’ve made the trade-off in favor of eventual consistency because it’s not crucial for my users to see the most fresh data, it’s fine if it’s a few hours stale.

Going back to the Challenge example: we can cut off many synchronous dependencies to the User service by just maintaining a local copy of users in the services:

Leaderboard service can maintain a local copy of users and avoid having to make requests to User service. No one really minds if the data is a bit old, it’s not a showstopper if someone sees a slightly old avatar.
Challenge service can do the same; say if it exposed a getChallengeDetails query and needed user display names and avatars to show the current challenge participants — it can also serve this eventually consistent data from its own materialized users table.
Notification service, although a bit more sensitive, can also utilize data sharing to remove its dependency on Users service. It can materialize users locally and maintain a best-effort updated state by listening to User updated events to ensure it has the most up-to-date emails.

Although we didn’t talk much about how services share this data (a topic for another time that I’m currently writing up!), a final example architecture would make use of a combination of event sourcing and caching.

If you want more examples, take a look at How to Share Data Between Microservices on High Scale by Shiran Metsuyanim, an engineer at Fiverr. It’s a great post that shows how to maintain robustness when adding a new service, by laying down the constraints, and then discussing the trade-offs between the possible synchronous, asynchronous, and hybrid solutions.

Conclusion

I wanted to get this point across to developers that, like me a few years ago, are stuck in the literal sense of “do not share data” but must realize that this only applies to not sharing the source of truth. Maintaining a copy of a service’s data in another service’s domain is perfectly fine, and embraces eventual consistency, a handy tool with its own trade-offs.

In a (very near) future post, I’ll be discussing the many possible solutions of sharing data between microservices.

The Benefits of Fail-fast Systems

Denat Hoxha — Mon, 30 May 2022 10:14:00 +0000

How do you handle an unexpected failure or system state?

Do you handle it gracefully and do a best-effort attempt at returning a response? (fail-safe)

Or, do you stop the failure in its tracks, let it blow up, and sound the alarms? (fail fast)

The general approach that has worked best for me is to always fail fast unless you’re close to the end-user.

The user interface (frontend) is the closest you can be to the end-user. When running into a failure, fatally crashing the UI is the absolute worst response. Any type of response would be better than a fatal crash: serving cached (but stale) data, partial responses, or a retry button. As a last resort, you should be showing a general “Whoops, something broke!” message.

It’s worth putting in the effort in determining the failure state combinations of our frontend to gracefully handle failures and ultimately deliver a good experience. A bad user experience causes churn and uninstalls.

However, in any other situation, failing fast is a vastly superior strategy to make your systems more robust.

Failing visibly makes bugs easier to find

When your service encounters an invalid state, failing fast means failing visibly. After halting execution, make it clearly visible that your system has encountered an invalid state.

Failing visibly makes defects much harder to miss. Even if you have logging in place (which you should have anyway), errors are much more likely to be noticed if all involved parties have it “blow up in their face”.

Easier debugging

Continuing execution after an invalid state makes your system much harder to debug. Instead of knowing exactly where execution stopped, you’ll now have to deal with stepping into your code, reproducing the scenario, and figuring out at what point your program diverged into the state.

Avoid cascading failures

Unless you’ve carefully controlled all possible continuation scenarios, allowing execution by failing safe means you’re effectively allowing your system to enter unknown territory. Unexpected invalid system states lead to more invalid states, and before you know it, they cascade into a much larger failure than if you had stopped it in time.

Less cognitive load and simple mental models

Failing safe means adding branches of possible code paths, resulting in more things to juggle around in your head and, consequently, a greater likelihood of mistakes.

Failing fast means predictable and deliberate programming. You write code confidently when you can rest assured that the system is in the state you expect it to be. By failing fast, you’ve effectively ruled out the possibility of unexpected states, allowing you to work with a simple mental model of your system.

Assertive Programming

There’s a whole software development methodology for fail-fast enthusiasts, called assertive programming, which I first read about in The Pragmatic Programmer.

Assertive programming follows the principle of failing fast by using assertions in the code to continuously validate the system’s state, throwing (crashing) if an assertion’s criteria have not been met.

Here’s an example:

const adult = generateAdult();
assert(adult.age >= 18);
sellItemTo(adult);

We add an assertion after generating an adult as a defensive check to ensure that the system will only ever get to line 3 if the statement adult.age >= 18 holds true. You might think, “but that won’t ever happen”. Well, although this example is an oversimplification, it’s always worth adding assertions to ensure that something that can’t happen, won’t.

Assertive programming is a good practice to validate your assumptions as you write code, especially if it’s rather clever and prone to mistakes:

function someCrazyCalculationThatReturnsAPositiveNumber(num) {
  // do something with num
  // ...
  return result;
}

for (let i = 0; i < total; i++) {
  const a = someCrazyCalculationThatReturnsAPositiveNumber(i);
  assert(a > 0); // I don't entirely trust my crazy calc code yet
  doSomethingWith(a);
}

Ultimately, it’s intended to be a tool to increase your confidence as you code and help you build robust, fail-fast systems.

I hope this post shed a bit more light on the many benefits of failing fast, and that you’ll be using this type of approach more often in your solutions. Thanks for reading, and I’d love to hear your thoughts. Feel free to connect with me on Twitter or LinkedIn!

How to Join Toptal: A Detailed Strategy Guide

Denat Hoxha — Fri, 10 Dec 2021 10:31:04 +0000

Toptal is one of the most exclusive freelancer networks out there, claiming to accept only the top 3% of applicants.

Once you're part of the network, you're part of it pretty much forever. You have the option of going for long-term full-time jobs, temporary full-time, or part-time gigs. You set your own hourly rate, but based on Toptal's price lists for companies, a good approximate is $50-$85+/hour (USD).

Wanting to give the contractor life a try, I joined the platform over a year ago and have just recently logged my 2000th hour. It's been wonderful so far. The clients are professional, the pay is excellent, and working remotely as a contractor is as flexible as it gets.

In this post, I'll be going through the steps of joining the platform as a developer while sharing some effective strategies, tips and tricks to maximize your chances of getting accepted.

If you're thinking of joining, apply to Toptal using my affiliate link and we'll both get $500 ($2500 for certain skill sets) when you land your first job. Feel free to DM me @denhox and I'll help you out however I can.

Step 1 - Brief intro and language evaluation

Difficulty: 🧡🤍🤍🤍🤍

This introductory step won't take long, about 5-10 minutes. You'll be invited to a video call where you'll give some details about your experience and what you're looking for at Toptal. The goal is to assess your English & communication skills. Be genuine and comfortable, this step is nothing to worry about.

Step 2 - Algorithmic Challenges

Difficulty: 🧡🧡🧡🧡🤍

Things get interesting from here. For your first technical assessment, you'll be given a link to complete a 90-minute coding test with 3 algorithm challenges.

Some info on the challenges:

Your solutions will be assessed by their correctness and performance.
You're not guaranteed to get the same problem set as another candidate.
The acceptance criteria is quite ambiguous. From my observations, you're not required to solve all of the problems, and there is no set score threshold.

There are no shortcuts to this step (or any of the next steps, to be fair), so a lot of it comes down to your experience in solving algorithmic challenges. Even experienced applicants will have to brush off some dust before jumping in. However, there is a clear roadmap on how to get to the level you need.

Preparation

Familiarize yourself with the algorithm challenge format. All challenges have the same structure: the problem statement, input, and expected output format. Go through the Codility practice test to see this structure, and also familiarize yourself with the platform.

Choose a programming language you're most comfortable with. Nail down its string & array operations, arithmetic, data structures (especially maps), and bitwise operations.

Practice on sites like LeetCode, HackerRank, and CodeWars. You can even combine them for more variety. Here's a brilliant curated list of 75 LeetCode questions that cover a large variety of exercises. Also, check out the Codility lessons.

Time yourself as you get deeper into practice. The real test is timed and has the pressure factor, so you should simulate this during practice as much as you can.

Strategy

The main goal is to get the highest overall score you possibly can.

The optimal way to do this is to gather as many points from all three challenges. Aim for breadth rather than depth by giving each task a fair shot, versus putting all your eggs in one basket by focusing your efforts on only one task.

Therefore it stands that to achieve the most gains you want to give an equal amount of time to each task and no more. Put your full focus into one task, but once its 30 minutes are up, drop it immediately and move on to the next one. If you solve a task in less than 30 minutes, congrats! You just added some bonus time to the remaining ones.

Solving a task against the test input doesn't necessarily mean your solution's correctness is going to be 100%. After submission, the platform will test your solution against some hidden "real" input, which will ultimately decide your correctness score. But don't let this bother you too much. If your solution passes the test cases and its time is almost up, move on.

The key is to aim for the correctness points of all three tasks first, and only put effort into improving your algorithm's performance if there's time remaining at the end. One little obstacle in scoring for performance is that the platform won't give you any performance measurement indicators, it only judges it after your official submission (which is too late anyway). So, for performance, you'll have to figure out the algorithmic complexity yourself and decide whether it's good to go.

Step 3 - Live Coding

Difficulty: 🧡🧡🧡🧡🧡

Welcome to what I believe is the most difficult step. According to Toptal, your chances of passing this phase are 48%. Looks pretty good to me!

This step is essentially a validation of your algorithm-solving skills, but by a human in a real-time video session. You'll have to solve two algorithm challenges, with 15 minutes each.

Have your development environment set up and ready with a barebones "Hello world" of your favorite language, ready to roll.

On joining the call, the Toptal technical screener will greet you, ask you a bit about the challenges you had in completing the previous step, and then ask you to open a webpage in your browser where they'll present you with your first problem statement.

You have less time to come up with a solution in this round, although the problems are of a slightly lower difficulty than the previous step. Correctness is everything and it's crucial for your solution to produce correct output. Performance does not matter as your solution won't be tested against huge input.

Preparation

Improve your speed by incorporating timing into your practice, and practice more. Use a timer for understanding the problem statement, and limit that to a maximum of 3 minutes. Then use a timer for the coding part, with a limit of 15 minutes.

Practice thinking out loud while you're understanding the problem statement, planning your solution, and while writing your code. Verbalize your thought process. See the strategy phase below for why this is important.

Strategy

The absolute number one tip for this step that I don't see people doing enough is to think out loud during the entire process. Most screeners will jump in and correct any misassumptions or give you a little towards the correct solution if they notice you might be wandering off the path. They will only be able to do this if you allow them to follow your thought process, and that's why you have to verbalize every decision you're making, from analyzing the problem statement to writing the code.

Spend as much time as possible understanding the problem statement before starting the timer. The screener won't start the timer until you say so, and you shouldn't say go until you've digested the problem statement entirely. There's an unwritten limit of course, but it's on you to prolong this as much as you need.

Do not make assumptions. One of the biggest reasons that candidates fail this step is that they make an incorrect assumption in the problem statement, eager to jump right into the code. Ask about any part of the problem statement that is not crystal clear to you, before starting the timer.

Identify the problem's edge cases and say them out loud too. As you might already know by now, edge cases in these algorithmic challenges represent some risky traps that require some special handling in your code.

Step 4 - Test Project

Difficulty: 🧡🧡🧡🤍🤍

You're almost at the finish line! According to Toptal, about 88% of applicants pass this phase. Looking good. 🙂

You will be sent a project assignment to complete within 2 weeks. The project depends on what you applied as (backend, frontend, or full-stack), but it'll typically take up to 40 hours. Plan accordingly.

The project assignment contains a list of functional and non-functional requirements that your project will have to fulfill. You'll be given access to a private Git repo to push your commits. Make sure you don't push any commits after the 2 weeks are up.

When you're done, you'll be able to set up a meeting to demo your project. During the demo, the screener will ask you to go through every requirement as they check them off the list.

The biggest factor for success in this step is (surprise, surprise) the sum of your technical project experiences so far in your career. A rough prerequisite for applying to Toptal is about 3 years of experience, which you most likely have if you've made it to this point.

Strategy

Use correct HTTP status codes and other semantics. Your project likely involves an API to build, consume, or both. The technical screener will verify that you're returning correct HTTP status codes for invalid input (400), non-existent resources (404), unauthorized requests (401), insufficient permissions (402), redirects (3xx), and a catch-all server error response (500).

Test extensively and make sure your project is clear of bugs and unhandled exceptions. An unhandled exception during the real demo will likely be fatal.

Rehearse the demo at least once. It's extremely likely you'll catch a missing piece of a requirement, or stumble upon an exception.

Conclusion

When the last requirement is ticked off the list, you're done! Congrats! 🎉

The screener will congratulate you and set your account and Toptal email up for onboarding. You're officially part of Toptal and can now apply for jobs, which in itself is a challenge that I'll be writing about in a future post.

If you're thinking of joining, apply to Toptal using my affiliate link and we'll both get $500 ($2500 for certain skill sets) when you land your first job. Feel free to DM me @denhox and I'll help you out however I can.

Got any questions? Have you had a different experience when joining? Anything I might have missed? I'd appreciate it if you could let me know @denhox.

Good luck!

Four Lessons Learned from Building Microservices

Denat Hoxha — Mon, 01 Nov 2021 14:14:23 +0000

I've had my fair share of projects based on microservices architecture. As fun as they are to work on, we fell into some nasty traps, some of which should've been painfully obvious. Here are four of them that I could gather from memory.

Services should hide their internal details

A trap I fell into pretty early in my microservices journey was allowing another team's microservice to access our microservice's database directly.

Their team was tasked with building a back-office internal tool capable of reading and writing data in all of the other microservices, including ours. Our public GraphQL API wouldn't suffice, it had very conservative queries and mutations intended only for our public clients. For what this team needed, we would need to add a bunch of administrational operations to our API.

Amid a very tight project schedule and with us thinking it would save us a lot of time, we proceeded with having their service access our service's DB directly. Yikes.

Not long after that, we realized that we had effectively given up our freedom to make changes to our database's schema, because the back-office tool had now been hard-coupled to it.

Getting things right was painful, but eventually we got their microservice's read paths to go through our API, and cut off their dependency to our database.

What we learned

Don't break a microservice's boundary by allowing another service to directly access one of its components (e.g. the DB). This covertly introduces a new contract that it'll have to adhere to. A microservice should (aim to) uphold one contract only: its API.

A microservice should hide as much details as possible. You can only easily change components that are hidden from the outside.

Invest early in the infrastructure required for service-to-service communication, i.e. service discovery and machine-to-machine tokens. You'll need it eventually, and this was a major factor that drove us away from the correct solution because the amount of work seemed significant.

Chatty microservices likely belong together

On what should've been a boring Tuesday, some QA reports began storming in that the application was hanging on certain screens. We confirmed that on our observability tool, Instana, where we saw a couple of GraphQL queries taking >30 seconds then ultimately failing.

The distributed trace in Instana revealed an infinite loop of requests as a result of a cyclic dependency between two of our services. We located the bug which was introduced in a recent commit: a new piece of middleware in service A was making a request to service B, which then would make a request to service A, triggering service A's middleware again to fire off a request to service B, and so on.

We realized we had been lucky to not run into this issue for as long as we did, considering that these two services were already very chatty and had pretty much always been dependent on each other.

As an urgent fix, we reverted the commit and decided to merge the two services into one.

What we learned

Observability tools and distributed tracing save the day. The long alternating sequence of requests in the request trace let us instantly recognize there was a cyclic dependency issue.

Two services being very chatty is a good indicator that they belong together. Most observability tools provide visuals of intra-service communication in your ecosystem that can help you spot which services are communicating excessively.

The best way to avoid cyclic dependency issues is to not have cyclic dependencies. Shallow cyclic dependencies like in our case (A → B → A) are manageable to a certain degree, but a poor design could end up introducing infinite request loops from transitive circular dependencies like A → B → C → A. Aim for no cyclic dependencies and a unidirectional request flow.

Minimize alerting noise. Ideally, an issue like this should have been noticed by us via our monitoring alerts, long before anyone had reported it. We had a Slack channel set up for automated monitoring alerts, but there was too much noise going on in there that we had gotten used to ignoring it. This incident made us take the time to revamp the alerts and be more attentive to the alert channel.

Maintain independent deployability

One of the core principles of microservices is independent deployability. You should be able to deploy a microservice at any time, without having to deploy anything else. Deviating from this rule brings trouble.

In one of my past teams, we had a tradition to gather once a week in a "war room" for a mass deployment of a usual set of 6-7 microservices to launch new features. To minimize downtime, these services had to be released in a strictly ordered sequence.

The biggest reason for these coordinated deployments was that teams were allowed to break their API contracts. There were no rules in place to require backward compatibility for every service. We assumed this just came with the territory, accepting it as a normal part of practicing microservices. However, over time the major downsides became very apparent.

If we had messed up one of the services and couldn't push a hotfix quick, we had to revert all the deployments. This was frustrating.

Allowing microservices to disregard backward compatibility effectively lowered our confidence in the stability of the ecosystem.

Getting many people to sync for the coordinated deployment was difficult since microservice teams usually had their own plan schedules. Also, any hiccup or failure during the process meant a lot of wasted time and productivity.

And, ultimately, it never felt right that teams had to concede their autonomy of deploying their microservices to production.

One of many things we did to address these was to add a rule: microservices must maintain backward compatibility and not break their API contracts. If breaking changes are inevitable, teams must be notified ahead of time and API features must go through a deprecation phase first.

What else we learned

Integrate automated checks in the CI/CD pipeline to report and prohibit breaking changes. We added a step to our pipeline that used static analysis to compare the new API contract to the previous one. Additive changes are OK, but any removal or modification of an API method failed the CI/CD job.

A culture of maintaining backward compatibility did wonders for our sanity and confidence in the system, knowing that you won't wake up one morning to find out a microservice's endpoint is now returning a completely new format or even disappeared completely.

Feature flags are a very useful tool to separate code deployment from feature deployment. If you want to deploy a breaking change, you can first hide it behind a feature flag, deploy the code, then switch the feature flag on at your convenience. More on feature flags below.

Adding end-to-end tests has a point of diminishing returns

The above statement admittedly applies to most systems regardless of architecture, but it was especially evident in a microservices project I took part in that had very healthy test coverage. The React frontend was full of integration and Cypress end-to-end tests, whereas the backend services had their own unit and end-to-end tests.

After a certain point, the number of bugs that made it through to staging and production began growing linearly with the number of new features added, even though we had strict measures in place that no code was to get merged without its accompanying tests.

We needed a strategy to complement the tests.

What we did (and learned)

The answer was testing in production, as so nicely explained in Cindy Sridharan's "Testing in Production, the safe way". This strategy involves applying post-release techniques to tackle encountered issues, rather than trying to prevent them all pre-release.

What this meant for us, was to crank up our monitoring and work on our knowledge sharing regarding our distributed tracing tools, making sure every developer can find their way around it.

But one of the techniques that proved most effective for us was the usage of feature flags. New feature code was expected to be wrapped in a feature flag, which we would then use to roll the feature out gradually, starting from our small team to the whole company, and finally to the entire userbase (we've had great success with LaunchDarkly). We piggybacked on this mechanism for error resolution as well: when our monitoring tools reported critical errors that originated from a particular feature, we could easily switch the feature off to buy us some time to understand the problem and apply a hotfix. This is a great example of how we did testing in production.

Endnotes

Some of the principles and rules I've mentioned above are brilliantly explained in the book Building Microservices by Sam Newman, which I've had the chance to read recently. It's a great resource with principles and best practices that will save you from the many pitfalls of microservices. The second edition has been released and I highly recommend it!