DEV Community

Bruno Paz
Bruno Paz

Posted on • Updated on • Originally published at

My experience and learnings from working in a microservices oriented project

In the past few years, Microservices have become a very hot topic in our industry and seen as the recommended way to build more decoupled, scalable and easy to maintain applications.

For more or less an year, I worked in a project that followed a Microservices oriented architecture from scratch.
In this post I will share my experience and learnings from it.

The project

To give you some context, the project I worked on was building a Flights booking web application, similar to Skyscanner.

The main user flow is really simple. The user is presented with a search form, where he can put where he wants to go, dates etc.

After he clicks the search button, a list of available flights are presented.

Then the user, can select one of the flights presented, and start the checkout process where he will enter his personal details, following by the selecting the payment method. In the end of the journey, he will get a confirmation email of his flight.

The start - Identify your service boundaries

The most important thing to start with a Microservices Architecture is do identify your service boundaries. Think about your Domain Model and the relationship between your Domain entities. This guide from Microsoft is a great resource, that explains this topic in more detail.

This is really important. If the service boundaries are not well defined, you might end up with services that are too coupled with each other. In case of doubt, start with a bigger boundaries. Its easier to break down into smaller units later on, than the other way around.

From the main user flow I described above, we can clear identify some potential services. A search service, that will be responsible for querying our flights providers and show a list of available flights. An "orders" service, that will manage the Booking / Checkout process. A payments service, that will handle all stuff related to Payments and even an Messaging service that will be responsible to send the confirmation emails to the users.

This separation of concerns makes a lot of sense in terms of Business domain, but you can also see advantages in terms of scalability and operations.

You will have much more searches than bookings, so the search service will probably need to scale a lot more than the orders service. Being a separate service allows that.

Also search service will need to do a lot of processing to fetch the available flights from multiple partners. Being a separate service allow us to choose the most appropriate language and tooling for the job. Besides different teams can work on these services separately.

That´s what we ended up building.

Flights business is really complex, so we ended up breaking down the Search service even more into smaller specialized services all working together.

We also had some other smaller services that I wont enter in detail.

In general, this structure was a great success, but of course there were also some less positive things that I will talk about next.

A state machine service that shouldn't have been born

In the company, we identified a common need across different projects. Every project needed to have some sort of State machine at some point. This project was not an exception. We needed to have a state machine for orders and payments.

So we decided to do a generic state machine service that could be potentially used by every other project in the company. While it sounds a great idea in theory, it wasn't that great in practice. Let me explain why.

The first problem is the term "generic" state machine. Generalization and Abstraction is of course a good idea in general, but it can often lead to over engineering and be a lot more complex and time consuming than doing something a little more specific.

To be able to properly generalize and abstract a solution, its vital to have a clear understanding of all the business requirements and how they will eventually evolve over time and the specific needs of each potential user of the service.

That was not the case. It took us a lot more time trying to anticipate all the possible features and needs than if we focused to build something more specific based on what we new from this particular project.

The second problem is that, if you think about our Domain model and what I talked about service boundaries, a state machine on his own is doesn't make much sense. It cant work on his own and has to be associated with some business entity, in this case orders or payments. It clear belongs to the orders / payments boundary.
The tight coupling between these was evident as both services ended up calling each other multiple times in a single request for every operation with all the associated overhead.

With the current implementation, Orders service cant work without State Machine service and State Machine service cant work without Orders service. This issue was even bigger as both services communicate synchronously, and that leaves to the next learning of this project, Prefer Asynchronous communication when possible.

What could have been done if we wanted to reuse some code, was to build a library and some SDK instead of creating a completely separate service.

Prefer Asynchronous communication

While each service should be an independent deployable unit, they work together to build your application, so of course there are some dependencies between them and they have to communicate in some way. But that dependency doesn't have to be an hard dependency that if some service is down, it will bring all your application down.
How could you avoid that? By using Asynchronous communication like message queues.

An example in this project was the Payments service. When the user made a booking, the Orders service will do a synchronous HTTP call to the Payments service to do the payment. The Payment service will then do another request to an external payment provider do effectively do the payment.

This process is slow and and there are many things that can fail in the middle. Using asynchronous communication makes your services more resilient to failure as you can implement mechanisms like retry in a case of error in an easier way.


The advantages of a Microservices architecture are clear, from a better separation of concerns resulting in more decoupled applications, the possibility to scale each service independently or to write each service into the most appropriate programming language.

But Microservices are not a silver bullet. The Orchestration / Deployment of the application is more complex, Debugging is more difficult and the way each service communicates with others needs to be well thought.

You need to plan very well your architecture if you want to follow that route. Correctly identifying service boundaries based on your Domain Model is the first step of a successful Microservices based implementation.

Due to the nature of the project I worked on, the service boundaries were clear defined and made total sense to go to a Microservices approach. If that´s not so clear or it is expected to change a lot, Its nothing wrong to start with a more monolith and extract some parts later if needed to.

The word Monolith has a bad connotation in general, but the truth is, nothing stops you from having a monolith that is modular, have a great separation of concerns, low coupling and high cohesion. Popular design patterns like SOLID can help a lot. If you do that, it will be relatively easy to start moving specific parts of the application to separate services as your application grows.

Every application is different. Its your job as a Software Engineer to analyze the requirements and choose the most appropriate path.

Hope you enjoyed the article. If you have any questions or comments, feel free to use the comment box below.


Top comments (12)

meysamsalehi profile image
Meysam Salehi

I'm in first steps of designing a migration plan from a big spaghetti monolith web app to microservices. In our architecture state machine play a key role and in the first thoughts like yours, we thinking about a generic state machine. but I agree with you about the costs of generalization. Thank you for sharing your experiences.

nverinaud profile image
Nicolas Verinaud

I advise against going from spaghetti monolith to microservices. It will be less risky to move toward a modular monolith first, best if you can refactor your current monolith instead of starting from scratch.

mamhaidly profile image

I am interested to learn more about asynchronous communication between services. Any recommendations of references on this topic?

thorstenhirsch profile image
Thorsten Hirsch

I'm also interested in this topic. Next year I might be able to write an article about a software system whose components use ActiveMQ for communication. However this system was not designed as a microservices system, so I'm eager myself to see if it works well.

I guess RabbitMQ might be a good decision in general, maybe even IBM MQ if you want to spend money on it. Kafka on the other hand is a bit special, but probably the messaging system of choice when you really have massive amounts of (small) messages and the publish-subscribe pattern matches your architecture.

brpaz profile image
Bruno Paz

Great question.

Dont remember any particular resource on the topic, but you might want to look at Messaging systems like RabbitMQ or Apache Kafka.

tyrotoxin profile image
Serge Semenov • Edited

You are not the only one, Bruno. Thanks for sharing - you just re-assured everyone in one of the most common problems developers solve over and over again.

For that very reason, I started the D-ASYNC open-source project, where your hierarchical state machines are represented merely by async methods. I.e. the programming language is the abstraction without a need for any special library or syntax.

I'm really curious about some details where your generic state machine library failed and what features were missing. And I totally understand how this complexity can derail developers from focusing on business logic (domain modeling). Even if you use an Actor Model framework, your code can blow up 5x in size. This is explained in details of Conquest of Distributed Systems series.

Share more, Bruno!

daveclarke profile image

As a dotnet developer my reference for building was Microservices in .NET Core by Christian Gammelgaard. It’s a really useful practical reference for developing a microservice system. It’s not the only way to build a microservice in .NET but it’s worth reading to get a perspective on one approach.

cheetah100 profile image
Peter Harrison

"Think about your Domain Model and the relationship between your Domain entities...
This is really important. If the service boundaries are not well defined, you might end up with services that are too coupled with each other." - perhaps the problem is basing services on the domain at all? Perhaps design services around what they can do and expel the domain into configuration/data. Why couple an application to the domain at all if can help it?

scottctr profile image
Scott C.

Might consider a state machine library. Here's one I created for similar projects

brpaz profile image
Bruno Paz

Yes, a library would be probably be the path I would take now.

Thanks for sharing.

david_j_eddy profile image
David J Eddy

Nice article Bruno, I look forward to more from you soon!

jordybaylac profile image
Jordy Baylac

Man this is epic!

...nothing stops you from having a monolith that is modular, have a great separation of concerns, low coupling and high cohesion

Great article.