Recently our development team had a small break in our feature delivery schedule. Technical leadership decided that this time would be best spent splitting our monolithic architecture into microservices. After a month of investigation and preparation, we cancelled the move, instead deciding to stick with our monolith. For us, microservices were not only going to not help us; they were going to hurt our development process. Microservices had been sold to us as the ideal architectural for perhaps a year now. So we were surprised to find out they weren't a good fit for us. I thought it would be interesting to present a case study of our experiences, and why our team decided against them.
Identifying finding problems and early compromises
We were heavily reliant on a third party
Our application is a custom UI over the top of an existing external product, integrating some of our custom business rules and presenting a touch-friendly user interface. Our client is a UWP app, and we have a range of back end services that transform between our domain and the third party's domain.
Building on top of a third party impacted how we could divide our domain into microservices. For example, our application occasionally has to convert features between domains. Making one part of the third party's domain act and feel like it was part of a different domain in our UI. This swap was not so bad when we had a single service between our front end and the third party. However, the domain switching caused us much confusion when we tried to split our domains into separate microservices. Did our microservices follow the same divisions as the 3rd party and we duplicate the front-ends requirements across both services? Or, did we divide the microservices according to our domains and have one microservice need to fetch from two separate areas of the third-party. Both felt like a violation of microservice guidelines and like they would lead to additional coupling.
We frequently worked in tandem with the external party, with features requiring both parties to make changes. Effectively, the third party was an additional team. Working so closely together meant we had to lockstep our release process with theirs. A benefit of microservices is that each team can be responsible for releasing their services independently and without coordination with other teams. Coordinating releases not just across teams, but across companies prevented us from gaining those advantages.
One of the central ideas of microservices is restructuring away from separate teams being responsible for separate layers. In a microservices architecture, each team is responsible for the full stack addressing their business concern. For us, since one of our layers was an entirely separate company, this restructuring was not possible.
We couldn't sufficiently isolate each microservice
We couldn't identify any obvious candidates in our monolith to be broken out into a microservice. So instead, we started drawing arbitrary lines between our domain models, and from this, we had the list of microservices we were to create. However, once we started investigating, we found a lot of shared business logic and implicit coupling between the soon to be separate microservice domains. Some further attempts were made to subdivide these microservices into smaller and smaller pieces, but that left us with even more coupling, messages buses everywhere, and a potential big bang of immediately going from one service to ten or more microservices.
The reason everything was so coupled and hard to break up was that the monolith we were trying to separate only served a single business concern. One of the overarching design goals of our client application was to bring the disparate concepts in the third party base application together. We are creating workflows that crossed domains and grouping features for the user's convenience. In essence, the UI had spent the last four years pushing everything together.
Somewhere along the way, we had misunderstood how microservices should be isolated and underestimated the importance of choosing the right boundaries between services. The only ways we could break down our monolith meant that implementing a standard 'feature' would involve updating multiple microservices at the same time. Having each feature requiring different combinations of microservices prevented any microservice from being owned by a single team.
Sharing microservices
We have approximately 12 developers spread across 2 feature teams and a support team. The work fluctuated enough that no team was locked to any area of the application. It was not uncommon to have both teams touching the same area of the code at once. We were not able to assign ownership of any potential microservice to a single team.
It is useful to bear Conway's law in mind when considering the shape of your architecture. It states that your software's architecture grows in a way that mimics how your organization and teams are structured. Lots of isolated microservices make sense if you have a bunch of isolated teams working on separate business concerns. However, few teams working on shared features better suit a single shared location.
The platform wasn't ready yet
Various issues meant that for at least 6 months, we would be hosting our new microservices next to our monolith in IIS. We wouldn't have access to many of the standard tools associated with microservices such as containers, Kubernetes, service buses, API gateways, etc. Not having these tools was going to make it more difficult for the microservices to communicate with each other. So instead, we decided that each microservice would duplicate any shared logic along with the common reads and transformations from our storage layer. Because we couldn't isolate any of our services properly, this was going to mean that we would be left with a significant amount of duplication. For example, we identified one particularly complicated and essential piece of business logic that would have to be copy-pasted and maintained across 4 of the planned microservices.
We didn't have a clear picture of the future
The development teams had a rough idea of the next 6 months and no information about what was beyond that. Further, the business changed it's mind frequently. It wasn't uncommon for requirements to change mid feature. This uncertainty made creating microservices more fraught, as we couldn't predict what new links would pop up, even in the short term. Would the connections and coupling between the planned microservices grow? Would we have to spend time in a few months joining them all back together again? We had already tried creating a proof concept microservice earlier this year, only to have it nixed as the business changed its requirements.
Time frames were tight
We had a tiny window, just large enough split our monolith into the list of microservices we had been given. What we didn't have was any extra time to allow us to reflect on what we had created or alter course if required. There was no time in the schedule for plan B. We were going to be stuck with whatever we created. Since we were discovering many issues and challenges in the planning stage, let alone the implementation phase, this caused the development team much concern.
Lacking experience
Compounding the risks and time pressures, none of the people responsible for architecting or implementing the microservices architecture had any specific prior experience. This was exacerbated by not having a lot of the standard tooling ready to use, meaning we would be implementing the platform ourselves. Conversations with some people with experience with microservices, but who weren't involved, raised more red flags. Suggesting infrastructure, we wouldn't have, pointing out the consequences of where we had drawn the lines between our domain models.
So far, our plan involved lots of compromises that deviated from standard microservice patterns, tight time-frames. There was no expert guidance and a strong likelihood of making many mistakes and learning lessons the hard way. The development team started to look nervous.
What were we trying to achieve again?
Is this addressing our pain points?
Once everything started getting hard, and the clear path forward started to get lost, we paused, and realized we didn't know why we were doing any of this. We didn't have a list of our pain points, and we had no clear understanding of how this would help solve any pain points we do have. Worse, microservices might be just about to create a whole set of new problems for us.
We started pressing these issues, what benefits are we supposed to be getting, and what problems are we trying to solve? We set more and more meetings trying to figure it all out, every coffee break and every conversation between developers was discussing and questioning microservices, and we still couldn't get a straight answer why.
As it turned out, we did have other, more pressing pain points, which had been ignored in the drive to towards microservices. Unfortunately, we might have run out of time to address those adequately, meaning we had neither microservices nor anything else.
What were the potential benefits?
Once we realized we had no idea why we were heading towards microservices, we paused and started to investigate for ourselves the benefits that microservices typically provide.
Autonomy
Microservices allow your team to have control over the full stack they require to deliver a feature. The benefit of this separation is a reduction in the amount of coordination you require with other teams. You won't be affecting their work, and they won't be affecting yours.
Allows your team to specialize
In a monolith, any team can end up working on anything. Ownership of any feature or area isn't a given. With each team owning their set of services, they can build expertise in that particular business concern. They get to understand the businesses rules and requirements in their domain. They know how their software stack is structured and implemented and can have greater confidence when making changes.
Easier to scale
With microservices, you can scale each service according to its performance needs. With a monolith, while you can also scale horizontally across more servers, you can't scale each component of the monolith separate to one another. Further, this granularity makes it easier to scale services up and down as required. Perhaps you are anticipating some additional load, or need some breathing room while you sort out performance issues.
Easier to Rollback
If each feature only requires a change to a single microservice, then that feature could be rolled back without affecting the work of other teams. Further, microservices help reduce the amount of your system that could be taken down by a single fault.
Easier to release and easier to release more frequently
If you have an extensive system, each release becomes time-consuming and risky. There is a lot that needs to be covered by regression testing, limiting your release cadence. You might need sign-off from multiple people and coordination between all of the teams involved in each release. A bug or regression from a team you've never even heard of can hold up time-sensitive features that you need to get out the door. Microservices limit the scope of changes and reduce the amount of coordination you require between teams. Teams can release according to their own schedule rather than being bound by the cadence of a monolith.
Use the most appropriate technologies
Microservices give your team the ability to choose the most appropriate technology for their team and the problems they are trying to solve. Perhaps they can use a modern technology, whereas monoliths can be hard to upgrade and stuck on outdated platforms.
Easier path to upgrading
Upgrading the framework used by a large application is never fun or risk-free in the best of circumstances. It is much much harder when you need to coordinate sweeping, interlinked changes across multiple teams. Smaller, isolated services give you the option of only upgrading the services that require the update or allowing you to perform the upgrade one service and one team at a time.
Protect from change
Different parts of your application change at different rates. Most of your application likely hasn't been changed in months, or even years. Separating rarely changed code away from areas with frequent churn allows you to reduce the risk of accidental regressions.
Smaller
A smaller service is much easier to reason about and understand. Further, being changed by only a single team means its design stays consistent. Its smaller size makes it easier to perform extensive refactoring. By comparison, a monolith may have an inconsistent, evolutionary architecture as the opinions of different teams cause it to vary over time.
Conclusion of benefits
There are a lot of potential benefits to adopting microservices. However, were we able to gain any of them?
Ultimately, parts of our architecture that we couldn't change and the compromises we had to make undermined the benefits. Microservices being a floating pool shared between all teams and features spread thinly across multiple shared microservices meant that we lost the benefits of isolation: reduced coordination, specialization and benefits that flowed on from these. The variation between microservices, instead of being a strength, became a disadvantage. Each feature would require learning how a new microservice worked and what changes other teams had made to it. Our reliance on a third party stopped us improving our reliance cadence and reduced the benefit we would get from independently scaling our services.
Weighing up the advantages and disadvantages
Killing a fly with an elephant gun
Adopting microservices isn't free. There is a vast list of additional concerns that you need to address. We would need to revisit many concerns that we had previously addressed in our monolith. For example, we would need to address or revisit: logging, monitoring, exception handling, fault tolerance, fallbacks, microservice to microservice communication, message formats, containerization, service discovery, backups, telemetry, alerts, tracing, build pipelines, release pipelines, tooling, sharing infrastructure code, documentation, scaling, timezone support, staged rollouts, API versioning, network latency, health checks, load balancing, CDC testing, fault tolerance, debugging and developing multiple microservices in our local development environment.
To make matters worse, without a microservices platform ready, we would have to be working a lot of the above list out for ourselves. We already had pain points and difficulties moving to microservices; we established that we weren't going to benefit from the advantages of moving to microservices, and we had a long list of additional work to set up and maintain to support microservices.
Microservices in name only
The following image demonstrates our current monolith, our planned architecture alongside a comparison of how microservice might look. Structurally, our new architecture still closely resembled our monolith, with everything still tightly linked together. Should we have even been using the microservices label to describe what we were doing?
Was our monolith that bad?
We were using monolith like a loaded term. As if saying "monolith" implies something terrible, and "microservices" implies something good. Once we looked past the stereotypes and branding, the development team had very few issues with our "monolith." It might have been one of the most pain-free parts of our entire system. It was straightforward to develop in and extend since it was mostly a passthrough to a third party. We didn't need to spend much time working on it. We had an excellent CI/CD setup, which made it easy to deploy and rollback. Our branching and testing strategies ensured that few issues that made it into production.
Realizing I had used microservices before
At this point, I realized that I did have experience with a microservice in a previous role. We had never referred to it as a microservice, and it probably didn't follow all of the "rules" of microservices, but it certainly solved the same problems and gave us the same benefits.
We were a small team of 5 in a company of about 200 developers. Maybe 5% of our back-end work was in the companies shared monolith, a vast C# application. The rest of our time, we were working within our two Node services.
We disliked working in the monolith. It was slow to work in, compile and run tests for, the architecture was varied to the point of unknowable, random stuff keep showing up in the build steps. Multiple times we had a high priority piece of work for a customer get delayed weeks because a team I had never heard of had regressed in functionality. Periodic technology updates took months as they required coordination across the entire company. Pull requests could be held up for weeks while we waited for approval from entirely separate teams.
Meanwhile, our two services were small; we had full control of their development, architecture and deployment. Once when we were having performance issues, we doubled the number of instances in production until we had resolved the underlying issue. We rarely had to coordinate with other teams. Having our service in TypeScript allowed our team of predominately front end developers to the same language on the front and back end. Best of all, it allowed us to include our complicated rule calculation engine in both our client and in our back-end validation and reporting services. Our team focused on a very narrow business concern, which we all became experts in.
More than just a technology problem
The more we looked into microservices, the more it seemed that it was less about technology and more about structuring teams and the work that came into them. Had we made a mistake approaching microservices as purely a technology problem?
Was restructuring the teams to be dedicated to separate business concerns practical? Could upcoming feature work be cleanly divided between the monoliths domains? Would there be enough work for all teams, or would a team be left without work? Would a team get slammed with mountains of high priority work that they couldn't share out? Would the same issues that made it hard for us to divide our monolith also prevent our management tier from dividing up incoming work? Was their appetite for this sort of transformation? There were many questions regarding the bigger picture that didn't have answers for.
Getting from a to b
Our plan to get to microservices was a big bang. Everyone stops feature work for a couple of months and starts splitting up our monolith. Even though many of the prerequisites weren't ready. We were forcing the way ahead rather than waiting for either the need to arise or natural candidates to emerge.
Not only was this not a very good way from getting from a to b, but it was also backwards. Create all of the microservices first, then set up the infrastructure for them and completely ignore the aspects structuring the teams and incoming work. Instead, if we had started by restructuring our teams around dedicated business concerns, then gotten the infrastructure ready, we set the stage for microservices to naturally emerge. If any new business concerns emerged, they could be placed directly into a new service.
By forcing microservices, it meant that we also had to choose upfront the size of each microservice. There is much conflicting advice about how large (or small) to make each microservice. Some articles suggested that each microservice should be large enough for one team. Others suggested that each microservice should be small enough you can keep the structure in your head or even so small that you could rewrite it in two weeks. Other suggested they should be the size of each business concern. Leadership decided to split our microservices up based on our domain models and then keep dividing them smaller if any issues came up. This lead to many of the issues mentioned above with teams and features needing to share microservices. In hindsight, if we had let microservices naturally emerge after everything else was in place, we might have ended up with microservices of a practical size.
Cancelling
As microservices day 1 drew closer and closer, our team just kept finding more and more issues. Creating more compromises and reducing the benefits further. Four days out from the first sprint of beginning the implementation of our microservices, we still couldn't identify any gains, and the list of problems and disadvantages was long enough to form the seed for this rather long blog post. We called a meeting, and despite what leadership wanted, the answer to microservices was written on every developer's face. Our move to microservices was cancelled.
So what did we do instead?
The fervour of moving to microservices had meant that the alternatives hadn't been investigated. Only after we abandoned microservices could we investigate other options. Ultimately, rather than separate our monolith into separate services, we started to break our solution into separate projects within the existing monolith. This division gave us a bit of additional structure and a better indication of where coupling and duplication existed, without the extra weight and challenges of microservices.
Further, this structure would make our domain models clearer, allowing us to evaluate candidates for any future microservices more easily. If something did turn out to be a suitable candidate, that project could then "fall out" of our monolith into a microservice, without having to be untangled.
Conclusion
Leadership set the direction of microservices without consideration for the challenges and state of our application. After evaluating it, we found that microservices weren't a fit for us, and required significant compromises. The compromises robbed us of any of the benefits and meant that moving to microservices was a net loss. Microservices had been decided on without evaluating non-technical concerns like team structure and incoming work. After months of investigation and work, we abandoned the project and spent the remaining time performing some minor refactors to our "monolith".
Top comments (0)