This article aims to highlight a data-informed decomposition of monolithic systems into microservices. Process Mining, algorithms, and techniques used to discover processes based on their digital footprint is a powerful tool that can come handy when migrating to microservices. Event logs generated by monolithic systems can be analyzed to gain an objective view on current flows and usage which can inform the design of better decomposition to microservices.
Process Mining can also be used to dynamically manage microservices and prioritize development efforts. But this paper sheds the light on process mining and its value during the decomposition design phase.
While writing this article with the help of my colleague Nizar Salman, we built on work done by several scholars cited throughout the article and in the reference section, but we were mainly influenced by the work of Professor Davide Taibi.
Why Microservices?
We have seen the use of microservices architecture grow tremendously in the past 5 years, from tech giants to smaller teams the gravitation of microservices is so strong in fact, it’s expected to become the default application architecture within the next five years. Which errors should be avoided in the migration journey while implementing the architecture in order to develop reactive and sustainable systems?
Microservices allowed developers to break down large, complex monoliths into smaller services or applications. Because of their independent deployment, microservices have a lot of advantages. They can be developed in different programming languages, they can scale independently from other services, and they can be deployed on the hardware that best suits their needs even when addressing intelligent IoT applications. Moreover, because of their size, they are easier to maintain and more fault-tolerant since the failure of one service will not break the whole system, which could happen in a monolithic system. Since every microservice has its own context and set of code, each microservice can change its entire logic from the inside, but from the outside, it still does the same thing (Taibi et al., 2017a).
Businesses and organizations are drifting further away from monoliths and closer towards a microservices architecture MSA for every app, website, and digital experience they develop to benefit from better agility, scalability, and resilience.
If you are new to microservices and want to get a hands-on feel of the migration from a monolith to this amazing application architecture, AWS did a great job in this tutorial, Break a Monolith Application into Microservices.
Decomposition Overview
Decomposing a monolithic system into independent microservices is one of the most critical and complex tasks and several practitioners claim the need for a tool to support them during the slicing phase in order to identify different possible slicing solutions (Taibi et al., 2017b)(Taibi et al., 2017c)(Taibi et al., 2018). The decomposition is usually performed manually by software architects (Soldani et al., 2018). Up to now, the only help that software architects can have is based on the static analysis of dependencies with tools such as Structure 101 while the slicing of the system commonly is delegated to the experience of the software architect itself. Moreover, static dependency analysis tools are not able to capture the dynamic behavior of the system and run-time dependencies like frequent method calls could have an influence on both maintainability and performance. Thus, this approach to slicing is based on runtime behavior instead of only considering static dependencies.
At the beginning of a transformation journey, the architects and the development team, with all their know-how and experience, develop a tailored architectural blueprint for the upcoming development. Common steps for migrating from a monolithic system to a microservices one are (Taibi et al., 2017d):
- Analysis of the system structure. All processes start by analyzing dependencies mainly with the support of tools (Structure 101, SchemaSpy 2, or others)
- Definition of the new system architecture. Architectural guidelines or principles, and the proposal of a decomposition solution into small microservices are defined. The decomposition is always done manually.
- Prioritization of feature/service development. In this step, all three processes identify and prioritize the next microservices to be implemented. Some processes prioritize microservices based on customer value; others according to components with more bugs; and yet others prioritize the development of new features as microservices, expecting that, in the long run, the new ecosystem of microservices will gradually replace each feature of the existing monolith.
- Coding and Testing are then carried out like any other software development project. Developers adopt the testing strategy they prefer. However, in some cases, testing of the different microservices is performed by doing unit testing at the microservices level and black-box testing at the integration level.
This article focuses mainly on the first two steps, supporting organizations in the analysis of the system structure and in the identification of decomposition alternatives. The architectural guidelines should be defined by the company based on their internal policies.
Data-Driven Strategy and Process Mining
Applications built from microservices should be as decoupled and as cohesive as possible! But how do we pick the best strategy for decomposition? Here are some common decomposition strategies (Richardson, 2017):
-> Decompose by business capability and define services corresponding to business capabilities.
-> Decompose by domain-driven design subdomain.
-> Decompose by verb or use cases and define services that are responsible for particular actions.
-> Decompose by nouns or resources by defining a service that is responsible for all operations on entities/resources of a given type.
The first two strategies are mostly abstract patterns of human decisions while the other two are based on predefined data-driven criteria which this method will focus on.
A data-driven microservices-oriented decomposition approach based on data flow diagrams from business logic. This could deliver more rational, objective, and easy-to-understand results thanks to objective operations and data extracted from real-world business logic (Chen et al., 2017). Similarly, you can adopt process mining to analyze the business processes of a monolithic system.
Tools and Techniques
However, while dependency analysis tools can support the identification of static dependencies, they do not enable the identification of the full execution path. This approach combines process mining techniques and dependency analysis to recommend alternative slicing solutions.
The approach requires the availability of an extended version of a log trace collected at runtime. For each user operation performed from the user interface (e.g., clicking on a button), or from any entry point of a system (e.g., APIs or command line), all the activities must be traced from the log files. Information about each class and method that is traversed for the execution of the operation must be included. The complete execution must be traced completely from the entry point (a click on a submission form or the beginning of an operation) to the access to the database (if any) and to the results returned to the client. An example of data reported in the log file is shown in Table 1.
Table 1 – Example of Log Traces
Here, you instrument the software to produce a log. The log trace must include events that indicate entrance and exit of functions as well as database accesses. Each entry in the log also has a timestamp and a session ID, to distinguish between the usage of the same class or method from different users.
The log trace could be collected by instrumenting the source code with Aspect-Oriented Programming, by adding the log trace into each method or with existing applications such as the Elastic APM or similar.
In case the data collection is not yet in place, you can use Elastic APM, since it allows to easily instrument the code with a minor effort. For some languages (e.g. Java and Node.js) the instrumentation requires the addition of one line of code to the application configuration, specifying the type of log trace required and the logging server URL.
Decomposition Process
Once the log files are created, organizations can start the decomposition following this 6-step process (Taibi et al., 2019).
Step 1: Execution Path Analysis
In the first step, you identify the most frequently used execution paths with a process-mining tool. In this case, DISCO 4 was used to graphically represent the business processes by mining the log files. The same result can be obtained by any other alternative process-mining tool. The result is a graphical representation of the processes, reporting each class and database table used in the business processes, with a set of arrows connecting each class based on the log traces. The result of this first step produces a figure that allows to understand:
• Runtime execution paths of the system. Paths never used, even if possible, are not represented in the figure.
• Dependencies between the classes of the system. The arrows represent the dependencies between methods and classes. External dependencies to libraries or web-services are also represented.
• The frequency of usage of each path. Process mining tools present the most used processes with thicker arrows.
• Branches and Circular Dependencies. The graphical representation allows easy discovery of circular dependencies or branches (e.g., conditional statement that led to different path based on the input provided), in case they exist.
The complete chain of arrows forms a candidate of a process. Figure 1 below represents a simplified example of one business process representing the data reported in Table 1 above.
Figure 1 – Simplified Process Example
Step 2: Frequency analysis of the execution paths
The thickness of the arrows created by the DISCO tool indicates the frequency of the calls between classes. This makes it possible to clearly understand which execution paths are used most frequently and which classes are rarely or never used during runtime. The output of this step is a table representing all the different execution paths with the frequency of their usage.
Table 2 – Frequency analysis of each execution path
Step 3: Removal of Circular Dependencies
In this step, you first find circular dependencies by analyzing the execution paths reported in the table generated in the first Step. This can be done with a simple algorithm to discover cycles in the execution paths. In the case of circular dependencies, software architects should discuss with the development team how to break these cycles. One example of the patterns that can be applied to break the cycles is Inversion of Control (Martin, 2003). However, every cyclic dependency could need a different breaking solution that must be analyzed carefully. The result is a refined version of the execution path table.
Step 4: Identification of decomposition options
Starting with the execution paths without cyclic dependencies obtained from Step 3, you identify different decomposition alternatives by visually inspecting the generated graphs. The candidate processes may have common sub-paths, i.e., the processes may merge or split.
Thus, different decomposition solutions are possible. This process could also be automated by developing an algorithm that provides all different decompositions based on the paths with fewer intersections. However, in this case, you rely on the expert-based decomposition.
As highlighted in Figure 2 below, the decomposition options need to deal with the duplication of some classes or methods. As example, the execution traces reported in Figure 2 below show that both the green and the orange execution traces use j(). Therefore, software architects could propose two decomposition alternatives. The first option includes the creation of three microservices where class E.java() is duplicated in microservice MS2 and MS3.
Figure 2 – Simplified Process Example
The second option includes the creation of two microservices, merging MS2 and MS3. Both options have pros and cons, but the decision of merging two execution traces or splitting into different microservices must be discussed with the team. If two microservices candidates for the splitting have different purposes, it is reasonable to consider the splitting. If they are doing the same thing, then it would be better to merge them into one single microservice for example.
Step 5: Metric-based ranking of the decomposition options
In this step, you identify three measures to help architects to assess the quality of the decomposition options identified in Step 4: Coupling, Number of classes per microservices, Number of classes that need to be duplicated.
A - Coupling
The decomposition to microservices should minimize coupling and maximize cohesion. Coupling and cohesion can be calculated with different approaches. While coupling can be obtained from the log traces, for all the cohesion measures we also need to know about the access to the local variables of each class, which makes it impossible to calculate them from the data reported in the log traces. However, coupling is commonly considered as inversely proportional to cohesion. Therefore, a system with low coupling will have a high likelihood of having high cohesion (Jabangwe et al., 2015). We define the Coupling Between Microservice (CBM) extending the well-known Coupling Between Object (CBO) metric proposed by Chidamber and Kemerer (Chidamber and Kemerer, 1994). CBO represents the number of classes coupled with a given class (efferent couplings and afferent couplings). This coupling can occur through method calls, field accesses, inheritance, arguments, return types, and exceptions.
You calculate the relative CBM for each microservice as follows:
Where Number of external Links represents the number of calls to external services used in each class of microservice. An external service linked several times by different classes of the same microservice is only counted once. External services could be other microservices, external APIs, etc. CBM is calculated for each microservice independently and presented in a table for the next step.
B - Number of classes per microservice
This measure helps to understand how big the microservice identified is and to identify if there are microservices that are too big compared to others. The number of classes should be minimized since the smaller the number of classes the more independent its development can be. Considering the example reported above in Figure 2, the MS decomposition option 1 has 7 classes while option 2 has six classes.
C - Number of classes that need to be duplicated
In some cases, several classes will be in common between two execution traces. As example, the method j in Class E (Figure 2) is used by two execution traces. In the example depicted in Figure 2, decomposition option 1 has one class that needs to be duplicated, while option 2 requires no classes to be duplicated. This measure helps to reason about the different slicing options, considering not only the size of the microservices but also the number of duplications, that will be then reflected in the development of the microservices. Duplicated classes should be avoided since the duplication adds to the size of the system and its maintenance, where they can be transformed into a library/dependency used by both microservices.
Step 6: Selection of the decomposition solution
This is the final step where, based on the different decomposition alternatives identified in Step 4, and on the measures collected in Step 5, software architects can decide which solution to adopt by merging or splitting existing processes. Software architects could consider the recommendations provided by this decomposition process and discuss with the team which solution is most suitable for them, considering the organizational structure. This process does not recommend the best decomposition solution but provides a reasoning framework on the possible decomposition options.
Conclusion
System logs offer a big opportunity to inform the decomposition of IT systems. By applying process mining to logs, main activities and transactions can be discovered. The architect is extended by invaluable and visual information that helps inform better design based on data and real needs.
References
Taibi, Davide & Systä, Kari. (2019). From Monolithic Systems to Microservices: A Decomposition Framework based on Process Mining.
Taibi, D., Lenarduzzi, V., and Pahl, C. (2018). Architectural patterns for microservices: a systematic mapping study.
Soldani, J., Tamburri, D. A., and Heuvel, W.-J. V. D. (2018). The pains and gains of microservices: A systematic grey literature review.
Taibi, D., Lenarduzzi, V., Janes, A., Liukkunen, K., and Ah- mad, M. O. (2017a). Comparing requirements decomposition within the scrum, scrum with kanban, xp, and banana development processes.
Taibi, D., Lenarduzzi, V., and Pahl, C. (2017b). Processes, motivations, and issues for migrating to microservices architectures: An empirical investigation.
Taibi, D., Lenarduzzi, V., Pahl, C., and Janes, A. (2017c). Microservices in agile software development: a workshop-based study into issues, advantages, and disadvantages.
Richardson, C. (2017). Pattern: Microservice architecture.
Jabangwe, R., Bo ̈rstler, J., Smite, D., and Wohlin, C. (2015). Empirical evidence on the link between object-oriented measures and external quality attributes: A systematic literature review.
Martin, R. C. (2003). Agile Software Development: Principles, Patterns, and Practices.
Chidamber, S. R. and Kemerer, C. F. (1994). A metrics suite for object oriented design.
Top comments (0)