From Monolith to Microservices: surfing the pipeline

In my previous post I quickly covered how I did the breakup of the majestic monolith into smaller services. In the end, I went for this:

1. An input streaming service (parse CSV input files)
2. An output streaming service (write CSV output files)
3. A service for sending payments to a 3rd-party
4. A service for polling for responses from that 3rd-party
5. A service to produce (decorated) output records.
6. An orchestrator service (to read folder and orchestrate pipeline steps), inspired by the Saga pattern. This service also holds the CLI application that it is used to kick-off the system.

However, there is a lot more to this than it might appear at first sight. I first started asking myself the following question.

Is there a tried-and-true methodology for breaking up a monolith into smaller microservices?

Domain-Driven Design

The number one methodology is called Domain-Driven Design (DDD). By carefully analysing your domain you are able to break up the monolith by business capability, not technical layers. Each microservice aligns to a so called "bounded context" (e.g., Billing, Inventory, User Management).

Domain logic and I/O

Whilst I had given DDD a shot at Commonplace, I did not see it as a great fit for my PoC project. I remembered watching this Scott Wlaschin's conference some time ago.

and finding it fascinating, as I saw so much synergy with what I was doing with the CSV Payments Processing PoC. I also remember this similar version called "Functional Core, Imperative Shell" (link to video), where DDD is also very much at play.

It's the pipeline, stupid!

Later on, when I was thinking about this "big breakup", I experienced this bulb moment when realised that, naturally, the boundaries were right there in front of me. A microservice per step of the pipeline was all I needed, give or take. And it was so much simpler than I thought it would be! I thought that was brilliant and gave myself a good pat on the back.

Benefits

Having a microservice per step of the processing pipeline has several benefits.

• Extensibility: very easy to add new processing steps as microservices
• Easy to orchestrate: as it plays along the original design
• Aligned with DDD: as each step in the pipeline takes care of a different business capability
• Simplicity: business logic and data flow in and out always in a single direction
• Loosely coupled: does not create any new dependencies. There are services that need to deal with the file system, whilst others just don't, and that is only dictated by the business logic and not forced by the architecture

Photo Credit: Pipeline - North Shore