DEV Community: Ticino Software Craft

Event storming, and then what?

Marco Sabatini — Thu, 11 Jul 2024 12:28:29 +0000

References

Overview

Some time ago, I read a fascinating book titled Introducing EventStorming: An Act of Deliberate Collective Learning by Alberto Brandolini.
This book, filled with concrete examples, discusses the event-storming technique for modeling the business processes underlying a digital product to be implemented through software.

What truly captivated me and sparked my curiosity about this methodology is its ability to model software architecture and reveal organizational limitations within a company. In this article, I will demonstrate with a concrete example how event-storming can effectively bridge the gap between business and software development.

Mars Rover Kata

The exercise I used as a reference is the mars rover Kata. I used Python to solve it.

Its requirements involve implementing software for a rover to receive commands for movement. The rover can move forward, rotate left and right, and has constraints related to the surface it can traverse and potential obstacles it might encounter.

Before starting to implement the software, I conducted an Event Storming session. Of course, this came with significant limitations: I was the only participant and not particularly experienced with the technique and this session was just to implement the software (there are several levels of event-storming abstraction we should consider).
I used basic elements to model a business process, including commands, events, aggregates, policies, and projections. The definitions of these components provided in the book are particularly enlightening:

Command: A decision made by the user.
Aggregate: Information necessary for making decisions.
Event: A state transition mapped somewhere.
Projections: Tools to support the decision-making process in the user's brain.
Policy: Triggers that respond whenever something happens

Regarding the flow, we can see how these components interact with each other in the picture below:

Everything will start with a command that triggers actions on a specific aggregate. This will generate an event and by listening to the event we can continue to trigger commands using Policies or create a view using Projections.

Using these building blocks, I attempted to model the Mars Rover design Event Storming.

Event Storming

Below, we can see the first iteration of the process.

First, I thought the rover needed to be powered on and set with a starting point and direction. The aggregate that came to mind here is the Mars Rover itself.
Once somebody starts it, it will be in "Started" mode and ready to move. Next, the rover can receive commands to turn right, turn left, and move forward. Depending on the presence of obstacles, the rover can either continue moving or encounter obstacles.
According to the exercise, in case of an obstacle, the rover should "shut down." Thus, I used a policy to react to the "ObstacleFound" event with a command instructing the rover to shut down.

From the first iterations, I noticed how intuitive it was to think in terms of Commands, Events, Aggregates, and Policies. I also used Projections to create datasets for future analysis, which I will discuss during the implementation phase.

From a modeling perspective, this technique is extremely useful. I could present the process to any product owner or domain expert or even implement it directly with them (as described in the book).
I am confident that in a very short time, we could create a dictionary of common terms understandable to all stakeholders involved (both business and software). Now, let's move on to the implementation phase.

Implementation

You can find the solution code here.

Upon opening the project, you'll find a package named ddd. In this package, I have included the basic elements described earlier:

class Command:
    pass

class Event:
    pass

@dataclasses.dataclass(frozen=True)
class AggregateId:
    value: str

    @classmethod
    def new(cls):
        return cls(value=str(uuid4()))

@dataclasses.dataclass
class Aggregate:
    id: AggregateId
    version: int

class CommandHandler:
    def handle(self, command: Command) -> Event:
        pass

class Policy:
    def apply(self, event) -> Command:
        pass

class Projection:
    def project(self, event):
        pass

I reused the same names to create a common base linking the modeling and implementation parts. I also implemented an in-memory repository responsible for loading and saving an Aggregate object and a very simple in-memory command dispatcher.

The command dispatcher receives a series of commands as input and applies command handlers, policies, and projections based on its construction.
For this exercise, the implementation is in-memory, but you could consider implementing it with remote queues.

Here is our command dispatcher:

class InMemoryCommandDispatcher:
    def __init__(self,
                 command_handlers: Dict[Type[Command], CommandHandler],
                 projections: Dict[Type[Event], List[Projection]],
                 policies: Dict[Type[Event], List[Policy]]):
        self.command_handlers = command_handlers
        self.projections = projections
        self.policies = policies

        self.commands: List[Command] = []

    def submit(self, commands: List[Command]):
        for c in commands:
            self.commands.append(c)

    def run(self):
        while self.commands:
            command = self.commands.pop(0)
            print(f"[COMMAND] {command}")

            event: Event = self.command_handlers[type(command)].handle(command)
            print(f"[EVENT] {event} generated")

            event_policies = self.policies.get(type(event), [])
            for policy in event_policies:
                new_command = policy.apply(event)
                if new_command:
                    self.commands.append(new_command)

            for projection in self.projections.get(type(event), []):
                projection.project(event)

At this point, I have all the necessary components to build the model defined during the Event Storming session. Therefore, I implemented the commands, events, aggregate, and command handlers, as shown in the function defined to build the process.


def create_command_dispatcher(mars_rover_repo: MarsRoverRepository,
                              mars_rover_storage: List[MarsRoverId],
                              path_projection_storage: List[Dict],
                              obstacles_projection_storage: List[Dict]) -> InMemoryCommandDispatcher:
    turn_right_command_handler = TurnRightCommandHandler(repo=mars_rover_repo)
    turn_left_command_handler = TurnLeftCommandHandler(repo=mars_rover_repo)
    move_command_handler = MoveCommandHandler(repo=mars_rover_repo)
    start_command_handler = StartMarsRoverCommandHandler(repo=mars_rover_repo)
    turn_off_command_handler = TurnOffCommandHandler(repo=mars_rover_repo)
    notify_obstacle_command_handler = NotifyObstacleCommandHandler()

    rover_path_projection = MarsRoverPathProjection(repo=mars_rover_repo, storage=path_projection_storage)
    rover_start_projection = MarsRoverStartProjection(repo=mars_rover_repo,
                                                      paths_storage=path_projection_storage,
                                                      mars_rover_storage=mars_rover_storage)
    rover_obstacles_projection = MarsRoverObstaclesProjection(storage=obstacles_projection_storage)

    obstacle_found_policy = NotifyObstacleFoundPolicy()
    turn_off_policy = TurnOffPolicy()

    return (InMemoryCommandDispatcherBuilder()
            .with_command_handler(TurnRight, turn_right_command_handler)
            .with_command_handler(TurnLeft, turn_left_command_handler)
            .with_command_handler(Move, move_command_handler)
            .with_command_handler(StartMarsRover, start_command_handler)
            .with_command_handler(TurnOff, turn_off_command_handler)
            .with_command_handler(NotifyObstacle, notify_obstacle_command_handler)
            .with_projection(MarsRoverStarted, rover_start_projection)
            .with_projection(MarsRoverMoved, rover_path_projection)
            .with_projection(ObstacleFound, rover_obstacles_projection)
            .with_policy(ObstacleFound, obstacle_found_policy)
            .with_policy(ObstacleFound, turn_off_policy)
            .build())

As you can see, this function creates the dispatcher by configuring the process modeled during the Event Storming session. Specifically, it associates commands with their respective command handlers, as well as policies and projections. It is straightforward to understand the actions associated with commands and events.

Command Handlers

Let's take a look at how I manage a command using a command handler for controlling the Rover's movement.

class MoveCommandHandler(CommandHandler):
    def __init__(self, repo: MarsRoverRepository):
        self.repo = repo

    def handle(self, command: Move) -> MarsRoverMoved:
        mars_rover: MarsRover = self.repo.get_by_id(command.id)
        event = mars_rover.move()
        self.repo.save(mars_rover)
        return event

The class uses the repository to load the MarsRover Aggregate into memory and then calls the "move()" function to change its state.
Following this, the event is emitted and the aggregate's state is saved

Domain

Regarding the domain part, the figure below shows the objects used to model it.

Let me show also the MarsRover Aggregate methods contract:

@dataclasses.dataclass
class MarsRover(Aggregate):
    actual_point: Point
    direction: Direction
    world: World
    status: MarsRoverStatus

    def start(self) -> MarsRoverStarted:
        ...

    def turn_off(self) -> MarsRoverTurnedOff:
        ...

    def turn_right(self) -> MarsRoverMoved | MarsRoverTurnedOff:
        ...

    def turn_left(self) -> MarsRoverMoved| MarsRoverTurnedOff:
        ...

    def move(self) -> MarsRoverMoved | ObstacleHit |  MarsRoverTurnedOff:
        ...

It contains all the possible actions the Rover can do and the logic to change its internal state, starting from the actual point and direction to the grid/world in which the rover is moving.

Services

To orchestrate everything, I implemented a service.
Below you can see the service being used in the end-to-end tests developed:

class TestE2E(unittest.TestCase):
    def test_execute_some_commands(self):
        repo = MarsRoverRepository()
        paths = []
        obstacles = []
        mars_rover_ids = []

        runner = (
            MarsRoverRunner(repository=repo,
                            path_projection_storage=paths,
                            obstacles_projection_storage=obstacles,
                            mars_rover_projection_storage=mars_rover_ids)
            .with_initial_point(x=0, y=0)
            .with_initial_direction(direction=Direction.NORTH)
            .with_world(world_dimension=(4, 4),
                        obstacles=[])
        )
        runner.start()

        id = mars_rover_ids[0]

        runner.run(id, "RMLMM")

        actual: MarsRover = repo.get_by_id(MarsRoverId(id))
        self.assertEqual("1:2:N", actual.coordinate())
        self.assertEqual("MOVING", actual.status.value)

        expected_path = ["0:0:N", "0:0:E", "1:0:E", "1:0:N", "1:1:N", "1:2:N"]
        self._assert_paths(expected=expected_path, actual=paths)

        self.assertListEqual([], obstacles)

In this test, I set the starting point, the grid on which the Rover moves, and its initial direction.
After that, the Rover is started calling runner.start() and receives commands in string format calling runner.run(id, "RMLMM"). I need to pass the rover id just to retrieve it using the repository. I can retrieve the RoverId from a projection created when we started it before.

Regarding assertions, I used the datasets generated by the paths projections, which are also developed with an in-memory version to facilitate interactions.

Below there are the logs of Commands and Events generated during the flow:

_e2e.py::TestE2E::test_execute_some_commands 
[COMMAND] StartMarsRover(initial_point=Point(x=0, y=0), initial_direction=<Direction.NORTH: 'N'>, world=World(dimension=(4, 4), obstacles=Obstacles(points=[])))
[EVENT] MarsRoverStarted(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[COMMAND] TurnRight(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[COMMAND] Move(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[COMMAND] TurnLeft(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[COMMAND] Move(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[COMMAND] Move(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='aea5e69c-4f08-40db-bf44-097b5ae36380'))

Basically the rover is getting as input this commands "RMLMM" and they are translated during the flow execution.

Path Projection

I used a paths projection in order to store the coordinates every time the rover is moved.
As we can see in the before session we are able to assert the entire Rover path:

expected_path = ["0:0:N", "0:0:E", "1:0:E", "1:0:N", "1:1:N", "1:2:N"]
self._assert_paths(expected=expected_path, actual=paths)

Therefore, for this exercise I implemented an in memory projection that is able to store path data in a list of dictionary:

class MarsRoverPathProjection(Projection):
    def __init__(self, repo: MarsRoverRepository, storage: List[Dict]):
        self.repo = repo
        self.storage = storage

    def project(self, event: MarsRoverMoved):
        mars_rover: MarsRover = self.repo.get_by_id(event.id)

        raw = {"id": mars_rover.id.value, "actual_point": mars_rover.coordinate()}

        self.storage.append(raw)

This class takes as input the event MarsRoverMoved because is triggered on this, loads the aggregate and is able to create a specific read model.
In this case the read model contains the paths traveled by the rover. The good thing of this design is that the projection is totally decoupled by the aggregate state change and could be easy bind on a specific event that represents the Aggregate state change.

Policy

At this point, when the Rover encounters an obstacle, it must automatically shut down.
To achieve this, I implemented a shutdown Policy. This Policy takes the ObstacleFound event as input and generates a shutdown event, which is then handled by its own command handler, as we saw earlier.
Below you can see the TurnOffPolicy implementation:

class TurnOffPolicy(Policy):
    def apply(self, event: ObstacleFound) -> Command:
        return TurnOff(id=event.id)

Additional requirement

I tried to push the design further by considering what would happen if I had an additional requirement, such as sending a notification when the Rover encounters an obstacle.
First, I reviewed the initial design event storming model and I integrated the notification feature, resulting in this new version:

As you can see from the diagram, all I had to do was define a new Policy based on the ObstacleFound event, which was responsible for creating the command to notify the obstacle found.

class NotifyObstacleFoundPolicy(Policy):
    def apply(self, event: ObstacleFound) -> Command:
        return NotifyObstacle(message=f"Rover {event.id.value} hit obstacle")

After the creation of the command we will have the command handler that will manage the notification flow.
Regarding the Policy this is initialized within the builder of the command dispatcher, which effectively configures the process.
Here, I simply added the policy without needing to change or refactor existing code.
The test below is just to show how we can make an end to end test setting up obstacles:

 def test_hit_obstacle(self):
        repo = MarsRoverRepository()
        paths = []
        obstacles = []
        mars_rover_ids = []

        runner = (
            MarsRoverRunner(repository=repo,
                            path_projection_storage=paths,
                            obstacles_projection_storage=obstacles,
                            mars_rover_projection_storage=mars_rover_ids)
            .with_initial_point(x=0, y=0)
            .with_initial_direction(direction=Direction.NORTH)
            .with_world(world_dimension=(4, 4),
                        obstacles=[(2, 2)])
        )

        runner.start()

        id = mars_rover_ids[0]

        runner.run(id, "RMMLMMMMMM")

        obstacles_found = [o["obstacle"] for o in obstacles]
        self.assertEqual([(2, 2)], obstacles_found)

It's easy setup obstacles and assert they are found checking the in memory obstacle projection storage. I created an obstacle projection in order to collect where is the obstacle found by the rover during the travel. It was easy because I had ObstacleFound event and I just needed to listen for this event and project it.

Below there are the logs of the Commands and Events generated during the test. As you can see during the travel the rover found an obstacle. After that it was turned off and the obstacle was notified.

test/test_e2e.py::TestE2E::test_hit_obstacle 
[COMMAND] StartMarsRover(initial_point=Point(x=0, y=0), initial_direction=<Direction.NORTH: 'N'>, world=World(dimension=(4, 4), obstacles=Obstacles(points=[Point(x=2, y=2)])))
[EVENT] MarsRoverStarted(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] TurnRight(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] Move(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] Move(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] TurnLeft(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] Move(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverMoved(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[COMMAND] Move(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] ObstacleFound(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'), coordinate=(2, 2))
[COMMAND] NotifyObstacle(message='Rover 649d914f-967c-41ae-b3ca-ffede9ca9e7d hit obstacle')
Rover 649d914f-967c-41ae-b3ca-ffede9ca9e7d hit obstacle
[EVENT] ObstacleNotified()
[COMMAND] TurnOff(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))
[EVENT] MarsRoverTurnedOff(id=MarsRoverId(value='649d914f-967c-41ae-b3ca-ffede9ca9e7d'))

Evolutionary Architecture

Regarding the notification, I did not proceed further in the exercise. This is a typical example of what happens every day in our work: we create something that may have follow-up actions if it proves valuable.

Interestingly, the model already decouples this new concept (the notification), which we could potentially develop into a dedicated product line, with its own aggregate. This could become an external system that reads the ObstacleFound event from a queue and generates the notifications.
If we need to scale up because the notification system requires product enhancements, we could create a dedicated team to handle this domain.

This example is just to illustrate how such a design approach not only helps evolve our architecture but also guides it to support product development and organizational changes.

Some time ago, we had Nicola Moretto speak at our meetup about product development consistency.

He showed us this slide, which I find very significant. He explained that Architecture, Business, and Organization should go hand in hand with product needs and must be easily modifiable to manage product increments.

How often do we encounter these situations?

Implement features on legacy architectures.
Implement features but can't do it end-to-end because we depend on other teams.
Implement features whose business value is uncertain.

To address these types of situations, the architect must act as the point of contact between the business and the organization needed to support it, by implementing an architecture that is most easily adaptable to the problem at hand.

The hidden trap of debugger

Mario Russo — Tue, 14 Dec 2021 19:28:41 +0000

"Debugging is like being the detective in a crime movie where you are also the murderer." - Filipe Fortes

How it started: a false illusion

When i had my first programming experience i was not aware of the possibility of using a debugger.
It happened that i had to understand why my software was not working as expected , so the only way i had to understand what was going on was to add some print statements all around the code.
Later I learnt that i can use the debugger as a tool to inspect variables and understand the status of my software in a specific time.
I thought

"I can troubleshoot without adding new code!!
I will be faster than before!"

How is going

After years working on many and more complex projects i almost totally change my mind, and realised that actually the debugger slowed me down!
To be clear i still think that in few specific simple cases it is worth to leverage the debugger but in complex cases i think there are better alternatives.
What happened ?
I was thinking that debugger was the only way to troubleshoot issues so i used it all the times.
If i come back to the my most stressful working times almost all of them were moment in which i was trying to understand the code ,spending hours in long debugging sessions inspecting tons of line of code variables and stack trace.
Can you remember your most stressful working time ?
Can you remember if in those moment you were in long debugging sessions ?
Probably you were trying to escape the legacy code matrix and you were stuck on the first phase: the understanding phase

Where is the trap ?

I will share now why in my opinion in complex use cases using the debugger actually slow us down and increase our stress and cognitive load

1. Give up to learn the code
Using the debugger is like using a navigator for your car. It actually helps you but sometimes for some reason it dint work and when it happens you are totally lost because you don't know the path.
Jumping soon to debugger prevent you to learn the code and to improve your reading and reverse engineering skills.
Those are skill that you can acquire only reading more and more code written by someone else. The more you do the more you get better to understand it. An remember there are cases in which you cannot rely use the debugger so it is better to have an alternative.

2. Manual actions
Inspecting a lot of lines of code and variables require manual and very repetitive action like put breakpoint , enter inside function , go to next line , inspect variable, add watches etc..
Manual action is much slower then automatic action that could performed by computers.
Keeping in mind a lot of variable values and application state require high cognitive load and energy.

3. Repetitive actions
Imagine that you have the following type hierarchy

data class University(val departments: List<Department>)

data class Department(val courses: List<Course>)

data class Course(val students: List<Student>)

data class Student(val name: String, val country: Country)

enum class Country {
    ITALY,
    SPAIN,
    FRANCE
}

and you need to check more than one time if an university has students from Italy
Every time you need to inspect the country of student in university you need to do at least 5 action like five clicks , inspect variable take notes or remember their values
If you have to it 10 times , you will have to do 50 click or debugger inspections!.

An alternative approach could be write a function like

fun logItalianPresenceInUniversity(university: University)

that prints what you want to check. Few additional code to write and just 1 click to execute and check the result.

3. Lack of trust in tests
Someone said

"A bug is just a missing test"

The purpose of our tests is to help us to define the behaviour of our system and help us to find issues.
An alternative way to debugging to detect an issue is to reproduce it by adding a new test.
Now if we found a bug thanks to debugger we can call in trap to think "test suite did not help me, so i will do the fix and forget the tests".
So we may enter in this dangerous loop

A sample case: microservice with many data transformations

Imagine a service exposing some functionality to the user that need to call other services to fulfil the request
In complex cases there are many data adaptation between the controller and the call to the other services

Now let's suppose that there is a bug in that service and some information that arrives from the external service is not sent back to the controller and the user.
Your task is to spot it and fix it and you need to deep dive in many adaptation layers.
Which options you have ?

1: Understand and document every little detail of the code
The safest option is to read carefully the code trying to understand it , taking notes and spot the missing point.
This option is good but may require a lot of time and effort to deep dive in the code structure. In legacy systems or when you don't have much time it can become an endless process. On other hand it on long term it improve your skills to read the code.

2: Use the application with debugger
We could put a breakpoint on controller and on client and use the application.
In our case having a lot of adaptation layers this will be take a lot of time with debugging session and so big effort.
For non trivial cases this approach is probably the worst one.

3: Use the application with probes in code
Here you can guess what is the idea.
You can write some small probe functions to automatically check what would manually ask the debugger to do. Remember the logItalianPresenceInUniversity function described above ? .

My personal approach in this cases is
1. identify the start and end of code section to check
in this case the controller and the client
2. Automatize checks by writing probes functions
Example add function that print if a data structure contains the data that i am looking for
3. Put probes in code and create a git patch for local changes
4. Run the application and check the probes
5. Repeat step 3 until issue is found
A binary search algorithm can limit the tries
6. Rollback local changes when done

In complex scenario i find this approach faster , incremental and it require less cognitive load.

Issue is found now what?

Regardless how i found my preference is:

highlight the issue by writing a new test and watching it fail***
Fix the text

Conclusion

When we have to find issues in code we have many option on the strategy to follow.
Using the debugger is only one of them and in my opinion in some cases is not the best approach.
Having more than one option we can choose the one that fit more our case.

*** Writing a new test can be hard in some cases you may need to break dependencies to isolate the section of code as described in working effectively with legacy code

Test Containers and Clean Architecture

Marco Sabatini — Thu, 25 Nov 2021 22:53:00 +0000

References

Overview

Separating the infrastructure from the domain is a smart move to define clear system boundaries, especially when dealing with components responsible for interacting with the external world, such as databases, external APIs, or web services. In this exploration, we'll delve into the TestContainers technique for constructing a robust test suite that aligns with these system boundaries.

According to the TestContainers documentation, this tool simplifies various types of tests, making them more manageable:

Data access layer integration tests: TestContainers allows us to use a containerized instance of databases like MySQL, PostgreSQL, or any other DBMS. This makes it easy to test data access layer code, ensuring compatibility without the need for intricate setups on developers' machines. The assurance that tests always begin with a known database state is a significant benefit. Additionally, any containerizable database type can be utilized.

Application integration tests: This involves running your application in a transient test mode with dependencies such as databases, message queues, or web servers. TestContainers facilitates this process, making it efficient and reliable.

UI/Acceptance tests: For automated UI tests, TestContainers provides containerized web browsers compatible with Selenium. Each test can spawn a fresh browser instance, eliminating concerns about browser state, plugin variations, or automated browser upgrades. Moreover, TestContainers records video sessions for each test, aiding in debugging and analysis.

Now, our approach will be a bit different. Instead of using the TestContainers framework directly, we'll leverage Docker along with some bash scripts for configuration. Let's dive into this pet code adventure!

Kata

Regarding my previous post, we want to write the DB access layer for this function:

loadEmployees: () -> Either<Error, Employees>

And hey, why not use Docker for testing the infrastructure? I've got these integration tests hanging around for the FileLoadEmployee function:

@Test
internal fun loadSomeEmployees()

@Test
internal fun employeeNotValidEmail()

No need to go into the nitty-gritty details of those tests—they're pretty straightforward.
Here's the plan: I want to repurpose them for the DB implementation, but let's not go down the road of duplicating code. My trick? Creating an abstraction for these tests that can serve both the File and DB implementations.

abstract class LoadEmployeeTest {

    @Test
    internal fun loadEmployee() {
        val loadEmployees = instance()

        assertThat(loadEmployees()).isEqualTo(
            Either.Right(
                Employees(
                    listOf(
                        Employee(
                            "Marco", "Sabatini", DateOfBirth(5, 3, 1983),
                            EmailAddress("address@email.com")
                        )
                    )
                )
            )
        )
    }

    @Test
    internal fun employeeNotValid() {
        val loadEmployees = wrongInstance()

        assertThat(loadEmployees()).isEqualTo(Either.Left(Error("Error For input string: \"wrong\"")))
    }

    abstract fun instance(): () -> Either<Error, Employees>
    abstract fun wrongInstance(): () -> Either<Error, Employees>
}

So, here's the deal: I'll whip up two abstract methods—instance() and wrongInstance(). These are the brainiacs responsible for conjuring up the function instances that handle loading employees. They'll serve up the goods for both the happy path and those pesky corner cases.

Now, for the File access layer:

override fun instance(): () -> Either<Error, Employees> =
        loadEmployeeFrom("./target/test-classes/employees.txt")

    override fun wrongInstance(): () -> Either<Error, Employees> =
        loadEmployeeFrom("./target/test-classes/employeesNotValid.txt")

    @Test
    internal fun fileNotFound() {

        val loadEmployeeFromFile = loadEmployeeFrom("NOT_EXIXSTING_FILE")

        Assertions.assertThat(loadEmployeeFromFile())
            .isEqualTo(Either.Left(Error("File NOT_EXIXSTING_FILE doesn't exist")))
    }

For DB access layer:

  @AfterEach
    fun cleanupTest() {
        execute("DELETE from employees")
    }

    override fun instance(): () -> Either<Error, Employees> {
        execute("INSERT INTO employees VALUES ('Marco', 'Sabatini','05/03/1983','address@email.com')")

        return loadEmployeeWith()
    }

    override fun wrongInstance(): () -> Either<Error, Employees> {
        execute("INSERT INTO employees VALUES ('', '','wrong','')")

        return loadEmployeeWith()
    }

    private fun execute(sql: String) {
        val stmt = connection().createStatement()
        stmt!!.executeUpdate(sql)
        stmt.close()
    }

You know, this trick I'm pulling off is called a contract test. It's like our secret sauce when we need to lay down the law for specific behavior in our code. Especially handy when we've got different implementations floating around.

So, what's cooking here? I'm defining the loadEmployees behavior between my domain code and the infrastructure code. It's like setting the rules of engagement. And the cool part? I'm testing different implementations to make sure they play nice. It's the go-to move if you want to throw in an in-memory implementation of an external 'port' and speed up those acceptance tests. Quick and snappy!

Docker && Docker Compose

Now it's time to move to the infrastructure part. I need a DBMS MySQL instance and a JVM maven runtime where my Kotlin test code can run. These containers have to communicate with each other.
This is the docker-compose file configuration that I used:

services:
  mysql:
    image: mysql
    container_name: mysql_server
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=pwd
    networks:
      - db-network

   db_client:
    build:
      context: ./containers/client
    networks:
      - db-network
    environment:
      - WAIT_HOSTS=mysql:3306
      - WAIT_HOSTS_TIMEOUT=300
      - WAIT_SLEEP_INTERVAL=30
      - WAIT_HOST_CONNECT_TIMEOUT=30

  maven:
    image: maven
    container_name: builder
    volumes:
      - ${PWD}:/tmp
    networks:
      - db-network

networks:
  db-network:

MySQL: DBMS
db_client: is a MySQL client command line interface. It is responsible for creating schema and could be used also for other purposes (ex. load demo test data etc.). It uses an entrypoint docker to create a schema or load demo data and this plugin helps to wait for MySQL instance to be started.

case "$@" in
  schema)
    /wait && mysql --host=mysql -uroot -ppwd < employees.sql
    echo "EMPLOYEES schema created!"
  ;;
  demo)
    /wait && mysql --host=mysql -uroot -ppwd < demo.sql
    echo "Could load demo data!"
  ;;
  *)
    exec "$@"
  ;;
esac

Here you can see docker container configuration details.

maven: is the container where my Kotlin test code is executed.

All the CI pipeline is orchestrated from this bash script:

#!/bin/bash
docker-compose up -d mysql
docker-compose build db_client
docker-compose run db_client schema
docker-compose run maven mvn --quiet -f /tmp clean install
docker-compose run maven mvn -f /tmp surefire-report:report -DshowSuccess=false
docker-compose down --remove-orphans

You can build the project directly executing it on your computer or we can use it to create a CI pipeline.

Setting CI pipeline

I have all the pieces to build my CI on github using GH actions.
Under .github/ folder I have this configuration:

name: docker-compose-actions-workflow
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: CI
        run: script/ci
      - name: DEMO
        run: script/demo

So, here's the lowdown: when we hit that push button, it kicks off a build that runs the CI and demo scripts. Just a smooth ride through the GitHub Actions tab, where you can catch all the action and check out the build outputs.

[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.kata.testcontainers.infrastructure.DBLoadEmployeeTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.561 s - in com.kata.testcontainers.infrastructure.DBLoadEmployeeTest
[INFO] Running com.kata.testcontainers.infrastructure.FileLoadEmployeeTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in com.kata.testcontainers.infrastructure.FileLoadEmployeeTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
containers ---

Demo output:

######## Employees ########
First Name: Marco
Last Name: Sabatini
Email: EmailAddress(value=address@email.com)
BirthDay: DateOfBirth(day=5, month=3, year=1983)
....

Keep in mind that I could add validation, performance, or quality gate steps.

Considerations

Having the infrastructure right there in the project codebase? It's like having a backstage pass to the entire ecosystem. No more long-distance vibes between dev and ops. Now, in my world, I can toss my container onto any cloud service—ECS on AWS, anyone? And the best part? I'm pretty darn confident the behavior will be just as groovy as what I see during CI or testing demos.

Now, here's the real talk from an organizational angle. Sure, having the infrastructure mingling with the code means having folks who can sling code, pick the right tools, and craft the necessary infrastructure magic. We're not just code jockeys; we're like the wizards who know their way around systems.

Clean Architecture and Either Monad

Marco Sabatini — Mon, 22 Nov 2021 14:27:17 +0000

References

Kata

I've been coding the birthday greeting in kotlin.
This exercise is very useful if you want to learn more about architect your code separating domain from infrastructure code.
I also had the purpose of making some practice with functional programming, in particular I wanted to use some functional data structure like monads.

Start

I started with some doubts: what kind of monad do I have to use? IO monad? Either monad? Both?
I can't find a response. I wanted something that made my code easily readable; I decided to use Either monad because my aim was to make the method signatures explicit.

I used the arrow implementation of the data structure. Docs is very clear and full of examples.

Either monad is like:

// This is the data structure
Either<ErrorType, ResponseType>

// Success case
Either.Right(ResponseType())

// Error case
Either.Left(ErrorType("This is the Error!"))

It helps to manage cases like a service that may result in a connection issue, or any unexpected JSON response.

Kata

Basically, my software will be a composition of functions.
The exercise asks for a software that is able to find employees whose birthday is today and send them emails.
I need to load employees from a data source (a file for this exercise), filter them, and send email.

My software could get an error when it's loading employees or sending Email.
It's curious to notice that when we filter employees (is your birthday?) nothing could go wrong because is something related our domain logic (pure function).

loadEmployees: () -> Either<MyError, Employees>

filterEmployees: (Employees) -> BirthdayEmployees

sendBirthdayNotificationTo: (BirthdayEmployees) -> Either<MyError, Unit>

These are the functions that I will compose into my use case. It's very clear what will be the purpose of code and when I could get an error.
I just need to inject the implementations of the functions using dependency injection.
As a result, I have a function responsibly to execute the composition (tadaaa!).

fun sendGreetingsWith(
    loadEmployees: () -> Either<MyError, Employees>,
    filterEmployees: (Employees) -> BirthdayEmployees,
    sendBirthdayNotificationTo: (BirthdayEmployees) -> Either<MyError, Unit>
): () -> Either<MyError, Unit> = {

    loadEmployees()
        .map(filterEmployees)
        .flatMap(sendBirthdayNotificationTo)
}

I can specify the types of errors I can get during execution, modelling them using sealed classes:

sealed class MyError {
    data class LoadEmployeesError(val msg: String) : MyError()
    data class SendMailError(val msg: String) : MyError()
}

Moreover in this situation the code is quite obvious and we don't need any comment to understand what is the purpose.

Testing

It's very curious to see how easily is testing after separating infrastructure from the domain.
For example: in this acceptance test I could inject the 'load employee logic' using in memory implementation and running the test without having additional configurational stuff for the infrastructure part.

    @Test
    internal fun oneEmployeeIsBornTodayAnotherIsNotBornToday() {

        val inMemoryLoadEmployee: () -> Either<Nothing, Employees> = {
            Right(
                Employees(
                    listOf(
                        employeeBirthday(dateOfBirth(TODAY), "TODAY_EMPLOYEE"),
                        employeeBirthday(dateOfBirth(TOMORROW), "TOMORROW_EMPLOYEE")
                    )
                )
            )
        }

        val sendGreetings: () -> Either<MyError, Unit> =
            sendGreetingsWith(
                inMemoryLoadEmployee,
                todayBirthdayEmployees,
                sendMailWith("localhost",9999)
            )

        val result = sendGreetings()

        val message = mailServer.receivedEmail.next() as SmtpMessage
        assertThat(message.body).isEqualTo("Happy birthday, dear TODAY_EMPLOYEE!")

        assertThat(result).isEqualTo(Right(Unit))
    }

Just inject the in memory behaviour and execute the test.

Do I need to test infrastructure?

Yes of course!
After separating domain logic from the infrastructure logic, It's very easy testing infrastructure code in isolated mode using an integration test like these:

    @Test
    internal fun loadEmployeeFromFile() {

        val loadEmployeeFromFile = loadEmployeeFrom("./target/test-classes/employees.txt")

        assertThat(loadEmployeeFromFile()).isEqualTo(Right(Employees(listOf(Employee("Marco","Sabatini", DateOfBirth(5,3,1983),
            EmailAddress("address@email.com")
        )))))
    }

    @Test
    internal fun employeeNotValid() {

        val loadEmployeeFromFile = loadEmployeeFrom("./target/test-classes/employeesNotValid.txt")

        assertThat(loadEmployeeFromFile()).isEqualTo(Left(MyError.LoadEmployeesError("Error For input string: \"address@email.com\"")))
    }

    @Test
    internal fun fileNotFound() {

        val loadEmployeeFromFile = loadEmployeeFrom("NOT_EXIXSTING_FILE")

        assertThat(loadEmployeeFromFile()).isEqualTo(Left(MyError.LoadEmployeesError("File NOT_EXIXSTING_FILE doesn't exist")))
    }

Imagine that we have to load data from database. We just could write the jdbc implementation of loadEmployees and we could use a docker container for making integration test and voila' (see on my post).
In this way all these integration tests will become a tests container or if we have to integrate with external services (like API) we could write just contract tests.

End

From a design point of view I have a use case responsible to orchestrate the flow.
We can see three main responsibilities:

Load data
Execute some domain logic
Send email messages

The software is quite generic and using dependency injection is very easy loading data from another data source or sending another kind of message like for example a mobile notification, just implementing these functions:

loadEmployees: () -> Either<MyError, Employees>

sendBirthdayNotificationTo: (BirthdayEmployees) -> Either<MyError, Unit>

This is the power of separating domain logic from infrastructure.

Regarding functional programming, we can say 'Either' monad make my code more readable even if I can achieve the same result using Object Oriented programming with exceptions.

I would think about monads like a meta-language on top of our programming language, able to create something understandable for developers (a sort of dev ubiquitous language).
If we had developed the exercise using 'Either' monad with another language probably we'll have the same result.

P.S.

This is an exercise. I could resolve it in a lot of different ways.
Don't focus on technical stuff but on design and testing!

Functional programming building blocks 2nd round

Marco Sabatini — Mon, 24 Aug 2020 13:25:56 +0000

References

OCP principle

In my previous post, I talked about functional programming's fundamental building blocks and introduced the Open-Closed Principle (OCP). Today, I want to explore software modularization and how we can implement the OCP using functional programming techniques.

The Open-Closed Principle (OCP) may sound simple, but it can be challenging to implement effectively: "Software should be open for extension but closed for modification." Achieving this principle is crucial for maintaining agility and adaptability in your development process.

Let's revisit the context of our Video Store Kata. We've already implemented the ability to print receipts in plain text. However, a new requirement has arisen: we need to print receipts in HTML format without altering our existing codebase. This is where the Open-Closed Principle comes into play.

To fulfill this requirement, we'll start by examining our receipt module:

class PrintableMovie {
    title: string;
    priceRepresentation: string;

    constructor(title: string, priceRepresentation: string) {
        this.title = title;
        this.priceRepresentation = priceRepresentation;
    }
}

const printableMovieWith =
    (calculateMoviePrice: (r: Rental) => number) =>
        (r: Rental) => new PrintableMovie(r.mc.title, calculateMoviePrice(r).toPrecision(2));

export const printableMovie: (r: Rental) => PrintableMovie =
    printableMovieWith(calculateMoviePrice);

This module is designed to be generic. It defines a PrintableMovie data type for items that need to be printed. It also includes two functions:

1) printableMovie transforms a Rental into a PrintableMovie.
2) printableMovieWith takes a price calculation function as input and prints the price with a precision of two digits.

This serves as a contract between the pricing module and the receipt module. Defining this contract using functions allows us to test pricing and receipt modules as separate black boxes.

With this foundation, we can now generalize the print receipt function:

export const genericReceipt =
    (header: (user: string) => string,
     body: (rentals: Rental[]) => string,
     footer: (rentals: Rental[]) => string,
     rentalPoint: (rentals: Rental[]) => string) =>

        (user:string, rentals:Rental[]) =>
            header(user) +
            body(rentals) + "\n" +
            footer(rentals) + "\n" +
            rentalPoint(rentals)

Now, we can implement both the plain text and HTML templates.

For plain text:

const textMovieReceipt = (m: PrintableMovie): string =>
     `- ${m.title} ${m.priceRepresentation}`

const textMoviesReceiptWith = (
    movieReceiptFunc: (x: Rental) => string) =>
     (rentals: Rental[]) => rentals.map(r => movieReceiptFunc(r)).join("\n")

const textFooterReceiptWith = (
    totalPrice: (rentals: Rental[]) => number) =>
     (rentals: Rental[]) => `Total ${totalPrice(rentals).toPrecision(2)}`

const textFooterRentalPointReceiptWith = (
    calculateRentalPoint: (rentals: Rental[]) => number) =>
     (rentals: Rental[]) => `Total Rental points ${calculateRentalPoint(rentals)}`

//WIRING HERE
const textFooterRentalPointReceipt =
    textFooterRentalPointReceiptWith(calculateRentalPoints);

const textFooterReceipt: (rentals: Rental[]) => string =
    textFooterReceiptWith(calculateTotalMoviesPrice);

const textMoviesReceipt: (rentals: Rental[]) => string =
    textMoviesReceiptWith(compose(
        printableMovie,
        textMovieReceipt))

const textHeader = (user: string) => `Hello ${user} this is your receipt\n`;

//WIRING THE PRINT FUNCTION WITH PLAIN TEXT BEHAVIOUR
export const printTextReceipt: (user: string, rentals: Rental[]) => string =
    genericReceipt(
        textHeader,
        textMoviesReceipt,
        textFooterReceipt,
        textFooterRentalPointReceipt)

And for HTML:

const htmlMovieReceipt = (m: PrintableMovie): string =>
    `<li>${m.title} ${m.priceRepresentation}</li>`

const htmlMoviesReceiptWith = (
    htmlMovieReceipt: (x: Rental) => string) =>
    (rentals: Rental[]) => `<ul>\n${rentals.map(r => htmlMovieReceipt(r)).join("\n")}\n</ul>`

const htmlFooterReceiptWith = (
    calculateMoviesTotalPrice: (rentals: Rental[]) => number) =>
    (rentals: Rental[]) => `<br>You owed ${calculateMoviesTotalPrice(rentals).toPrecision(2)}`

const htmlFooterRentalPointReceiptWith = (
    calculateRentalPoint: (rentals: Rental[]) => number) =>
    (rentals: Rental[]) => `<br>You earned ${calculateRentalPoint(rentals)} frequent renter points\n</body>\n</html>`

//WIRING HERE
const htmlFooterRentalPointReceipt: (rentals: Rental[]) => string =
    htmlFooterRentalPointReceiptWith(calculateRentalPoints);

const htmlFooterReceipt: (rentals: Rental[]) => string =
    htmlFooterReceiptWith(calculateTotalMoviesPrice);

const htmlMoviesReceipt: (rentals: Rental[]) => string =
    htmlMoviesReceiptWith(compose(
        printableMovie,
        htmlMovieReceipt))

const htmlHeader = (user: string) =>
    `<!DOCTYPE html>\n` +
    `<html>\n` +
    `<head>\n` +
    `<title>Video store - statement for ${user}</title>\n` +
    `</head>\n` +
    `<body>\n` +
    `<h1>Rental Record for ${user}</h1>\n`

//WIRING THE PRINT FUNCTION WITH HTML TEXT BEHAVIOUR
export const printHtmlReceipt: (user: string, rentals: Rental[]) => string =
    genericReceipt(
        htmlHeader,
        htmlMoviesReceipt,
        htmlFooterReceipt,
        htmlFooterRentalPointReceipt)

The code structure remains consistent for both templates. We've successfully achieved the Open-Closed Principle – our code is open for extension (to add new templating formats) but closed for modification, which ensures robustness.

A key takeaway is the importance of an emerging design. My initial version was different from the current one, and I had to refactor my code before implementing the new HTML receipt feature. Continuous refactoring is essential for maintaining a resilient architecture.

Functional programming building blocks

Marco Sabatini — Sun, 16 Aug 2020 14:18:10 +0000

References

Overview

Functional programming is fascinating, but how can you start practicing it? If you're a developer who has been pondering this question, I'd like to share my personal journey and experiences.

I firmly believe in the "learning by doing" approach. That's why I decided to use a coding exercise known as a kata to practice functional programming. Specifically, I chose the Martin Fowler kata for the video store, not the refactoring version but the one starting from scratch. This approach provided me with a clean slate to work with, enabling me to focus on the application's domain rather than getting bogged down in technical details. I opted for TypeScript to leverage the functional capabilities offered by the language.

The chosen kata is rather straightforward. My aim was to concentrate on the application's domain logic rather than dealing with technical intricacies like database persistence or external HTTP service integration. The primary objective of this kata was to create a system capable of renting different types of movies and generating receipts in various formats, such as plain text and HTML.

Test first

My journey began by writing a test suite to calculate the price for specific movie types:

it('rent new Release Movie for one day', () => {
        expect(moviePriceFor(new Rental(1, newReleaseConfiguration("UNUSED")))).toEqual(3.0)
});
it('rent Children Movie for four day', () => {
        expect(moviePriceFor(new Rental(4, childrenConfiguration("UNUSED")))).toEqual(3.0)
});

While writing these tests, I discovered key concepts like rental, movie types, additional price calculations for extra days, and individual movie price calculations.

I implemented the following production code to pass these tests:

const additionalCostFor = (rental: Rental): MoviePrices => {
  let additionalCost = 0.0;
  if (rental.rentalDays > rental.mc.minRentDays) {
    const additionalDays = rental.rentalDays - rental.mc.minRentDays
    additionalCost = rental.mc.additionaCostPerDay * additionalDays;
  }
  return new MoviePrices(additionalCost, rental.mc.price);
}

const priceFor = (moviePrices: MoviePrices): number => {
    return (moviePrices.movieBasePrice + moviePrices.additionalCost).toPrecision(5) 
};

The first function calculates the additional cost, while the second function adds the base price and rounds it to five decimal places.

At this point, I realized that I had the essential building blocks needed to compose a function that calculates the total price for a single movie type.

Let's go and apply composition!

Composition

Next, I decided to implement a compose function, but of course, I wrote a test first:

it('compose two function', () => {

  let f = (x: string): string => `f(${x})`
  let g = (x: string): string => `g(${x})`

  let gfx: (x: string) => string = compose(f, g)

  expect(gfx("value")).toEqual("g(f(value))")
});

In this test, I defined two functions, 'f' and 'g', which take an input parameter and return a string with that parameter interpolated. By composing them, I achieved string concatenation.

The production code for the compose function looks like this:

export const compose = <A,B,C>(
  f: (x: A) => B,
  g: (y: B) => C):
  (x: A) => C => {

    return (x) => g(f(x))
};

Using TypeScript generics, I created a versatile compose function that can be used for any pair of functions where the output type of one matches the input type of the other.

With this compose function in place, I was able to compose the additionalCostFor and priceFor functions like so:

const additionalCostFor = (rental: Rental): MoviePrices => {...}

const priceFor = (moviePrices: MoviePrices): number => {...}

const moviePriceFor: (x: Rental) => number = compose(additionalCostFor, priceFor)

Thanks to the type system, I didn't even need to write a test for this specific composition because it naturally emerged, and the compiler confirmed that the functions could be composed successfully.

Try to compose again!

Leveraging Curry

By creating these basic building blocks, I could easily compose them to create more complex functions. This approach encourages clear and isolated responsibilities, leading to excellent cohesion and loose coupling.

For the total price calculation, I reused the calculation for individual movies by currying the function and applying it using map and reduce:

const additionalCostFor = (rental: Rental): MoviePrices => {...}

const priceFor = (moviePrices: MoviePrices): number => {...}

const moviePriceFor: (x: Rental) => number = compose(additionalCostFor, priceFor)

export const totalPrice = (moviePriceFor:(r:Rental) => number):(rentals:Rental[])=> number =>{
  return (rentals) => rentals.map(r=>moviePriceFor(r)).reduce((x,y)=>x+y);
}

Currying allowed me to partially apply the function and return a configured function, making composition even more powerful.

Modularization of Software

To maintain clean and modular code, I exported the total price calculation function from the pricing module. This function was used by modules responsible for printing receipts in HTML and plain text formats.

By doing so, I defined a clear public interface between the modules. I also had the flexibility to mock this function to facilitate testing of the printing modules (HTML and plain text).

Final Thoughts

In functional programming, functions are the fundamental building blocks. Each function can be thought of as a Lego brick, and pure functions are inherently isolated. Unlike encapsulation, where an object tries to hide information, pure functions can only do what they declare in their interface or signature, making them "honest."

This paradigm shift encourages problem-solving by breaking them down into small, isolated functions and then reassembling them at the application's entry point. While this approach may seem counterintuitive initially, it fundamentally changes how you think about building software.

If you'd like to explore this topic further, check out the second round of my journey.

Consumer-Driven Contract Testing with Pact and Java - Part II

mos_ — Wed, 17 Jun 2020 12:00:00 +0000

The previous post explains the principles and motivations behind contract testing. Today we take a look to how write consumer-driven contract tests with pact and Java in a SpringBoot application.

Pact foundation provides junit5 integration for creation and verification of contracts.

Let’s start!

What you need

The Java Pact Testing framework used in this example is the 4.0.0, based on v3 specification.

Furthermore, you should have:

jdk 1.8 or later
maven 3.2+
a bit of testing knowledge in Spring

You can find all the presented code on github.

The example

The proposed example is similar to the previous one seen in part I: we have a provider service which expose anAPI that given a sentence and a timestamp, it replies an echo response enriched with local timestamp. This API _is consumed_by another service. To summarize:

endpoint POST /api/echo

request body

{
    "timestamp": 1593373353,
    "sentence": "hello!"
}

response body

{
    "phrase": "hello! sent at: 1593373353 worked at: 1593373360"
}

Consumer side

We are driven by Consumer, so then we start working on consumer side: we add the pact maven dependency in consumer pom.xml

<dependency>
    <groupId>au.com.dius</groupId>
    <artifactId>pact-jvm-consumer-junit5</artifactId>
    <version>4.0.10</version>
</dependency>

create a contract

Let’s start creating a junit5 test with PactConsumerTestExt junit extension:


import au.com.dius.pact.consumer.junit5.PactConsumerTestExt;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(PactConsumerTestExt.class)
class ConsumerContractTest {

in @BeforeEach method we can assert that the mockServer which will serve the contracts is correctly up:

    @BeforeEach
    public void setUp(MockServer mockServer) {
        assertThat(mockServer, is(notNullValue()));
    }

ok, now we can create a contract. A contract can be defined with a method annotated with @Pact that returns a RequestResponsePact and provides as parameter PactDslWithProvider. All methods annotated with @Pact are used toinstrument the mock server through the PactDslWithProvider in this way:

    @Pact(provider = "providerMicroservice", consumer = "consumerMicroservice")
    public RequestResponsePact echoRequest(PactDslWithProvider builder) {

        return builder
                .given("a sentence worked at 1593373360")
                .uponReceiving("an echo request at 1593373353")
                .path(API_ECHO) /* request */
                .method("POST")
                .body(echoRequest) 
                .willRespondWith() /* response */
                .status(200)
                .headers(headers)
                .body(echoResponse)
                .toPact();
    }

The Pact DSL provides a fluent API very similar to Spring mockMvc: Here we are saying that when the mock server receives an echoRequest, it should return 200 and an echoResponse. The given and the uponReceiving method, define the specification in bdd approach and the Pact testing framework, uses the given part to bring the provider into the correct state before executing the interaction defined in the contract.

Matchers: build a response with PactDslJsonBody

in the previous step we created an interaction using two json object(echoRequest and echoResponse). On the provider side, the test verify that the generated response is perfectly equal to the one defined in the contract.

The Pact testing framework provides also a DSL that permits the definition of different matching case in this way:

    @Pact(provider = "providerMicroservice", consumer = "consumerMicroservice")
    public RequestResponsePact echoRequestWithDsl(PactDslWithProvider builder) {

        PactDslJsonBody responseWrittenWithDsl = new PactDslJsonBody()
                .stringType("phrase", "hello! sent at: X worked at: Y") /* match on type */
                .close()
                .asBody();

        return builder
                .given("WITH DSL: a sentence worked at 1593373360")
                .uponReceiving("an echo request at 1593373353")
                .path(API_ECHO)
                .method("POST")
                .body(echoRequest)
                .willRespondWith()
                .status(200)
                .headers(headers)
                .body(responseWrittenWithDsl)
                .toPact();
    }

Here we created a response with PactDslJsonBody DSL that defines a match case based on type instead of value. It’s possible with PactDslJsonBody different match case based on regex or array length.

Verify the contract

Now we can create the real test which verify the contract on the consumer side:

    @Test
    @PactTestFor(pactMethod = "echoRequest")
    @DisplayName("given a sentence with a timestamp, when calling producer microservice, than I receive back an echo sentence with a timestamp")
    void givenASentenceWithATimestampWhenCallingProducerThanReturnAnEchoWithATimestamp(MockServer mockServer) throws IOException {

        BasicHttpEntity bodyRequest = new BasicHttpEntity();
        bodyRequest.setContent(IOUtils.toInputStream(echoRequest, Charset.defaultCharset()));

        expectedResult = new EchoResponse();
        expectedResult.setPhrase("hello! sent at: 1593373353 worked at: 1593373360");

        HttpResponse httpResponse = Request.Post(mockServer.getUrl() + API_ECHO)
                .body(bodyRequest)
                .execute()
                .returnResponse();

        ObjectMapper objectMapper = new ObjectMapper();
        EchoResponse actualResult = objectMapper.readValue(httpResponse.getEntity().getContent(), EchoResponse.class);

        assertEquals(expectedResult, actualResult);
    }

if we run mvn test and we don’t have errors, we will see in ./target/pacts a json file that use the pact formalism for contracts. We use the generated contract in the provider-side.

Provider side

For the provider, we have a different dependency to add in pom.xml:

        <dependency>
            <groupId>au.com.dius</groupId>
            <artifactId>pact-jvm-provider-junit5</artifactId>
            <version>4.0.10</version>
        </dependency>

Verify the contract on provider side

Here is the thing: we need to verify the contract against provider implementation. In the Spring world, it’s sounds like an integration test which verify the web layer. So here the magic:

@ExtendWith(SpringExtension.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT, classes = ProviderApplication.class)
@EnableAutoConfiguration
@AutoConfigureMockMvc
@TestPropertySource(locations = "classpath:application-contract-test.properties")
@Provider("providerMicroservice")
@PactFolder("../consumer/target/pacts")
public class ProviderContractTest {

    @Value("${server.host}")
    private String serverHost;
    @Value("${server.port}")
    private int serverPort;

    @BeforeEach
    void setupTestTarget(PactVerificationContext context) {
        context.setTarget(new HttpTestTarget(serverHost, serverPort, "/"));
    }

    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    @State("a sentence worked at 1593373360")
    public void sentenceWorkedAt1593373360() {
        when(phraseService.echo(1593373353, "hello!"))
                .thenReturn(new Phrase("hello! sent at: 1593373353 worked at: 1593373360"));
    }

}

That’s it. As you can see, we have a @SpringBootTest with a fixed port and a @TestPropertySource that defines it in order to attach the pact context to the application context with host and port info.Obviously there are other ways, like random ports and so on, but the main thing here is to bind both context together.

Another thing here is the @PactFolder annotation that points to contracts generated by the consumer. The Pact Framework search for contracts that belong to the service, and run the verification.

The `@State` annotation

    @State("a sentence worked at 1593373360")
    public void sentenceWorkedAt1593373360() {
        when(phraseService.echo(1593373353, "hello!"))
                .thenReturn(new Phrase("hello! sent at: 1593373353 worked at: 1593373360"));
    }

As previously mentioned, the given statement in the consumer contract, define with a business expression, the state in which the system-under-test, should be during the execution. Following this approach, we define in the provider, a method with @state annotation, that contains the commands necessary for the correct execution. In our case, we mock the business service delegated to execute the eco logic. The framework executes the state method before calling the API defined in the contracts. The real test, in this way, is “reduced” to a simple call:


    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

Use a broker

If you have a broker that stores the contracts, you can change the @PactFolder annotation with @PactBroker one and define the following plugin in the pom.xml:

    <build>
        <plugins>
            <plugin>
                <groupId>au.com.dius</groupId>
                <artifactId>pact-jvm-provider-maven</artifactId>
                <version>4.0.0</version>
                <configuration>
                    <pactDirectory>target/pacts</pactDirectory>
                    <pactBrokerUrl>${pact.broker.protocol}://${pact.broker.host}</pactBrokerUrl>
                    <projectVersion>${contracts.version}</projectVersion>
                    <trimSnapshot>true</trimSnapshot>
                </configuration>
            </plugin>
        </plugins>
    </build>

What’s Next

We have covered how to develop and verify a simple contract with java starting from the consumer. We have used the Pact DSL and matchers introduced in v3 spec which is an interesting feature during design & testing. As previously mentioned, you can find a complete working example in this github repo.

In the next posts(I hope), we will see how deploy a broker server to store contracts and how integrate the entire flow witha CI.

The original post can be found here

Consumer-Driven Contract Testing - Part I

mos_ — Wed, 03 Jun 2020 12:00:00 +0000

Introduction

This is the first of a series of blog posts about Contract Testing which cover the minimum set of theory and practice necessary for an effective adoption in your team, from design to code integration.

Contract Testing is a category of testing activity where the data formats and conventions defined by two systems (services) which communicate a business value, is tested against a Mock called “Contract”. A service provides a callable API that can be consumed by another (or many) service which create an interaction between parties that needs to be satisfied during the evolution and developing of services which are now coupled. The interaction between services probably relies on a communication layer which can be slow or not reachable which can affect the test results. For this reason, sometimes the best solution is to verify the interaction with a TestDouble which describe the expectations between the parties.

For the sake of clarity, I have strong opinion.

Context

Let’s suppose that we have a Product Service which has an HTTP method that provides product information in a JSON format upon receiving a product id:

HTTP GET: api/v1/product?id=123

{
  "name": "nexus",
  "type": "smartphone",
  "price": "21.03"
}

The product information are consumed by Customer Service.

Conventions are defined in this way:

Customer Service is the Consumer
Product Service is the Provider

During developing, the Product service can evolve the API in a way that the HTTP response change the data format. For example, the type field from a string value change to object:

{
  "name": "nexus",
  "type": {
    "category": "smartphone",
    "weight": 0.2,
    "color": "blue"
  },
  "price": "21.03"
}

this is defined as a breaking change in which Provider doesn’t respect the defined interaction between Consumer.

In the same way, the Customer service can evolve the request call. For example, instead of call the API with a query parameter id, it can use a field in the header to query the Product Service. Also, this example is a sort of breaking change in which Consumer doesn’t respect the defined interaction between Provider.

Therefore, the interaction between services can be broken in many ways, and the cause can be triggered by both sides which makes necessary some checks to avoid regression during the development.

e2e Testing

The simplest thing to do, is to create an end-to-end test that cover the entire calls flow between Consumer Service and Provider Service.

Despite what I previously said about simplicity, e2e testing give the best confidence on software behaviour but with some problems.They are flaky tests and often fail with false positive furthermore e2e tests are hard do maintain, slow, and it’s necessary to span multiple services in testing environment with terrible slowness caused by deploy flow and environment nightmares. There are some solutions (1, 2) that can provide the right balance between simplicity and maintainability which are a step back e2e tests but often you need a staging environment to execute tests verification or add a level of complexity to remove the environment need, which personally I found with these solutions a tedious process. Moreover, all these solutions are reactive approaches which in the simplest case, doesn’t avoid the integration of code that breaks the services interactions, but they provide a sort of alerting like open a Jira defect or github issue. It’s possible to avoid code integration in case of breaks but imply an overcomplicated CI or some experimental branch.

Mocks

~~Developers~~ (I) usually dislike maintaining things which depend on environments or that needs hours to have a result. If something is faulty and slow, it’s often untrusted and consequently useless. Furthermore, works with other services means deal with other teams which are often too much busy doing new ~~bugs~~ features. So then, to avoid this annoying stuff, it’s necessary to adopt a strategy which imply a level of complexity which usually developers are used to see: Mocks.

Mocks are a type of TestDouble that define a sort of specification based on expectations and, in this particular case, mocks can substitute APIs or clients reducing in this way, parties to set up and run.

Even if the image show deployed services, there are many solutions to load part of an application which doesn’t imply a real execution. however, you can simulate an interaction through a mock or stub based on Contracts. As showed before, we can have two categories of breaking change based on the side that doesn’t maintain expectations and in the same way, we can identify two categories of contracts:

Provider Contracts
Consumer Contracts

Contracts characteristics

A Provider service exposes a set of business functionalities which can be used or not by one or many consumers and ingested in different ways. A change in the provider interface can break interactions with many consumers, but it isn't the same for the consumer. In the same way, a Provider contract cover completely the functionalities exposed by the service with only one definition. On the contrary, a Consumer contract can cover only a set of functionalities that are interesting for the service.

We can say that a provider contract is a closed and complete expression of business functionalities. Instead, consumer contract is an open and incomplete definition of business expectations. When a provider accepts the expectations defined by the consumer contract, confirms that the expectation is a functionality that supported for a period of time.

Consumer-driven Contracts

Ok, so now? Well, If you take a look on internet, ~~you will not find~~ any trace of provider driven contract testing ~~or question about it~~.On the contrary, the web is full of posts about Consumer-driven contract testing.This should be enough for you and me to choose a consumer-driven solution but, as I said at the beginning, the idea of this post is to give the minimum set of knowledge about Contract testing, and to have an answer in case of your colleagues are free mind enough to ask if exists an alternative meanwhile you are exposing why Contract tests are so cool.

In short terms, it’s all about business. Consumer contracts point the finger on current supported business value exposed by the provider. In a consumer-driven approach, the sum of all contracts generated by the consumers and asserted by the provider is closed and complete respect the functionalities requested.

Anyway Consumer-driven Contracts drive the evolution of the services keeping the focus on what really matter with a feedback loop on the changes that will arrive during the life and death of the services which cannot be achievable with a Provider-driven approach which not take in account the feedback from the consumer.

In my experience, it’s easier and natural to think in a Provider-First manner than Consumer-first, and the reason behind this mindset, it might be connected to the fact that a breaking change made by the Provider is more detectable than the one made by the Consumer. In a real world example where your team isn’t the owner of both services, you have to deal with meetings, longer meetings and extravagant, informal, hermetic, long-winded design docs and other mythical beasts. A contract is a formal expression of needs and duties which can be another tool in your pocket that can be used in order to reduce the background noise during the process of evolution of services. Probably you know the sensation of powerlessness when a design doc or swagger definition reaches the mailbox or a shared folder and it’s necessary another meeting or mail tread to have a change in the definition or worse, get an HTTP response status error because someone made a little change in the API which isn’t versioned.

To summarize, Consumer-Driven contracts can give you a process that presents an iterative way of proceeding with a formalised format which can help large organization with services which are owned by different teams that can be in different locations.

Contract Testing with Pact

Pact.io is an implementation of Consumer-driven contract testing which actually support many languages. At the moment of writing this post, pact foundation released v3 specification which cover the following features:

pact format for message queues
regular expression and type matching
specification shared between Ruby, JVM and .Net versions

Furthermore, it’s available a broker which can be used to publish and share contracts between services.

What’s Next

We have covered the minimum set of principles and motivations that are necessary in my opinion to work with contract testing. In the next posts we will take a look to how:

create consumer-driven contracts tests with Pact and Java
setup broker server
integration with Continuous Integration

The original article can be found here

References

Consumer-Driven Contracts: A Service Evolution Pattern

ContractTest

TestDouble

The curious case for the Provider Driven Contract

The Practical Test Pyramid

Patterns for Managing Source COde Branches

Mocks Aren’t Stubs

7 Reasons to Choose Consumer-Driven Contract Tests Over End-to-End Tests

DEV Community: Ticino Software Craft

Event storming, and then what?

References

Overview

Mars Rover Kata

Event Storming

Implementation

Command Handlers

Domain

Services

Path Projection

Policy

Additional requirement

Evolutionary Architecture

The hidden trap of debugger

How it started: a false illusion

How is going

Where is the trap ?

A sample case: microservice with many data transformations

Issue is found now what?

Conclusion

Test Containers and Clean Architecture

References

Overview

Kata

Docker && Docker Compose

Setting CI pipeline

Considerations

Clean Architecture and Either Monad

References

Kata

Start

Kata

Testing

Do I need to test infrastructure?

End

P.S.

Functional programming building blocks 2nd round

References

OCP principle

Functional programming building blocks

References

Overview

Test first

Composition

Leveraging Curry

Modularization of Software

Final Thoughts

Next

Consumer-Driven Contract Testing with Pact and Java - Part II

What you need

The example

Consumer side

create a contract

Matchers: build a response with PactDslJsonBody

Verify the contract

Provider side

Verify the contract on provider side

The @State annotation

Use a broker

What’s Next

Consumer-Driven Contract Testing - Part I

Introduction

Context

e2e Testing

Mocks

Contracts characteristics

Consumer-driven Contracts

Contract Testing with Pact

What’s Next

References

The `@State` annotation