DEV Community: Angelo Hulshout 🇮🇹 🇳🇱

The Closed Loop

Angelo Hulshout 🇮🇹 🇳🇱 — Mon, 28 Dec 2020 23:12:46 +0000

In software engineering, and specifically in software architecture, we spend a lot of time talking about patterns, architectural styles and other rules and guidelines. Over the past 25 years, and before, a lot of books and articles have been written about those, covering all aspects of software engineering. All of this is very useful, but it can also be overwhelming and sometimes appear quite complex. That's why every once in a while, I enjoy going back to very basic principles. Basic, as in simple, effective, and often so obvious that they often get overlooked.
Recently, during discussions of the sofware architecture for a new project, I came across one of these principles again. One that is so obvious, and yet so often overlooked in software design: the closed loop.

The closed loop is a principle that is very common in in electrical engineering, and specifically in control systems. In such systems an actuator, e.g. a motor, is given an input signal. That is supposed to trigger a certain action that results in an output signal. Part of that output signal is fed back to the component that generated the input signal, to ensure the system is behaving as expected.
Depending on the type of actuator, it may be needed to add a sensor to detect the action and provide the output and thus also the feedback signal, but the principle stays the same. In the case of the motor that starts moving on an input signal, the feedback could for example come from a motion sensor that detects movement on whatever the motor is supposed to drive.
In software, and cerainly in larger systems, I see the results of the lack of such closed loops. All to often we trigger an action, set something in motion and never check the result. This leads to unexpected behaviour - not just in control applications, but also in web applications, or even (the tax agency will love this) in book keeping software. This, while all it takes is following one simple rule in design and implementaition:
Close the loop: trigger an action, and verify whether it is executed correctly.

Some very simple examples from different application domains:
Signal the start of a motor: verify it is moving by checking the output of a motion sensor or cycle counter
Trigger a database transaction: check the result of the commit (Yes, I know it's obvious, so why do so many applications skip this check?!)
Stop a process: verify it is no longer running after a specified timeout (even Microsoft Teams forgets to do this - just try switching between accounts a few time)
Send a message on a message queue: check the result of the queueing function, and design your protocol to return an ACKnowledge message to be returned

It's basic, it's simple and feels like an open door, but as said, I keep running into violations of this very basic principle. The excuse is often: yes, but we tested every possible situation. That's wrong - no manual test, nor any automated (unit) test framework will ever cover every situation until the day the software becomes end of life. Don't fool yourself!
And now I'm curious to who shares this experience. Would you care to have a look at your own last project and let me know in the comments if you are convinced that you closed every loop?

Domain Driven Design - method and process agnostic

Angelo Hulshout 🇮🇹 🇳🇱 — Thu, 14 May 2020 12:59:14 +0000

This is the second article in a series that covers the contents of a training I developed (and taught) for over 7 years now. The training covers software design, using the Domain Driven Design approach as it’s basis.

On processes, methods and methodologies

As every experienced software professional knows, every organisation, be it a multi national, a startup, a small business, or even an open source project, has its own way of working. Often you find a mix of methodologies and best practises that has developed over time into something specific for the organisation.

What those exact mixes are is beyond the scope of this article, but let's say that in most organisations you find a bits of waterfall, agile or iterative processes, mixed with functional or object oriented design, some form of requirements engineering and bits of various testing methodologies. The exact combination is beyond this article (and different in every case) but I have to address it. It was one of the reasons I first got in touch with Domain Driven Design, DDD for short. When I was asked by a customer if I could set up a training for software designers, I was confronted with the question which design method to follow. All candidates for the training were working on projects for different companies, on secondment basis. The implicaiton of that was that, although my customer would love ot promote a specific design approach, the course participants would be stuck to the way of working of the project and organisation they were working for.

In disussions with a fellow trainer working at my customer, we concluded that the work a software designer needs to do, and the deliverables coming out of it, should not be dictated by this company specific approach. Instead, they should always be more or less the same, even if not all steps would be executed always in the same order - i.e. in a waterfall or in iterations. the work and deliverables in the end should be roughly the same. That's why, after a short research into existing approaches to software development, we settled on Domain Driven Design. DDD covers every element that we consider important for making a software design, in terms of activities, deliverables and patterns, and it can be fit in with any development process, no matter how modern or archaic.

What these are? It's a short list:

Problem analysis - identifying stakeholders, system context and requirements
Creating models - of the problem and solution space
Creating the design itself
Combining specific problems with generic solutions - technology choices, design patterns, re-use etc.

The web site DDDCommunity.org summarized it in a way that made it easy for us to select DDD - it literally said at that time (in 2012):

__"The premise of Domain Driven Design is twofold:

for most software projects, the primary focus should be on the domain and domain logic, and
- complex domain designs should be based on a model

Domain Driven Design is not a technology nor a methodology, it is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains."__

So what are these domains DDD talks about?

Domain is a word with two meanings when talking about software development using DDD, as far as I'm concerned. It can either mean problem domain or solution domain. If you do things the right way, the two almost become one in the end, as we'll see later.

Problem domain: the organisation, process, environment in which the software is going to be used
Solution domain: the organisation, process, environment in wich the software is going to be created

When we start a project, the funny thing here is that, as software engineers, we know a lot about the solution domain, but often far less about the problem domain.
Let me use myself as an example. in over 25 years in software engineering I learned over 15 programmming languages, countless frameworks and design patterns, and many different technologies relatd to software development. At the same time, when I started working on a project (that is now about to end) 3.5 years ago for setting up the software to control a petfood factory, I knew close to nothing about petfood manufacturing.

That changed a lot over the past 3.5 years, thanks in part to DDD. We used ideas from DDD (remember it's a way of thinking and not a methodoloy?) to clarify terminology, to set up functional descriptions, do design work, and now I not only know how the software works, but I can also explain how we get from raw chicken, cranberries, bone meal and barley to your dogs favourite food.

So what is so magical about DDD here? Well, plain and simple it comes down to is this. At the core of Domain Driven Design is a model - the domain model. It is a model of the problem domain, expressed in whatever notation you want. I use a derivative of UML often, but text or blocks-and-lines are equally possible. This model, and that's where the dark magic happens, is defined in such a way that eventually, you can implement it in code. So that means, your problem domain model becomes part of the software solution.

At the same time, working on the model, which in DDD is something you should do together with your customer as much as possible, builds up a common understanding of the problem, and later on the solution, with anyone involved. That means, you understand the problem customer and user are dealing with, and they understand your solution. Not that they can write software, but it is clear to them what you are creating and why - even in the externally invisible parts.

And to top it of, the terminology used in the model becomse a shared language between customers and engineer. When a customer or a user wants a change, he can express that in terms of the model, and similarly an engineer can explain a change being made using the same. This is what DDD refers to as the Ubiquitous Language.

Example of what it looks like

The diagram below shows something from a first talk I had with the customer of the petfood factory 3.5 years ago. The terminology changed a bit in later talks, but this represents what a customer can tell you.

At a factory, new material comes in Trucks. Each truck has a load of material, which it has picked up at a certain origin. A lot is referred to as a Lot. Lots and origins are identifiable, because in case of problems (e.g. material contamination) all materials and products in which they are used need to be traced and dealt with. There are a lot of traceability related requirements that are serviced by making this things identifiable. Upon delivery, the truck dumps its load into an intake pit, and from there it is transferred to silos.

This diagram is an object diagram, which contains very limited information. To add more information, what I usually do, is transform it into a class diagram after constructing it together with the customer, and keep only the parts that will be relevant in the software later on. This is what is shown in the class diagram below. In this case, the truck is not relevant in the software, it is an external entity passing by, while it's load is referred to as a lot and there is no need to keep both. Similarly, the intake pit is just a physical location where material passes by, that we are not really interested in.
Purists may say that you have to keep these things anyway, but for me this is a matter of being practical.

DDD itself in a domain model

As an exercise in my trainings, I often ask trainees to look at Domain Driven Design itself, and create a basic domain model for it. This often results in something that resembles the diagram below.

I have no software based on this one, but one day I may put it into a wizard that guides newbies through the steps. Who knows...

And how do you fit DDD into any process or method?

That of course is the main question - how does this fit in with any process or method that a company uses? The clue is in the four elements of the design activity we mentioned earlier:

Problem analysis
Creating models
Creating the design itself
Combining specific problems with generic solutions

As depicted in the diagram below, we can do DDD in an iterative way. The Problem Analysis is basically performed in interviews with stakeholder experts (customer, end user, representative), and captured in the domain model. The domain model is then extended with items from the solution domain into a design model, which is the basis for implementation. Finally the implementation gets tested, evaluated and used as input for further stakeholder talks.

Now, only if someone would implement a pure top to bottom waterfall approach, this would be impossible to fit into their way of working. In an interative (Agile optional) approach this will fit in seamlessly, in a more waterfall like appraoch, there still is a feedback cycle somewhere that can be used to do this - and if not on project scale it can still be done on team scale or even between two or three developers and a stakeholde representative in some form.

That basically explains how DDD ended up being the basis of my design training, but also of my own work since 2012. More in follow up articles.

Register for my newsletter here to be kept up to date with my articles and trainings, and interesting references to what else is going on in the field of software engineering.

References

Domain Driven Design - Eric Evans, ISBN 0-321-12521-5 Addision Wesley 2003
Implementing Domain Driven Design - Vauhgn Vernon, ISBN 978-0321834577 Addison Wesley 2013
DDD Community web site: https://dddcommunity.org

What would be your training needs?

Angelo Hulshout 🇮🇹 🇳🇱 — Mon, 04 May 2020 20:27:04 +0000

So, I've been running in company trainings for years, and recently decided that open subscription trainings and online trainings may be at least as useful and maybe also more fun to do. I started by offering my Domain Driven Design training online.

That's an easy one, because I have all the material of 7+ years of running it. However, I am curious to see what your training needs would be - given that dev.to covers a lot more topics on technology (libraries, frameworks, languages) than on design approaches and the process side of software development. So, feel free to comment and let me know - you may spark a new training that I can offer, to help some of you forward.

Other ideas are also welcome, I've been considering the option of short pre-recorded mini-courses and online Q&A sessions as well.

Fire away, I'm listening!

Software design — is it different from architecture and why do we need it?

Angelo Hulshout 🇮🇹 🇳🇱 — Tue, 28 Apr 2020 08:46:04 +0000

This is the first article in a series that covers the contents of a training I developed (and taught) for over 7 years now. The training covers software design, using the Domain Driven Design approach as it’s basis. Why I chose Domain Driven Design is explained in a seperate article.

At the start of this first article of the series, I have to thank Ewald de Bruijn and Peter Schipperen who have provided a lot of help in developing and teeaching this material over the past 7+ years.

An example

So, what are we talking about when we talk about software design? Let’s start by looking at an example of a software development problem.
Quite some years ago, I had a colleague who was working on software for a weighbridge system for a transport company. This is a relatively simple system, where a truck drives onto a weighbridge, it’s weight is determined, and a ticket with the weight is printed and handed to the driver.

The main actors on this system are the operator, who is responsible for weighing the truck, and the truck driver, who is driving the truck on and off the weigbridge, and gets handed the ticket before driving off.
The whole flow related to the picture above is described as follows:

Trucks drive onto the weighing bridge. The operator checks the weight and prints a ticket with the weight. After receiving the ticket, the driver drives the truck of the weighing bridge so that the next truck can come in.

The software for this system consisted of 4 main packages

User interface
Printer Queue
BridgeInterface
Printer Driver

The customer of this simple software application asked to extend the text driven user interface with a simple menu to replace hand type commands, and a reporting function to get an overview of the weighings of the past 4 hours.
This turned out to be a small problem, because the controller board had only 2kb of memory left and we needed 20kb for the software change. Yes kb, this was the 1990s, when a 1 megabyte RAM module still cost around 200 euros.

Looking at the software, which is abstracted in the diagram below, my colleague wondered what could done to reduce the amount of memory used by the existing application. It took a little bit of thinking and discussion, but in the end the solution was very simple.

Looking at the usage scenario, it’s pretty obvious that the printer queue has no purpose here. At any point in time there is only one print job in progress, since no truck can enter the weigh bridge before the one on there receives its ticket and drives off. So removing the queue and the related software easily solved the memory problem.
This is an example of how easy it is for a designer to thinking in solutions, which isn’t a bad thing in itself, but at the same time loose track of what is really needed. It pays off to keep in mind who you develop the software for, your stakeholders, and what problem they try to solve.

This is one of many pitfalls in software design, all of which are avoidable as long as we are aware of them and apply the best practises for avoiding them.

Design

To clarify that a bit further, let’s have a look at what software design actually is. There are many definitions, but the one that works best for me is the following:

“Software design is the process of defining software structure and the related behaviour that satisfy both functional and non-functional requirements of our stakeholders”

That definition covers, for me, the most important aspect of software design.

First of all, design is a process, more than (what is commonly assumed) the outcome of a process — ‘the design’. ‘The design’ is never finished, that’s what is most important to realise here: during creation of the software, the design will change due to new requirements, new insights.

After creation, there will again be requirements that lead to more changes. As such, the process is more important than the result. In fact, there is nothing that can be called the results, all so called resulting designs are intermediate designs at best. I put that a bit black and white, but in essence it’s true.
If we do want to talk about the result — and of course we do — I talk about the design model, which is an abstraction of the software solution expressed in tables, text or (preferably) diagrams that help software developers to create software that covers the issues addressed during design.

Second, design should cover both structure and behaviour of the solution. A block diagram, class diagram, package diagram by itself is NOT the design model of the software. It has to be clear how the software behaves, if only because the days of simple, sequential mainframe or BASIC programs has long past. Most modern software systems handle data streams and events, and deal with things happening in specific order, or in parallel. If that behaviour is not matched to the structure, there is no design model that will help us implement the actual software.

Third, the functional requirements provide little need for explanation. Of course, every software solution has to provide the functionality required by it’s users. However, we have to make sure that we cover the required functions, but also the necessary functionality to deal with errors of any kind that may realistically occur.

That brings us to the fourth and final element. We have to deal with non-functional requirements, or quality attributes. These cover what some people call the ‘-ilities’ even though not all of them have a name that ends in ‘-ility’: functionality, reliability, usability, robustness, fault tolerance and so on. The notion of dealing with errors as described above shows that functionality and quality attributes (fault tolerance in this case) go hand in hand, and practise shows that this goes for any combination of quality attributes.
In the end, in our design process and the resulting design model, we will cover choices that allow the resulting solution to balance all quality attributes that are relevant for the system at hand.

Architecture

So, now that we know what software design is, how does it differ from software architecture?

Well, let’s start with the definition of architecture as given by the IEEE 42010 recommended practise for architecture description:

“Architecture is the fundamental organisation of a system embodied in its components, their relationship to each other and to the environment, and the principles guiding its design and evolution.”

In essence, despite the heavier wording, this is not so different from design. The main difference is that in larger systems, an architecture defines the overall structure of a software system, including guidelines for more detailed design work within that structure.
This is mainly put in place to translate higher level requirements, sometimes (almost) on the meta level that the designs of all parts of the system must adhere to.

As an example:

A large company has a software system that has been developed over a period of more than 20 years. It covers millions of lines of code, and in order to keep things manageable, their lead architects have set up a basic structure and a set of guidelines that should be followed by all designers when extending or modifying the system:

The software is divided into layers, where higher layers are further removed from the underlying hardware than lower layers — say the highest layer is the user interface, while the lowest is at the device driver level
Within each layer, they defined building blocks, because the layers are cross cutting across serveral functional areas (which are orthogonal to the layers)
Within the building blocks, which may be quite large, they defined the concept of components, which are small, dedicated software pieces
Layers, building blocks and components have well defined, managed interfaces
Each layer, building block and component has an owner, typically an architect or a senior software engineer
Modifications to the interfaces cannot be made without consent of the owner, and this is actively managed through authorisations on the version control system

Of course, this is a very simple example, in modern architectures there are much more complex rules and guidelines than this. Just think of the security related guidelines that (should) apply to systems that manage our personal data and/or are connected to the internet. It’s those guidelines that are prescribed by the architecture, and that are to be followed in the designs that detail the parts of the architecture.

On a closing note, I’d like to stress that architecture and design are both needed to develop proper software that stands the test of time and quality. In the past 20 years, I’ve seen a lot of people trying to without either one of the two, and the results are not exactly positive. Doing architecture, and doing design does not necessarily mean you have to do a lot of paper work. It’s mainly about setting up structural and behavioural rules for your solution, a framework that matches your needs and those of your customer. In follow up articles, I’ll elaborate on that — combining 25 years of working on software and systems architectures with the ideas of Domain Driven Design.