Software Development as an Emergent System

Emergent systems are sometimes described as systems where the whole is greater than the sum of the parts.

I personally prefer to think of an emergent system as something that is complex to reason about as a whole but simple(r) to understand when you look at its individual components.

Probably one of the most given examples of an emergent system in nature is that of a flock of birds flying in perfect coordination.

If you try to model the birds' movement as a whole it becomes difficult and convoluted. The movement they make seems almost unpredictable.

But if instead you model this system by modeling one bird, then it becomes simple to understand.

This has been done exactly for this problem (modeling the behavior of a flock of birds) and the solution is called Boids.

In Boids, the complex behavior of a flock of birds is achieved by having each bird follow three simple rules.

The first one is separation, which states that each bird will try to keep a reasonable distance from other birds. The second rule is alignment which means that a bird will try to align itself with the birds in its vicinity. And finally, cohesion which means that each bird will try to move to the average position of other nearby birds.

It's impressive how lifelike a simulation looks using just these three simple rules. You can watch them in action in this video that does a very good job of explaining each rule in isolation.

But what does this have to do with software development?

Software is developed by individuals and any software system of reasonable size becomes hard to reason about as a whole.

Unfortunately it isn't possible to come up with simple rules that allow us to predict how a particular piece of software will evolve. However looking at software development from this perspective surfaces some aspects that might help determine if a software product will be a quality product, or if it will derail into something that no developer wants to work on.

One of the things that comes to mind are "initial conditions". In the Boids example this would be the birds' initial position in the simulation. Different starting positions might lead to one big flock or to multiple individual flocks.

In software there are several things that can be thought of as initial conditions, for example the type of technology used or even the project's initial structure.

I've worked on large projects where the default MVC structure (one folder for controllers, one folder for views and one folder for models) for a web application was blindly taken. Having a controller folder with hundreds of controllers quickly gets out of hand. At some point things become hard to name, to find, and a pain to work with.

An aspect that is implicit from the Boids' simulation is that every bird follows its rules perfectly. If they didn't the flock would break apart.

People can't be expected to follow rules perfectly, especially because there are no perfect rules in software development.

However, if we have some way to measure how things are going, like we can look at a flock and see where it's breaking apart, we can nudge things back to harmony.

But to do that we need to be able to measure the health of the system.

If you develop in .Net and you use Visual Studio you have access to a few Code Metrics, namely a Maintainability Index, Cyclomatic Complexity, Depth of Inheritance and Lines of Code.

Those can be useful (although the Maintainabilty Index may be a bit questionable). However, they are a bit limited and they can't be neither changed nor added to.

Recently I've discovered a tool that seems to have been made to solve this problem. It's called NDepend and it not only has these metrics and hundreds more, it also allows you do change them to your needs and even and add your own.

NDepend

NDepend is a paid extension for Visual Studio (with a trial period).

It analyzes your solution each time you build it and provides insights in the form of interactive graphs and a dashboard that shows passing/failing rules.

The rules in NDepend are similar to Linq queries (they are called CQLinq in NDepend where CQ stands for Code Query) and there are more than 200 available.

They are easy to read as well, for example this rule creates a warning for methods that have more than 10 lines of code:

// <Name>Keep methods short and concise</Name>
warn if count > 0 from method in Methods where method.NbLinesOfCode > 10 select method

This is what you'll see when there are methods that are picked up by this rule:

You can click any of the methods and NDepend will take you to where they are in the source code.

You can find the predefined and also your custom rules in NDepend's Rule Explorer Panel:

You've probably noticed that I've used select method in the rule definition. You can add additional information by returning an anonymous object with a method property and the extra information that we want to add. That extra information will be displayed in the UI.

For example, let's add the method's visibility and how many methods are called from the offending method:

// <Name>Keep methods short and concise</Name>
warn if count > 0 from method in Methods where method.NbLinesOfCode > 10 select new {
    method,
    method.Visibility,
    method.MethodsCalled
}

You can access this information when you click the rule in the Rule Explorer. You can even hover over each of the "returned" fields and get extra information:

Additionally to being able to customize how the rules are displayed you can define how they affect the project in terms of technical debt. Namely, through an arbitrary amount of time.

Let's say that we think that it would take on average 5 minutes to fix each line over 10 lines of code (by breaking the method in smaller methods, for example) in a method and we want that to be quantified.

We can include a Debt value in the rule's select statement that describes this:

// <Name>Keep methods short and concise</Name>
warnif count > 0 from method in Methods 
where method.NbLinesOfCode > 10 
select new { 
    method,
    method.Visibility,
    method.MethodsCalled,
    method.NbLinesOfCode,
    Debt = (5 * (method.NbLinesOfCode -10)).ToMinutes().ToDebt()
}

This will then add the the project's overall Debt that you can consult in NDepend's dashboard.

This is just a sample of what you can do with rules. I've only mentioned methods, but you can query assemblies, types, fields, code created by you (you can fine tune what this means in NDepend), third party code, etc. You can also cherry pick which rules you want considered in case you don't agree with some of them.

Also, you can integrate NDepend with a build server and have the build fail when you detect something you consider serious going on. This is achieved with a set of rules named Quality Gates that you can also customize and add/remove to.

Additionally to the rules you can produce graphs that provide visual insights over the code base.

For example, you can create a "Code Metric View" graph that can show you two dimensions simultaneously. For instance the number of lines of code (LoC) in a class and its cyclomatic complexity. The LoC is represented by how big an area the class takes and the complexity by the color:

You can see from the example above that the ManageController is the "worst" class in terms of cyclomatic complexity and it's the second largest in terms of lines of code.

NDepend has other graph types that you can use to better understand your project's dependencies, inheritance hierarchies and test code coverage just to name a few.

However, all of this only allows you to see one picture of how your project is doing in terms of quality right now.

My favorite feature of NDepend is that you can create a baseline and then see how the project evolves in relation to that baseline (you can configure this from the Dashboard).

For example, if you add or remove Debt or have more or less rules passing/failing this is highlighted and you can truly have a feeling of seeing the direction the project is going.

Conclusion

Software development is a complicated discipline, especially when you consider that it is performed by several people working together.

Comparing it to emergent systems is useful because it provides a perspective where we can think of software as something that evolves.

Being able to measure that evolution is then crucial if we want to be able to tell if the product we are building is holding up in terms of quality.

I also describe a tool named NDepend that serves exactly this purpose (and as far as I know has no competitors). It provides extensive metrics and allows for the creation of custom rules, all of this while supporting integration with a continuous integration workflow.