DEV Community: Adrian Kuper

Communication is key (to good software)

Adrian Kuper — Mon, 11 May 2020 19:31:33 +0000

What I came to realize lately is this: Communication is the single key factor of creating high quality software. Why? Let me try to explain.

Stages of software development

The very first thing to acknowledge when building a software is: Software Development is not just about writing code (maybe for many this was obvious, but it wasn't for me as a coder). If your team is solely build from "Software Developers" (as in people who write code) the project is bound to fail. Instead each software project must go through the following stages, which each require a unique set of skills, that are pretty much impossible for a single person to aquire. These stages in their most basic form are:

Planning
Building
Testing
Deploying
Monitoring/Maintenance

In a corporate environment it is pretty much inevitable for any software to avoid any of these stages, which all build upon and are interdependent on each other. You can't build software without planning and you can't test, deploy or monitor software that has not been build. At the same time any built software that is not tested, deployed or maintained is obviously very unlikely to generate any long-term value.

To emphasize the importance and the vast set of skills necessary for each stage I would like to explain each in greater detail:

Planning

The planning stage sets the foundation for the software project and is therefore from utmost importance. Just like any house built on a loose foundation will sooner or later collapse, any software project is destined to fail without proper planning. No code should be written before planning.

The planning stage requires specifying the software project from a user perspective. There must be a clear vision for the software, which is understood by all team members and that - when executed through code - solves a clear problem and generates value to the company.

This stage requires strong communicational skills in order to understand the needs of the customer, how those needs can be served by the software product and communicate that to the other team members. Basic technical skills from each other stage can be hugely beneficial in order to balance the needs of the customer and the rest of the development team, while team leading skills are needed in order to gain the trust and backing of the other team members when making decisions.

Building

With a proper plan, and clear vision of the product, it's time to start writing code. This is done most of the time by people called "software developers", although (as should be clear by now) much more people and roles with different skill sets are involved during the development of software.

Writing code is a complex, lengthy and expensive task. In order to create a code-base that is maintainable, scalable, perfomant, testable and adaptable many skills must be aquired, including:

Programming language(s) for front-end, back-end
Database design and query languages
Software design and patterns
IDEs and version control (Visual Studio, IntelliJ, Git, etc.)
A multitude of ever-changing frameworks
Understanding of computers work in general
Clean code principles

Since technology and frameworks keep evolving and changing, the art of writing code is a life long task of learning, adapting, trying and refactoring.

Testing

To prevent bugs, potentially business-critical failures, slow performance and ensure a high quality in general, each change to the product must be tested thoroughly. With a growing code base, this can only be done by automating as much of the test suite as possible. The software must be planned and designed for making automated tests easy, which requires knowledge in:

Software design
Unit, integration and system testing
Test automation frameworks (like JUnit, Selenium, etc.)
TDD, BDD and many others

Since a lot of time can be spend on testing, it's doubly important to have clear specifications for the software and features in development that are as likely as possible to deliver value to the customer. Otherwise a lot of time can be wasted on creating a automated test suite for software features that are unneccesary, unwanted or even contraproductive.

Deploying

No software can generate any value to the company, if isn't deployed aka rolled out to the customer. Sadly this stage is more often than not overlooked during planning and therefore unnecessarily causes stress, frustration, overtime for the employees and downtime for the customers.

In order to prevent this, the software must be planned and developed with an easy and automated deployment in mind. This requires knowledge of technologies such as:

Server administration (Linux, or if you're crazy: Windows Server)
Shell scripting (bash, Powershell, etc.)
Computer Networks
Containers and Orchestrators (Docker, Kubernetes, etc.)
Virtual Machines
Configuration Managment Tools (Ansible, Chef, Puppet, etc.)
Cloud Services (AWS, Azure, Google Cloud, etc.)
Build Server (Jenkins, Azure Pipelines, Gitlab CI, etc.)
Continuous Integration/Delivery/Deployment in general

Monitoring and Maintenance

To keep software running, identify problems with the product early and keep in touch of the needs of the customer, every deployed piece of software must maintained and monitored. Otherwise performance problems might cause downtimes with a growing customer base, or customer might become increasingly unhappy with the product and stop using it.

Mechanism for effective monitoring and maintance must planned and developed during the early stages. In addition - as with any other stage - there a lot of technologies to learn and skills to aquire:

Server administration
Software monitoring tools
General customer support and relations

The key to build a successful product

With all stages of software development in mind, what is the key to build a successful product? In my mind the answer can be clearly summed up in a single word: communication. In order to build a successful product, you must build a orgisational structure that makes it as easy as possbile for information to flow between each and every the stage of developemnt at all times.

The reason is simple: All stages depend and influence each other, while requiring unique sets of skills, talents and knowledge, which are impossible to be obtained by a single person. So learnings and information from each stage must be continuously be synched between the team members through communication.

It is important to keep the communication as immediate as possible. Each detour from where the information is actually needed, bears the risk of losing or altering that information, simply because we - as humans - are imperfect. Each form of communication between humans is error-prone and always carries the possibilty of misunderstandings, delays or just forgotten memories.

One easy thing to do to improve existing teams is to track how information is transported between the stages and where that information is actually transformed into actual work. Track the source of the information, count the amount of people involved in transporting the information and the amount of time needed.

In typical, siloed work environments its not untypical for information to make its way through 2 or 3 delegates (often in the form of managers) over the course of many days or even weeks ("let's wait for next Jour Fix!"), before it can make its way to the person, that actually transforms that information (or a very likely altered version of it) into work. Often the work is also delayed further, since each silo plans work independently and is very unlikely to have time for the problem at hand.

If you can cut down on the time and persons necessary to transform the information into actual work, the software product will improve.

Sadly, more often than not, organizational structure discourages frecent communication by building building barriers, which make it very hard and frustrating to build software. In my experience bad communication tends to be the main cause for frustrated employees and the single root cause for bad products.

Bad communication as a system: The waterfall model

A very infamous process framework, that systematized bad communication, is the waterfall model (as beautifully drawn by me):

At first glance it doesn't look too bad since every stage is accounted for, right? But the big problem is how communication is supposed to flow by the system: One-directional from one stage to another. Communicating "back up" or from any stage that is not next to each other is not encouraged. This means that each error made in the previous stage gets worse by the following, while its made artifically hard for information about problems and their solutions to make their way back in order to be prevented in the first place, e.g. during planing or code writing.

A little bit better is the extended waterfall model, which allows information to flow back between the stages.

Still, any problem encountered during deployment or monitoring has to make its way back through each and every preceeding stage. This is bad because the information has to make "detours" and does not get where its needed immediatly. The ("Chinese whispers")[https://en.wikipedia.org/wiki/Chinese_whispers] effect takes place and with each detour through a unneeded stage the information is delayed, falsified or might even be lost due to shifting priorities.

What to do instead

That the waterfall model is flawed isn't a new insight by any means. And people have been starting to adopt more "agile" frameworks like Kanban or Scrum, which - when implemented properly - can eaze the communication by making work visible (through kanban boards for example) and suggesting teams consisting of specialists for different stages (Product Owners for the planning stage and Software Developers for writing code).

But instead of blindly following "Scrum" or "DevOps" while spending many dollars for expensive courses, you could simply ask yourself the following question when building your software development team:

What can I do to improve the continuous communication between all stages of development?

Things to do include:

Build a autonomous and interdisciplinary team - starting with the planning stage - that covers all stages and is focussed on a single, well-defined project
Have daily team meetings to keep everyone in sync. The meeting ensures that every new piece of information or problem will be discussed no later than the following day
Cut meetings that serve no clear purpose or consist of disfunctional groups (meaning that not all people needed to solve a problem are present)
Keep at least 50% idle time (as in time without meetings) for everyone involved in general. It must be possible for everyone to ask questions and get feedback from any stage. Busy calendars discourage questions - even though there might be some idle time here and there.
Keep work visible by using a single kanban board for all team members
Have a single set of tools used by all team members. Have at least readable access for everyone in the team.
Release frequently for a tight feedback loop (which is communication) from the customer. Keep in mind that stages repeat.
Automate as much as possible, e.g. tests, deployments. This will allow you to release often and reduces the amount of handoffs and communication needed. Communicate how the automatic release process works instead.
Plan with technical debt and learning in mind

I feel that, as long improvements are focused on communication, the software quality and value will improve. It is not necessary to do X just because a process framework told you so. But Kanban, Scrum, DevOps, the agile movement in general all try to formulate frameworks, that make communication easier and more immediate on an organizational level. And that makes them a good starting point, although improvement is not achieved by following any named methodology, but by focussing on communication instead.

How-To: Setup a unit-testable Jenkins shared pipeline library

Adrian Kuper — Sat, 05 Jan 2019 15:17:58 +0000

(Originally posted on loglevel-blog.com)

In this blog post I will try to explain how to setup and develop a shared pipeline library for Jenkins, that is easy to work on and can be unit tested with JUnit and Mockito.

NOTE: This blog post is kinda long and touches many topics without explaining them in full detail. If you don't feel like following along a lengthy tutorial you can have a look at the complete example library on GitHub. Also if you have questions, or really any kind of feedback on what I could improve, please leave a comment and I will come back to you asap ;) Additionally, if you are completely unfamiliar with Jenkins Shared Libraries, you should probably read about them first in the official docs.

Let's get going!

Basic Development Setup

First, let's create a new IntelliJ IDEA project. I suggest using the IntelliJ IDEA for Jenkins shared pipeline development, because it is the only IDE I know of, that properly supports Java and Groovy and has Gradle support. So, if you don't have it installed it yet, you can download it here for Windows, Linux or MacOS. Also make sure to have the Java Development Kit installed, which is available here.

When everything is ready, start IntelliJ, create a new project, select Gradle and make sure to set the checkbox on Groovy.

Next up, enter a GroupId and an ArtifactId.

Ignore the next window (the defaults are fine), click "Next", enter a project name and click "Finish".

IntelliJ should boot up with your new project. The folder structure in your project should be something like the following.

This is cool for usual Java/Groovy projects, but for our purpose we have to change things up a bit since Jenkins demands a project structure like this:



(root)
+- src                     # Groovy source files
|   +- org
|       +- somecompany
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- org
|       +- somecompany
|           +- bar.json    # static helper data for org.foo.Bar

So:

add a vars folder to your project root folder
add a resource folder to your project root folder
delete all files/folders inside src and add a new package like org.somecompany
edit the build.gradle file:



group 'somecompany'
version '1.0-SNAPSHOT'

apply plugin: 'groovy'
apply plugin: 'java'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
    testCompile group: 'junit', name: 'junit', version: '4.12'
    testCompile "org.mockito:mockito-core:2.+"
}

sourceSets {
    main {
        java {
            srcDirs = []
        }
        groovy {
            // all code files will be in either of the folders
            srcDirs = ['src', 'vars'] 
        }
    }
    test {
        java {
            srcDir 'test'
        }
    }
}

After saving, import your changes to the Gradle project:

At this point our project has the right structure to be used as a shared library by Jenkins. But, as you might have seen in the code snippet above, we also added a source directory for unit tests called test. Now is the time to create this folder at the root level of the project and add a package org.somecompany like we did with src. The final structure should look like the following.

Cool, it's time to implement our shared library!

The General Approach

First a quick run-down on how we build our library and on why we do it that way:

We will keep the "custom" steps inside var as simple as possible and without any real logic. Instead we create classes (inside src) that do all the work.
We create an interface, which declares methods for all required Jenkins steps (sh, bat, error, etc.). The classes call steps only through this interface.
We write unit tests for your classes like you normally would using JUnit and Mockito.

This way we are able to:

Compile and execute our library/unit tests without Jenkins
Test that our classes work as intended
Test that Jenkins steps are called with the right parameters
Test the behaviour of our code when a Jenkins step fails
Build, test, run metrics and deploy your Jenkins Pipeline Library through Jenkins itself

Now let's get really going.

The Interface For Step Access

First, we will create the interface inside org.somecompany that will be used by all classes to access the regular Jenkins steps like sh or error.



package org.somecompany

interface IStepExecutor {
    int sh(String command)
    void error(String message)
    // add more methods for respective steps if needed
}

This interface is neat, because it can be mocked inside our unit tests. That way our classes become independent to Jenkins itself. For now, let's add an implementation that will be used in our vars Groovy scripts:



package org.somecompany

class StepExecutor implements IStepExecutor {
    // this will be provided by the vars script and 
    // let's us access Jenkins steps
    private _steps 

    StepExecutor(steps) {
        this._steps = steps
    }

    @Override
    int sh(String command) {
        this._steps.sh returnStatus: true, script: "${command}"
    }

    @Override
    void error(String message) {
        this._steps.error(message)
    }
}

Adding Basic Dependency Injection

Because we don't want to use the above implementation in our unit tests, we will setup some basic dependency injection in order to swap the above implementation with a mock during unit tests. If you are not familiar with dependency injection, you should probably read up about it, since explaining it here would be out-of-scope, but you might be fine with just copy-pasting the code in this chapter and follow along.

So, first we create the org.somecompany.ioc package and add an IContext interface:



package org.somecompany.ioc

import org.somecompany.IStepExecutor

interface IContext {
    IStepExecutor getStepExecutor()
}

Again, this interface will be mocked for our unit tests. But for regular execution of our library we still need an default implementation:



package org.somecompany.ioc

import org.somecompany.IStepExecutor
import org.somecompany.StepExecutor

class DefaultContext implements IContext, Serializable {
    // the same as in the StepExecutor class
    private _steps

    DefaultContext(steps) {
        this._steps = steps
    }

    @Override
    IStepExecutor getStepExecutor() {
        return new StepExecutor(this._steps)
    }
}

To finish up our basic dependency injection setup, let's add a "context registry" that is used to store the current context (DefaultContext during normal execution and a Mockito mock of IContext during unit tests):



package org.somecompany.ioc

class ContextRegistry implements Serializable {
    private static IContext _context

    static void registerContext(IContext context) {
        _context = context
    }

    static void registerDefaultContext(Object steps) {
        _context = new DefaultContext(steps)
    }

    static IContext getContext() {
        return _context
    }
}

That's it! Now we are free to code testable Jenkins steps inside vars.

Coding A Custom Jenkins Step

Let's imagine for our example here, that we want to add a step to our library that calls the .NET build tool "MSBuild" in order to build .NET projects. To do this we first add a groovy script ex_msbuild.groovy to the vars folder that is called like our custom step we want to implement. Since our script is called ex_msbuild.groovy our step will later be callable with ex_mbsbuild in our Jenkinsfile. Add the following content to the script for now:



def call(String solutionPath) {
    // TODO
}

According to our general idea we want to keep our ex_msbuild script as simple as possible and do all the work inside a unit-testable class. So let's create a new class MsBuild in a new package org.somecompany.build:



package org.somecompany.build

import org.somecompany.IStepExecutor
import org.somecompany.ioc.ContextRegistry

class MsBuild implements Serializable {
    private String _solutionPath

    MsBuild(String solutionPath) {
        _solutionPath = solutionPath
    }

    void build() {
        IStepExecutor steps = ContextRegistry.getContext().getStepExecutor()

        int returnStatus = steps.sh("echo \"building ${this._solutionPath}...\"")
        if (returnStatus != 0) {
            steps.error("Some error")
        }
    }
}

As you can see, we use both the sh and error steps in our class, but instead of using them directly, we use the ContextRegistry to get an instance of IStepExecutor to call Jenkins steps with that. This way, we can swap out the context when we want to unit test the build() method later.

Now we can finish our ex_msbuild script:



import org.somecompany.build.MsBuild
import org.somecompany.ioc.ContextRegistry

def call(String solutionPath) {
    ContextRegistry.registerDefaultContext(this)

    def msbuild = new MsBuild(solutionPath)
    msbuild.build()
}

First, we set the context with the context registry. Since we are not in a unit test, we use the default context. The this we pass into registerDefaultContext() will be saved by the DefaultContext inside its private _steps variable and is used to access Jenkins steps. After registering the context, we are free to instantiate our MsBuild class and call the build() method doing all the work.

Nice, our vars script is finished. Now we only have to write some unit tests for our MsBuild class.

Adding Unit Tests

At this point writing unit tests should be business as usual. We create a new test class MsBuildTest inside the test folder with package org.somecompany.build. Before every test, we use Mockito to mock the IContext and IStepExecutor interfaces and register the mocked context. Then we can simply create a new MsBuild instance in our test and verify the behaviour of our build() method. The full test class with two example test:



package org.somecompany.build;

import org.somecompany.IStepExecutor;
import org.somecompany.ioc.ContextRegistry;
import org.somecompany.ioc.IContext;
import org.junit.Before;
import org.junit.Test;

import static org.mockito.Mockito.*;

/**
 * Example test class
 */
public class MsBuildTest {
    private IContext _context;
    private IStepExecutor _steps;

    @Before
    public void setup() {
        _context = mock(IContext.class);
        _steps = mock(IStepExecutor.class);

        when(_context.getStepExecutor()).thenReturn(_steps);

        ContextRegistry.registerContext(_context);
    }

    @Test
    public void build_callsShStep() {
        // prepare
        String solutionPath = "some/path/to.sln";
        MsBuild build = new MsBuild(solutionPath);

        // execute
        build.build();

        // verify
        verify(_steps).sh(anyString());
    }

    @Test
    public void build_shStepReturnsStatusNotEqualsZero_callsErrorStep() {
        // prepare
        String solutionPath = "some/path/to.sln";
        MsBuild build = new MsBuild(solutionPath);

        when(_steps.sh(anyString())).thenReturn(-1);

        // execute
        build.build();

        // verify
        verify(_steps).error(anyString());
    }
}

You can use the green play buttons on left of the IntelliJ code editor to run the tests, which hopefully turn green.

Wrapping Things Up

That's basically it. Now it's time to setup your library with Jenkins, create a new job and run a Jenkinsfile to test your new custom ex_msbuild step. A simple test Jenkinsfile could look like this:



// add the following line and replace necessary values if you are not loading the library implicitly
// @Library('my-library@master') _

pipeline {
    agent any
    stages {
        stage('build') {
            steps {
                ex_msbuild 'some/path/to.sln'
            }
        }
    }
}

Obviously there is still a lot more I could have talked about (things like unit tests, dependency injection, Gradle, Jenkins configuration, build and testing the library with Jenkins itself etc.), but I wanted to keep this already very long blog post somewhat concise. I do hope however, that the general idea and approach became clear and helps you in creating a unit-testable shared library, that is more robust and easier to work on than it normally would be.

One last piece of advice: The unit tests and Gradle setup are pretty nice and help a ton in easing the development of robust shared pipelines, but unfortunately there is still quite a bit that can go wrong inside your pipelines even though the library tests are green. Things like the following, that mostly happen because of Jenkins' Groovy and sandbox weirdness:

a class that does not implement Serializable which is necessary, because "pipelines must survive Jenkins restarts"
using classes like java.io.File inside your library, which is prohibited
Syntax and spelling errors in your Jenkinsfile

Therefore it might be a good idea to have Jenkins instance solely for integration testing, where new and modified vars scripts can be tested before going "live".

Again, feel free to write any kind of questions or feedback in the comments and take a look at the completed, working example library on GitHub.

CSS grid intro for WPF/XAML developers

Adrian Kuper — Fri, 21 Dec 2018 14:51:39 +0000

This post tries to explain the CSS grid feature to those, who are already familiar with the grid layout in WPF/XAML. The XAML grid layout is pretty cool, because it makes defining complex UIs pretty easy and straight-forward. With CSS on the other hand, I always felt the need the use CSS frameworks in order to somewhat feel confident in making properly looking UIs. But no more! CSS grids brought the awesomeness of the XAML grid layout to the web almost one-to-one, which makes building complex layouts with HTML + CSS manageable at last. So let's get started.

Example mock-up

Let's take a look at a simple mock-up for a comment entry form, which we will use as an example:

First we will see how to implement this using the XAML grid layout. Then we will build the same UI with HTML + CSS grids, in order to easily compare both methods.

Using XAML

Whenever I am presented a mock-up for a user interface I have to implement with XAML, I first try to figure out "the grid" of the interface and draw it into the mock-up. For our example, the grid could look like this:

From this point on translating this picture using the XAML grid layout is straightforward and should be understandable even for those, who don't really know about XAML:

<Grid>
    <!-- First, we define our grid -->
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto" />
        <RowDefinition Height="30" />
        <RowDefinition Height="Auto" />
        <RowDefinition Height="Auto" />
        <RowDefinition Height="60" />
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*" />
        <ColumnDefinition Width="*" />
    </Grid.ColumnDefinitions>

    <!-- then we add all ui items at their respective position -->
    <Label Grid.Column="0" Grid.Row="0" Content="Name:" />
    <Label Grid.Column="1" Grid.Row="0" Content="Email:" />

    <TextBox Grid.Column="0" Grid.Row="1" />
    <TextBox Grid.Column="1" Grid.Row="1" />

    <Label Grid.Column="0" Grid.Row="2" Content="Comment:" />

    <!-- something like an text area -->
    <TextBox Grid.Column="0" 
             Grid.ColumnSpan="2" 
             Grid.Row="3"
             Height="200"
             TextWrapping="Wrap"
             AcceptsReturn="True" />

    <Button Grid.Column="1" Grid.Row="4" Content="Submit" />
</Grid>

The result:

And now with CSS grids

The cool thing about CSS grids is that they can be used remarkably similar to the XAML grid layout, which means if you are good with XAML you will be comfortable with CSS grids in no time.

First, let's take a look at the HTML that is necessary to replicate the UI we want to create:

<div id="container">
  <div id="name-label">Name:</div>
  <div id="email-label">Email:</div>

  <input id="name-txtbox" type="text" />
  <input id="email-txtbox" type="text" />

  <div id="comment-label">Comment:</div>

  <textarea id="comment-txtarea" rows=10></textarea>

  <button id="submit-btn" type="button">Submit</button>
</div>

Doesn't look complicated, but unlike XAML there is nothing about grids in the markup. Well, it's not called CSS grid for nothing. The rows and columns of the grid, and the position of the UI items inside the grid are all defined inside the css file:

/* "#" selects an item inside the html by their id */
#container {
  display: grid; /* with this we declare our container as a grid */

  grid-template-columns: 1fr 1fr; /* two columns, equal width */
  grid-template-rows: auto 30px auto auto 60px; /* 5 rows */
}

#name-label {
  grid-column: 1;
  grid-row: 1;
}

#email-label {
  grid-column: 2;
  grid-row: 1;
}

#name-txtbox {
  grid-column: 1;
  grid-row: 2;
}

#name-txtbox {
  grid-column: 2;
  grid-row: 2;
}

#comment-label {
  grid-column: 1;
  grid-row: 3;
}

#comment-txtarea {
  grid-column-start: 1;
  grid-column-end: span 2; /* span both columns */
  grid-row: 4;
}

#submit-btn {
  grid-column: 2;
  grid-row: 5;
}

First, we select our "container"-div element, declare it as a grid and set the rows and columns almost identically to XAML (1fr pretty much equals *, auto equals Auto and 30px and 60px are self-explanatory). After that we only have to select our UI items inside the grid and give them their place inside the grid using grid-column and grid-row.

Take a look at this CodePen I created to see the result.

To summarize, the following table comprehensively shows how certain elements known from the XAML grid layout "translate" into CSS grids:

XAML	CSS grid
`<Grid>`	`<div id="foo">` + `#foo { display: grid; }`
, 2	1fr, 2fr
Auto	auto
`<RowDefinitions>`	grid-template-rows:
`<ColumnDefinitions>`	grid-template-columns:
Grid.Row	grid-row:
Grid.RowSpan	grid-row-start: + grid-row-end:
Grid.Column	grid-column:
Grid.ColumnSpan	grid-column-start: + grid-column-end:

Pretty easy, right? CSS grids allowed me to reuse the knowledge and experience I had with XAML grid layouts and finally made me feel comfortable in building complex UI layouts using HTML + CSS without the need for CSS frameworks like Bootstrap. Plus, they have some neat extra features that XAML grid layouts lack, e.g. named areas, which you should definitely check out. Once you're convinced by CSS grids and want to know all about them, take a look at this awesome and complete guide.