DEV Community: SCHREIBER Christophe

Proposition of manifesto for green software development

SCHREIBER Christophe — Mon, 29 Mar 2021 20:33:56 +0000

Nobody can ignore the environmental crisis we are facing and the consequences that global warming is having on our lives. Unfortunately, situation is only going to get worse if we do nothing.
Digital services are a significant part of the greenhouse gas emissions, being around 4% of the global emissions in 2018, as much as civil aviation, and it is expecting to raise until 5.6% in 2025 (source greenit.fr).

For decades, we have been relying on the progress of the hardware following Moore's law to compensate the lack of optimization of our software. Why should we care about memory consumption ? Memory is so cheap. Why should we care about the size of our webpages ? Networks are always faster.
In the same time, the number of devices we own and the number of applications we use every day has exploded in a few years. And because software is not efficient, we are tempted to renew our devices frequently for our own comfort even though our current one is still working, despite the dramatic environmental (and societal, given the conditions of life of people working to extract the mineral necessary to build a smartphone or a laptop) cost of producing a new device.

Bad news is that this digital luxury can not last forever: resources are limited, this way of life is very energy costly, and we can't ignore the risks that our consumption of digital services imply for our planet.
Fortunately, we as developers can have a direct impact on how we produce software, by integrating those environmental aspects in the way we design and write software.
For instance, by taking care of the optimization of our applications, we can make sure our users won't require a new device to use our services.

For me, this green mindset of development is the logical sequel to the software craftsmanship culture that has been driving developers aspiring to raising the bar for software quality for more than ten years. Now, it's time to raise the bar of environmentally friendly development.
That's why I'm proposing an equivalent to the Software Craftsmanship Manifesto, which I'm calling the Green Digital Manifesto:

As aspiring ecologically responsible Software Craftsmen, we want to go further the standards of quality to enable digital services compatible with the environmental crisis we are facing. Through this work, we have come to value:

Not only well-crafted software but also resources efficient digital services

Not only steadily adding value but also controlling the evolution of resources required by our digital services

Not only a community of professionals but also a common goal for greener digital services

Not only productive partnerships but also guiding our customers in their environmental transformation

That is, in pursuit of the items on the left we have found the items on the right to be an even greater goal for the benefits of the planet.

Please tell me what you think of this proposition, and don't hesitate to propose your own green principles for green development, and of course share it with people involved in software development!

Credits: Cover image

Spreading craft practices by teaching my managers how to code

SCHREIBER Christophe — Thu, 19 Dec 2019 21:44:02 +0000

Introduction

Legacy software is everywhere. Many of us are struggling everyday with code that was written in a hurry to meet a deadline by someone who is long gone and nobody really understands their code.

Business guys are angry because every tiny feature that they ask takes weeks to implement and deliver, and production incidents arise too often. Unfortunately, for many developers this is the true life of a software maintainer.

More than 10 years ago, the Software Craftsmanship movement was born to tackle these issues and raise the bar of quality software and professionalism. For many, this was the light at the end of the tunnel, a ray of hope that could transform the software industry.

For some companies, it worked, and they were able to reach a higher level of quality and development culture. But for too many others, this movement just didn't work as expected. Sure, developers liked the concepts and methodologies promoted by the Craftsmanship movement, but they just couldn't put them in practice because of other intervenants : either their product owners or their managers were still pushing them to deliver as fast as possible, they didn't understand the benefits of craft practices, they were reluctant to let developers learn new practices like Test Driven Development, were too afraid of the possible regressions that can be introduced by refactoring, thought of pair programming as a waste of time (2 developers working on the same subject whereas each one can handle one story ? non sense !)...

This was the context in my current team. We had been paying Craft and Agile coaches for years, but the Craft culture was just not working as promised. My manager decided to try another approach : he asked me to prepare a training session targeting non technical people, to learn them the basics of modern software development and make them realize the benefits of Craftsmanship and, more generally, software quality.

One training to rule them all, and in Craftsmanship bind them

When I started building this training, it quickly became clear that the main difficulty will be to allow people that never wrote a line of code to feel satisfied by the results of their work, while still interesting former developers. Just doing some small "Hello World" examples was not an option. So, I've thought about one crazy idea : what if students attending the course could develop a real application fulfilling one of our business needs, reaching the quality standards of our company, that we could deliver on the last session of the training ?

This seemed challenging at the time. I had many subjects that I wanted to share with students, but we wanted this course to be accessible to as many people as possible. So, first decision was the format of the training : it would be made of 12 sessions of 2 hours, with about half of the time dedicated to a theoretical lesson and the rest would be some practical exercises. We'd need to start small, and then add complexity and features little by little as students would feel more confident.

After a bunch of brainstorming sessions with my manager, we decided to build an API to handle Swift messages, a kind of flow that our applications need to handle very often. We've prepare a program of 12 thematic sessions:

Introduction and installation of the tools
Object Oriented Programming Basics
Automated tests (2 sessions)
Clean Code principles
Continuous Delivery
Creating an API
Working with databases
Security
Preparing our application for production
Live production delivery
Retrospective

In order to know where we were going, I had to develop a prototype of the application we were going to develop, to make sure it was feasible for people discovering development. To make things easier and to follow the guidelines given to our developers wanting to develop APIs, I decided to use Spring Boot's ecosystem: we will build our API using Spring Web MVC, rely on Spring Security to handle our authentication, use Spring Cloud to register through our service registry, communicate with the database with Spring Data... Thanks to Spring's magic, it would allow the wow effect of having a whole application with just very few code to write, and moreover it would allow us to build it brick after brick, which was perfect for my planning !

But there was still one issue : Swift messages are quite complex, and there is a lot of messages type. At first, I thought that each student could model one type of message, but I quickly understood it was not going to be possible for newcomers in just a few hours. So, I've started looking for an opensource library that we could use. I found one (Prowide), and it greatly simplified the work for students while promoting reusability instead of reinventing the wheel.

Registration time

With the skeleton of my course ready, it was time to find students. We talked about our idea to our managers, to business analysts, and we even asked Ops team what they thought about it. It appeared that many people were curious about the initiative and wanted to be part of it. We originally planned to have a 8 students class, we finished with 13 students divided into 2 promotions !

The profiles were very diversified: we had business analysts that never learned how to code, others that did a bit of development at school, we had managers that didn't code since more than 15 years, Ops that knows a lot about scripts coding but never practiced OOP...

While this successfull enrollment was a joy, it lead to the greatest difficulty I was going to face : how to satisfy such various people ? How to teach to the full beginners while not boring the ones with some prior experience ?

Adapting to everyone

The first decision to mitigate this issue was simple : since we needed 2 promotions, we've defined the groups depending (as much as possible) on their existing knowledge : the first group would be composed of people already having a developer experience, whereas the second one would be for beginners. Since both groups will have their weekly sessions in 2 different days, it also allowed me to adapt my lessons and practical exercises: if "experienced" students were struggling to understand one aspect or implement a feature, I would change the way I was using to present the subject and give more detailed instructions for exercices, and even live code with students for the most difficult parts. And of course, the most important point was to make sure that everyone understood as much as possible what we just learned before leaving the session. I allowed them to interrupt me everytime they had a question, I didn't want them to let go about something that would not be clear.

For the theory, lessons needed to be as clear as possible: I needed to focus on the key messages that I wanted them to remember. Of course, some subjects were only briefly mentioned, but every time someone wanted more details, I tried to vulgarize as much as possible (sometimes giving them a bit more details to tease them ;-) ). The overall idea was to keep the lessons short and to infuse the key takeaways about coding and quality that we want to spread in our teams.

Feedbacks of the experience

For the beginners, the first sessions were painful : since we didn't have a lot of time to cover the basics of programming, especially the syntax of the language (Java), they struggled to write simple lines of code. After a few weeks, they started to understand better the concepts we were manipulating and were a bit more confident.

In the end, they all realized that the developer's job was very different from what they imagined or what they knew in the beginning of their career : they were impressed by the number of topics and technologies a developer had to understand and master in the context of a modern application. But they were also relieved by the quality of the tools (IDE for instance) and the incredible choice of libraries that are available and ready to use. Most importantly, they understood the benefits of craft practices : when a manager told me that the test harness showed him that his refactoring was causing a regression, I was happy. When they told me that the code they just refactored was easier to understand and to evolve, I was happy. When they told me that they understand better what being a developer means, how they work, what are their constraints and why they need to focus on quality, I was super happy.

It is still a bit soon to know if this experience will change how our teams work and how we tackle the issue of our technical debt. But from the conversations I'm hearing in the openspace, some progress are already there :-)

But this initiative is not over : people from other entities of the company have heard about it and want to join the next promotion. We even proposed our top management to join us. We want our offshore teams to reproduce the course within their teams as well.

The next step is to make sure that we can leverage on this "craft trend" : we need to keep insisting on the benefits of craftsmanship, not only with developers, but with anyone involved in the development of our systems, hoping that this time the quality culture that we want will not fade after the departure of motivated developers.

Mutation testing in 1000 characters

SCHREIBER Christophe — Fri, 07 Dec 2018 13:58:30 +0000

Code coverage is an common metric to measure code quality but it doesn't guarantee that tests are really testing the expected behavior.

Mutation testing can be described as tests of tests. It consists of introducing mutations to your codebase and checking if at least one test fails for each mutation (we say that mutation is killed in this case, otherwise it lived):

inversion of conditions
change boundary conditions (< becomes <=)
change incrementation of counters
change mathematical operators
return null instead of returned value
don't call void methoids, constructors...
many other type of transformations

If a mutation is killed, that means that your tests are covering some edge cases, giving you more confidence about the quality of your tests.

Mutation testing can be an heavy process, so it doesn't need to be executed as often as unit tests, but it can be a step in your release process.

Java projects can use Pitest library.

Working on legacy code should not be a grief

SCHREIBER Christophe — Tue, 13 Nov 2018 14:00:07 +0000

Working on legacy software is difficult. We've all faced this issue in our career, and that's why there are so many books about working with legacy code, improving an existing code base using unit tests and clean code principles...

The lack of tests implies that code is hard to understand, hard to test, long to deliver and costly to support because of bugs that will inevitably happen in production.

For too many teams, testing legacy code leads to the same 5 phases as those of grief and loss:

denial: "we don't need automated tests, we can still maintain our product, we know how it works"
anger: "there was still an incident after our delivery! this product is s**t, we should rewrite it from scratch"
bargaining: "ok, we could write some tests for new features. Let's try this for the next release"
depression: "our code is not easily testable, the release is late because it takes too much time to write tests, and moreover they are fragile, there is nothing we can do"
acceptation: "let's hire more people for support, this will help us since we can't add automated tests. We'll also release less often so that we can plan non regression testing phase"

Sure, testing a legacy monolith is not an easy task, even for experienced and cautious developers. In fact, testing spaghetti code is almost guaranteed to lead to poor tests, very coupled with implementation details, and every change will break them.

That's why it is important to improve the code step by step, with small refactorings that are unlikely to break existing behaviors, starting from very small steps and going on with more complex ones as the code improves:

rename variables, methods or classes so that their purpose is clearly expressed and easier to understand
remove unused or dead code
testing a long method or class is difficult, so break them into smaller pieces. By simply extracting them using your IDE's refactoring capabilities, you shouldn't break anything. The code will become easier to tests, since the dependencies and the responsibility of each block will be reduced.
once you start to have a comfortable tests harness, you can start performing more deeper refactorings (using abstractions instead of concrete implementations to favor testability and evolutiveness, using design patterns...)

Don't forget the importance of team work during these difficult times. If you can't solve a problem alone, pair programming or even mob programming will help you and your pairs. This will ensure that every developer of the team will know how to face similar problems when they'll want to improve another piece of code.

During the challenging tasks, clean code principles, SOLID principles and all the usual craftsmanship toolkit will help you, so be sure to understand them and to be able to apply them when necessary !

SOLID Principles in 1000 characters

SCHREIBER Christophe — Thu, 08 Nov 2018 14:32:02 +0000

SOLID Principles are 5 concepts described by Uncle Bob to improve software design quality in object oriented programming:

Single Responsibility Principle: group together the components (classes, modules...) that will change for the same reason
Open-Closed Principle: software should allow easy extension (opened) without needing to change existing code (closed to modification)
Liskov Substitution Principle: every subtype of a given type can be used wherever the parent type is required
Interface Seggragation Principle: one interface for one purpose, a component should only depend on components/behaviors it requires to fulfill its purpose
Dependency Inversion Principle: components should depend on abstractions, not implementation details

These principles help the software to be easier to maintain, evolve and test, and are considered good practices and lead to a cleaner architecture.

SOLID principles are to design and architecture what Clean Code is to code writing.

Test Driven Development in 1000 characters

SCHREIBER Christophe — Mon, 05 Nov 2018 21:32:51 +0000

Test Driven Development (TDD) is a practice of Extreme Programming consisting in writing tests before production code to guarantee the code quality.

Development process is divided in 3 steps:

Red: write a failing test
Green: write the minimum production code required to make the test pass, all shortcuts are allowed
Refactor: clean the mess of the code you just wrote to make it clean, improve the design

Since no production code is added before a test covers a usecase, test coverage is as close as possible to 100%.

Moreover, TDD promotes the usage of clean code principles thanks to the refactoring stage.

It can also help to make a clean design appear progressively. As Uncle Bob says: "as the tests become more specific, the code becomes more generic".

TDD helps team to write code that is easier to maintain and less risky to deliver.

TDD is often difficult to learn, especially on real life projects. Code katas are helpful to master this practice progressively with more simple use cases.

Talking about craftsmanship in 1000 characters

SCHREIBER Christophe — Mon, 05 Nov 2018 21:32:12 +0000

Time is precious, this is a resource we would all be glad if we could have more. Although long detailed articles are great to learn a lot about a subject, I think some very small posts to present a topic briefly can be a good kickstarter.
So, I’d like to try a format of very short articles, to give an overview of a subject related to software craftsmanship in only 1000 characters. This may be a difficult constraint, but summarizing complex topics in this very small format will also help me to only talk about the core principles, the ones that should be remembered even if readers don’t go deeper with other posts.
And I also hope that it will help me write more on dev.to ;-)
My first article in this « Less than 1000 characters » series will be about Test Driven Development. Don’t hesitate to propose some topics you would like to read about !

Architecture as a burden

SCHREIBER Christophe — Tue, 09 Oct 2018 20:24:21 +0000

Update : french people can find a talk I did in Paris Java User Group inspired by this article here

Introduction

During the various missions I've been sent to, I've worked on a variety of legacy software projects which suffered from various kind of flaws.

Of course, poor software quality (no unit tests, clean code principles not being used...) was often a major issue, but there were also problems coming from architectural decisions taken in the early days of the project, or even at the dawn of the enterprise system. This kind of issues is, from my point of view, the greatest cause of pain for many projects.

As a matter of fact, improving code is quite easy, especially now that the software craftsmanship movement is spreading good practices across teams. But changing the core of the systems, the constraints that were imposed at the very beginning of its lifecycle, is very challenging.

I'll talk about several types of architectural decisions that I've encountered, and that can be real burdens for the teams maintaining these systems.

Sharing your database with the whole company

This is probably one of the most common issue I've seen. When several applications need to use common data, why can't we simply share the database? After all, repetition is a bad thing in software development, right? Well, not eveytime, and especially not when database is involved. Venkat Subramaniam said it in a way that can't be forgotten: "A database is like a toothbrush, you should never share it". What's so wrong about sharing a database? Many things in fact...

First thing that I can think of is obviously the coupling in the datamodel. Imagine that 2 applications A and B are dealing with cars. Application A is used in the team responsible of repairs, and so they need to store a lot of technical data about the mechanics, the failures, the history of interventions on the car... Application B is used to handle appointments for the technical team, so it only needs basic information about the car to be able to identify it. In this case, using the same datastructure for both applications makes no sense: they use very different data, so they should use their own data strucuture. This is made even easier since a car can be easily identified, so there is no need to share a common referential.

The second issue also comes from this coupling of the datamodel. Imagine that B wants to rename the identifier of the car because it makes more sense from a domain point of view. In this case, A should also be updated to handle the new column name... So, to avoid disturbing A's team, B's developers will start duplicating the information in a different column since they can't change the existing name... Of course, A will say that they will plan this changes in the future to avoid having 2 columns containing the same data, but we all know this will most probably never be achieved...

Things get even uglier when applications are not just reading data from the same source, but they also modify them! In this case, who is the owner of the data? Who should be trusted? How can the integrity of the data be guaranteed? This is already difficult when several parts of the same applications are modifying the same information, and this becomes much worse when several applications are involved...

The last case I've seen is 2 applications sharing the same data structure to store information about 2 relatively close business objects, but with just enough differences to make understanding which data belongs to which application really hard. In this case, both applications were using this table to model financial market executions, but with different levels of aggregation. Nothing indicated that there were 2 types of data in this table, so we had to look in another table (owned by the second application) to identify the lines generated by each applications... Each new developer having to work on this table would inevitably fall in the same pit as every of their predecessorq and use incorrect (sensible) data, with all the risks involved for the company.

Building your system around a business software

Not every company can develop the system to handle all its business usecases. In fact, in many cases, this would just be reinventing the wheel, since these usecases are common to many companies and so you can easily find software already supporting them on the market.

So, buying the product is often cheaper than building it. But of course, the software you just bought can't integrate with that other piece of software you also use, so you need to develop a connector between 2 (proprietary, most of the time) applications. You will probably build your own tools to handle specific part of business, and since this expensive software you've bought already has a convenient model, you'll be tempted to just use its database and add your informations to its own tables...

A few years pass, dozens of developers or teams do the same, and then you're stuck: you just can't use another software if its editor closes, or if the product is no longer supported, or if another new product suits your needs better. In some cases, you can even have technical dependencies on an external software. If the editor of the solution wants you to use each version of the language/framework/server/whatever, then you don't own the architecture of your own system. If they want to sell you a new version to provide you a feature that you absolutely need, but if this version implies a change on the technical requirements, you'll be forced to update all your technical stack to align with their recommendations. I've been there, this is not a forced migration that you want to face often...

I've worked on a project where the editor of the software we were using didn't want to develop new features for all their clients, because it became too complicated for them to handle concurrent modifications and several current versions (each client having a specific version with features only them wanted). So, they decided to sell us a Software Development Kit (SDK) so that we can implement our own features. Of course, they didn't provide much documentation about how to do it, and moreover we had to use their business entities, which we needed to decompile to understand their structures since we had neither the sources nor the documentation... The simplest feature would take days to implement, and it was barely testable since everything was very complicated and introduced scripting langages no one knew about in the team to an already complicated stack...

Tight coupling between dozens of applications

Remember the early 2000s and the joy of using Enterprise Java Beans (EJB) to handle remote calls between applications in your information system. At this time, this may have looked like a good idea. Sharing your codebase with other teams to avoid duplication seems ok too. Yes, every teams were forced to deliver their applications at the same time to make sure there was no broken binary dependencies, but these were fun evenings, eating pizzas with colleagues while waiting for the 2 hours delivery process to be completed, isn't it?

Well, in fact it wasn't that fun. And being unable to refactor a single class in your own codebase because someone in the company liked your code and decided to use it in their untested application isn't a pleasure neither.

Once you realize the mess that these early decisions caused, the effort required to decouple your application from the rest of the world is overwhelming. It litteraly take years to cut down your project into different components so that other applications won't be able to use your core domain, your client or your cache mechanism anymore, to remove every use of external classes that are tight coupling to other projects, to replace all EJB calls with REST APIs... But the reward for everyone involved in the project is huge: easier development and testing, faster delivery process since there is no need to synchronize with everyone else anymore, better separation of concerns in your own code, easier dependency management, no more issues of transitive dependencies because your are importing a ton of other applications' dependencies in your classpath... These expensive changes are really a life saver for the team, and they would have been much cheaper to implement at the dawn of the project!

Building your project over someone else's project

This problem may be the one you're most unlikely to face, but this can still happen and this is the worst case scenario, since it cumulates several of the previous issues. In fact, I've faced this issue in one of the first project I've worked on in my career.

When I arrived on the project, I was told this was a total rewrite of the company system and that the project had just started 2 months ago. So, when I saw a complex webapplication with a full adminstration module, a complex business feature already implemented and a mature framework to help developing other modules, I was surprised. I quickly learned that all this stuff has mostly not been developed by the team: it was decided to reuse the framework developed by another company inside the group to avoid starting from scratch. The problem is that this framework had not been isolated from the project it was developed for. So, our team just got an archive containing all the source code of the other company's project, including their business code, which had nothing in common with our own business. Even worse, we've also inherited from their database schema and data...

As a newcomer in the team, it was difficult to know what code was related to the framework, to our project and to the other company's business. The team wanted to clean this mess, but many attempts ended with severe regressions because of dependencies between parts of the code (I can't talk about modules since there was only one!), and of course there was no automated tests at all. Moreover, we had to abandon the idea of using a different application server because there was code specific to the one used by the other company everywhere in the system, making this migration too expensive for our small team.

At some point, we wanted to add some nice features to the framework, but we were told this had already been done in the other company. So, we were asked to merge our current version with the current version of the other company... The team managed to avoid this nightmare by just cherry picking a part of the new feature, but it was still way more complex and rich than what we needed...

We managed to finish this project, but the quality of our project was a real pain. At least 40% of the code and the database contents was useless, and it never became a priority to clean this dead code. I hope the team has finally the occasion to isolate their own code since I left the team !

All your business logic is in a rule management engine

Putting a bit of your business logic in a rule management system is a common practice. This is for instance useful when some of your business rules need to be updated frequently but your monolithic application's delivery process requires long testing phase before being able to validate a release candidate, making it impossible to adjust some of your "volatile" rules. Eventhough I prefer that all the domain rules to be located in the code, I can understand that sometimes a rule management system can help.

But I've faced a case where almost ALL the business logic was located in a rule management system, with sometimes rules being generated from an Excel file! Moreover, rules were not supposed to change very often, since the project was basically an ETL batch. The Java project behind all this was just made of technical details about the batch framework and raw read/write from source and target systems, with absolutely no reference to the domain.

As a consequence, all the rules were written in a specific language that nobody really mastered in the team, was hard to write (our IDEs didn't handle it) and almost impossible to debug or test. When a new rule or a change to an existing one was requested, most developers in the teams just copied/pasted an existing rule, leading to whole identical files except one specific change (often, it was the field on which the rule applied).

If this already seems troubling, there was absolutely no clue in each rule about its purpose. Rules were named Rule1, Rule2 with more than 100 of them! And each rule was basically checks and assignment on hard coded values without any business term. Even the name of the project didn't explain the purpose of the whole ETL.

Conclusion

As Uncle Bob explains in his book "Clean Architecture", when thinking the architecture of a project, some decisions must be postponed until we really need to make a choice unless we can't continue to add value to our product (like choosing a database for instance). Other decisions must be taken really early, do not wait until it gets ugly. Fortunately, this kind of critical decision can easily be spotted, because they are what can be called architectural smells: when you think about it, they can only be bad ideas that will come back and haunt you at one point or another. Unfortunately, when working on legacy software, this kind of burden is often burried deep in the code, making them very expensive to eliminate.

We shouldn't be afraid. Yes, cleaning years or even decades of mess is not an easy task, but as software professional, we just can't let it continue to rot and kill the developers' motivation and the trust our users put into our product and our capacity to deliver business value to them.

Of course, each of the architectural burdens I described can be solved in many ways, so there is no silver bullet to resolve each issue. But I'm sure that every team can come up with propositions to finally be free of their burden. So, let's face our issues together and start cleaning this mess!

How to write a tail recursive list partioner in Kotlin

SCHREIBER Christophe — Sun, 04 Feb 2018 19:37:20 +0000

Writing a list partioning function is a classic programming interview exercise. I'd like to show you a solution to this problem, written in Kotlin. My solution will make use of tail recursive recursion, a feature of the language that is very interesting and that we have missed for so long in Java.
I will proceed step by step using Test Driven Development (TDD).

Prerequisites

If your are working with IntelliJ IDEA, Kotlin is compatible out of the box since version 15 of the IDE. If you are using another IDE, please follow the instructions on how to get your environment up and running on Kotlin's tutorial page.

Explanation of the exercise

The problem is very simple: we want to write a method that allows to partition a list into a list of lists of a given size (called the partition size).
For instance, the list made of {1, 2, 3, 4, 5} partitioned with a size of 2 will be { {1, 2}, {3, 4}, {5} }.
Of course, list size should be strictly greater than 0, otherwise this is a non-sense call. The method should be able to handle lists of any type, like lists of strings, lists of integers...

First step : a failing test

As always in TDD, I'm first going to write a failing test.

class ListPartitionerKtTest {

    @Test
    fun empty_list_should_return_an_empty_list() {
        assertEquals(emptyList<List<String>>(), emptyList<String>().myPartition())
    }
}

As you can see, I've chosen to implement my partitioning method as a myPartition function extension for Kotlin List type. I've chosen the name myPartition to avoid confusion with the existing List.partition method (which has not the same purpose).
Let's make this test pass:

fun List<String>.myPartition() = emptyList<List<String>>())

Being generic

Now, let's implement one of the exercise's constraint, by handling multiple types of list:

@Test fun should_also_handle_integers() {
    assertEquals(emptyList<List<Int>>(), emptyList<Int>().myPartition())

We have to change our implementation so that it becomes generic:

fun <T> List<T>.myPartition() = emptyList<List<T>>()

Declaring a partition size

Our tests are passing, but the signature of our myPartition function is incorrect regarding specification: we have to be able to choose the partition size. So, let's write a failing test:

@Test fun should_accept_different_partition_size() {
    assertEquals(emptyList<List<Int>>(), emptyList<Int>().myPartition(3))
}

Obviously, the code doesn't compile, so let's add a parameter to myPartition, with a default value to avoid having to modify our previous tests:

fun <T> List<T>.myPartition(partitionSize: Int = 2) = emptyList<List<T>>()

Checking the partition size

Now that we have a partition size parameter, we have to check that it is a valid argument. Partition size must be stritly greater than 0.

@Test(expected = IllegalArgumentException::class) fun partition_size_should_not_be_negative() {
    emptyList<Int>().myPartition(-1)
}
@Test(expected = IllegalArgumentException::class) fun partition_size_should_be_strictly_greater_than_0() {
    emptyList<Int>().myPartition(0)
}

We have several things to do to fix our implementation:

Add a control on the parameter and throw an exception
Since we are going to write a full method body, we'll need to write an explicit type return and a return statement

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<T> {
    if (partitionSize < 1) throw IllegalArgumentException("Partition sizz should be strictly greater than 0, ${partitionSize} is invalid")
    return emptyList<List<T>>()
}

Our test is passing, so let's refactor a bit to avoid having a long line of exception initializing:

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<List<T>> {
    validatePartitionSize(partitionSize)
    return emptyList<List<T>>()
}

private fun validatePartitionSize(partitionSize: Int) {
    if (partitionSize < 1) throw IllegalArgumentException("Partition sizz should be strictly greater than 0, ${partitionSize} is invalid")
}

Everything is still green, so let's move forward.

Partioning a list smaller than partition size

We'll write writing the actual list partitioning. First step is easy: if the list is smaller than the partition size, we'll just have to return a list containing the list itself:

@Test fun given_list_smaller_than_partition_size_should_return_the_a_list_containing_source_list() {
    val list = listOf(1)
    assertEquals(listOf(list), list.myPartition(2))
}

As always in TDD, let's write the most simple code to make this test pass:

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<List<T>> {
    validatePartitionSize(partitionSize)
    return listOf(this)
}

We can add the case of a list with exactly the partition size:

@Test fun given_list_size_equals_to_partition_size_should_return_the_a_list_containing_source_list() {
    val list = listOf(1, 2)
    assertEquals(listOf(list), list.myPartition(2))
}

No code to write since the test is passing !

Let's (finally) partition a list!

The next test is obvious: we pass a list of 3 elements with the default partition size (2), and we expect to have a list containing two elements :

a list containing the first 2 elements
a list containing the third element

@Test fun given_1_2_3_and_partition_size_2_should_return_1_2_and_3() {
    val list = listOf(1, 2, 3)
    assertEquals(listOf(listOf(1, 2), listOf(3)), list.myPartition(2))
}

Again, let's implement the minimum code to fix this test:

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<List<T>> {
    validatePartitionSize(partitionSize)
    if (this.size <= partitionSize)
        return listOf(this)
    else
        return listOf(this.take(partitionSize), takeLast(this.size - partitionSize))
}

The test passes, but this looks ugly, so let's refactor it using Kotlin's when operator:

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<List<T>> {
    validatePartitionSize(partitionSize)
    return when {
        this.size <= partitionSize -> listOf(this)
        else -> listOf(this.take(partitionSize), takeLast(this.size - partitionSize))
    }
}

We can add a test that should already pass: the case of a list of size 4, with a partition size of 2:

@Test fun given_1_2_3_4_and_partition_size_2_should_return_1_2_and_3_4() {
    val list = listOf(1, 2, 3, 4)
    assertEquals(listOf(listOf(1, 2), listOf(3, 4)), list.myPartition(2))
}

All green, so we don't change anything :)

Partitioning a list several times

Our previous tests were involving only one partitioning, we will now complicate things a bit and move towards a real partitioning implementation:

@Test fun given_1_2_3_4_5_and_partition_size_2_should_return_1_2_and_3_4_and_5() {
    val list = listOf(1, 2, 3, 4, 5)
    assertEquals(listOf(listOf(1, 2), listOf(3, 4), listOf(5)), list.myPartition(2))
}

The simplest way to make this test pass is to add a bit of recursion to our implementation:

fun <T> List<T>.myPartition(partitionSize: Int = 2) : List<List<T>> {
    validatePartitionSize(partitionSize)
    return when {
        this.size <= partitionSize -> listOf(this)
        else -> listOf(this.take(partitionSize))
         + takeLast(this.size - partitionSize).myPartition(partitionSize)
    }
}

Aplying the myPartition recursively to the last elements of the list and adding it to the result list allows our test to pass. We now have a working list partitioning method !

The dangers of recursion

Of course, any programmer that has played with recursive calls will have spotted the weakness of our implementation: with big lists, we will surely be facing a StackOverflowException error.
Let's add a test to test the limits of our method:

@Test fun should_handle_big_lists() {
    val list = getListOfSize(1000)
    list.myPartition(2)
}

private fun getListOfSize(size: Int): List<Int> {
    val range = 1..size
    var list = emptyList<Int>()
    for (i in range) list = list + i
    return list
}

Apparently, our big list is not big enough, our test is passing. Let's use a real big list:

    @Test fun should_handle_really_big_lists() {
        val list = getListOfSize(100000)
        list.myPartition(2)
    }

We are finally facing our stack overflow error, nice !
In order to fix this test, we are going to use the tail recursion optimization. This will allow the compiler to reuse the last stack element instead of creating a new one, and so the stack overflow error will disappear.
To implement tail recursion, our code must respect the following constraints:

the last call of the function (the return statement) should be the recursive call
we have to indicate the Kotlin compiler that it should use the tail recursive optimization by using the tailrec keyword in our function definition Tail recursion is typically done using a private method, that will take the current parameters of our call (here, the list and partition size) and the result of all previous steps, usually called the accumulator. Let's implement this for our partitioning method:

fun <T> List<T>.myPartition(partitionSize: Int = 2): List<List<T>> {
    validatePartitionSize(partitionSize)
    return myRecursivePartition(this, partitionSize, emptyList())
}

private tailrec fun <T> myRecursivePartition(list : List<T>, partitionSize: Int = 2,
     accumulator : List<List<T>>): List<List<T>> {
    return when {
        list.isEmpty() -> accumulator
        list.size <= partitionSize -> accumulator + listOf(list)
        else -> myRecursivePartition(list.takeLast(list.size - partitionSize),
                partitionSize,
                accumulator + listOf(list.take(partitionSize)))
    }
}

Our myPartition function is now just a call to the recursive function with the correct parameters:

the current list
the partition size
the initial state of the accumulator: an empty list of lists

The recursive method has 3 distinct return conditions:

if the current list is empty, that means we have finished the partitioning, the result is then the accumulator, which has been computed step by step
if the current list's size is smaller or equals to the partition size, we return the accumulator with the addition of the list. This is also a terminal operation
the last case is where the recursion occurs: we call the same method with the remaining elements, and we add the current partition to the accumulator

And our test with a very big list is now passing !
Tail recursion is a very useful tool supported by functional languages such as Scala, but many Java programmers don't use it often since Oracle's language doesn't currently support this optimization.
Thanks to Kotlin, we can now use this tool :)
I hope this article was useful to you. Don't hesitate to send me feedbacks if you think this implementation can be improved or if you need more information about this example.
You can find the code on my Github.

My feedback on XebiCon 2017 - Agile Smells

SCHREIBER Christophe — Thu, 01 Feb 2018 17:00:13 +0000

On November, 30th 2017, I've attented to the XebiCon conference, a conference organized by Xebia society and its partners, in which many subjects were discussed, such as technical subjects (Kotlin, Big Data, IoT, Blockchain, DevOps...), organisational aspects (Feature teams...) and of course several talk about Agile methodologies.
One very interesting talk was the one from Julien Rossignol about Agile Smells, those signs that indicate something is wrong in your team organisation.
This talk was based on a series of articles he published on Xebia's blog (in French) in the beginning of 2017.

Lack of visual management

As a reminder, visual management consists in using the walls of your office to display information about your team's life: in progress tasks, its objectives, its potential issues... This allows to everyone passing by to quickly know what the team is currently working on.
For Julieu Rossignol, an Agile team's office with blank walls (no post-it, burndown chart or any other information) is a real sign of important problem, for instance:

External pressure from the management, leading to the team hiding information to protect themselves
A lack of product perspective and of medium to long term objectives, which can lead to loss of motivation for team members
Usage of digital only tools, which can not be suitable for everyone and may lead to members being less involved during ceremonies Here is an example of good visual management that Julien provided during the conference : As a conclusion for this part, Julien expose the 3 characteristics of a good workspace:
audibility: anyone should be able to easily hear and speak to the rest of the team
visibility: every team members should see each other
isolation: you should be able to discuss without disturbing the rest of the team

Taskboard : user stories stay in the In Progress column

Julian described a simple example: since development tasks stay in the In Progress column after the development has ended, let's add a Validation column.
After that, we see that a Review column would be useful, then a Test one, then Deploy...
Multiplying the columns in the taskboard shows again a malfunction in the team:

the need for multiple validation steps could indicate that the Product Owner is not present enough
there is no testing best practices
there is no team spirit, explaining why code reviews are so long to be done
there is no OPS profile, which makes deployment slower

This also has consequences on the Sprint proceeding:

there are a lot of ongoing user stories
stories come back and forth between columns(for instance In Progress -> Validation -> In Progress -> Validation...)
these are in fact disguised waiting lines
collaboration between team members is not encouraged, since no one makes the effort to take stories out of In Progress column

Julien proposed several options to solve this problem:

remove unnecessary columns. According to him, 4 columns are more than enough for most teams : Ready | Todo | In Progress | Done
limit the work in progress (like in Kanban method) to favor team spirit. Developers will have to help each other in order to add new stories, using pair programming, code reviews...
if possible, it is best to have teams where everyone can work on any topic, rather than having dedicated experts for each domain

Being late for the Morning Meeting

This is a classic issue that every Scrum team must have faced :-P
The bad idea is to postpone the meeting to a later time so that everyone can attend it. After a few weeks, you'd probably need to postpone it even later because of new delays of the teams members. By doing this, you're only treating the symptoms, not the problem itself. Delays are often a sign of lack of interest for this ceremony.
Again, Julien proposed several ways of solving this issue:

control the duration of the meeting by limiting each member's speaking time and postpone discussions after the meeting
change the format : instead of letting everyone speak successively, you can try to focus on each story, and everyone involved in this story can talk about what is meaningful to say, avoiding repetitions between persons. You can also try a more informal meeting, in front of a coffee for instance, but from experience this is not a very good alternative according to Julien.
change the organisation of the Sprint: try to have fewer objectives and stories that are coherent with them, limit the work in progress, encourage pair programming...
if the team really think there is no point in this meeting, you can even cancel it definitely, for instance when team members already know what the rest of the team is working on and have good communication during the day (especially for small teams)

Loss of attention during meeting

According to some studies, the average attention time for an adult is 52 minutes. So, long meetings have negative effects on the participants, as they can lose their focus, take their phones... Julien advises to ask the participants to disconnect so that everyone can stay focused.
As said previously, duration of the daily meeting should be limited
Sprint Planning has to be prepared by working on the Backlog. You should avoid having too technical discussions and focus on the business aspects, with fast estimation decisions.
Sprint Review must be prepared, practiced, so that it can be efficient, with as few slides as possible. Don't detail everything that has been done, tested or not...
During retrospective, it is better if you focus on one or two problems instead of solving them all.

Retrospectives where problemms are not raised or are too vague

Beware of the routine ! If few issues are raised, maybe it means that everything is alright, or maybe developers have forgotten the issues faced during the Sprint, or worse they think the retrospective is useless.

You should put the retrospective in context with what happened during the Sprint, by using adapted formats.

Assign a role to each member of the team
Define real actions (they should be SMART)
Evaluate the retrospective to know if the meeting was useful

Backlog with too many details

If the backlog becomes very long, with a lot of details, we are close to a waterfall methodology, because we will tend to follow the initial plan, losing the advantages of Agile. Moreover, if items are discarded or replaced with other stories because of what has already been achieved or new constraints, every business or technical analysis previously done is lost, which can be frustrating for the team.étude et les analyses techniques effectuées, ce qui peut être frustrant.
Julien's advice is to have a pyramidal backlog:

detailed stories for the next 2 or 3 iterations
less detailed features for the content of the current release
epics with few details for next release's features Backlog and user stories will be refined during iterations.

Roles are not clear

The most frequent example is the role of the Scrum Master, which can be difficult to understand for the team. This is a sign of an issue of the team's structure and it can lead to conflicts and tensions.
You should redefine the role to make it clear to everyone:

you can use the Stop / Start / Continue methode to know what is fine, what is wrong or what should be done
build your own Scrum Master : talk with the team to know what they think the role of the Scrum Master should be

Another role that can be unclear is the Technical Lead. If is omnipresent, it can lead to a decreased motivation of the developers because they will lack autonomy or learning...

Difference between Agile Coach and Scrum Master

The Agile Coach:

observs the team
speaks only with the Scrum Master
may lead a community of Scrum Masters
lead the ceremonies alongside the Scrum Master

The risk for the Scrum Master is losing his legitimity. The can indicate that efforts are only directed on organisation and management. In such a case, we can wonder if being Agile is enough.
Indeed, the technical part should not be forgotten:

you can add a Craft Coach to improve development practices
improving the design and the tests is also important
you also need to have better tools is delivery is an issue (DevOps)

Conclusion

You shouldn't blindly follow a methodology and its ceremonies, you absolutely have to adapt it to your team and to the project's context.
You need to measure the efficiency of the process, for instance the time required for a feature to change column on the taskboard, and review during the retrospective what can be improved according to these measures.

Simplifying our POJOs with Lombok

SCHREIBER Christophe — Sun, 30 Jul 2017 19:48:58 +0000

Everybody knows how to write POJO (Plain Old Java Objects). Moreover, our modern IDEs handle this repetitive and not so rewarding job by generating the boilerplate code such as accessors, constructors and often overriden methods such as toString, equals or hashCode.
By the way, which developer can tell he never had long debates with his team about how to write a satisfying equals or hashCode method, who never received a notification from his static code analysis tool about an inconsistency between the fields used in these methods' implementation ?
The Lombok project's goal is to make developer's life easier by handling this dirty work and cleaning the POJOs from all this boilerplate code that doesn't have much value and that, more importantly, loses the important information, the business code, in the middle of technical layers. How does this work in a nutshell : Lombok integrates with your favorite IDE and will generate this code automatically without showing it to you. You will only see fields definition, even though all accessors and "utility" methods will be available. This magic is performed by using one or more annotations.
We will also talk about some other repetitive and/or verbose tasks that Lombok can handle for us.

Let's start !

Installing Lombok is easy. You can either download its self-executable jar on the official site and execute it, or install its plugin in your IDE (I personnaly use IntelliJ, so I've used this solution), then add lombok.jar to your Java project's classpath. If you are using Maven to handle your build and dependencies, add the following dependency in your pom.xml :
<dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.16.8</version> <scope>provided</scope> </dependency>

Once we are ready to work, let's create a simple POJO :

As expected, the compiler is note very satisfied with our code... Let's just add Lombok's @data annotation to our class definition :

Problem solved ! But what is really interesting is that our POJO now has much more to offer :

We can quickly test the results :

We now have a pretty human readable toString method without having to write several lines of code !

Special needs

This was the simpler use case of Lombok, but I'm sure some of you are thinking that it's a bit limited and that it won't work for your special cases. Fear not ! Lombok allows you to customise everything it's generating.
Instead of using the @data to generate all the code, Lombok provides annotations for each of its features. For instance, if we want to expose accessors for only some fields, or with a specific visibility, we can achieve it by using the @Getter and @setter annotations and their parameters on the desired attributes.

Regarding the features we've seen earlier, we can annotate the class to generate the toString, equals and hashCode methods, and for each method we can specify the attributes to include or exclude for code generation :

We can also generate a private constructor and its associated factory :

Another interesting annotation provided by Lombok is @NonNull. It allows to automatically check that a parameter of a setter or a constructor is not null, and throw an exception otherwise :

Keeping things simple is important !

Lombok also contains more annotations, but I find them less interesting.
For instance, @CleanUp allows to automatically call the close methode of a variable defined inside a try/catch block. It can be useful for those who are still using Java 6 and can't enjoy the try with resourses introduced in JDK 7.
The @synchronized annotation allows to define locks. I personaly prefer defining locks using the standard synchronized keyword that every Java developer knows.
The last annotation I want to talk about is the trickies : @SneakyThrows allows throwing an unchecked exception inside the body of a methode without specifying it inside the method signature. It can mask potential errors to the developer and so can be very error prone, I don't recommend using it.

Final thoughts

As a conclusion, I think Lombok is an interesting tool to simplify the writing of the POJO. It is not a killer tool, but having less boilerplate code to maintain is a good thing to me, it helps making the business information more visible.
The technical annotations of Lombok are less interesting according to me, and they can even be quite dangerous from my point of view. Despites these few drawbacks, I'm ready to give Lombok a try inside my personal projects.

You can find the code snippets of this article on my my Github repository

Writing microbenchmarks with JMH

SCHREIBER Christophe — Thu, 27 Jul 2017 19:26:29 +0000

Today I'd like to talk about a subject that cares for me : performance. I'm going to give a quick overview of an aspect that it not often treated : micro-benchmarking. We will see how we can implement micro-benchmarking using the JMH tool.

Why using micro-benchmarking and why JMH ?

When we are coding, we often want to have a quick estimatin of the performance of an algorithm, a utility method or any component we are writing or using. What we often do is writing some simple code in a very basic main class, calling the test we want to inspect and measuring elasped time of the call using system clock. Unfortunately, this is a very unprecise approach, and the only case when it can be a sufficient indicator is when it clearly shows very poor performance. This kind of performance testing is too different from the context in which we want to execute our production code:

such single measure is very error prone and can't supply reliable statistics
simple tests with just a few executions can't enjoy the benefits of JVM optimization, such as the JIT (Just in Time) compiler that compiles frequently executed bytecode into native code to optimize execution durations. Such optimizations can only be applied after numerous calls, so that the JIT compiler can detect possible gains.
it is difficult to reproduce a production environment, with multithreading concurrent access and all the impact it can have on performance In order to adress these issues, the OpenJDK project has developed a tool dedicated to benchmarks for languages running on the JVM, called JMH. This framework aimes at executing automatically repeated executions of our code, simulating production context and collecting statistics about code performance.Of course, JMH can be used with the official JDK from Oracle, not only with an OpenJDK installation.

Project setup

A JMH Maven archetype is available, making it very easy to setup a project using the command line:

mvn archetype:generate -DinteractiveMode=false -DarchetypeGroupId=org.openjdk.jmh -DarchetypeArtifactId=jmh-java-benchmark-archetype -DgroupId=com.cs -DartifactId=poc-jmh -Dversion=1.0

The module will contain all necessary dependencies and Maven plugins, and a default MyBenchmark class will also be created. This class will contain en empty annotated method.
Like many other Java frameworks, JMH is using annotations. Benchmark classes are similar to the unit tests classes we are writing everyday (I hope you do ! ;-) )
Executing the benchmarks (meaning executing classes annotated with @Benchmark) requires building our Maven project with the well-known command:

mvn clean install
Then we just have to execute the generated jar file:
java -jar target/benchmarks.jar

This will launch a series of iterations (continuous execution of our code during a predefined duration) and will generate a report at the end.

Usage example

Let's consider a simple usecase everyone knows : strings concatenation inside a loop. We will compare the naive implementation using the + operator on String objects and the StringBuilder usage:

This very simple benchmark indicates the average operation numbers (execution of our benchmark code) for each method during the duration, and also other intesting statistics like min/max/average, standard deviation and the 99th percentile...

Handling more complex cases

After implementing a very simple JMH benchmark using default execution parameters, we will see how we can customize our benchmarks. As explained in the beginning of this article, JMH aims at providing executions close to the production context. This is achieved with some parameters tuning, like:

the number of iterations
the duration of each iteration
the benchmark type (we can for instance measure the number of executions during a time frame or the average execution time)
the number of thread to be used to simulate concurrent access
the number of JVM instances on which we want to execute our benchmarks, and their parameters
the number of warmup iterations to be executed in each scenario. These iterations don't appear in the computed statistics, but this is a very useful feature when we want to have an idea of the behaviour of our code once the JIT compiler will have optimized it This is not an exhaustive list of JMH features. If you want to go deeper, I invite you to have a look at the numerous example of its [official documentation]official documentation.

Conclusion

JMH is very useful and powerful tool to build, execute and monitor our code by using micro-benchmarks. This is a very interesting approach that completes the more traditional performance tests like load testing. It can give us very valuable data about some precise parts of our code.
Like unit tests, the more decoupled the code to be tested is, the simpler is the writing of the benchmarks, and results will be more accurate too !
You can find the sources of this article on my Github repository