DEV Community: Raúl Ávila

You should go to the Global Day of Coderetreat

Raúl Ávila — Fri, 29 Sep 2017 23:34:06 +0000

First of all, if you have attended a Global Day of Coderetreat before, you can make a better use of your time, so please stop reading here.

Every year, a global event takes place on a Saturday in dozens of locations around the world. During this day, lots of enthusiastic developers gather together to improve their craft and learn tons of things.

The format is actually quite simple: a challenge is proposed to the attendees at the beginning of the day, and they have to start implementing a solution. There are only three constraints: you must work in pairs, you have to use TDD, and you don't have much time to finish your implementation.

Maybe you're thinking how this can be possible...we have a whole day, haven't we? Well, that's true, but the day is organised in short sessions (45 minutes each). In each session, pairs rotate and they start implementing the solution again...from scratch. Also, you have to throw away all the code you just wrote in the previous session, yikes!

You can try different approaches every time you start over, and of course you can change languages, IDEs, etc. In fact you should try to use languages you don't know, specifically you should find people who know that language and pair with them. Pair Programming is one of the best ways to absorb knowledge very quickly, and you'll hardly find a place having such a diverse mix of skills and the will to share them.

I lied a bit before when I said there were only three constraints. The facilitators usually add different additional constraints in every session, because...well, because this way it's more fun :).

You may find that in some sessions you don't learn much but you teach new things to your pair. That will give you an amazing feeling too, and hey, sometimes teaching things is the best way to refresh what you know, challenge your assumptions...it's clearly a win-win.

If I haven't convinced you of attending yet, I'll give you one final reason: the atmosphere is fantastic, and I'll tell you why. The event happens on a Saturday, and it starts very early. Everybody in the room has woken up very early to be there on time, and rest assured that their intention will be getting the best experience and enjoy the day as much as possible.

I'm writing this post in 2017, and this year the event will be on November 18. So you have plenty of time to find the one closest to you. Of course I would be happy to read your thoughts in the comments section after your experience!

Comparing Spock and JUnit

Raúl Ávila — Wed, 31 May 2017 10:39:41 +0000

JUnit is considered the standard framework to test Java applications. I don't think anybody will challenge its position in the short term. In addition JUnit 5 will be released soon, and it will support the shiny Java 8 features, something that has been a bit of a handicap recently with JUnit 4.

However, a new testing framework is pushing hard. It was initially created to test Groovy code, but given that Groovy runs on the JVM it can be used to test Java code too!

Spock

Spock is an acronym for "Specification and Mock". I guess now it's easy to figure out what it allows us to do: creating specifications of our systems, adding capabilities to create mocks of our dependencies. I say specifications and not tests, something that can be a bit confusing at the beginning, but hopefully will be well understood with the examples. In few words, a specification is just a test class on steroids.

My intention with this post is showing in a clear example the main differences between Spock and JUnit. I don't want to go into too many details about what Spock has to offer, you have the official documentation for that.

The class we need to test

We're going to create a naive class called Calculator, that we want to test:

public class Calculator {

    private Audit audit;

    public Calculator(Audit audit) {
        this.audit = audit;
    }

    public long add(int operand1, int operand2, Mode mode) {
        audit.register(String.format("%d + %d (%s)",
                        operand1,
                        operand2,
                        mode));

        if (mode == Mode.ABSOLUTE) {
            operand1 = Math.abs(operand1);
            operand2 = Math.abs(operand2);
        }

        return operand1 + operand2;
    }

    public static enum Mode {ABSOLUTE, STRAIGHT;}
}

The features of this class (which is not a great example of best practices, in any case) are clear.

Tests in JUnit

This would be the JUnit class that would cover all the features in Calculator:

@RunWith(Parameterized.class)
public class CalculatorTest {

    @Mock
    private Audit audit;

    @InjectMocks
    private Calculator calculator;

    @Before
    public void setUp() throws Exception {
        MockitoAnnotations.initMocks(this);
    }

    private int operand1;
    private int operand2;
    private long expectedResultStraight;
    private long expectedResultAbsolute;

    @Parameterized.Parameters
    public static Collection data() {
        Object[][] data = new Object[][] {
                { 2, 2, 4, 4 },
                { -2, 2, 0, 4 },
                { -3, -3, -6, 6 },
                { 0, 0, 0, 0 }
        };

        return Arrays.asList(data);
    }

    public CalculatorTest(int operand1,
                          int operand2,
                          long expectedResultStraight,
                          long expectedResultAbsolute) {

        this.operand1 = operand1;
        this.operand2 = operand2;
        this.expectedResultStraight = expectedResultStraight;
        this.expectedResultAbsolute = expectedResultAbsolute;
    }


    @Test
    public void testAddStraight() throws Exception {
        long sum = calculator.add(operand1, operand2, STRAIGHT);
        assertThat(sum, is(expectedResultStraight));

        verify(audit).register(
                String.format("%d + %d (STRAIGHT)", operand1, operand2));
    }

    @Test
    public void testAddAbsolute() throws Exception {
        long sum = calculator.add(operand1, operand2, ABSOLUTE);
        assertThat(sum, is(expectedResultAbsolute));

        verify(audit).register(
                String.format("%d + %d (ABSOLUTE)", operand1, operand2));
    }

}

I see several inconveniences here:

We're adding two libraries on top of JUnit to improve our test code: Hamcrest, to improve readability, and Mockito, to create mocks of our dependencies.
In order to use Data Driven Testing appropriately we need our tests to be run by the Parameterized runner, instead of the JUnit default runner.
The quantity of boilerplate code that we need to write out of our tests methods is considerable. We need to prepare a data matrix using a certainly ugly approach (in the data method), we need a setUp method to initialize dependencies, a constructor to initialize the data in every test, etc.

All this issues are not bad by themselves, we all know that Java is not popular for being a concise language. Its verbosity can be good in some occasions actually, because it forces good practices and thinking carefully about our designs. But when we need to write tests, wouldn't it be better to skip some of these problems?

Spock to the rescue

Let's see how we would implement our specification for Calculator in Spock:

class CalculatorSpec extends Specification {

    Audit audit = Mock()

    @Subject
    Calculator calculator = new Calculator(audit)

    def "Calculator can add operands in straight mode"() {
        when: "We add two operands in straight mode"
        long sum = calculator.add(operand1, operand2, STRAIGHT)

        then: "The result of the sum matches the expected one"
        sum == expectedResult

        where:
        operand1 | operand2 || expectedResult
        2        | 2        || 4
        -2       | 2        || 0
        -3       | -3       || -6
    }

    def "Calculator can add operands in absolute mode"() {
        when: "We add two operands in absolute mode"
        long sum = calculator.add(operand1, operand2, ABSOLUTE)

        then: "The result of the sum matches the expected one"
        sum == expectedResult

        where:
        operand1 | operand2 || expectedResult
        2        | 2        || 4
        -2       | 2        || 4
        -3       | -3       || 6
    }

    def "Audit object intercepts all calls to the Calculator"() {
        when: "We add two operands in any mode"
        calculator.add(2, 2, ABSOLUTE)
        calculator.add(2, 2, STRAIGHT)

        then: "The Audit object registers the call"
        1 * audit.register("2 + 2 (ABSOLUTE)")
        1 * audit.register("2 + 2 (STRAIGHT)")
    }
}

We can see that:

The tests are written in Groovy.
The name of the test class has the Spec suffix and extends Specification. This names actually means that our class is not only testing code, it's actually generating specifications readable by a human being.
We can create a mock invoking the method Mock, which is part of the framework.
The Subject annotation tells us which class we're testing / describing
Our methods can be named using a String without any restriction. This means we don't need to follow any convention given by programming languages, so it's much easier to describe what we're going to test without being afraid of creating long and unreadable method names.
The test methods are clearly divided in sections, following the Arrange Act Assert pattern or "Given When Then". Our example is so simple that we don't need to do anything in the given: section, so we skip it.
Every section can have its own description, so we can explain exactly what we're preparing, running or verifying there.
The then: section contains assertions which will be verified using Groovy Truth. We don't need to use assert methods, just boolean expressions.
The verifications using mocks are very clear: 1 * audit.register("2 + 2 (ABSOLUTE)") verifies that we're invoking the register method in audit one single time with those parameters, and it will fail if that doesn't happen. Simulating behaviors with mocks is very simple too, take a look at the official documentation for further details.
The data matrices are very concise, as we can see in where: sections. No need to create two dimensional arrays anymore.

Finally, let's see how the framework shows assertion errors, as it does it in a graphical way, where we can see the error, the values of all the variables involved, etc. Here's an example:

Condition not satisfied:

sum == expectedResult
|   |  |
4   |  5
    false

Spock vs JUnit

I'm not trying to convince anybody here about using one framework against the other, just wanted to compare the differences between both. The benefits of using Spock are clear, but it's true that it forces us to add a new language to our technology stack, for example. In my experience, companies are quite reluctant to do this, as it makes hiring a bit more complicated. I don't think learning Groovy and Spock if you're a Java developer is very challenging, anyway.

To use Spock in a Maven project it's necessary to configure our POM file properly (more info here). It's of course easier if we use Gradle.

What is an Agile Developer?

Raúl Ávila — Wed, 24 May 2017 11:22:07 +0000

I guess nobody will be surprised if I say that everybody is talking about Agile these days. The topic is even reaching another dimension with the grandiose name of "Digital Transformation". Therefore, I think it won't be necessary to explain in this post what Agile development is.

It's not hard to find job offers where companies are trying to find "Agile Developers". And here's when I think that this agile madness starts to be a bit ridiculous. From my perspective, I don't think most of the companies publishing this kind of offer knows exactly the skills that an Agile Developer must have. And I'm saying this because I've barely seen mentions to what I believe are the main skills a developer working with agile needs.

Kent Beck said in Quora that coining "Agile" was probably a mistake when a group of professionals decided to use this name almost two decades ago. What has happened over time is that big enterprises have started to confuse "Agile" with "Fast", so "Agile Developer" is confused with "Fast Developer". Indeed, companies just want people being able to deliver more features in less time, without having in mind the skills that such developer must have. It is frequently thought that following Scrum ceremonies will be enough to reach this higher speed magically.

In my opinion, what defines Agile Development is sustainable pace and constant feedback. In order to achieve both things we need to improve communication and achieve maximum visibility of the work done, but most importantly, we need to have a technical excellence in our development process.

It is with this technical excellence how we achieve, in the long run, to increase the number of features delivered per time unit. Indeed, I'm talking about advocating for the maximum quality in what we do.

Having said this, what are the main skills that a good agile developer must have? I think we just need to master two things:

Testing
Refactoring

If a developer is able to deliver software perfectly covered and described by tests, such software won't have any regression in the future (or it will have a very small one), so making the system grow will be much easier, without a notorious decrement in velocity. Having these tests as a safety net will allow us to shape our code using refactoring without being afraid to break anything. And it's this constant refactoring, the adaptation of our code to make it more readable and more maintainable what actually has a huge impact in the quality of the software we deliver.

What's more important, testing or refactoring? I think none is more important than the other, both need each other to create a virtuous circle. Maybe you're thinking about TDD as the best way to accomplish this circle. Personally I'm a big advocate of TDD, and I find it extremely useful, but lastly I don't think is that important to use it as long as we don't stop adding tests and refactoring all the time. This is the only way to achieve the sustainable pace that I mentioned above.

Conclusion: if, in the future, you come across a job offer where somebody is looking for "Agile Developers", but the offer does not mention testing and refactoring as important skills, be suspicious of that offer :).

Don't leave broken windows

Raúl Ávila — Mon, 10 Apr 2017 19:30:07 +0000

“Don’t leave broken windows”. The first time I read this phrase, in the seminal book The Pragmatic Programmer, was probably the most important aha moment of my career. Do you know what it means?

It is (scientifically?) demonstrated that, when a building is abandoned, it can resist a long period of time without people noticing it. But there is a moment when a crucial event happens…that changes everything. This event is a broken window. The broken window won’t be fixed, because the building is not maintained by anybody, so in a short period of time everybody will know about this situation, vandals will start ruining new windows, breaking into the building, and it will be a complete mess very, very soon.

The same idea can be applied to Software development. When we take certain shortcuts to deliver something in the shortest period of time, and our code reflects how careless we’ve been, developers coming after us (from the same team, a future team, and even ourselves!), will reach an important conclusion: it’s not important to pay enough attention to the code we produce. Little by little our application will start deteriorating in what it becomes an unstoppable process.

Levels of broken windows

Not everything is black or white, while there are certain issues that must be addressed immediately, some others can be considered later. Following the metaphor of the building, it’s not the same to have a broken window in the first floor or in the 20th floor of a skyscraper. This is why I think there are two levels of broken windows:

Stop the world: you have absolutely no excuse to fix this. Not doing it will have serious consequences sooner rather than later.
Next iteration: assuming we work following an agile methodology, this problem is something that should be addressed as soon as we begin the next iteration / sprint. In order to remember it, we need to add it as an action to the tool we use to track our work (Trello, Pivotal Tracker, JIRA…).

Examples

I’ll review a series of problems that I’ve been seeing over my last few years working as a developer, and which can be classified clearly as broken windows.

README file

Having a README file in our codebase is essential. Sometimes people refers to the Agile Manifesto, which says we should favour "Working Software over comprehensive documentation”, and they translate it as “We don’t need to document”.

This is atrocious. Please, from day one, write and maintain a README file containing basic information such as:

Brief description of our application (two or three lines, what it’s doing and what problem it solves).
How to build the project.
How and when to deploy the project.
How to test the code automatically (and, if necessary - I hope it’s not - manually).
What dependencies our system has (database, message broker, etc), and how to connect to them.
Versioning strategy.
Conventions to follow (commit messages format, coding style guide, etc).

I’m probably missing some things. The idea is that, if somebody joining our project is not able to ramp up quickly without a lot of support from the rest of the developers (a minimum support will be always required, of course!), it’s because something is missing in our README file.

Too much documentation (and outdated)

It’s the counterpoint to the previous example. In the first stages of the project we may write tons of documentation, containing requirements, design, etc, in a Wiki, Word document, etc, and as the workload increases we don’t maintain those documents, which will be automatically transformed in a source of confusion.

In general, I think that it’s important to have a document containing the high level overview of the architecture of our system, but the implementation details must be in just one place: the code. Every time you start writing a new document that you know will take some time, think about the value you’re adding and the “time debt” you’re acquiring. A “time debt” is nothing but a concrete action that will require additional time from us in the future. You can read more about this here.

Dead Code

This is code that remains in our codebase but it's not invoked by anybody, therefore it's useless. Many people decide to leave that code "just in case I need it in the future". I did this a lot in the past, and the truth is that I never saw myself recovering this code. In addition, there's something called source control :), which facilitates enormously the work to recover code implemented some time ago.

With the current tools it's very simple to detect dead code, and removing it is super simple. So no excuses.

Â Commented out code

Being a subset of "dead code", this is even more aberrant. Please, never ever leave commented out code in your software.

Â Ignored tests

These are tests that failed one day, and in order to continue doing what we were doing without breaking our CI environment, we added a pretty @Ignore without clearly specifying the reason. In the future, when somebody sees the @Ignore, it can happen that he removes it, and after seeing that the test fails the annotation is added again. So there it remains, forever.

I only see a case where this can be reasonable, and it's when we want to push code "in progress" while we work in a new feature. This code won't be exposed to the final user but it will be integrated in master, and the new feature won't be activated until the test passes. Any other case is just a broken window.

Several dependencies to do the same thing

A developer decides to add a library to do a certain task, let's say working with JSON, and she chooses Jackson.

Later on, another developer who need to work with JSON, decides to add GSON. So we have two libraries to do the same thing!

This transmits a clear message, lack of communication within the team. To avoid this kind of problem, it's very important to carry out Code Reviews. If it's too late, and both libraries are living together in the same project, we should choose one and replace the other for consistency as soon as possible.

Inconsistent code formatting

Who hasn't seen code like this?

public class DishImpl implements Dish {

    private final String name;
    private boolean mixed = false;
    private boolean cooked = false;
    private final List<Ingredient> ingredientList = Lists.newArrayList();

    public DishImpl(String name)
    {
        this.name = name;
    }

    public void add_ingredient(Ingredient ingredient)
    {
      Validate.notNull(ingredient);

      System.out.printf("%s - Adding ingredient %s%n",
                        name,
                        ingredient.getName());
      ingredientList.add(ingredient);
    }

    public void mix()
    {
        if(ingredientList.isEmpty())
        {
            throw new IllegalStateException(
                "There are no ingredients to mix");
        }

        System.out.printf("%s - Mixing ingredients: %s%n",
                            name,
                            ingredientList.toString());

        mixed = true;
    }

    public void cook() {
        if(!mixed)
            throw new IllegalStateException(
                "Ingredients are not mixed, please call mix first");

        System.out.printf("%s - Cooking...%n",
                            name);

        cooked = true;
    }

    public void serve()
    {
      if(!cooked)
        throw new IllegalStateException(
            "Dish is not cooked");

      System.out.printf("%s - Serving...%n", name);

    }
}

(I know, the code is quite stupid)

We see several inconsistencies in this code, like two different ways of using the curly braces (new line in some places, same line in others), two different ways to indent the code, two different approaches to organize if statements with a single instruction (using curly braces vs. not using curly braces), different naming conventions (camel case vs. snake case), etc.

It's extremely important that all developers agree in the style guide to follow, and respect it. My opinion on this is that you should never know who wrote a particular piece of code based on its style.

It's quite easy to find good style guides on the Internet, for example the one from Google.

Â Insufficient test coverage

If you practice TDD this shouldn't be a big issue, but in case you don't it can happen that our tests only cover 60-70% of our production code (and even less than that!!).

In the acceptance criteria of our user stories it should be clear that a story won't be delivered if the test coverage decreases. This test coverage should never be below the threshold 90-95%. And if this happens, we should analyze the reasons (too much pressure maybe or just slopiness?).

In any case, the test coverage shouldn't be a goal by itself, but a measure of quality, so please take care with this!

Â Manual actions in deployments

The deployment of a new version of our project should be 100% automated. Ideally we should have a task in our CI environment that performs all the necessary actions, and that only requires to push a button from us. Any additional action is just a waste of time that should be analyzed for automation.

Â Unstable CI

When our CI environment is in read more time than in green, it's because we're doing something wrong. With this I mean that 95% of the times our tasks (in Jenkins, Bamboo, Concourse, etc) should be executed successfully. If this doesn't happen we have to stop anything we're doing to make it green again.

Â Unused imports

This is very typical and easy to avoid. We add an import statement to our class file because we need it at a certain point in time. Later on, we remove the code related to that import (because we refactor the code and we move it somewhere else, for example), leaving the import, which is now completely useless. It's not rare to see a block of 25 imports where only 5 or 6 are actually used.

Removing unused imports is quite simple in the current IDEs. There are shortcuts that can do it instantly, and there's even the option to activate some options that permanently scan our code and remove the unused imports for us.

Unused dependencies

While the previous example can probably be included in the second category ("next iteration"), having unused dependencies in our code that are not used anymore opens the door to potential problems that can be incredibly hard to detect, due to conflicts with transitive dependencies, etc.

As soon as you stop using a dependency, remove it from your project!

Â Unclear Git strategy

Without having clear guidelines of how to work with Git, every developer can make his / her own decisions, and we may find ourselves with a Git repository full of branches (many of them dead after being merged or rebased into master), lots of merges, etc.

I'm not going to propose an ideal solution to work with Git here. There are several strategies (aka workflows), but in any case you should leave the decision to every individual in your team.

Dirty Code

We all know what dirty code is, and as professionals we should never deliver features implemented in an unsustainable way, making the maintenance extremely hard due to unreadable code. If you haven't done it yet, please read Clean Code or Code Complete to know exactly what I'm talking about.

This is not Technical Debt

In this post I haven’t talked about Technical Debt, which is a completely different thing. I’m talking about being professionals and taking care of our work! So please don’t forget this mantra every time you sit in front of a screen: “Don’t leave broken windows”.

Multithreading in Java for dummies (part 3)

Raúl Ávila — Mon, 09 Jan 2017 08:50:09 +0000

This will be the last post of this series about multithreading. In the second post, we left our Ping Pong game in a good position, with the threads (players) being blocked while waiting for their turn, and with the possibility of scaling up the number of players as desired. However, there was something that wasn't very elegant, the class Game was in charge of creating, starting and finishing the threads, as well as synchronization with them to avoid the main thread from finishing before the players. Let's recap:

//...
Thread thread2 = new Thread(player2);
thread2.start();
Thread thread1 = new Thread(player1);
thread1.start();

//Let the players play!
try {
   Thread.sleep(2);
} catch (InterruptedException e) {
   e.printStackTrace();
}

//Tell the players to stop
thread1.interrupt();
thread2.interrupt();

//Wait until players finish
try {
   thread1.join();
   thread2.join();
} catch (InterruptedException e) {
   e.printStackTrace();
}
//...

Thread management / Thread pools

Apart from being tedious, if we don't encapsulate the creation of threads conveniently, our code won't be very cohesive, because we're coupling the logic of the game itself to the concurrency management. In addition, creating threads is expensive, and in complex applications this will have an important cost in terms of performance.

The Java concurrency API exports a series of classes and interfaces allowing us to encapsulate the thread creation management with high flexibility: the Executor framework. Its three main elements are:

Executor: it's an interface with a single method, execute(Runnable). The idea with this framework is that we handle tasks instead of threads, so we're asking the instance of Executor to run the task (which is an instance of Runnable) when possible
ExecutorService: interface extending Executor, which publishes a series of more advanced methods, to control the whole lifecycle of the work to be done (shutdown, awaitTermination), and to run tasks of the type Callable, which are just a Runnable returning a value (more info here). In the official documentation you can read more about all the possibilities of this interface
Executors: the two previous components are interfaces, and we can create our own implementations if we want. However, the most common use cases are already implemented in the JDK. To use these implementations we need to request an instance using the static factory methods exposed by Executors

In general, the term "Thread Pool" is used to refer to the implementations of Executor/ExecutorService that we use to handle our threads. The most common types, which can be obtained invoking the static methods exposed by Executors are:

Single Thread Executor (newSingleThreadExecutor): it contains a single thread, and it's not widely used
Fixed Thread Pool (newFixedThreadPool): it contains a constant number of "alive" threads, which are waiting to receive tasks to run
Cached Thread Pool (newCachedThreadPoool): it maintains a thread pool that can grow or shrink depending on the demand
Scheduled Thread Pool (newScheduledThreadPool): it's used to scheduled the execution of tasks. It returns an instance of ScheduledExecutorService, given that ExecutorService doesn't expose the appropriate methods to schedule future tasks, only to execute them as soon as possible

Ping Pong, version 5: Thread Pool

Without modifying the class Player, we can adapt our class Game to use a thread pool instead of taking charge itself of the tedious tasks of creating, starting and stopping the threads. Let's see how:

public class Game {

    public static void main(String[] args) {
        Lock lock = new ReentrantLock();

        Player player1 = new Player("ping", lock);
        Player player2 = new Player("pong", lock);

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setPlay(true);

        ExecutorService executor = Executors.newFixedThreadPool(2);

        executor.execute(player1);
        executor.execute(player2);

        sleep(2);

        executor.shutdownNow();

        System.out.println("Game finished!");
    }

    public static void sleep(long ms) {
        try {
            Thread.sleep(ms);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

We're using a pool containing two threads (one per player), and we send to it the tasks to run. We sleep the main thread for 2ms, so both players can play for a while, and we invoke the method shutdownNow(), which is equivalent to interrupting the threads as we did in previous versions, but encapsulating all the logic in the pool. It's necessary to invoke shutdownNow instead of shutdown, because the latter waits until the running tasks finish, returning a list with the pending tasks. Our players play indefinitely until they're interrupted, so if we try to finish with shutdown, the application will never end!

Well, if we run this version several times, we'll see that it works as expected some times, while others it shows an output like this:

Game starting...!
ping
pong
//...
Game finished!
pong

What happened? After requesting both threads to interrupt, it's possible that the main thread finishes before the rest of the threads. This is why the text "Game finished!" appears before the last "pong" turn. Exploring the ExecutorService API, we can see a method called awaitTermination. This method blocks the thread who invokes it until all the tasks in the pool have finished, or until a timeout passed by parameter expires:

//...
ExecutorService executor = Executors.newFixedThreadPool(2);

executor.execute(player1);
executor.execute(player2);

sleep(2);

executor.shutdownNow();

try {
    executor.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
    System.out.println("Main thread interrupted while waiting for players to finish");
}

System.out.println("Game finished!");
//...

We finally get the expected output, and the game behaves as expected, with a cleaner and more readable main class. Have we finished? Not yet!

Barriers

Entry / Exit barriers are synchronization mechanisms facilitating the simultaneous execution of a thread group (entry barrier), or the waiting until a whole thread group finalizes.

We have seen the idea of exit barrier before in this post, with awaitTermination. This method allows us to create an exit barrier, but it forces us to set up a timeout (it can be hours, but it's a timeout!). We would like to create an exit barrier without a timeout.

In order to understand what an entry barrier is, we're going to add an instruction to Game:

//...
executor.execute(player1);
sleep(1000);
executor.execute(player2);
//...

We send the main thread to sleep for a second before sending the second player to play. Even though the result is hard to reproduce here, because it's related with the timing of the application, something like this happens:

Game starting...!
ping
// Waiting 1 second
pong

That is, the player "ping" plays, but during a second there isn't anybody to play with! So the game is "suspended" for a second, which could be minutes (the time the main thread takes to start the second thread/player). This situation is not ideal, because we're starting a concurrent process requiring the presence of several threads before all of them are ready. To avoid this we need an entry barrier.

There are several classes in the Concurrency API that can be used as barriers, but the simplest one, and the one we'll use as entry and exit barrier is CountdownLatch. The usage of this class can be summarized in three points:

We create a barrier with a counter initialized to N
The threads that must continue depending on the barrier will invoke await(), and will be blocked until the counter reaches zero. There's an await method with timeout too
The actors that can influence the opening of the barrier will invoke countDown() when necessary conditions are met. In general, N conditions must be met for the barrier to be opened

Version 6: Entry / Exit barriers

In this new version we'll have to modify both Game and Player. Let's see how they would look like:

public class Player implements Runnable {

    private final String text;

    private final Lock lock;
    private final Condition myTurn;

    private final CountDownLatch entryBarrier;
    private final CountDownLatch exitBarrier;

    private Condition nextTurn;

    private Player nextPlayer;

    private volatile boolean play = false;

    public Player(String text,
                  Lock lock,
                  CountDownLatch entryBarrier,
                  CountDownLatch exitBarrier) {
        this.text = text;
        this.lock = lock;
        this.myTurn = lock.newCondition();

        this.entryBarrier = entryBarrier;
        this.exitBarrier = exitBarrier;
    }

    @Override
    public void run() {
        if(entryBarrierOpen())
            play();
    }

    public boolean entryBarrierOpen() {
        try {
            entryBarrier.await();
            return true;
        } catch (InterruptedException e) {
            System.out.println("Player "+text+
                                " was interrupted before starting Game!");
            return false;
        }
    }

    private void play() {
        while (!Thread.interrupted()) {
            lock.lock();

            try {
                while (!play)
                    myTurn.awaitUninterruptibly();

                System.out.println(text);

                this.play = false;
                nextPlayer.play = true;

                nextTurn.signal();
            } finally {
                lock.unlock();
            }
        }

        exitBarrier.countDown();
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
        this.nextTurn = nextPlayer.myTurn;
    }

    public void setPlay(boolean play) {
        this.play = play;
    }
}

The class doesn't start until the entry barrier is open, and when it's interrupted to finish, it invokes countDown on the exit barrier, which will be how Game knows that both players are done.

Think for a second the values we have to initialize the counters in entryBarrier and exitBarrier before you keep reading...

public class Game {

    public static void main(String[] args) {
        CountDownLatch entryBarrier = new CountDownLatch(1);
        CountDownLatch exitBarrier = new CountDownLatch(2);

        Lock lock = new ReentrantLock();

        Player player1 = new Player("ping", lock, entryBarrier, exitBarrier);
        Player player2 = new Player("pong", lock, entryBarrier, exitBarrier);

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setPlay(true);

        ExecutorService executor = Executors.newFixedThreadPool(2);

        executor.execute(player1);

        sleep(1000);

        executor.execute(player2);

        entryBarrier.countDown();

        sleep(2);

        executor.shutdownNow();

        try {
            exitBarrier.await();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }

    public static void sleep(long ms) {
        try {
            Thread.sleep(ms);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Indeed, the entry barrier has a counter set to 1, because it will be opened as soon as the main thread has passed all the tasks to the thread pool, while the exit barrier, which here it's used as an alternative to awaitTermination, has a counter set to 2, which is the number of actors that must finish the execution before the main thread can progress.

This way, the timing in our application is the desired one, although we have complicated the code a bit. The thing is that concurrency itself is quite complex, so it's hard to encapsulate perfectly all the mechanisms used.

Before finishing the post, I would like to mention that the exit barrier has been added to this version with didactic goals. The best mechanism to wait for the finalization of a group of threads is through awaitTermination, using a reasonable timeout, so if we reach this timeout it will be because a failure has happened in one of the tasks we're waiting for its termination. I have added a version 7 to GitHub, which uses an entry barrier, and awaitTermination as exit barrier. This version could be considered the optimal version of the application.

I hope this series of posts has been helpful to clarify many of the concepts used in concurrent applications implemented with Java. If you want to learn more, the best book ever bout this topic is Java Concurrency In Practice by Brian Goetz.

(All the code can be found in this GitHub repository)

Multithreading in Java for dummies (part 2)

Raúl Ávila — Mon, 09 Jan 2017 07:31:23 +0000

In the first post of the series we saw some basic multithreading concepts using a practical approach. With the implementation of a Ping Pong game in mind, we'll continue introducing new improvements that we'll use to explain some additional concepts that everybody should know when implementing concurrent applications in Java.

The starting point of this post will be one of the last versions we discussed in the first post:

public class Player implements Runnable {

    private final String text;

    private Player nextPlayer;

    private volatile boolean mustPlay = false;

    public Player(String text) {
        this.text = text;
    }

    @Override
    public void run() {
        while(!Thread.interrupted()) {
            while (!mustPlay);

            System.out.println(text);

            this.mustPlay = false;
            nextPlayer.mustPlay = true;

        }
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
    }

    public void setMustPlay(boolean mustPlay) {
        this.mustPlay = mustPlay;
    }
}

This version is quite horrible actually. We can never justify doing something like this in our code:

while(!mustPlay);

Busy waiting

This statement is an example of Busy Waiting, and it's nothing but an infinite check of a condition, avoiding the progress of the app until the condition is true. The problem of this approach is that our thread is overloading the CPU, because the Thread Scheduler does not detect anything blocking the execution of such thread, so there will be always resources keeping the thread in "Running" state (you can find a good thread state diagram here). The result is an excessive and unjustified use of resources.

I'll tell you a fun story about this. When I implemented the sample code for this series, I left my IDE open with the application running (and the busy waiting, of course). The result was that my battery, which usually lasts between 6 and 8 hours was drained in less than 2. Let's think about the consequences of this kind of faulty design in serious enterprise applications!

Locking

The easiest way to get rid of this busy waiting is using Locks. In a few words locking is a mechanism that allows us to implement exclusion policies in concurrent applications when there are resources which state can be shared an modified by different threads.

This state which can be modified by more than one thread must be protected using a critical section. Java offers different mechanisms to implement critical sections, and we'll see the most important ones in this post.

Version 3: Intrinsic Locking

The oldest mechanism in Java to create critical sections is known as Intrinsic Locking, or Monitor Locking. Each object created in Java has a lock associated (intrinsic lock or monitor lock), which can be used with exclusion goals in our threads through the use of the keyword synchronized:

//...
Object myObject = new Object();
//...
synchronized(myObject) {
    //critical section
}

In this example we're using an instance of Object as lock, so that every thread that wants to access the critical section must obtain the lock, which is what we attempt to do in the synchronized statement. If the lock is available, the thread obtains it and it won't be available for any other thread. If a new thread attempts to obtain the lock, it will fail, and its state will be set to "Blocked" by the Thread Scheduler.

Internet is full of examples about the use of synchronized, so I won't get into many details here in relation to best practices. I'll just add a few points to consider:

It's quite usual to synchronize on this (synchronized(this)), so the own instance is using itself as lock to protect its clients from concurrency problems. However, we must be very careful if we do this because our clients could synchronize in the same instance, causing a DeadLock
A better practice would be using a private lock (like the one we used in the code snippet above). This way we're not exposing the locking mechanism used to the outside world, because it is encapsulated in the own class
synchronized has another goal apart from exclusion , and it's visibility. In the same way that keyword volatile ensures the immediate visibility of the modified variable, synchronized ensures the visibility of the state of the object used as lock (so the scope if bigger). This visibility is guaranteed by the Java Memory Model

Waiting mechanisms

If we use locking mechanisms only we won't be able to remove the busy waiting completely in our application. We need something else, and this is known as waiting mechanisms.

Every object exposes a method, wait(). When this method is invoked by a thread it makes the Thread Scheduler to suspend it, changing its state to "Waiting", i.e.:

//the thread state at this point is Running
i++
lock.wait(); // => thread state changes to Waiting

This example is a bit flaky, because we should never invoke wait this way. The appropriate idiom when implementing waiting mechanisms is:

synchronized (lock) {
    try {
        while (!condition)
            lock.wait();

        //Execute code after waiting for condition

    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
    }

In the code we see that:

It's necessary to acquire the lock on the object we want to invoke wait
That wait implies that we're waiting for "something". That something is a condition (condition predicate), which can be true before having to wait. Therefore, we check that condition before invoking wait
The waiting is done in a loop and not in an if sentence for several reasons. The most important one is known as "spurious wakeups". From its name it's easy to deduce what it is, sometimes a thread wakes up from "Waiting" state without anybody asking it to do it, so it can happen that the condition is not true yet and it must wait again
Last, but not least, wait throws InterruptedException, which we handle as discussed in the first part of this series

Having seen this, we have a thread changing to "Waiting" state expecting a condition to be true, but somebody should inform that one or more threads should wake up...Well, this is done through the methods notify and notifyAll, which, as you have probably deduced, asks to one or all the threads waiting on a lock to wake up and check the condition. The idiom is:

synchronized(lock) {
    //....
    condition = true;
    lock.notifyAll(); //or lock.notify();
}

Again, we need to be in possession of the lock to invoke the methods on the object. There are tons of articles written about the use of notify and notifyAll, it depends on the intention of each application. Precisely, the usage of notifyAllis one of the reasons why waiting for the condition to be true is done in a loop, in occasions just a single thread from all the threads waiting can progress when the condition is true.

Let's see finally how our Ping Pong app would look like after applying all the concepts we have seen:

public class Player implements Runnable {

    private final String text;

    private final Object lock;

    private Player nextPlayer;

    private volatile boolean play = false;

    public Player(String text,
                  Object lock) {
        this.text = text;
        this.lock = lock;
    }

    @Override
    public void run() {
        while(!Thread.interrupted()) {
            synchronized (lock) {
                try {
                    while (!play)
                        lock.wait();

                    System.out.println(text);

                    this.play = false;
                    nextPlayer.play = true;

                    lock.notifyAll();

                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        }
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
    }

    public void setPlay(boolean play) {
        this.play = play;
    }
}

The lock in this app could be considered the ball in the game, which in every turn can be in possession of one player only. We also see that, after printing the text in the standard output, the player notifies the other one that he can play. I have used notifyAll, but it could be notify too.

The main class doesn't vary too much over the last version of the first post in the series:

public class Game {

    public static void main(String[] args) {

        Object lock = new Object();

        Player player1 = new Player("ping", lock);
        Player player2 = new Player("pong", lock);

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setPlay(true);

        Thread thread2 = new Thread(player2);
        thread2.start();
        Thread thread1 = new Thread(player1);
        thread1.start();

        //Let the players play!
        try {
            Thread.sleep(2);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //Tell the players to stop
        thread1.interrupt();
        thread2.interrupt();

        //Wait until players finish
        try {
            thread1.join();
            thread2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }

}

Version 4: Explicit locks and conditions

Java exposes in its concurrency API an interface, Lock, which allows us to implement the same exclusion mechanisms that we have seen before using intrinsic locks, but through a different approach.

The main implementation of Lock is ReentrantLock. It has this name because the locks in Java are reentrant. This means that, once a lock is acquired by a thread, if the same thread attempts to acquire the same lock it succeeds. We're going to implement the same examples seen above using this API.

Critical sections

Lock lock = new ReentrantLock();
//...
lock.lock();
try {
    //critical section...
} finally {
    lock.unlock();
}

Easy, we only have to bear in mind that we must invoke the method unlock in the finally clause to ensure that the lock is released even in case of error.

Personally, I wouldn't say that this mechanism is better than the one offered by synchronized, as the latter is more compact. The biggest advantage of using Lock is that it comes with a bunch of methods enabling the implementation of more complex locking policies, like:

tryLock(): we try to acquire the lock, but the thread is not blocked if it doesn't succeed
fairness: we can create a lock as "fair". By default locks in Java are not fair, so a thread waiting can be chosen to acquire the lock even though is the last one to arrive. With a fair lock a FIFO locking will be implemented

I would advice you to have a look at the API, for further info.

Waiting mechanisms

The implementation of these mechanisms is done using the class Condition. The creation of a Condition instance must be done from a Lock:

Condition condition = lock.newCondition();

The class Condition exposes two methods, await() and signal(), with are kind of equivalent to wait() and notify() in intrinsic locks. Moreover, we can use methods such as:

await(long time, TimeUnit unit): it waits for a condition a maximum time
awaitUninterruptibly(): non-interruptible version of await(). This means that, if the tread that is suspended waiting for a condition is interrupted, this method won't throw the well known InterruptedException. The only way to activate it is through signal() / signalAll() on the condition (spurious wakeups apart)

In general, for waiting mechanisms I would say that the usage of Condition offers a series of very interesting features. In addition, it allows us to create different conditions associated to the same lock, which is not possible with intrinsic locks.

Let's see the aspect of our applications after adding Lock and Condition API:

public class Player implements Runnable {

    private final String text;

    private final Lock lock;
    private final Condition myTurn;
    private Condition nextTurn;

    private Player nextPlayer;

    private volatile boolean play = false;

    public Player(String text,
                  Lock lock) {
        this.text = text;
        this.lock = lock;
        this.myTurn = lock.newCondition();
    }

    @Override
    public void run() {
        while(!Thread.interrupted()) {
            lock.lock();

            try {
                while (!play)
                    myTurn.awaitUninterruptibly();

                System.out.println(text);

                this.play = false;
                nextPlayer.play = true;

                nextTurn.signal();
            } finally {
                lock.unlock();
            }
        }
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
        this.nextTurn = nextPlayer.myTurn;
    }

    public void setPlay(boolean play) {
        this.play = play;
    }
}

We can see that the use of Condition makes the code more readable. We have used the method awaitUninterruptibly, this ensures that both players play the last turn when the main thread interrupts the threads (NOTE: There's an issue with this approach, which is discussed in the comments section).

public class Game {

    public static void main(String[] args) {
        Lock lock = new ReentrantLock();

        Player player1 = new Player("ping", lock);
        Player player2 = new Player("pong", lock);

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setPlay(true);

        Thread thread2 = new Thread(player2);
        thread2.start();
        Thread thread1 = new Thread(player1);
        thread1.start();

        //Let the players play!
        try {
            Thread.sleep(2);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //Tell the players to stop
        thread1.interrupt();
        thread2.interrupt();

        //Wait until players finish
        try {
            thread1.join();
            thread2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }
}

Bonus, scaling to N players

Let's see how easy it is to scale the game to many players, so they pass the ball between them in a particular order (this wouldn't be Ping Pong anymore :)). The output of the application would be something like this:

Game starting...!
player0
player1
player2
player3
player4
player5
...
Game finished!

It turns out we don't need to modify the class Player at all! Indeed, every player must be aware of the next player in the game only, so the only changes will have to be done in the class Game:

public class GameScale {

    public static final int NUM_PLAYERS = 6;

    public static void main(String[] args) {
        Lock lock = new ReentrantLock();

        int length = NUM_PLAYERS;

        Player[] players = new Player[length];

        for (int i=0; i < length; i++) {
            Player player = new Player("player"+i, lock);
            players[i] = player;
        }

        for (int i=0; i < length - 1; i++) {
            players[i].setNextPlayer(players[i+1]);
        }
        players[length - 1].setNextPlayer(players[0]);

        System.out.println("Game starting...!");

        players[0].setPlay(true);

        //Threads creation
        Thread[] threads = new Thread[length];
        for (int i=0; i < length; i++) {
            Thread thread = new Thread(players[i]);
            threads[i] = thread;
            thread.start();
        }

        //Let the players play!
        try {
            Thread.sleep(2);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //Tell the players to stop
        for (Thread thread : threads) {
            thread.interrupt();
        }

        //Don't progress main thread until all players have finished
        try {
            for (Thread thread : threads) {
                thread.join();
            }
        }  catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }

}

The code is a bit more complex, but I think it's easy to understand. Changing the constant we can scale the game as much as we want, and concurrency will ensure the turns are taken in the right order.

In the last post of the series we'll focus on the creation and management of threads, so that the class Game is less cryptic than it is now.

(All the code can be found in this GitHub repository)

Multithreading in Java for dummies (part 1)

Raúl Ávila — Mon, 09 Jan 2017 06:19:54 +0000

In this series we'll review several basic concepts to ramp up with concurrent programming in Java. I know, Internet is full of references, tutorials, articles...about the topic, but my intention here is to use a different perspective. Instead of explaining the theory in detail and / or using overly complex examples, I'll try to use a practical approach, developing a simple app from scratch which will grow while introducing different concepts and APIs that we need to know to implement concurrent applications in Java along the way. Hopefully I'll reach my goal :)

The application: Ping Pong

As simple as that. Our Java application will show in the standard output, and alternately the texts "ping" / "pong", along with a header and footer to begin and finish the game:

Game starting...!
ping
pong
ping
pong
//....
Game finished!

There will be two players (or actors), who will print the texts "ping" and "pong". The actor "ping" will play first.

Version Zero: a single thread

The first version will run in a single execution thread, so there won't be any concurrent programming here :). The first versions we'll implement will finish after both players have participated a fixed number of times, let's say 10 (configure in the constant MAX_TURNS).

This is the code implementing the class Player for our first version:

public class Player {

    private final String text;

    private int turns = Game.MAX_TURNS;

    private Player nextPlayer;

    public Player(String text) {
        this.text = text;
    }

    public void play() {
        if (!gameFinished()) {
            System.out.println(text);
            turns--;
            nextPlayer.play();
        }
    }

    private boolean gameFinished() {
        return turns == 0;
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
    }

}

Each player prints the text and asks the other player to play, so they're alternating. The class that wires both players and starts the game is quite simple too:

public class Game {

    public static final int MAX_TURNS = 10;

    public static void main(String[] args) {

        Player player1 = new Player("ping");
        Player player2 = new Player("pong");

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.play();

        System.out.println("Game finished!");
    }
}

We can see that the Player constructor does not have a parameter for the other player. This is because we have a circular reference, so the second player may not be instantiated yet, and we have to inject it through setter.

The attribute text in Player is declared as final. It's a good practice in concurrent applications (in all apps really) to declare class attributes as final if we know they won't be modified. Not only our code will be more reliable, this guarantees the visibility of our attributes between threads, a concept known as "Safe publication". You can read a good discussion about safe publication here. Going a bit further, we should always try to design our classes as immutable, even though this is not possible in our example.

Version One: Players as Threads

Let's improve our application to make it work concurrently. We'll add one small improvement at a time, discovering that our first approaches don't work as expected.

First, let's make our Player classes implement Runnable (more info here):

public class Player implements Runnable {

    private final String text;

    private int turns = Game.MAX_TURNS;

    private Player nextPlayer;

    private boolean mustPlay = false;

    public Player(String text) {
        this.text = text;
    }

    @Override
    public void run() {
        while(!gameFinished()) {

            while (!mustPlay); //Busy Waiting

            System.out.println(text);
            turns--;

            this.mustPlay = false;
            nextPlayer.mustPlay = true;

        }
    }

    private boolean gameFinished() {
        return turns == 0;
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
    }

    public void setMustPlay(boolean mustPlay) {
        this.mustPlay = mustPlay;
    }
}

The important method is run, it contains a loop that iterates until the number of turns for the player is over. In addition, after every iteration there is a Busy Waiting until it's the turn of the player to act. When this happens the player prints the text, it sets its own mustPlay to false and tells the other player to play.

A big difference between this version and the previous one is that, in the first version, one player told the other to play using direct method invocation (i.e. play), while here it happens modifying the value of a flag (mustPlay), that each player checks individually and constantly.

Let's see how our class Game would look like now:

public class Game {

    public static final int MAX_TURNS = 10;

    public static void main(String[] args) {

        Player player1 = new Player("ping");
        Player player2 = new Player("pong");

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setMustPlay(true);  //Plays first!!!

        Thread thread2 = new Thread(player2);
        thread2.start();
        Thread thread1 = new Thread(player1);
        thread1.start();

        System.out.println("Game finished!");
    }

}

The biggest difference is that the threads are started separately, and we're only responsible of setting the flag mustPlay adequately. In fact, I have started the player2's thread first on purpose to confirm that even in this case the first message printed is "ping".

Let's see what happens when we start the application:

Game starting...!
ping
Game finished!

This is not what we expected...what happened? Our application has three threads now:

Main thread (Game.main)
player1's thread
player2's thread

The problem is that the main thread finishes as soon as the rest of the threads are started, so although the rest of the threads continue their execution and finalize correctly our IDE doesn't reflect the output generated in these two additional threads, creating double confusion because we see the message "Game finished!". To avoid this, there is a simple solution, using the method join:

public class Game {

    public static final int MAX_TURNS = 10;

    public static void main(String[] args) {

        Player player1 = new Player("ping");
        Player player2 = new Player("pong");

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setMustPlay(true);

        Thread thread2 = new Thread(player2);
        thread2.start();
        Thread thread1 = new Thread(player1);
        thread1.start();

        //Wait until both threads finish
        try {
            thread1.join();
            thread2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }

}

This method conditions the progress of the thread in execution to the finalization of the thread where it's invoking the method join. join is a blocking method (i.e. it blocks the thread which invokes it), and blocking methods can be interrupted, so it throws the ckecked exception InterruptedException. We'll talk more about the complications to interrupt threads later on this post.

Therefore, in theory our main thread will wait now until both players run out of turns...or maybe not? Well, let's run the app a couple of times, depending on the luck we have it's possible that everything is correct, but if we run it several times it's quite likely that sooner or later we get an output like this:

Game starting...!
ping

Our app is blocked and does not progress at all!

What happened? One of the main problems in concurrent programming is "visibility". Java only guarantees the visibility of attributes between threads if we follow a set of guidelines regulated by the Java Memory Model, concretely by the "happens-before" relationship. According to Wikipedia, in Java this relationship says that:

In Java specifically, a happens-before relationship is a guarantee that memory written to by statement A is visible to statement B, that is, that statement A completes its write before statement B starts its read

In our code we're not following any of the conventions that ensures the visibility of the modification in the attribute mustPlay between threads. Obviously, the modification of the own player's mustPlay attribute is visible for the player's thread, who does not continue playing, but, in the way we're doing it, the modification of the attribute mustPlay of the other thread (nextPlayer.mustPlay = true) is not automatically visible for the affected thread, so both players have mustPlay set to false, and our app gets blocked (aka deadlock).

To fix this problem we'll have to introduce the volatile modifier. This modifier indicates to the JVM that the attribute is likely to be shared between threads, therefore read operations mustn't be cached in any way, accessing to the main memory in all cases. In addition, write operations must be done atomically and made visible immediately.

Our code, then, should be changed this way:

public class Player implements Runnable {
    //...
    private volatile boolean mustPlay = false;
    //....
}

And now, finally, the app works in a deterministic way in every execution. One of the main problems of thread visibility in concurrent applications is that it fails randomly, so, if we're not aware about the guidelines to follow, debugging these problems can be extremely complicated.

Version 2: Infinite game

Instead of playing a fixed number of turns we're going to make both players to play forever. Or better said, until the main thread wants. To achieve this, we'll have to use one of the features that Java offers to interrupt a thread. Let's see how our Game class would look like:

public class Game {

    public static void main(String[] args) {

        Player player1 = new Player("ping");
        Player player2 = new Player("pong");

        player1.setNextPlayer(player2);
        player2.setNextPlayer(player1);

        System.out.println("Game starting...!");

        player1.setMustPlay(true);

        Thread thread2 = new Thread(player2);
        thread2.start();
        Thread thread1 = new Thread(player1);
        thread1.start();

        //Let the players play!
        try {
            Thread.sleep(2);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //Tell the players to stop
        thread1.interrupt();
        thread2.interrupt();

        //Wait until players finish
        try {
            thread1.join();
            thread2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        System.out.println("Game finished!");
    }

}

We see that, once both player threads are started, the main class goes to sleep for a while (2ms), and as soon as it wakes up (it returns to "running" state actually), it requests both player threads to finish.

I repeat, it requests. The only thing that happens when the method interrupt is invoked on a thread is that a flag interrupted is set to true in that thread. It's responsibility of the own thread to act when it considers necessary, perform cleaning tasks, and finalize. But it's perfectly valid if the thread decides not to do anything and continue with its execution (even though that wouldn't be very correct, of course). The way to know the value for this flag is using the method Thread.interrupted(), so our class Player would change to:

public class Player implements Runnable {

    private final String text;

    private Player nextPlayer;

    private volatile boolean mustPlay = false;

    public Player(String text) {
        this.text = text;
    }

    @Override
    public void run() {
        while(!Thread.interrupted()) {  //Was I interrupted?

            while (!mustPlay);

            System.out.println(text);

            this.mustPlay = false;
            nextPlayer.mustPlay = true;

        }
    }

    public void setNextPlayer(Player nextPlayer) {
        this.nextPlayer = nextPlayer;
    }

    public void setMustPlay(boolean mustPlay) {
        this.mustPlay = mustPlay;
    }
}

Instead of checking after every iteration if we have run out of turns we check the state of the flag interrupted, and we finish if it's true. As simple as that.

Version 2b: More about interrupt

Before finishing this first post of the series, let's delve into the implications of interrupting a thread.

In some occasions, we've seen that some of the methods in the class Thread (join, sleep,...) throw InterruptedException. This happens when a thread is interrupted being in blocked state after invoking any of these methods (e.g. anotherThread.join()). In that case, what happens is that the method (join, to follow with the example), sets the flag interrupted to false in the thread and throws InterruptedException. I'm not a big fan of checked exceptions, but I think this is one of the few cases where their use is more than justified.

Let's modify the class Player slightly, so once it's its turn to play it goes to sleep for 1ms before printing the text:

public class Player implements Runnable {
    //...
    @Override
    public void run() {
        while(!Thread.interrupted()) {
            while (!mustPlay);

            try {
                Thread.sleep(1);
            } catch (InterruptedException e) {
                e.printStackTrace(); //1
            }

            System.out.println(text);

            this.mustPlay = false;
            nextPlayer.mustPlay = true;

        }
    }
    //...
}

If we keep using the same version of the Game class, it's very likely that the game lasts forever. Why? Because if the interruption happens when the player thread is sleeping, the method sleep will swallow the "interrupted" state before throwing the exception, and given that we're only printing the error in //1, the loop does not detect the state interrupted and continues forever.

The solution to this problem is reestablishing the Thread state to interrupted:

@Override
public void run() {
    while(!Thread.interrupted()) {
        while (!mustPlay);

        try {
            Thread.sleep(1);
        } catch (InterruptedException e) {
            //I was interrupted, propagate the state  
            Thread.currentThread().interrupt();
        }

        System.out.println(text);

        this.mustPlay = false;
        nextPlayer.mustPlay = true;

    }
}

In general, we have to be very careful when we handle InterruptedException. Another recommended strategy, which would imply modifying the logic in our run method, is rethrowing the exception, so it will be handled somewhere else. We should never swallow the exception.

There are many improvements to carry out, the application is far from being optimal (that horrible Busy Waiting...). In the next post we'll optimize the use of the CPU using locks and conditions.

(All the code can be found in this GitHub repository)

Review your auto-generated code

Raúl Ávila — Sat, 12 Nov 2016 15:49:20 +0000

In this post I'll review several mistakes that, based on my experience, may be frequent that we make developing code in Java, and particularly using IDE's tools to auto-generate code.

Auto-generating code is fantastic, it saves us lots of keystrokes and facilitates focusing on the domain problem instead of the nuances of boilerplate code. However, sometimes we don't reflect conveniently about the code that it's generated for us by the IDE, and we leave small defects that should be fixed. In this post I'll list some of them.

Public scope

The IDEs consider all classes public by default. So, if we tell IntelliJ to create a new class, what will happen is:

public class MyClass {
}

In my opinion, classes should have package scope by default, i.e.:

class MyClass {
}

This way, they are not visible outside of the package they belong to. If, at some point, we need to change the scope of the class we will need to think why, and make the convenient design decisions. If the class is public by default, we'll use it straight away, increasing the coupling level of our code.

The best way to avoid this problem is editing the template in our IDE. In IntelliJ IDEA, for example, this option is in Preferences > Editor > File and Code template > Templates > Class.

One of the best practices when we use the package scope by default, it's creating an interface as public, but without exposing its implementations. This way clients of the interface won't have any idea about the implementation they're using, because they don't even have access to it, favoring practices like dependency injection.

As another example of this preference for public scope, IntelliJ by default creates the constants as public:

public static final String CONSTANT = "value";

Most of the times, constants are only used by the class owning them. Please pay attention to this, or change your template too.

Immutable classes

This is a detail without a direct association with auto-generated code, but that we skip 99% of the times. In general, when we want to design a class as immutable, the result will be something like this:

public class MyImmutableClass {

    private final int id;
    private final String name;

    public MyImmutableClass(int id, String name) {
        this.id = id;
        this.name = name;
    }

    public int getId() {
        return id;
    }

    public String getName() {
        return name;
    }
}

Private and final fields, no setters and initialization by constructor. OK, this is quite good, but we're missing something. Let's see what happens if we do this:

public class MyImmutableDerivedClass extends MyImmutableClass{

    public String address;

    public MyImmutableDerivedClass(int id, 
                                   String name, 
                                   String address) {
        super(id, name);
        this.address = address;
    }

    public String getAddress() {
        return address;
    }

    public void setAddress(String address) {
        this.address = address;
    }
}

An instance of this second class, which inherits from MyImmutableClass could act as a surrogate of it without clients noticing, i.e.:

MyImmutableClass myImmutableClass
                = new MyImmutableDerivedClass(1, "name", "address");

which is not ideal, because this derived class is not immutable!

The solution is simple, let's make our immutable class final:

public final class MyImmutableClass {
    //...
}

And now it won't be possible to create derived classes.

Getters and setters

It's very frequent to create a class with all its fields, and add, in an auto-generated code frenzy, its constructor, getters and setters for all the fields:

Do we really need to do this?

public class MyClass {

    private Collaborator1 collaborator1;
    private Collaborator2 collaborator2;

    public MyClass(Collaborator1 collaborator1,
                   Collaborator2 collaborator2) {
        this.collaborator1 = collaborator1;
        this.collaborator2 = collaborator2;
    }

    public Collaborator1 getCollaborator1() {
        return collaborator1;
    }

    public void setCollaborator1(Collaborator1 collaborator1) {
        this.collaborator1 = collaborator1;
    }

    public Collaborator2 getCollaborator2() {
        return collaborator2;
    }

    public void setCollaborator2(Collaborator2 collaborator2) {
        this.collaborator2 = collaborator2;
    }
}

A class should expose the minimum number of internal details. So, having said in the first section that a class should have package scope until it requires to be public, now I say that a class shouldn't have getters or setters until it's needed. And when it happens, think if that's the best decision or if there's something in your design that is not appropriate.

Equals method

In general, IDEs expose an option to generate the methods equals and hashCode together. This makes sense, because we usually need a right and consistent implementation for both methods (I won't get into many details here, assuming we all know the equals and hashCode contract).

I don't have many objections to the implementation that IDEs generate for hashCode, they're usually quite optimal, even though maybe we could optimize them even more. There's a small problem with the equals implementation though:

public class MyClass {

    private final int id;
    private final String name;

    //Constructor, equals, etc

    @Override
    public boolean equals(Object o) {
        if (this == o)
            return true;

        if (o == null ||
            getClass() != o.getClass())
                return false;

        MyClass myClass = (MyClass) o;

        if (id != myClass.id)
            return false;

        if (name != null ?
            !name.equals(myClass.name) :
            myClass.name != null)
                return false;

        return true;
    }
}

Everything seems correct here, but something serious is happening, this class is violating the Liskov Substitution Principle. This principle is part of the SOLID principles, and in few words it says that "a derived class can act as its base class without clients noticing it". This is a bit rough definition (this principle would require an entire post), so I think I'll illustrate the problem better with an example, so that we can see why our equals method is broken.

Let's create a derived class:

public class MyDerivedClass extends MyClass {
    private final String address;

    public MyDerivedClass(int id, String name, String address) {
        super(id, name);
        this.address = address;
    }

    public String getAddress() {
        return address;
    }

    // equals and hashCode
}

I omit equals and hashCode implementations, as they're not necessary for the example.

public static void main(String[] args) {
    MyClass object1 = new MyClass(1, "name");
    MyDerivedClass object2 = new MyDerivedClass(1, "name", "address");

    System.out.println(areEquals(object1, object2));
}

public static boolean areEquals(MyClass object1, MyClass object2) {
    return object1.equals(object2);
}

The method areEquals receives two instances of MyClass. Both instances are identical based on the information they contain, so we should expect areEquals to return true. But this is not the case, because the auto-generated equals method is doing this:

if (o == null ||
    getClass() != o.getClass())
        return false;

getClass returns MyClass for the first object and MyDerivedClass for the second, so our method equals returns false.

We could discuss about the conveniency of this to happen, because both are instances of different classes. But, does it really make sense when the values for all the fields in both instances are exactly the same? In fact, one alternative implementation of areEquals could be:

public static boolean areEquals(MyClass object1, MyClass object2) {
    return object1.getId() == object2.getId() &&
            object1.getName().equals(object2.getName());
}

In this case, the result is true.

This is what Liskov Substitution principle is about. One of the implications of this principle is that, if we overwrite a method it will be to add some behaviour without removing anything the base class is doing. Something that breaks this principle dramatically is the practice of overwriting a method to leave it empty (I guess many of us have done this at some point in our careers).

So, we need a better implementation for our method equals. And it would be something like this:

@Override
public boolean equals(Object o) {
    if (this == o)
        return true;

    if (!(o instanceof MyClass))
        return false;

    MyClass myClass = (MyClass) o;

    if (id != myClass.id)
        return false;

    if (name != null ?
            !name.equals(myClass.name) :
            myClass.name != null)
                return false;

    return true;
}

Instead of interrogating the objects to know their class using getClass, what we do is asking the object passed by parameter to equals if it's an instance of the class where equals lives through the operator instanceof. This operator returns true for instances of that class and its derived classes too. And now, finally, our areEquals method works the way we want. All this (and more) is explained perfectly in the seminal book Effective Java, book that every Java developer should have on his library.

The truth is that IntelliJ exposes a way to auto-generate this version of equals, but we must be careful with the options of the wizard, because it's unchecked by default, and we must check it ourselves. It is "Accept subclasses as parameters to equals() method":

The dialog box itself mentions that the implementation by default breaks the equals contract. It seems the instanceof implementation may not be compatible with some frameworks. In that case, we should switch to the first version, but always being aware of the consequences. So, let's use the correct version in the first place, and let's switch to plan B if necessary :)

My Experience with Pair Programming

Raúl Ávila — Sun, 06 Nov 2016 18:52:56 +0000

Four years ago or so, I was working in Madrid (Spain), and I did an interview for a company that was using Extreme Programming as a methodology. At that time, I had no idea about what that was, so they explained it quickly: "Every morning, we have a stand-up meeting, where we speak about what we did the day before. Then, we review that whiteboard full of post-its, which represent tasks, and each pair, because here we do Pair Programming, chooses a task and works on it".

That sounded extremely different to everything I knew (I was still doing waterfall...), and, to be honest, I was a bit reluctant, mainly because I didn't see myself working full-time with somebody else and sharing the same computer. "I won't be able to listen to music!", I thought. This, along with the fact that the technologies they used didn't catch my attention, and that I was quite comfortable in the company I was working for, made me reject the offer they made.

It's 2016. I've been living in London since 2014, and when I made the difficult decision of moving to a different country it was crystal clear to me that I would try to make the most of the experience. So, since day one, I began to attend meetups, workshops, etc, as in London you can find plenty of these events. In many of these workshops we did Pair Programming in case studies with the objective of familiarizing ourselves with technologies, practices, etc. Last year I attended the Global Day of Code Retreat, which was amazing, and where you basically spend the whole day pairing with different people.

I started to think that Pair Programming wasn't that bad. I enjoyed it a lot, I realized I learnt a lot too, and, given my increasing interest in TDD, and technologies like Spring Framework, Cloud platforms and many others, I found a company where I was going to be able to use all this and much more.

After one year of pairing, I think I'm now in a position to give my opinion about full-time Pair Programming. Yes, we work from 9 to 6 sharing a computer with another person, but we don't share keyboard and mouse, since each member of the pair has his/her own one. The usual approach to pairing is that one person takes the role of driver, being responsible of coding, while the other person should avoid touching the devices and just review and talk about what the pair is doing. To avoid tedium both roles are interchanged frequently.

The pros

There are lots of pros, and they defeat the cons completely, in my opinion:

Since two people are working on the same problem, and every single aspect of the implementation is discussed, the pair doesn't make the same number of "stupid" mistakes that we make when we work alone. This kind of mistakes makes us waste a lot of time, and I guess all of you know what I'm talking about
You learn tons: this happens because your pair will always know something that you don't know, starting with simple things like command line tools, and more generics aspects like patterns or features of the language we're using. This point reaches its maximum potential when you're working in a language you don't know well, but the other member of the pair masters. I can't think of a better way to learn a new programming language that pairing. Believe me, the experience is awesome!
You teach tons: same as the other person will know things you don't know, the inverse is true, and you'll find yourself explaining to your pair how to do certain things. Apart from how satisfactory teaching is, you'll see that teaching is a good way to review your knowledge, and discover holes in it
The knowledge is shared: pair rotation is frequent if you stick to Extreme Programming practices, so the "single point of failure" problem is effectively removed completely, and nobody will start shaking if one member of the teams is not working one day. Indeed, through pair rotation it's quite hard that a single developer is absolutely essential in the team. This can break the traditional way of thinking in many companies, because it challenges the corporate ladder model slightly. You know, you own a niche so that promotion is easier. If you think this way, stop to think if you're looking for a personal benefit or a team benefit in your office when you go to work. A team benefit will always be much better for your company, which is paying you good money to receive the maximum value in return
It's fun: people are social beings, we like to talk to people, discuss things, make jokes. Pair Programming encourages this all the time, so the idea of the isolated programmer hearing music during the most part of the day disappears completely. Personally, I think this is very beneficial for our mental health in the long run too :)
It helps to get more cohesive teams. In the book Peopleware we can read about the concept of "jelled teams", teams that achieve cruise velocity and are unstoppable. Reaching this ideal is not something we can do in one day, and I can't think of a better way of achieving this than Pair Programming
The code generated has less mistakes: when we do Pair Programming we're doing code reviews all the time, so the percentage of bugs decreases considerably

It's quite likely that I'm missing something, but I think that after reading all these points you should have a good idea about all the benefits of pairing

The cons

As everything in life, there are some cons, even though I believe most of them can be mitigated:

It's exhausting: my first weeks doing Pair Programming I left the office exhausted, and not because I'm a weak person, everybody in my company mentioned that it was very common. Spending the whole day speaking to another person and completely focused on your work (because distractions are less common than when you work alone) makes your mind be drained at the end of the day. However, we usually adapt quickly to new environments, and in three or four weeks you get used to it. In addition, breaks are highly recommended, and we have started to adopt the Pomodoro technique, which has proven to be very useful to keep us focused and less tired
Keyboard and mouse collisions: in occasions, the person that in theory shouldn't touch the keyboard and mouse, can't avoid doing it to comment something that it's complicated without using the devices. Most of the times we take control unconsciously and it can be a bit annoying for the other member of the pair, but this situation it's hard to avoid. When it happens, the best thing we can do is taking it with humor if you're interrupted, and if you want to interrupt, try to alert politely (can I take over for a while?)
Searching something / Googling is a bit awkward: when we're stuck and we go to Google to look for a solution, I have noticed that every person reads the contents randomly, so we're actually following two different lines of thought. From a point of view, this can be beneficial, because maybe we find a solution faster, but many times I think I would do it more comfortable if I did it alone. When this happens, it's perfectly valid to propose the alternative of investigating in parallel using two different computers
Personality clashes: each person has a different character, and so many hours working together can lead to some tension if there are disagreements. I have experienced this situation working with client developers more than with colleagues of my company, as our recruitment process is very focused on finding the right people for our way of working. Patience is super important when you pair, and you should never be rude with the other
Disconnection: if you feel that you're disconnecting from the task you're doing, or this is happening to the other one, the best thing you can do is proposing a break. We have ping pong tables in our offices to facilitate having proper breaks, as we disconnect completely from the task we're doing to play ping pong for a while

Conclusion

I don't know if this article will convince anybody to do Pair Programming, but at least I expect it's clear now why it is such a recommended practice. If you want to learn more, have a look at Kent Beck's Extreme Programming Explained.

Hello World!

Raúl Ávila — Sun, 06 Nov 2016 16:53:32 +0000

Hello, I'm RaÃºl Ãvila, and I have been coding professionally for more than 11 years.

I live in London, and I work for Pivotal Labs, the consulting division of Pivotal.

My main language is Java, but I can code in many other languages, such as Javascript, Python, Groovy, a bit of Scala...

My goal as a professional is making the code I produce as readable and maintainable as possible. In the last year I have embraced all the principles and practices of Extreme Programming. IMO, I think there isn't a better way to deliver software of the highest quality than applying XP.

You can find me on Twitter as @_Raul_Avila, and if you speak Spanish, you can read my personal blog too: http://raulavila.com/.

Nice to meet you!