Dependency Hell

Resolving dependency conflicts is not fun. I personally have not had to pleasure of dealing with conflicts until more recently. Bundler and the entire Ruby community does an amazing job making sure I didn't have to. In the JVM world, things are less clear. For example, with SBT, the latest version of a dependency always wins, but if the latest version is higher by a major version, things will most likely break. SBT won't tell you when it replaces dependencies, you have to ask! I covered more details of the strange behavior in a previous post. What I didn't go over is how we can get into this situation in the first place.

Setting the stage

The best way I learn is by doing, tweaking things, and breaking it, then fixing it...maybe. Let's come up with a contrived example (follow along by cloning this repo). Below we have 3 files describing classes depending on each other Life > Person > Cat.

// com/kaoruk/Life.java =============================
package com.kaoruk;

import com.kaoruk.Person;

class Life {
  public static void main(String args[]) {
    Person person = new Person("Kaoru");
    if (args.length > 0) {
        person.adoptCat();
    }
    System.out.println(person.sayHello());
  }
}


// com/kaoruk/Person.java =============================
package com.kaoruk;

import com.kaoruk.Cat;
import java.util.Objects;

class Person {
  private final String name;
  private Cat cat;

  public Person(String name) {
    this.name = name;
  }

  public void adoptCat() {
    this.cat = new Cat();
  }

  public String sayHello() {
    if (Objects.isNull(cat)) {
      return this.name + ": Life has no meaning without a cat";
    } else {
      return this.name + ": Oh hai, I haz kitty! " + cat.sayHello();
    }
  }
}


// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String sayHello() {
    return "meow!";
  }
}

Now let's compile each file and place them in a jar:

#!/bin/bash

set -e

mkdir jars || true

echo "Compling Cat..."
javac com/kaoruk/Cat.java
jar -cvf jars/Cat.1.0.0.jar com/kaoruk/Cat.class
rm com/kaoruk/Cat.class

echo "Compling Person..."
javac -cp jars/Cat.1.0.0.jar com/kaoruk/Person.java
jar -cvf jars/Person.1.0.0.jar com/kaoruk/Person.class
rm com/kaoruk/Person.class

echo "Compling Life..."
javac -cp jars/Person.1.0.0.jar com/kaoruk/Life.java
jar -cvf jars/Life.1.0.0.jar com/kaoruk/Life.class
rm com/kaoruk/Life.class

If all goes well we should be able to run Life:

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life Foo

Kaoru: Oh hai, I haz kitty! meow!

Great! Okay so let's say we want to teach our Cat how to say "Can I haz job?" but also we decide that the method name "sayHello" doesn't really accurately portray the message. We change Cat.java look like:

// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String beg() {
    return "Can I haz job?";
  }
}

Great, let's compile it, replace Cat.1.0.0.jar and run it:

$ javac com/kaoruk/Cat.java
$ jar -cvf jars/Cat.2.0.0.jar com/kaoruk/Cat.class

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat


$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life foo

Exception in thread "main" java.lang.NoSuchMethodError: com.kaoruk.Cat.sayHello()Ljava/lang/String;
    at com.kaoruk.Person.sayHello(Person.java:22)
    at com.kaoruk.Life.main(Life.java:11)

Oh hai, dependency hell!

Detection is difficult

You might be thinking to yourself, well duh Life is broken because you updated a method in Cat and replaced it with a new JAR. This is exactly what build tools do! To detect the problem above, before running the code, we would need to recompile Person.1.0.0.jar. But more importantly, we would need to recompile it using Cat.2.0.0.jar, ignoring its explicit reliance on Cat.1.0.0.jar. As a build tool author, not that I am one, I would imagine my users would not expect the tool to recompile the entire world, because it would take too long.

The logic to recompile the entire world is no easy task. First, you'd have to list all dependencies and transitive dependencies, override dependencies if there is a higher version, create a graph, find the leaves, and the compile JARs all the way back. I'm not even sure this would work, I'm just spitballing!

Tests are not thou Savior

In our example, if our tests only cover running the main method without an argument, we wouldn't catch this bug, one of our poor users will. So that means we have to have tests for every single branch of our code? Yes, regardless if it doesn't work in this scenario! Having a test for every branch might not work because you'd also have to test every branch of your dependencies and transitive dependencies! In our example, if we depended on Life, our tests probably would not have hit the bug, because why would we test every branch of Life?

It ain't so bad...right?

Yes, if we follow a standard. In this case, the standard is semantic versioning, or semver. If a library's major version has changed, we know that something will break, we don't have to go digging into the source code of the library or every dependee of the library. Unfortunately, not all library maintainers follow semver. For example, Scala Akka, does not follow this pattern. Their minor versions introduce breaking changes that are incompatible with older minor versions. The reality is we have to keep track of which library follows semver and which follows a different pattern.

Okay so what can we do about it?

Bundler solves this problem by allowing engineers to specify an acceptable range. For example, gem 'thin', '~> 1.1' means that Bundler is allowed to install newer versions of thin < 2.0. If I bring in a gem which requires thin 2.0, Bundler throws an exception forcing me to deal with the problem. Maven also has this functionality, so Coursier shouldn't be too far behind? But it still doesn't solve the problem if library maintainers ignore semver. So I'm not entirely sure there's a solid solution to this problem yet. I think Bazel, rules_jvm_external to be specific, at least, makes a step towards the right direction informing us when there is a conflict.