fixing a (transitive) dependency mess with gradle

the transitive dependency mess

While extracting openapi-processor-core from openapi-processor-spring I noticed duplicate dependencies with different and sometimes unexpected versions in the external libraries section o the project view (in IDEA).

In this case jackson packages.

Running gradle dependencies to see where they are coming from prints the following output (look for jackson):

gradle dependencies (simplified)

+--- com.fasterxml.jackson.module:jackson-module-kotlin:2.11.0
|    +--- com.fasterxml.jackson.core:jackson-databind:2.11.0 <1>
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0
|    |    \--- com.fasterxml.jackson.core:jackson-core:2.11.0
|    \--- com.fasterxml.jackson.core:jackson-annotations:2.11.0
+--- org.openapi4j:openapi-parser:1.0
|    \--- org.openapi4j:openapi-core:1.0
|         +--- com.fasterxml.jackson.core:jackson-databind:2.9.10 -> 2.11.0 (*) <2>
|         \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.10 -> 2.10.2 <3>
|              \--- com.fasterxml.jackson.core:jackson-core:2.10.2 -> 2.11.0 <4>
+--- io.swagger.parser.v3:swagger-parser:2.0.20
     +--- io.swagger.parser.v3:swagger-parser-v3:2.0.20
          +--- io.swagger.core.v3:swagger-models:2.1.2
          |    \--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0 <5>
          +--- io.swagger.core.v3:swagger-core:2.1.2
          |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0
          |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.11.0 (*)
          |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.1 -> 2.10.2 (*) <6>
          |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 <7>
          |         +--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0
          |         +--- com.fasterxml.jackson.core:jackson-core:2.10.1 -> 2.11.0
          |         \--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.11.0 (*)
          +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 -> 2.11.0
          +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.11.0 (*)
          \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 (*) <8>

There is a bit of a mess regarding the jackson packages and their versions. Here are a couple of entries:

<1> some with version 2.11.0
<2> upgrade from 2.9.10 to 2.11.0
<3> upgrade from 2.9.10 to 2.10.2
<4> upgrade from 2.10.2 to 2.11.0
<5> upgrade from 2.10.1 to 2.11.0
<6> upgrade from 2.10.1 to 2.10.2
<7> version 2.10.1
<8> version 2.10.2

The 3 top-level dependencies are direct dependencies (as gradle calls them) of my project and all of them use different jackson versions:

. com.fasterxml.jackson.module:jackson-module-kotlin:2.11.0
. org.openapi4j:openapi-parser:1.0
. io.swagger.parser.v3:swagger-parser:2.0.20

The first one is simple. Because com.fasterxml.jackson.module:jackson-module-kotlin is a jackson package itself, gradle selects 2.11.0 for the other transitive jackson dependencies.

jackson-module-kotlin (simplified)

    +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.11.0
         +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0
         +--- com.fasterxml.jackson.core:jackson-core:2.11.0
         \--- com.fasterxml.jackson.core:jackson-databind:2.11.0

Next one is org.openapi4j:openapi-parser:1.0. This one depends on jackson 2.9.10 and gradle upgrades everything to 2.11.0 because that is the newest version it found.

openapi-parser (simplified)

+--- org.openapi4j:openapi-parser:1.0
     +--- com.fasterxml.jackson.core:jackson-core:2.10.2 -> 2.11.0
     +--- com.fasterxml.jackson.core:jackson-databind:2.9.10 -> 2.11.0 (*)
     \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.10 -> 2.10.2

Ok, not for all jackson packages. It doesn't use 2.11.0 for jackson-dataformat-yaml.

It is not using 2.11.0 because gradle has not seen a newer version of jackson-dataformat-yamlthan 2.9.10. So why is it not 2.9.10? It is not because the third direct dependency io.swagger.parser.v3:swagger-parser:2.0.20 is using 2.10.1 and 2.10.2.

swagger-parser (simplified)

+--- io.swagger.parser.v3:swagger-parser:2.0.20
     +--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0
     +--- com.fasterxml.jackson.core:jackson-core:2.10.1 -> 2.11.0
     +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.11.0
     +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.1 -> 2.10.2
     \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1

2.10.2 for jackson-dataformat-yaml. This is the highest version for this package so gradle picks it.

Gradle finally selects the following versions:

package (stripped com.fasterxml.jackson.)	version
core:jackson-annotations	2.11.0
core:jackson-core	2.11.0
core:jackson-databind	2.11.0
dataformat:jackson-dataformat-yaml	2.10.2 => want 2.11.0
datatype:jackson-datatype-jsr310	2.10.1 => want 2.11.0

Gradle uses the highest version it does find for each package. It doesn't know that the jackson packages belong together and that all of them should use the same version.

There are two possible solutions to make all jackson packages use the expected version.

solution

According to the gradle documentation the preferred solution is to add a constraints block to the dependencies. Adding a direct dependency is not the correct solution:

gradle-constraints

Often developers incorrectly fix transitive dependency issues by adding direct dependencies. To avoid this, Gradle provides the concept of dependency constraints.

Modifying the dependency resolution rules is not an option either:

gradle dependency resolve rules

If you are authoring a library, you should always prefer dependency constraints as they are published for your consumers.

constraints

The project is a library and the constraints solution look like this in the build.gradle:

ext {
    jacksonVersion = '2.11.0'
}

dependencies {
    // .. dependencies ..

    constraints {
        implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:$jacksonVersion") {
            because 'use the same version for all jackson packages'
        }
        implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$jacksonVersion") {
            because 'use the same version for all jackson packages'
        }
    }

}

Ok, this fixes the issue. :-)

Too bad each single package requires its own constraint (the closure with because ist not required). It would be nice to use wildcards to match all jackson packages with a single constraint:

ext {
    jacksonVersion = '2.11.0'
}

dependencies {
    // .. dependencies ..

    constraints {
        // does NOT work
        implementation("com.fasterxml.jackson.*:*:$jacksonVersion") {
            because 'use the same version for all jackson packages'
        }
    }

}

That would be a bit simpler but unfortunately it does not work.

platform

I found another solution in the gradle documentation that even uses jackson as an example:

gradle dependency version alignment.

First step is to create a metadata rule that joins all jackson packages into a platform. Jackson has a platform bom that lists all packages belonging to a platform version, and it is used to create the platform rule:

class JacksonPlatformRule implements ComponentMetadataRule {
    void execute (ComponentMetadataContext ctx) {
        ctx.details.with {
            if (id.group.startsWith ("com.fasterxml.jackson")) {
                belongsTo ("com.fasterxml.jackson:jackson-bom:${id.version}", false)
            }
        }
    }
}

I simply added the class at the end of the build.gradle for testing, but it will probably move to gradles buildSrc folder.

Second step is to activate the rule in the dependencies block:

dependencies {
    components.all(JacksonPlatformRule)

    // .. dependencies ..
}

By grouping the packages gradle now selects the jackson platform with the highest version and uses the versions listed in the bom for any jackson package.

This is also visible in the output of gradle dependencies (starting at line 4). Gradle selects the 2.11.0 platform module and upgrades all jackson packages to the platform version:

gradle dependencies (simplified)

+--- com.fasterxml.jackson.module:jackson-module-kotlin:2.11.0
|    +--- com.fasterxml.jackson.core:jackson-databind:2.11.0
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0
|    |    |    \--- com.fasterxml.jackson:jackson-bom:2.11.0
|    |    |         +--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
|    |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0 (c)
|    |    |         +--- com.fasterxml.jackson.core:jackson-core:2.11.0 (c)
|    |    |         +--- com.fasterxml.jackson.core:jackson-databind:2.11.0 (c)
|    |    |         +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.11.0 (c)
|    |    |         +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.11.0 (c)
|    |    |         \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.11.0 (c)
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.11.0
|    |    |    \--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
|    |    \--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
|    +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0 (*)
|    \--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
+--- org.openapi4j:openapi-parser:1.0
|    \--- org.openapi4j:openapi-core:1.0
|         +--- com.fasterxml.jackson.core:jackson-databind:2.9.10 -> 2.11.0 (*)
|         \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.10 -> 2.11.0
|              +--- com.fasterxml.jackson.core:jackson-databind:2.11.0 (*)
|              +--- org.yaml:snakeyaml:1.26
|              +--- com.fasterxml.jackson.core:jackson-core:2.11.0 (*)
|              \--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
+--- io.swagger.parser.v3:swagger-parser:2.0.20
     +--- io.swagger.parser.v3:swagger-parser-v3:2.0.20
          +--- io.swagger.core.v3:swagger-models:2.1.2
          |    \--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0 (*)
          +--- io.swagger.core.v3:swagger-core:2.1.2
          |    +--- com.fasterxml.jackson.core:jackson-annotations:2.10.1 -> 2.11.0 (*)
          |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.11.0 (*)
          |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.1 -> 2.11.0 (*)
          |    \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.11.0
          |         +--- com.fasterxml.jackson.core:jackson-annotations:2.11.0 (*)
          |         +--- com.fasterxml.jackson.core:jackson-core:2.11.0 (*)
          |         +--- com.fasterxml.jackson.core:jackson-databind:2.11.0 (*)
          |         \--- com.fasterxml.jackson:jackson-bom:2.11.0 (*)
          +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 -> 2.11.0 (*)
          +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.11.0 (*)
          \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 -> 2.11.0 (*)

conclusion

The constraints solution is easy to use on single dependencies. The platform rule is the way to go for groups of packages like jackson.

That's it.