Gradle's leaky abstractions: Declarative(ish) shell, imperative core: Implementing a safe(ish) global configuration DSL

Gradle is very aware they have a complexity problem. Fundamentally, the problem is that Gradle build scripts use an Actual Programming Language (either Groovy or Kotlin), and therefore provide users access to the complete Java/Groovy/Kotlin ecosystem—the JDK, the standard libraries, and all the other libraries too.

Gradle wants you to write your scripts like this:

// build.gradle.kts
// so declarative!
plugins {
  id("foo")
}

dependencies {
  implementation(libs.coolThing)
}

myExtension { ... }

but what that boils down to is just this:

// er, imperative?
project.pluginManager.apply("foo")

project.dependencies.add("implementation", libs.coolThing)

project.extensions.getByType(MyExtension::class.java).run { ... }

which in turn is this:

// yeah, definitely imperative
FooPlugin().apply(project) {
  ... all the code in FooPlugin.apply() ...
}

... etc ...

Where one of the most important takeaways here is that build scripts are evaluated top-down, one "declarative block" after the other. Although, even that is not necessarily true. Kotlin DSL confuses this. Here's a valid settings script:

// settings.gradle.kts
rootProject.name = "..."
pluginManagement { ... }
buildscript { ... }
dependencyResolutionManagement { ... }
plugins { ... }

Gradle will not complain if you do this, but actually some of these blocks are Special and get evaluated not in top-down order. buildscript is the Primordial Special block; it is always evaluated first (which makes sense, as it contributes dependencies to the script itself). pluginManagement is evaluated next, followed by plugins, followed by everything else. Sometimes, when I'm working on a build, I will reorder blocks to match actual evaluation order because I want users to remember this (implicitly), but mostly I've given up. Note that, in a Groovy DSL script, Gradle will actually complain and refuse to compile a script that is written out-of-order. ¯\_(ツ)_/¯

What about afterEvaluate and various other lazy callbacks?

This isn't a master's thesis.

Sigh

Anyway.

Can I create a custom extension that is configured in the root project and which configures all subprojects, and is safe for isolated projects?

Yes, but not easily. Here's what we want to do, from a "UI" perspective:

// root build.gradle.kts
plugins {
  id("my-root-plugin")
}

myExtension { ... }

// subproject/build.gradle.kts
plugins {
  id("my-sub-plugin")
}

// values set in rootProject.myExtension are
// available here, safely

Where "safely" means these projects aren't, uh, unsafely coupled, and in particular don't access one another's mutable state. In very particular, they should follow the Isolated Projects contract (which is currently a WIP). I'll elaborate on "unsafely coupled" in a moment.

Naively, what we'd like to do is something like this:

class MySubPlugin : Plugin<Project> {
  fun apply(target: Project) {
    val myExtension = target.rootProject
      .extensions
      .getByType(MyExtension::class.java)

    // do things with values from rootProject's
    // myExtension extension.
  }
}

That, however, is not safe. A project's ExtensionContainer is mutable and accessing another project's mutable state is not safe (maybe the API should forbid this 🤔).

So instead of getting access to that state directly, we will instead use one of Gradle's few blessed ways to manage global mutable state for access at configuration time: the Shared Build Service.

The data flow will be as follows:

1. Users configure the extension in the root project.
2. Method calls on the extension immediately push data into mutable objects held by the build service.
3. Subprojects query the data in the service during configuration.
4. With a little bit of cleverness, we could also have extensions in subprojects configure data just for their subprojects. (This is left as an exercise for the reader.)

Important! Note that subprojects are coupled to parent projects! (The root project, in this case.) But so long as we don't access the parent projects' mutable state directly, it's ok! I promise! Gradle promises to always configure parent projects before child projects, so there's always this temporal coupling. (The main caveat to this is if you use an API like evaluationDependsOn()).

Finally, evaluate each Project by executing its "build.gradle" file, if present, against the project. The projects are evaluated in breadth-wise order, such that a project is evaluated before its child projects. This order can be overridden by calling evaluationDependsOnChildren() or by adding an explicit evaluation dependency using evaluationDependsOn(String).

So, uh, don't do that or everything goes sideways.

Implementing the root extension

The following demonstrates a custom DSL that's configured at the root level. The existence of an inner DSL is strictly unnecessary, but included as a demonstration because it's often useful. From the user perspective, it would be configured like so:

// root build.gradle.kts
myExtension {
  handler {
    foo(/* true or false */)
    bar(/* a String */)
  }
}

It's important that the inner handler expose methods, not properties. Methods let us do whatever we want most easily. (A Kotlin property with custom setter would also work, but I think that's unnecessarily complicated.)

abstract class MyExtension @Inject constructor(project: Project) {

  private val objects: ObjectFactory = project.objects

  private val service = MyService.of(project)
  private val myHandler = objects.newInstance(MyHandler::class.java, project, dslService)

  fun handler(action: Action<MyHandler>) {
    action.execute(myHandler)
  }

  internal companion object {
    fun register(project: Project): MyExtension {
      return project.extensions.create(
        "myExtension",
        MyExtension::class.java,
        project,
      )
    }
  }
}

abstract class MyHandler @Inject constructor(
  private val project: Project,
  private val service: Provider<MyService>,
) {

  private val objects = project.objects

  private val foo = objects.property(Boolean::class.java)
  private val bar = objects.property(String::class.java)

  fun foo(value: Boolean) {
    foo.set(value)
    foo.disallowChanges()

    service.get().configureHandlerFor(project) { config ->
      config.foo = value
    }
  }

  fun bar(value: String) {
    bar.set(value)
    bar.disallowChanges()

    service.get().configureHandlerFor(project) { config ->
      config.bar = value
    }
  }
}

We could actually skip the Property creation entirely, but I like it because of the disallowChanges() API, which ensures the method can only ever be called once, and I want to ensure a single source of truth for most cases.

Implementing the build service

A ("shared") build service is kind of like a singleton, in that when you register one in any project, it's available in all projects as a single instance. (This unfortunately turns out not to be true, in some cases, when using composite builds, but can be worked around.) An actual singleton (global static instance) doesn't work at all, for the record—try it if you want to lose some sanity. Anyway, use a build service whenever you need global mutable state in your build.

Please note that the examples in this post demonstrate a project extension having a reference to the build service. Do not attempt to do this in the other direction. You don't want a globally-available object (the build service) having access to a mutable object "owned by" any individual project.

abstract class MyService : BuildService<BuildServiceParameters.None> {

  private val handlerConfigurations = createConfigMap<HandlerConfiguration>()

  internal fun configureHandlerFor(
    project: Project, 
    action: Action<HandlerConfiguration>
  ) {
    handlerConfigurations.merge(
      project.path,
      HandlerConfiguration().apply(action::execute)
    ) { acc, _ ->
      acc.apply(action::execute)
    }
  }

  internal fun findHandlerConfig(project: Project): HandlerConfiguration? {
    return handlerConfigurations.findConfig(project)
  }

  private fun <T> createConfigMap(): MutableMap<String, T> = mutableMapOf()

  private fun <T> Map<String, T>.findConfig(project: Project): T? {
    return get(project.rootProject.path)
  }

  internal companion object {
    fun of(project: Project): Provider<MyService> {
      return project
        .gradle
        .sharedServices
        .registerIfAbsent(
          "myService",
          MyService::class.java
        ) {}
    }
  }
}

project.rootProject.path is okay because project.path is immutable state and safe to access from another project.

HandlerConfig is just a mutable value object.

internal class HandlerConfig(
  var foo: Boolean = false,
  var bar: String? = null
)

Consuming the values in a subproject

In your convention plugin that you apply to your subprojects (you're doing that, right?), you can get a reference to the shared build service, and then query the configuration object and do whatever you like with it.

class MySubPlugin : Plugin<Project> {
  fun apply(target: Project) {
    val service = MyService.of(target).get()
    service.findHandlerConfig(target)?.let { config ->
      // do something with config.foo
      // do something with config.bar
    }
  }
}

And there we have it. If you want the ability to configure a big project from the root project, with complex data, in a way that keeps your projects mostly decoupled (sigh) from each other, I have a variant of the above running in a large production system.

Is there another way?

Sure. You could rely entirely on Gradle properties and organize your configuration knobs as a set of key-value pairs. This is much easier to implement but much harder to use because Gradle properties don't support code completion, don't have easily accessible Javadoc, don't have any IDE support really, and it's shockingly easy to scatter your properties across your entire codebase, making them practically undiscoverable by feature engineers.

Comments

[Martin Bonnin]

TIL about a new footgun, evaluationDependsOn, thanks!

[Hunter LaFaille]

I've been building a new Java build tool (and package registry) to fix these problems. I know for sure there are organizations that need the flexibility Gradle provides, but every Java project I've worked on is a single codebase with a single build step. I spend more time at work screwing with Gradle then actually writing code, which is ridiculous in my book.

github.com/hlafaille/espresso
github.com/hlafaille/espresso-regi...