BYOG (Build Your Own Gradle)

#showdev #kotlin #gradle

Gradle is an incredibly powerful tool for build automation and dependency management in the Java ecosystem. Almost too powerful, if you know what I mean. Sometimes working with Gradle just feels a bit too magical, and depending on what you're wanting to do, it may seem a bit overkill.

Well this week, while working to integrate Dokka with Orchid so I can document my Kotlin code, I found that I needed to download some Maven artifacts and execute them, just as if I were running a JavaExec task. But for various reasons I didn't want to use Gradle for this, and I also couldn't bundle them within Orchid itself, so I decided to figure out exactly how Gradle works by building the same thing for myself, and if you follow along, you'll be able to do the same for yourself too!

Step 1: How to run a Java program

This first thing I needed to figure out in order to run Dokka from Orchid, was how I even execute a jar from the command-line.

For those unfamiliar with Java, this basically involves collecting a bunch of JARfiles (Java ARchives of compiled class files and related resources) and passing them to the java command using the -classpath flag. You concatenate the paths to all the jars needed to run a program with : (or ; on Windows), provide it with the class containing the main() method, and it magically works!

For running Dokka, this meant something along the lines of:

java -classpath /path/to/dokka.jar:/path/to/my-dokka-formatter.jar:/path/to/kotlin.jar org.jetbrains.dokka.MainKt

Here, I wanted to run the Dokka program, which is contained in the dokka.jar jar, using a custom formatter I compiled to my-dokka-formatter.jar, which depends on kotlin.jar. After specifying the Dokka main class, org.jetbrains.dokka.MainKt, Java will boot up a JVM instance, load the classes it needs from the jars I provided to it, and execute the main method. Now we're in business!

Step 2: How To Get Those Jars

With a small executable like this, it is technically feasible to just to to Bintray and download all these jars manually. But this can quickly get unreasonably difficult when one of these jars depends on classes defined in another jar, which depends on classes defined in another jar, which depends on classes defined in another jar.....

You see how this could get very tricky to make sure you've downloaded all the right jars so that the entire program can run correctly. That's why tools like Maven and Gradle exist and are so popular, because they make this process almost painfully simple.

Among many other things, these tools are able to take a simple declaration of dependencies and figure out how to download them and run them together. Gradle uses Groovy and provides a very simple DSL where to can simply declare your dependencies, and it will do the rest.

repositories {
  jcenter()
}
dependencies {
  compile 'io.github.javaeden.orchid:OrchidAll:0.10.1'
}

The Maven dependency consists of 3 sections, in Gradle syntax separated by :. They are:

groupId: the individual or organization who produced the artifact
artifactId: the name of the project
version: the version of the project, typically declared using semantic versioning

These 3 sections guarantee a unique jar to use for compilation and execution, so that using them in your builds can be deterministic (nothing changing from one build to the next).

The process for fetching all these is surprisingly simple. Essentially, they are combined into a URL pointing at a file on some web server, such as JCenter, and you can request that file over HTTP. These servers are typically called "Maven repositories", but realistically you could use any server to host your artifacts (even Github Pages if you really wanted to). But that's really all there is to it. You can try it yourself, by visiting the following link to download the OrchidCore jar declared above.

https://jcenter.bintray.com/io/github/javaeden/orchid/OrchidAll/0.10.1/OrchidAll-0.10.1.jar

The generalized format is: [repository URL]/[groupId (replace dots with /)]/[artifactId]/[version]/[artifactId]-[version].jar

Step 3: Resolving Jar Dependencies

Now that we know how to get the jars, we need some way to figure out the other jars that one depends on. Well the smart guys who created Maven made this really simple, because every Jar hosted in a Maven Repository also has a corresponding POM, or "Project Object Model".

The POM is an XML document which contains some basic metadata about that jar, such as its name and description, list of contributors, VCS url, etc. But it also contains a special section for dependencies, which looks like the following:

<dependencies>
  <dependency>
    <groupId>io.github.javaeden.orchid</groupId>
    <artifactId>OrchidCore</artifactId>
    <version>0.10.1</version>
    <scope>compile</scope>
  </dependency>
  ...
</dependencies>

Does this look familiar? It should, because it is a list of dependencies, with the same artifactId, groupId, and version that were used to download this POM! There is also the additional scope tag in the dependency, which is usually either compile or test. Compile dependencies are required in order for this jar to be compiled and run, and test dependencies are required to run the tests for that project.

No matter whether you build your project with Maven or Gradle, the published artifact will have this POM file. The nice thing is that you can download this file in the same way as the jar, at nearly the same URL. You just need to use the .pom extension instead of .jar.

https://jcenter.bintray.com/io/github/javaeden/orchid/OrchidAll/0.10.1/OrchidAll-0.10.1.pom

Step 4: Putting It All Together

That's really all that you need to know for the Maven specification, you're now ready to write your own Gradle!

First, we need to get a list of all the Maven artifacts we need to run our Java program. I'll just be using kotlin-y pseudocode here, but I'll include a link to the Kotlin file where I actually implemented this at the end of this article.

This is an inherently recursive process (getting jars for jars for jars...), but it is quite naturally accomplished by processing a queue rather than using recursion. Essentially, we need to pop a Maven artifact from the queue, download its JAR and its POM, cache the jar on disk, and then add the dependencies from its POM back into the queue. When the queue is empty, we're finished!

val queue = queueOf("com.github.copper-leaf:dokka-json:0.1.0")
while(queue is not empty) {
  val artifact = queue.pop()
  val jar = downloadJar(artifact)
  val pom = downloadPom(artifact)
  queue.addAll(pom.dependencies)
}

That's great, we've downloaded all the jars we needed to, including all our dependencies! Now we just need to run our Dokka program. Let's extend the example just a bit, to show how we accomplish this.

val queue = queueOf("com.github.copper-leaf:dokka-json:0.1.0")
val jars = emptyList()
while(queue is not empty) {
  val artifact = queue.pop()
  val jar = downloadJar(artifact)
  val jarPath = cacheJar(jar)
  jars.add(jarPath)
  val pom = downloadPom(artifact)
  queue.addAll(pom.dependencies)
}

ProcessBuilder()
  .command(
    "java",
    "-classpath", jars.joinToString(File.pathSeparator),                         
    "org.jetbrains.dokka.MainKt"
  )
  .start()

And there you have it! In less than 20 lines of code, we have fully implemented our own Gradle, complete with dependency resolution and Jar execution!

Now of course, this is not a very robust solution. Gradle does a lot of work behind the scenes to make sure it's not downloading files it has already downloaded, and determining which jar to use when multiple versions are requested, and a million other things as well.

But still, I find it very surprising just how simple the process really is. In just a couple hours and about a 100 lines of code total, I actually implemented this in Kotlin so that I can run Dokka within Orchid, and it works surprisingly well. You can go check out the source here, or just stick around until the full integration is complete and you can seamlessly document your Kotlin projects with Orchid.

In the meantime, you might want to go check out the cool stuff going on with Orchid and learn how to use it for yourself, so when the Kotlindoc plugin is ready, you'll be ready to pick it right up! You can also check out one of the many sites I've already built with Orchid to get inspired for your next project.

And by the way, Orchid isn't just for documenting Kotlin, you can also do Javadoc and Swift code docs, as well as maintaining your project's wiki, and much more!