DEV Community: Devyn Goetsch

PipelineKt - Jenkinsfiles and Kotlin

Devyn Goetsch — Fri, 03 Jan 2020 17:17:23 +0000

When we introduced Jenkinsfiles into our ecosystem, we got some very nice features out of the box. Pipelines now could execute arbitrary code, and the builds would be versioned along with the source they were building.

Our initial implementation of Jenkinsfiles had its problems. The Jenkinsfile would share code by calling the load step on the same scripts repeatedly, which had several problems.

loading these scripts took processor time to interpret them
loading these scripts takes network time to move the script from the worker node to the master (where is is interpreted)
Pipelines could not recover after system restart and replay did not work due to the usage of load
These scripts were generally unpleasant to develop, there was poor IDE integration, and it was a subset of groovy functionality. Code completion and autocorrect did not work well.

It wasn't good, but it was better than it used to be.

The architects approached me about improving the developer experience for these pipelines. They mentioned using Kotlin to generate pipelines, as Jenkins developers have stated they will not support kotlin for pipelines. I was doubtful of this approach, but also intrigued.

Meanwhile, I wanted to distribute common code in a way that eliminated a developers need to worry about common details such as scm notifications, influx statistics, workspace cleanup, and scm optimization. I had become frustrated trying to communicate these issues, and Jenkinsfiles didn't give me good tools to do that, but if I was generating the Jenkinsfile I could build such code into the generation engine.

After several days of thinking I was still doubtful of the approach to generate Jenkinsfiles. However, I couldn't resist the temptation to experiment. The opportunity to develop in kotlin was tantalizing - code completion, discoverability, and strong compile time guarantees all drew me in. So I set out to prove I could do it.

I was simultaneously elated and afraid of the idea that I was creating a Rube Goldberg machine.

At first, I experimented with compiling kotlin scripts on the fly with the kotlin interpreter. I wrote a simple bit of kotlin code to output a jenkinsfile in a library jar. I ended up getting this to work by loading the correct jars into jenkins and interpretting a .kts script on the fly in jenkins, but I ran into several problems.

Local develepment was not supported very well
Using it required an indepth understanding of how jenkins manages libraries
The kotlin script engine was broken in the current version of kotlin

I had generated a pipeline from kotlin, but I was off the mark, and I wasn't convinced this approach would work out. Could this actually make developing Jenkinsfile easier, or was I chasing my tail? Naturally I didn't care about the answer to this question because I was having fun.

For the next week I kept working, I converted the development and generation process to use gradle, which meant that local development was now supported, the artifact resolution was standardized, and I didn't have to worry about the script engine.

Now that I had a good development environment, and could dynamically generate a jenkinsfile from kotlin source code I began to work on my library. And this was indeed fun. The tooling got out of my way, as is expected for kotlin development and I could focus on my code. I was dealing with kotlin, and the IDE was able to tell me immediately if my code was valid. I had some bumps a long the way, and occasionally I was able to generate bad Jenkinsfiles, but broadly my pipelines began to shore up. Common code fell into place as I built up my pipelines, and patterns became apparent. Isolating common code became trivial, since I was dealing with kotlin types and the IDE could refactor for me and tell me when I made a mistake.

After the first week I felt reassured that I was on the right path. My kotlin code became more concise every day as I moved functionality into the shared library, and I was able to properly share code both across projects in the common library, and within pipelines that had many repetitive steps (such as our monolith).

Now, I can express my gradle pipeline build in just a few lines of kotlin without having to worry about specifically configuring all of the esoteric Jenkins features:

val pipelineDsl = PipelineDsl()

val myPipeline = pipelineDsl.pipeline {    
    agent {
        label("linux")
    }
    stages {
        stage("Build") {
            steps {
                echo("Hello, World!")
            }
        }
    }
}

generatePipeline(myPipeline, "path/to/Jenkinsfile.generated")

I am able to express my pipeline the same way as I could in groovy, but with improvements provided by kotlin, and configured defaults provided on the PipelineDslobject.

Of course, all of these default settings could be overriden, as the entire pipeline object is nothing but simple data classes. This also makes my pipelines easy to unit test as assertions become simple kotlin object equality assertions. The best part is I can use any tool that integrates with gradle to develop my pipelines. Interested in trying out kotlin pipelines? Check out the repo for documentation and examples.

Kotlin type Extensions

Devyn Goetsch — Sat, 14 Dec 2019 17:18:34 +0000

Assumptions

General understanding of kotlin

The kotlin type system is extremely powerful. It provides a wealth of useful tools: data classes, interfaces with default implemetnations, sealed classes, object types, final by default on concrete types - the list goes on.

However, there is one feature that in my mind, that stands out as being overlooked but incredibly valuable - Extension Types.

What is an extension type?

Lets take a simple class as an example:

import java.util.concurrent.LinkedBlockingQueue

class Context<T> {
    private val items: Queue<T> = LinkedBlockingQueue()

    fun add(item: T) {
        items.add(item)
    }

    fun drainAll(): List<T> {
        val result = items.toList()
        items.clear()
        return result
    }
}

Lets say you had some common methods that you had to run on this class, for example a "merge" method.

In java, you would create static methods like so:

fun <T> merge(context1: Context<T>, context2: Context<T>): Context<T> {
    val newContext = Context<T>()
    context1.drainAll().forEach { newContext.add(it) }
    context2.drainAll().forEach { newContext.add(it) }
    return newContext
}

You could then invoke this method like so:

val context1: Context<T>
val context2: Context<T>

merge(context1, context2)

This implementation works, even if it isn't the most elegant it is pragmatic. Its easy to test, and the semantics are clear.

Extension methods allow us to express the exact same code a little bit more elegantly:

fun Context<T>.merge(context: Context<T>): Context<T> {
    val newContext = Context<T>()
    this.drainAll().forEach { newContext.add(it) }
    context.drainAll().forEach { newContext.add(it_) }
    return newContext
}

You now invoke merge on a context object like you would any normal method on the Context<T> class

val context1: Context<T>
val context2: Context<T>

context1.merge(context2)

The function definitionfun Context<T>.merge(...) tells the compiler within this method, the keywordthis references the Context<T> on which merge was invoked.

Extension type implementation and limitations

At compile time, extension types become a static method that takes in an extra preceding type argument for the first type. In fact, in the above example, the static method example is conceptually what the compiler creates from the extension method example.

Extension methods have a few limitations:

extension methods cannot access protected or private members
extension methods cannot override existing methods
extensions can be messy and can make code much much worse if misused.

Extension Functions

Above we saw a method with an extension type. Kotlin takes this one step further by also providing the ability to also apply this same principle to functions.

fun <T> withContext(body: Context<T>.() -> Unit): List<T> {
    val context = Context<T>()
    context.body()
    return context.drainAll()
}

val result: List<String> = withContext<String> {
    add("hello")
    add("goodbye")
}

Just like extension methods, extention functions expose the type on the left side of the function parameter declaration as "this" within the function. The following screenshot shows the IDE identifying this in the block inside of withContext:

Extension methods on type parameters

The kotline standard library is full of excellent extension methods like the kotlin use function

First, lets naively attempt to implement use:

inline fun <R> Closeable?.uselessUse(block: (Closeable?) -> R): R { ... }

var stream: BufferedInputStream = BufferedInputStream(FileInputStream(File("my-file.txt")))
it.uselessUse { closable: Closeable? ->
    //doesn't know closable is a BufferedInputStream
    false
}

In this implementation, the block loses context of what type of closable uselessUse was called on. This makes it difficult to interact with the Closeable inside of block, which undermines the utility of this method.

inline fun <T : Closeable?, R> T.use(block: (T) -> R): R { ... }

var stream: BufferedInputStream = BufferedInputStream(FileInputStream(File("my-file.txt")))
stream.use { bufferedInputStream: BufferedInputStream ->
    //knows that the closable was a BufferedInputStream
    String(bufferedInputStream.readAllBytes())
}

Notice how the use function extends a type parameter T: Closable? instead of a concrete type Closable?. This allows the block to have context of what type of Closeable is being used, and allows the user to take full advantage of auto-closable functionality.

Okay, so what?

In Java, auto closable was implemented as a language feature in Java 7. It worked, and it provided some syntactic sugar, but it was incredibly limited in scope. With extension functions, the Kotlin standard library provides an implementation for auto closable and describes a reproducible pattern for extending the language itself.

Extension functions are the main mechanism through which kotlin enables DSL implementation. In my next post I'll go into detail about how this can be done.