DEV Community: Sebastian Aigner

Idiomatic Kotlin: Solving Advent of Code Puzzles, Passport Validation

Sebastian Aigner — Wed, 01 Sep 2021 14:52:59 +0000

Today in “Idiomatic Kotlin”, we’re looking at day 4 of the Advent of Code 2020 challenges, in which we tackle a problem that feels as old as programming itself: input sanitization and validation.

Day 4. Passport processing

We need to build a passport scanner that, given a batch of input text, can count how many passports are valid. You can find the complete task description at https://adventofcode.com/2020/day/4.

Like many challenges, we first inspect our input:

ecl:gry pid:860033327 eyr:2020 hcl:#fffffd
byr:1937 iyr:2017 cid:147 hgt:183cm

iyr:2013 ecl:amb cid:350 eyr:2023 pid:028048884
hcl:#cfa07d byr:1929

hcl:#ae17e1 iyr:2013
eyr:2024
ecl:brn pid:760753108 byr:1931
hgt:179cm

The input is a batch of travel documents in a text file, separated by blank lines. Each passport is represented as a sequence of key-colon-value pairs separated by spaces or newlines. Our challenge is finding out how many passports are valid. For part one, “valid” means that they need to have all the required fields outlined by the security personnel: byr, iyr, eyr, hgt, hcl, ecl and pid (we conveniently ignore their request to validate the cid field).

Solving Day 4, Part 1

Like many challenges, we start by reading our puzzle input as text and trim off any extraneous whitespace at the beginning and the end of the file. As per the description, passports are always separated by blank lines. A blank line is just two “returns”, or newlines, in a row, so we’ll use this to split our input string into the individual passports:

val passports = File("src/day04/input.txt")
    .readText()
    .trim()
    .split("\n\n", "\r\n\r\n")

(Note that depending on your operating system, the line separator in text files is different: On Windows, it is \r\n, on Linux and macOS, it’s \n. Kotlin’s split method takes an arbitrary number of delimiters, allowing us to cover both cases directly.)

We now have a list of passport strings. However, working with lists of raw strings can quickly get confusing. Let’s use Kotlin’s expressive type system to improve the situation and encapsulate the string in a very basic Passport class.

class Passport(private val text: String) {

}

We then just map the results of our split-up input to Passport objects:

// . . .
.map { Passport(it) }

From the problem description, we remember that key-value pairs are either separated by spaces or newlines within a single passport. Therefore, to get the individual pairs, we once again split our input. The delimiters, in this case, are either a space or one of the newline sequences.

fun hasAllRequiredFields(): Boolean {
    val fieldsWithValues = text.split(" ", "\n", "\r\n")
}

We then extract the key from each passport entry. We can do so by mapping our combined fieldsWithValues to only the substring that comes before the colon:

fun hasAllRequiredFields(): Boolean {
    val fieldsWithValues = text.split(" ", "\n", "\r\n")
    val fieldNames = fieldsWithValues.map { it.substringBefore(":") }
    return fieldNames.containsAll(requiredFields)
}

The result of our function will be whether the fieldNames we extracted contain all required fields. The requiredFields collection can be taken directly from the problem statement and translated into a list:

private val requiredFields = listOf("byr", "iyr", "eyr", "hgt", "hcl", "ecl", "pid" /*"cid"*/)

To calculate our final number, and get our first gold star for the challenge, we need to count the passports for which our function hasAllRequiredFields returns true:

fun main() {
    println(passports.count(Passport::hasAllRequiredFields))
}

With that, we have successfully solved the first part of the challenge and can set our sights on the next star in our journey.

Find the full code for the first part of the challenge on GitHub.

Solving Day 4, Part 2

In part two of the challenge, we also need to ensure that each field on the passport contains a valid value. We are given an additional list of rules to accomplish this task, which you can again find in the problem description. Years need to fall into specific ranges, as does a person's height depending on the unit of measurement. Colors need to come from a prespecified list or follow certain patterns, and numbers must be correctly formatted.

A refactoring excursion

Before we start building the solution for part 2, let’s briefly reflect on our code and find possible changes that will make adding this functionality easier for us. At this point in the challenge, we know that our Passport class will need access to the different field names and their associated values. The classical data structure to store such kind of associative-dictionary information is a map. Let’s refactor our code to store passport information in a map instead of a string.

Because turning an input string into a map is still a process that’s associated with the Passport, I like encapsulating such logic in a companion object “factory” function. In this case, we can aptly call it fromString:

companion object {
    fun fromString(s: String): Passport {

The implementation for fromString partially reuses the normalization logic we had previously used in the first part of this challenge and expands it to create a map directly via Kotlin’s associate function:

fun fromString(s: String): Passport {
    val fieldsAndValues = s.split(" ", "\n", "\r\n")
    val map = fieldsAndValues.associate {
        val (key, value) = it.split(":")
        key to value
    }
    return Passport(map)
}

A Passport object now encapsulates a map of string keys and string values:

class Passport(private val map: Map<String, String>) {

Interestingly enough, this change makes the implementation of the first part of our challenge trivial. We can simply check that the key set of our map contains all required fields:

fun hasAllRequiredFields() = map.keys.containsAll(requiredFields)

Returning to solving part 2

For the second part of the challenge, we consider a passport valid if it contains all the required fields and has values that correspond to the official rules.

To ensure that all fields have valid values, we can use the all function to assert that a predicate holds true for every single key-value pair in our map:

fun hasValidValues(): Boolean =
    map.all { (key, value) ->

We can distinguish the different types of fields using a when expression. In this first step, we distinguish the different cases based on the keys in our map:

when (key) {

Each key we know gets a branch in this when statement. They all need to return a boolean value – true if the field is okay, false if the field violates the rules. The surrounding all predicate will then use those results to determine whether the passport as a whole is valid.

The byr (Birth Year), iyr (Issue Year), and eyr (Expiration Year) fields all require their value to be a 4-digit number falling into a particular range:

"byr" -> value.length == 4 && value.toIntOrNull() in 1920..2002
"iyr" -> value.length == 4 && value.toIntOrNull() in 2010..2020
"eyr" -> value.length == 4 && value.toIntOrNull() in 2020..2030

Note that our combined use of toIntOrNull together with the infix function in allows us to discard any non-numeric values, and ensure that they fall in the correct range.

We can apply a very similar rule to the pid (Passport ID) field. We ensure that the length of the value is correct and ensure that all characters belong to the set of digits:

"pid" -> value.length == 9 && value.all(Char::isDigit)

Validating ecl (eye color) just requires us to check whether the input is in a certain set of values, similar to the first part of our challenge:

"ecl" -> value in setOf("amb", "blu", "brn", "gry", "grn", "hzl", "oth")

At this point, we have two more fields to validate: hgt (Height) and hcl (Hair Color). Both of them are a bit more tricky. Let’s look at the hgt field first.

The hgt (Height) field can contain a measurement either in centimeters or inches. Depending on the unit used, different values are allowed. Thankfully, both “cm” and “in” are two-character suffixes. This means we can again use Kotlin’s when function, grab the last two characters in the field value and differentiate the validation logic for centimeters and inches. Like our other number-validation logic, we parse the integer and check whether it belongs to a specific range. To do so, we also remove the unit suffix:

"hgt" -> when (value.takeLast(2)) {
    "cm" -> value.removeSuffix("cm").toIntOrNull() in 150..193
    "in" -> value.removeSuffix("in").toIntOrNull() in 59..76
    else -> false
}

The last field to validate is hcl (Hair Color), which expects a # followed by exactly six hexadecimal digits – digits from 0 through 9, and a through f. While Kotlin can parse base-16 numbers, we can use this case to show off the sledgehammer method for validating patterns – regular expressions. Those can be defined as Kotlin strings and converted using the toRegex function. Triple-quoted strings can help with escape characters:

"hcl" -> value matches """#[0-9a-f]{6}""".toRegex()

Our hand-crafted pattern matches exactly one hashtag, then six characters from the group of 0 through 9, a through f.

As a short aside for performance enthusiasts: toRegex is a relatively expensive function, so it may be worth moving this function call into a constant. The same also applies to the set used in the validation for ecl – currently, it is initialized on each test.

Because the whole when-block is used as an expression, we need to ensure that all possible branches are covered. In our case, that just means adding an else branch, which simply returns true – just because a passport has a field we don’t know about doesn’t mean it can’t still be valid.

else -> true

With that, we have covered every rule outlined to us by the problem statement. To get our reward, we can now just count the passports that contain all required fields and have valid values:

fun partTwo() {
    println(passports.count { it.hasAllRequiredFields() && it.hasValidValues() })
}

We end up with a resulting number, which we can exchange for the second star. We’re clear for boarding our virtual flight. Though this was probably not the last challenge that awaits us…

Find the complete solution for the second part of the challenge on GitHub.

Conclusion

For today’s challenge, we came up with an elegant solution to validate specific string information, which we extracted using utility functions offered by the Kotlin standard library. As the challenge continued, we reflected on our code, identified more fitting data structures, and changed our logic to accommodate it. Using Kotlin’s when statement, we were able to keep the validation logic concise and all in one place. We saw multiple different ways of how to validate input – working with ranges, checking set membership, or matching a particular regular expression, for example.

Many real-world applications have similar requirements for input validation. Hopefully, some of the tips and tricks you’ve seen in the context of this little challenge will also be helpful when you need to write some validation logic on your own.

To continue puzzling yourself, check out adventofcode.com, whose organizers kindly permitted us to use their problem statements for this series.

If you want to see more solutions for Advent of Code challenges in the form of videos, subscribe to our YouTube channel and hit the bell to get notified when we continue our idiomatic journey. More puzzle solutions are coming your way!

Exploring Kotlin Lists in 2021

Sebastian Aigner — Thu, 08 Jul 2021 14:46:58 +0000

This blog post accompanies a video from our YouTube series which you can find on our Kotlin YouTube channel, or watch here directly!

Today, we're talking all about lists! Lists are the most popular collection type in Kotlin for a good reason, and we’ll find out why together.

Lists

What’s a list?

If you've written Kotlin code before, you've definitely seen a list – they're collections of ordered elements, where each element is accessible via an index. As such, they're one of the basic building blocks for a lot of Kotlin code.

Creating lists

If you’re creating lists on your own, you’re most likely using the listOf function, which takes a variable number of arguments, and those become the elements of your list. Even in this blog post series, we've created a list like that about a hundred times:

listOf(1, 2, 3, 4, 5)
// [1, 2, 3, 4, 5]

A little lesser known is the ability to create lists via the List constructor function. Here, we pass two parameters – the size of the list, and an init function that creates each of the elements in our list. That function we pass gets the element index as its parameter, which we can use to adjust the item content:

List(5) { idx -> "No. $idx" }
// [No. 0, No. 1, No. 2, No. 3, No. 4]

Of course, lists can come from other places as well: types like collections, iterables, and others often feature a toList method.

For example, in the case of a string, we get a list of its characters:

"word-salad".toList()
// [w, o, r, d, -, s, a, l, a, d]

Given a map of placements and the associated medals, we can call toList on that to get a list of key-value pairs:

mapOf(
    1 to "Gold",
    2 to "Silver",
    3 to "Bronze"
).toList()
// [(1, Gold), (2, Silver), (3, Bronze)]

Sequences, ranges, and progressions behave similarly. They materialize their values, and put them in a list when calling toList. As an example, we can consider a random sequence of numbers, or the inclusive integer range from zero to ten:

generateSequence {
    Random.nextInt(100).takeIf { it > 30 }
}.toList()
// [73, 77, 69, 79, 71, 64]

(0..10).toList()
// [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

An extra case worth mentioning is calling toList on something that already is a list. This creates brand-new copy of the original list. We can see this in the following example, where we create a mutable list with a few numbers. By calling toList, we obtain a new working copy:

val list = mutableListOf(1, 2, 3)
val otherList = list.toList()

list[0] = 5
println(list)
// [5, 2, 3]

println(otherList)
// [1, 2, 3]

As we can see, when the original list is changed, the working copy we just created does not contain any of the changes applied to the original collection.

Accessing list items

To get items out of our lists, we have multiple options. The most basic way of doing so is using the get function, together with an index:

val myList = listOf("🍔", "🌭", "🍕")
myList.get(1) // 🌭

But if you ever type out .get manually, you’ll see that IntelliJ IDEA already gives you the helpful hint to use some much more popular syntactic sugar for it – the indexed access operator, denoted by the brackets with an index:

myList[1] // 🌭

There are also some additional flavors of the get function which we can explore. Two of those that come to mind are getOrElse and getOrNull. They help us handle cases where we might be accessing an index that falls out of bounds (which can either be a negative index, or an index that’s larger than the last index in our collection.)

Using the default indexed access causes an exception when provided a parameter that's out of bounds:

myList[3]
// Index 3 out of bounds for length 3

We can use getOrNull to short-circuit our return value to null. Alternatively, we can use getOrElse to compute a default value to be used instead. The default value is computed based on a passed lambda, which also receives the index:

val myList = listOf("🍔", "🌭", "🍕")
myList.getOrNull(3)
// null

myList.getOrElse(3) {
    println("There's no index $it!")
    "😔"
}
// There's no index 3!
// 😔

These special functions are only necessary to work with indexes that might fall out of bounds, though. Nullability, for example, is handled the same way as you would in any other situation in Kotlin: using the power of the Elvis operator, smart-casts and friends.

val listOfNullableItems = listOf(1, 2, null, 4)
val x: Int = listOfNullableItems[0] ?: 0

Slicing

Of course, we can go beyond getting individual items out of our list. Because a list is a collection like any other, we have access to the same take and drop functions that were introduced in the Diving into Kotlin collections post.

But lists have a special way of retrieving multiple items - the slice function!

When we give this function a bunch of indexes, it returns the elements at those places in our collection. In this example, we’re passing a list with index 0, 2, 4, and get those items from our list of letters:

val myList = listOf("a", "b", "c", "d", "e")
myList.slice(listOf(0, 2, 4))
// [a, c, e]

Instead of writing out all the indices by hand, we could also use IntRanges or progressions to specify the indexes. For example, we could request “all items from 0 through 3”, or specify a custom step-size of 2. We could even pull out some items in reverse order, if we create a progression that uses downTo:

myList.slice(0..3)
// [a, b, c, d]

myList.slice(0..myList.lastIndex step 2)
// [a, c, e]

myList.slice(2 downTo 0)
// [c, b, a]

As you may suspect, this list of list features is not quite exhaustive – as always, there’s some more to explore even on this subject. But let’s put that on the back burner for a bit, and move on to a special kind of list – it's time to talk about mutable lists!

Mutable Lists

What's so special about mutable lists? Well, you can mutate them! That, of course, doesn’t mean that these lists will turn into zombies (🧟‍♂️), but that you can change their content. If we consult an excerpt of a class hierarchy, we can see that MutableList specializes List, meaning everything we’ve learned about lists so far also works for their mutable counterpart, plus some extra functionality.

It's precisely that extra functionality that we’re interested in right now!

Creating mutable lists

Once again, mutable lists are commonly created via the mutableListOf function, with a bunch of values as arguments. And, wherever you were able to find a toList method, as discussed previously, you’ll probably also find a toMutableList. That also includes other lists and mutable lists – where you’ll get a fresh copy when calling toMutableList:

mutableListOf(1, 2, 3)
// [1, 2, 3]

(0..10).toMutableList()
// [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

listOf(1, 2, 3).toMutableList()
// [1, 2, 3]

Add / Remove / Update

Let's move on to the core of this subject – the ability to change content. That starts with adding something to the collection. If we want to add an extra number to the end of our mutable list we can do so via the add function, or by using the += operator shorthand, both of which append an item to the end of the list.

If we know where in the collection we want our item to go, the add function also accepts an index, which inserts the new element at that position and moves the surrounding elements to accommodate it. In the same way, we can also add a whole other collection to our mutable list:

val m = mutableListOf(1, 2, 3)
m.add(4)
m += 4
println(m)
// [1, 2, 3, 4, 4]

m.add(2, 10)
println(m)
// [1, 2, 10, 3, 4, 4]

m += listOf(5, 6, 7)
println(m)
// [1, 2, 10, 3, 4, 4, 5, 6, 7]

We’re of course not constrained to just adding elements to our list – we can also remove them. If we know what element we want to get rid of, we can do that via the remove function or the -= operator shorthand, which removes from our collection a single instance of the element we provide. In this example, after calling -= and remove, we got rid of two of the 3s in our original collection – because each invocation removed one of them.

Alternatively, we can also pass the -= operator a collection of elements. In this case, the operator acts as a shorthand for the removeAll function. Here, it looks at every element in the collection we pass, and removes all instances of them in our original, mutable collection. (This is an important distinction to make!) So, by passing 1 and 4 as a collection, we remove all instances of those numbers from our mutable list, and we’re left with only 2 and 3 at the end.

val m = mutableListOf(1, 2, 3, 3, 3, 4, 4, 4)
m -= 3
m.remove(3)
println(m)
// [1, 2, 3, 4, 4, 4]

m -= listOf(1, 4)
println(m)
// [2, 3]

If we know the index where we want to kick an item out, we use the removeAt function instead. For example, we could remove the second element in our list, which resides at index 1.

val m = mutableListOf(1, 2, 3, 3, 3)
m.removeAt(1)
println(m)
// [1, 3, 3, 3]

To update an item, we most commonly use the indexed access operator – so the brackets – together with an assignment. That one calls the set function with that index and element under the hood, and switches out the item at the specified index – in this case, trading a "b" for an "a".

val m = mutableListOf("a", "b", "c", "d", "e")
m[1] = "a"
println(m)
// [a, a, c, d, e]

Fill and Clear

In certain situations, we might want to turn all elements of our list into the same element – like zeroing out a buffer before reusing it. This is something we can do using the fill function, which replaces each element with the same value we specify. If we look at a list of fruits, for example, and suddenly realize that all of them are really just sugar, we use fill to replace them with candy (🍬). While that metaphor may not be entirely scientifically accurate, it's tasty nonetheless!

And when we want to wipe our collection clean, the clear function can help with removing all elements from a collection – in our case, getting rid of all the candy:

val fruits = mutableListOf("🍉", "🍊", "🥝")
// wait, it's all sugar?
fruits.fill("🍬")

println(fruits)
// [🍬, 🍬, 🍬]

// ... nom nom
fruits.clear()

println(fruits)
// []

Perhaps unsurprisingly, mutable lists grow and shrink automatically to accommodate all your items, so you can have an arbitrary number of elements in your collection. This might be obvious, but it’s so darn convenient, so I figured I’d mention it. The things we take for granted!

In-place modifications

Thinking back to some of the previous entries of this series, we’ve seen a number of neat functions which we wouldn’t want to miss for mutable collections either – things like sorted, shuffled, and reversed. However, those don’t modify the original collection.

Luckily for us, these functions also have a mutable counterpart. So, when we want to sort, shuffle, or reverse a mutable list in place – instead of creating a new, separate copy with the effects applied – we use the sort() instead of sorted(), shuffle() instead of shuffled(), and reverse() instead of reversed() functions:

val list = listOf(3, 1, 4, 1, 5, 9)
list.shuffled()
list.sorted()
list.reversed()

println(list)
// [3, 1, 4, 1, 5, 9]

val m = list.toMutableList()
m.shuffle()
println(m)
// [5, 1, 1, 3, 4, 9]

m.sort()
println(m)
// [1, 1, 3, 4, 5, 9]

m.reverse()
println(m)
// [9, 5, 4, 3, 1, 1]

Mutable lists also offer the possibility to remove or keep all elements that fulfill a certain predicate. The removeAll function can remove all elements that match the predicate we specify. Let’s say we’re not a fan of small numbers in our collection, and only want to keep numbers that are 5 or above – removeAll helps us do exactly that.

val numbers = mutableListOf(3, 1, 4, 1, 5, 9)
numbers.removeAll { it < 5 }
println(numbers)
// [5, 9]

The retainAll function is the opposite, and only keeps those elements in the mutable list that match. If we want to retain every character in our collection that is a letter, we do that with the retainAll function:

val letters = mutableListOf('a', 'b', '3', 'd', '5')
letters.retainAll { it.isLetter() }
println(letters)
// [a, b, d]

This might feel a bit familiar to you, and rightfully so, because these are essentially the mutating equivalents of the filter and filterNot functions.

Views on Lists

The last topic on today’s agenda is views on lists. That name already hints at what they allow us to do – they allow us to look at the elements in our list from a different perspective – let’s see what that means.

Let’s assume we have a collection of fruits. To create a view, we can use the subList function, which takes a beginning and end index, which determines which elements should be “visible” in the view. By having a look at an example sublist, we can see that it contains the elements from our original collections based on the indices we specify (with the upper bound being exclusive):

val fruits = mutableListOf("🍉", "🍊", "🥝", "🍏")
val sub = fruits.subList(1, 4)
println(sub)
// [🍊, 🥝, 🍏]

Because this is only a view, and not a copy of our original collection, changes are automatically visible. That means if we change the orange to a banana in the underlying fruits list, then our sublist will reflect that change:

fruits[1] = "🍌"
println(sub)
// [🍌, 🥝, 🍏]

What may be even more interesting is that this sublist is in itself mutable, as well! If we change the green apple in our sublist to a pineapple, and have a look at our original fruits collection again, we see that the change is visible from here as well:

sub[2] = "🍍"
println(fruits)
// [🍉, 🍌, 🥝, 🍍]

Or, we can use the fill function which we’ve learned about earlier to turn an interval inside of our fruit-list back into candy, again:

sub.fill("🍬")
println(fruits)
// [🍉, 🍬, 🍬, 🍬]

To reiterate; all of that works because these aren’t two different collections – there is only one collection, and subList has just given us a different perspective on that list!

An important note on the topic of sublists: They are only well-defined as long as the underlying, original list is not structurally changed. Changes affecting the size of the list, for example, automatically cause any views that were previously returned by invoking subList to have undefined behavior.

For a common case, which is looking at a list backwards, the Kotlin standard library also comes with the asReversed function. It provides a backwards view of the underlying list. Once again, changes made in the view are visible in the original collection, and vice versa. As you can see in the following example, turning the orange into a banana in our original list also changes what we see in the reversed view. Altering it back to a pineapple via our reversed view also alters our original mutable list:

val fruits = mutableListOf("🍉", "🍊", "🥝", "🍏")
val stiurf = fruits.asReversed()

println(stiurf)
// [🍏, 🥝, 🍊, 🍉]

fruits[1] = "🍌"
println(stiurf)
// [🍏, 🥝, 🍌, 🍉]

stiurf[2] = "🍍"
println(fruits)
// [🍉, 🍍, 🥝, 🍏]

These types of “views” are actually available for non-mutable lists, as well, and allow you to pass around different sub-selections of your collections without having to create new copies every time – however, this seemed like a topic that would be nicer to illustrate with the mutable variant, to drive the point home that there really is only one underlying collection.

Outro

With that, we have reached the end of today’s expedition! I hope some of the stuff you’ve seen today is helping you strengthen your understanding of Kotlin lists. When you’re writing Kotlin code the next time, see if you can apply some of the stuff we’ve talked about today – whether it’s slicing a collection, using sub-lists, or handling out-of-bounds situations for lists elegantly with the getOrNull and getOrElse functions.

If you’ve learned something new, and want to see more like this, make sure to follow us here on dev.to, and check out some of the interesting content we publish on our official YouTube channel, as well!

Now, it’s time for all of you to go and explore some more Kotlin! Take care, and see you in the next one!

Advanced Kotlin Collection Functionality

Sebastian Aigner — Mon, 14 Jun 2021 17:49:10 +0000

This blog post accompanies a video from our YouTube series which you can find on our Kotlin YouTube channel, or watch here directly!

Today, we are learning about advanced functions that we can use to work with and manipulate all kinds of Kotlin collections!

Checking predicates: `any`, `none` and `all`

Let’s warm up by having a look at a selection of functions that allow us to check conditions for our collection elements.

They’re called any, none, and all. Each of them takes a predicate – so a function that returns true or false – and checks whether the collection fits this predicate.

Let’s say we have a group of friends (which is really just a List<Person>, each featuring a name, age, and maybe a driversLicense):

data class Person(val name: String, val age: Int, val driversLicense: Boolean = false)

val friendGroup = listOf(
    Person("Jo", 19),
    Person("Mic", 15),
    Person("Hay", 33, true),
    Person("Cal", 25)
)

When we want to check if the group can travel by car, we want to check if any of them have a driver's license – so we use the any function. It returns true if there is at least one element in our collection for which the predicate returns true.

val groupCanTravel = friendGroup.any { it.driversLicense }
// true

As another example, let’s say we want to check if this group of friends is allowed to enter a club – for this, we would need to make sure that none of the folks in the group are underage!

Here, we can use the none function, which only returns true when there is not a single element in our collection that holds true for our predicate:

val groupGetsInClub = friendGroup.none { it.age < 18 }
// false

The third function in the bunch is the all function. At this point, you can probably spot the pattern – all returns true, if each and every element in our collection matches our predicate. We could use it to check whether all names in our friend group are short:

val groupHasShortNames = friendGroup.all { it.name.length < 4 }
// true

Predicates for empty collections

While on the topic, let's have a little brain teaser: How do any, none, and all behave for empty collections?

val nobody = emptyList<Person>()
// what happens here?

Let’s look at any first. There is no element that can satisfy the predicate, so it returns false:

nobody.any { it.driversLicense }
// false

The same goes for none – there is no function that can violate our predicate, so it returns true:

nobody.none { it.age < 18 }
// true

The all function, however, returns true with an empty collection. This may surprise you in the first moment:

nobody.all { it.name.count() < 4 }

But this is quite intentional and sound: You can't name an element that violates the predicate. Therefore, the predicate has to be true for all elements in the collection – even if there are none!

This might feel a bit mind-bending to think about at first, but you’ll find that this concept, which is called the vacuous truth, actually plays very well with checking conditions, and expressing logic in program code.

Collection parts: `chunked` and `windowed`

With our brain freshly teased, let’s move on to the next topic, and learn about how to break collections into parts!

The `chunked` function

If we have a collection that just contains a bunch of items, we can cut up the list into individual chunks of a certain size by using the chunked function. What we get back is a list of lists, where each element is a _chunk _of our original list:

val objects = listOf("🌱", "🚀", "💡", "🐧", "⚙️", "🤖", "📚")
println(objects.chunked(3))
// [[🌱, 🚀, 💡], [🐧, ⚙️, 🤖], [📚]]

In the example above, we break our list of random objects (represented with emojis) apart, using a chunk size of 3.

The first element in our result is in itself a list which contains our first three objects – [🌱, 🚀, 💡].
The second element is once again a chunk, and contains the three elements that follow after that – [🐧, ⚙️, 🤖].
The last element is also a chunk – but since we ran out of elements to fill it with three items, it only contains the book stack - [📚].

In typical standard library fashion, the chunked function also provides a little bit of extra power. To immediately transform the chunks we just created, we can apply a transformation function. For example, we can reverse the order of elements in the resulting lists, without having to do another map call separately:

println(objects.chunked(3) { it.reversed() })
// [[💡, 🚀, 🌱], [🤖, ⚙️, 🐧], [📚]]

To summarize: the chunked function cuts our original collection into lists of lists, where each list has the specified size.

The `windowed` function

Closely related is the windowed function. It also returns a list of lists from our collection. Instead of cutting it up into pieces, however, this function generates a “sliding window” of our collection:

println(objects.windowed(3))
// [[🌱, 🚀, 💡], [🚀, 💡, 🐧], [💡, 🐧, ⚙️], [🐧, ⚙️, 🤖], [⚙️, 🤖, 📚]]

The first window is once again the first three elements – [🌱, 🚀, 💡].
The next window is [🚀, 💡, 🐧] – we simply “moved” our window of size 3 over by one, which includes some overlap.

The windowed function can also be customized. We can change both window and step size, the latter being the number of elements that the window should “slide along” for each step:

println(objects.windowed(4, 2, partialWindows = true))
// [[🌱, 🚀, 💡, 🐧], [💡, 🐧, ⚙️, 🤖], [⚙️, 🤖, 📚], [📚]]

As you can see in the example above, we can also control whether our result should contain partial windows. This changes the behavior when we’ve reached the end of our input collection, and we’re running out of elements.

With partial windows enabled, we just keep sliding, and we get the last elements trickling in, in the form of smaller windows, until we get a window which once again only contains the last element from our input collection – [⚙️, 🤖, 📚], [📚].

windowed also allows us to perform an additional transformation at the end, which can modify the individual windows directly:

println(objects.windowed(4, 2, true) {
    it.reversed()
})
// [[🐧, 💡, 🚀, 🌱], [🤖, ⚙️, 🐧, 💡], [📚, 🤖, ⚙️], [📚]]

Un-nesting Collections: Flatten and Flatmap

The chunked and windowed functions, along with some others all return nested collections – lists of lists. What if we want to un-nest these, turning them back into flat lists of elements? As usual, we do not need to fear, because the standard library has got us covered.

We can call the flatten function on a collection of collections. As you may suspect, the result is a single list of all the elements that were originally contained inside of our nested collections:

val objects = listOf("🌱", "🚀", "💡", "🐧", "⚙️", "🤖", "📚")
objects.windowed(4, 2, true) {
    it.reversed()
}.flatten()
// [🐧, 💡, 🚀, 🌱, 🤖, ⚙️, 🐧, 💡, 📚, 🤖, ⚙️, 📚]

This is also a good point to talk about the flatMap function. flatMap is like a combination of first using map, and then using flatten – It takes a lambda which generates a collection from each of the elements in our input collection:

val lettersInNames = listOf("Lou", "Mel", "Cyn").flatMap {
    it.toList()
}
println(lettersInNames)
// [L, o, u, M, e, l, C, y, n]

In the example above, the function that we provide creates a list for each element in our input collection, containing the letters of the original string. Next, that collection of collections gets flattened. As desired, we end up with a plain list of elements – the list of characters from the names of the original collection.

If you are doing an operation on a list, which in turn generates a collection for each one of the input elements, consider if flatMap can help you simplify your code!

Combining collections: `zip` and `unzip`

So far, we have always looked at a single collection, and what we can do with it. Let's learn about a way to combine two collections, and create a new one from them – it's time to zip!

The `zip` function

Assume we have two collections, where the elements at each index are somehow related. For example, this could be a list of cities in Germany, and we have another list of German license plates that correspond to those cities:

val germanCities = listOf(
    "Aachen",
    "Bielefeld",
    "München"
)

val germanLicensePlates = listOf(
    "AC",
    "BI",
    "M"
)

println(germanCities.zip(germanLicensePlates))
// [(Aachen, AC), (Bielefeld, BI), (München, M)]

As you can see, by zipping these two collections, we get a list of pairs, where each pair contains the elements with the same index from the original two collections.

Metaphorically, this is similar to a zipper on a jacket, where the teeth match up one by one. We zip together the elements of our collection, and we get pairs of each city and its corresponding license plate.

For an extra bit of flair, we can also call the zip function using infix notation:

println(germanCities zip germanLicensePlates)
// [(Aachen, AC), (Bielefeld, BI), (München, M)]

zip can also take a transformation function. We can pass a lambda that receives the values of the individual zipped pairs, and we can apply a transformation:

println(germanCities.zip(germanLicensePlates) { city, plate ->
    city.uppercase() to plate.lowercase()
})
// [(AACHEN, ac), (BIELEFELD, bi), (MÜNCHEN, m)]

The `unzip` function

The standard library also contains the inverse function, called unzip, which takes a list of pairs, and splits them back into a pair of two separate lists:

val citiesToPlates = germanCities.zip(germanLicensePlates) { city, plate ->
    city.uppercase() to plate.lowercase()
}
val (cities, plates) = citiesToPlates.unzip()

println(cities)
// [AACHEN, BIELEFELD, MÜNCHEN]

println(plates)
// [ac, bi, m]

The example above uses a destructuring declaration to easily access both of them.

The `zipWithNext` function

In a way, zipWithNext is really a specialized case of the windowed function we got to know today: Instead instead of zipping together two separate lists element by element, this function takes one collection, and zips each of its items with the one that follows it:

val random = listOf(3, 1, 4, 1, 5, 9, 2, 6, 5, 4)
println(random.zipWithNext())
// [(3, 1), (1, 4), (4, 1), (1, 5), (5, 9), (9, 2), (2, 6), (6, 5), (5, 4)]

In the example above, we're zipping together a list of numbers. If we want to check the “change” – how much the value increments or decrements each step – we can express this quite elegantly using zipWithNext. We provide a lambda that receives a pair of one number and the one that follows immediately after:

val random = listOf(3, 1, 4, 1, 5, 9, 2, 6, 5, 4)

val changes = random.zipWithNext { a, b -> b - a }
println(changes)
// [-2, 3, -3, 4, 4, -7, 4, -1, -1]

Custom aggregations: `reduce` and `fold`

We have finally arrived at the grand finale for this post – functions that help us build custom aggregations.

The `reduce` function

Let’s set the scene with a small callback – in the previous post, we learned about functions like sum, average, count, and functions to receive the minimum and maximum elements inside a collection. All of these reduce our collection to a single value.

It's possible that we find ourselves in a situation where there’s no out-of-the-box function for how we want to generate a single value for our collection. For example, we may want to multiply all numbers in a list, instead of summing them.

In this case, we can rely on the reduce function as a more generic version for aggregating a collection:

val random = listOf(3, 1, 4, 1, 5, 9, 2, 6, 5, 4)
val multiplicativeAggregate = random.reduce { acc, value -> acc * value }

println(multiplicativeAggregate)
// 129600

As seen in the example above, we call the reduce function with a lambda block which receives two parameters:

An accumulator, which has the same type as our collection, and
An individual item from our collection.

The task of the lambda function is to return a new accumulator. Each invocation, one after the other, receives not only the current element, but also the result of the previous calculation, inside the accumulator.

The function starts with the first element of our collection in the accumulator.
Then it runs our operation – in this example, we multiply the accumulator (which right now is the first number) with the current element (which is the second number).
We’ve calculated a new value, which will be stored in the accumulator, and used when our function is called once more with the third element

This cycle repeats, and we continue to gradually build up the final result in our accumulator. One might even say we’re accumulating that result!

Once we’ve gone through all the elements in our collection, reduce returns the final value that’s inside the accumulator.

As you can see, with reduce, we can hide a lot of mechanics for aggregating our collection behind one function call, and stay true to Kotlin’s concise nature.

The `fold` function

But we can actually go beyond this, and can take this versatility one step further – with the fold operation. Remember – when we used reduce, the iteration starts with the first element of our input collection in the accumulator.

With the fold function, we get to specify our own accumulator – and in fact, it can even have a different type than the items in our input collection! As an example, we can take a list of words, and multiply the number of their characters together using fold:

val fruits = listOf("apple", "cherry", "banana", "orange")
val multiplied = fruits.fold(1) { acc, value ->
    acc * value.length
}
println(multiplied) // 1080

The underlying mechanism is the same – the lambda passed to the fold function gets called with an accumulator and a value, and calculates a new accumulator. The difference is that we specify the initial value of the accumulator ourselves.

(Note that we pass 1 as an initial value for our accumulator, and not 0. That’s because for multiplication, 1 is the neutral element)

Both fold and reduce come in a number of other flavors, as well:

– the sibling functions reduceRight and foldRight change the order of iteration

reduceOrNull allows you to work with empty collections without throwing exceptions.
runningFold and runningReduce don’t just return a single value representing the final state of the accumulator, but instead return a list of all the intermediate accumulator values as well.

That's it!

This concludes my overview of some advanced collection operations in Kotlin – I hope you found this post useful, and have learned something new!

Maybe you can find a point in your code where a predicate, some zipping, chunking or windowing could come in handy! Or maybe you want to explore by defining your own aggregations functions based on the reduce or fold functions.

To get reminded when new Kotlin content is released, follow us here on dev.to/kotlin, and make sure to follow me on Twitter @sebi_io.

Also, use this opportunity sure to find the subscribe button and notification bell on our YouTube channel!

Take care!

Tips & tricks for building a game using Compose for Desktop

Sebastian Aigner — Thu, 06 May 2021 13:03:41 +0000

In the first part of my blog post series about building a small clone of the classic arcade game Asteroids on top of Jetpack Compose for Desktop, we saw how to implement the main game loop, as well as manage state and draw basic shapes. In this post, we will explore some more details of the game implementation. This includes:

Rendering details – making sure game objects don't escape our play area, and using a device-independent coordinate system for rendering
Geometry and linear algebra – the secret sauce that makes the space ships fly
Frame-independent movement – so that our game works consistently.

Let's learn about these topics!

Rendering: Clipping and Coordinate Systems

In the context of rendering, there are two areas that still need our attention – we need to make sure that our game objects are constrained to the game surface, and we need to make a conscious decision about the units of the coordinates we use to describe the position of a game object. We'll discuss both in this section.

Clipping

By default, Compose naively draws your objects without any clipping. This means game objects can poke outside the "play surface", which produces a weirdly fourth-wall-breaking effect:

We constrain the game objects to the bounds of our play surface by applying Modifier.clipToBounds() to the Box which defines our play surface:

Box(modifier = Modifier
                .fillMaxWidth()
                .fillMaxHeight()
                .clipToBounds()
                // . . .

Because all our game elements are drawn as children of this play area Box, using this modifier causes the rendered entities inside it to be cut off at the edges (instead of being drawn over the surrounding user interface):

Device-Independent Pixels and Density

Something else to be aware of when doing any kind of rendering tasks in Compose for Desktop is to keep the units of measurement in the back of your mind.

Wherever I worked with coordinates, I decided to work in device-independent pixels:

The mouse pointer position is stored as a DpOffset
Game width and height are stored as Dps
Game objects are placed on the play surface using their .dp coordinates.

This helps the game work consistently across high-density displays and low-density displays alike. However, it also requires some operations to be performed in the context of Density.

For example, the pointerMoveFilter returns an Offset in pixels – and they are not device-independent!. To work around this, we obtain the local screen density in our composition:

val density = LocalDensity.current

We then use with(density) to access the toDp() extension functions to the Offset into a DpOffset, allowing us to store our targetLocation in this device-independent pixel format:

.pointerMoveFilter(onMove = {
    with(density) {
        game.targetLocation  = DpOffset(it.x.toDp(), it.y.toDp())
    }
    false
})

For storing the play area's width and height, we do a very similar thing, just without wrapping it in a DpOffset:

.onSizeChanged {
    with(density) {
        game.width = it.width.toDp()
        game.height = it.height.toDp()
    }
}

A Game of Geometry and Linear Algebra

Underneath the visualization, the "Asteroids" game builds on just a few basic blocks to implement its mechanics – it is really a game of vectors and linear algebra:

The position, movement, and acceleration of the ship can be described by position, movement, and acceleration vectors.
The orientation of the ship is the angle of the vector between the ship and the cursor.
Circle-circle collisions can be tested based on distance vectors.

Instead of reinventing the ~~wheel~~ vector, I decided to use openrndr-math, which includes an implementation of the Vector2 class including all common operations, like scalar multiplication, addition, subtraction, the dot product, and more. (Ever since listening to the Talking Kotlin episode, I've been meaning to explore OPENRNDR in detail, but that will have to happen in a separate project.)

As somebody who happens to be a bit rusty with their linear algebra skills, I extended the functionality of the class a bit. For example, I defined the following extension function to allow me to access the angle a Vector2 in degrees between 0-360:

fun Vector2.angle(): Double {
    val rawAngle = atan2(y = this.y, x = this.x)
    return (rawAngle / Math.PI) * 180
}

Thankfully, I did not have to spend too much time on figuring out the call to atan2, because I previously watched one of Leland Richardson's live streams where he also uses this function to calculate some angles.

Extensions like this one help me express ideas in ways I understand them myself – and hopefully still will a few months down the road.

I also made use of properties with backing fields to make it possible to access a GameObject's movement vector in different representations:

As a combination of length (speed) and angle
As a vector with x and y coordinates

In the context of a GameObject, that can look like the following, for example:

var speed by mutableStateOf(speed)
var angle by mutableStateOf(angle)
var position by mutableStateOf(position)
var movementVector
    get() = (Vector2.UNIT_X * speed).rotate(angle)
    set(value) {
        speed = value.length
        angle = value.angle()
    }

If we're using this functionality outside of the GameObject class a lot, we could also consider defining additional length / angle getters and setters as extension properties on the Vector2 class, directly.

For our simulation, we still need to do a bit more – we haven't yet addressed the problem of how to update location and speed based on the elapsed real time. Let's talk about the approach for that next.

Frame-Independent Movement With Delta Timing

When building game logic, we need to keep one essential point in mind: Not all frames are created equal!

On a 60 Hz display, each frame is visible for 16ms.
On a 120 Hz display, that number drops to 8.3ms.
On a 240 Hz display, each frame only shows for 4.2ms.
On a system under load, or while running in a non-focused window, the application frame rate may be lower than 60 Hz.

That means that we can't use "frames" as a measurement of time: If we define the speed of our spaceship in relation to the frame rate, it would move four times faster on a 240 Hz display than on a 60 Hz display.

We need to decouple the game logic (and its rudimentary "physics simulation") from the frame rate at which our application runs. Even AAA games don't get this right all the time – but for our projects, we can do better!

A straightforward approach for this decoupling is to use delta timing: We calculate the new game state based on the time difference (the delta) since the last time we updated the game.
This usually means we multiply the result of our calculations with the time delta, scaling the result based on the elapsed time.

In Compose for Desktop, we use withFrameMillis and withFrameNanos. Both of them provide a timestamp, so we just need to keep track of the previous timestamp to calculate the delta:

var prevTime = 0L

fun update(time: Long) {
    val delta = time - prevTime
    // . . .

In my case, a GameObject has an update function that takes a realDelta: Float:

val velocity = movementVector * realDelta.toDouble()
obj.position += velocity

As demonstrated in the code above, I use it to scale the velocity of game objects.

Closing Thoughts

This concludes our tour of building a small game with Compose for Desktop! To see how all the pieces fit together, read the source code (~300 lines of code) on GitHub!

Building Asteroids on Compose for Desktop was great fun! I am always surprised by the iteration speed that Compose for Desktop provides: Getting from a first rectangle to a full game in just one long evening.

Of course, implementing a retro game like Asteroids on modern hardware comes with the luxury of not having to think too hard about performance optimizations, allocations, entity-component systems, or more. When building something more ambitious, these points likely need addressing, and you might find yourself using a few additional libraries besides a Vector2 implementation.

For the next Super Hexagon, pixel roguelike, or other 2D game, however, you can definitely give Compose a shot.

Once again, you can find all 300 lines of source code for this project on GitHub.

If you're looking for additional inspiration, take a look at some other folks building games with Compose!

Vivek Sharma built everybody's favorite dinosaur game
vitaviva built Tetris with Compose
John O'Reilly made a Compose for Desktop CHIP-8 frontend
theapache64 pushes the limits of Compose's builtin components to implement Switch, Check, and Radio Snake

How I built an "Asteroids" game using Jetpack Compose for Desktop

Sebastian Aigner — Thu, 06 May 2021 13:03:23 +0000

A while ago, I tweeted about a small game I had created on top of Jetpack Compose for Desktop: A small clone of the classic arcade game Asteroids, in which you control a space ship with your mouse, and navigate the vastness of space, avoiding and breaking asteroids in the process.

// Detect dark theme var iframe = document.getElementById('tweet-1382668779377762305-462'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1382668779377762305&theme=dark" }

Today, it's time to take a look under the hood and understand how I built a basic version of this game, and how Compose for Desktop helped me achieve it in just one evening!

We will take a look at parts and structures in the code that I find the most interesting. To see how it all fits together, I suggest exploring the whole code on GitHub. The whole implementation is only 300 lines of code, which I hope makes studying and understanding it easy.

The Game

If you're not caught up on your 80s arcade trivia, Asteroids was a popular arcade game where you try to steer your space ship through space, avoiding and destroying asteroids with your ship.

Because of the limitations of the hardware at the time, the game is quite simplistic in appearance: a triangular spaceship moves across a plain background and avoids simple displays of asteroids on a 2D surface.

What makes this a challenge is the interia: Just like a real space ship, your spaceship moves along its course in a straight line at constant speed, and you need to make corrective maneuvers by turning your ship and directing your thrust.

Asteroids has achieved cult status in the arcade game scene. Because of that, I wanted to see what it would take to recreate this experience using Jetpack Compose for Desktop!

The Building Blocks

I have roughly divided the project into a few building blocks that make up the project, and that we will talk about. Namely, those are:

The Game Loop
Game State Management
Rendering to the Screen

In the second part of this series on building a game with Compose for Desktop, we will also look at additional rendering details, the geometry and linear algebra behind the game, and frame-independent movement.

Let's dive right in!

The Game Loop

At the center of most games stands the game loop. It acts as the entry point that calls the game logic code. This is a fundamental difference between implementing typical declarative user interfaces and building games:

Declarative UI is usually mostly static, and reacts to user actions (clicking, dragging) or other events (new data, computation progress...)
Games run their logic many times per second, simulating the game world and its entities one frame at a time.

That is not to say that these two approaches are incompatible! All we need to run a main "game loop" is to get our function to execute once per frame. In Jetpack Compose, we have the withFrame family of functions (withFrameMillis, withFrameNanos), which can help us achieve exactly that.

Let's assume we already have a game object – we will talk about state management shortly. We can then create a LaunchedEffect which asks Jetpack Compose for Desktop to call our update function whenever a new frame is rendered:

LaunchedEffect(Unit) {
    while (true) {
        withFrameNanos {
            game.update(it)
        }
    }
}

withFrameNanos is a suspending method. Its exact implementation is described in the documentation:

withFrameNanos suspends until a new frame is requested, immediately invokes onFrame with the frame time in nanoseconds in the calling context of frame dispatch, then resumes with the result from onFrame.

The frame time, which it provides us with, will also come in handy, as we will see in the second part of this blog post series, when we talk about frame-independent movement.

Game State Management

Jetpack Compose is excellent at managing state, and when building a game like Asteroids, we can use the same mechanisms to keep track of the data attached to game objects or the current play session, to name just two examples.

As suggested in the previous section, my Asteroids game has a Game class, an instance of which is wrapped in a remember call in the main composition.

val game = remember { Game() }

It acts as a container for all game-related data. For example:

Game object information (in a mutableStateListOf)
The current game phase (RUNNING / STOPPED)
The size of the playing field (based on window dimensions)

Inside the Game object, we treat the state data as mutable, and make any state changes as we see fit.

Game Objects

Individual game objects once again group the state belonging to an individual game entity: a spaceship, an asteroid, or a bullet, and provide methods to modify their state, spawn new game objects, or check their relation to other game objects.

In my implementation of Asteroids, all game objects share a lot of behavior, from the way they move through the environment to how they check their collision – we'll talk about the geometry and linear algebra that goes into that a bit later.

The GameObject class provides implementations for these shared behaviors:

sealed class GameObject(speed: Double = 0.0, angle: Double = 0.0, position: Vector2 = Vector2.ZERO) {
    var speed by mutableStateOf(speed)
    var angle by mutableStateOf(angle)
    var position by mutableStateOf(position)
    var movementVector /* ... */
    abstract val size: Double

    fun update(realDelta: Float, game: Game) {
        val velocity = movementVector * realDelta.toDouble()
        position += velocity
        position = position.mod(Vector2(game.width.value.toDouble(), game.height.value.toDouble()))
    }

    fun overlapsWith(other: GameObject): Boolean {
        return this.position.distanceTo(other.position) < (this.size / 2 + other.size / 2)
    }
}

For example, the ShipData class inherits speed, angle, position and its update method from GameObject, but defines its own size, angle, and a function to fire a bullet:

class ShipData : GameObject() {
    override var size: Double = 40.0
    var visualAngle: Double = 0.0

    fun fire(game: Game) {
        val ship = this
        game.gameObjects.add(BulletData(ship.speed * 4.0, ship.visualAngle, ship.position))
    }
}

Note that the ShipData (or even a GameObject in general) does not include any logic on how to render this item to the display – with Jetpack Compose, keeping state and presentation separated is quite easy.

Because a lot of behavior is shared between all types of entities in the game, our main game loop can treat them as the supertype GameObject for the most part, and only specific interactions between certain types of objects, like bullet-asteroid or asteroid-player collisions, need to be handled specifically.

Rendering to the Screen

I found that in Jetpack Compose, separating game data from the visual representation comes quite naturally. Game objects like a ship, an asteroid, or a bullet are all represented in two parts:

A class holding the state associated with the game object (in terms of "Compose state" – via mutableStateOf and friends) – We briefly talked about this in the previous section.
A @Composable, defining the rendering based on the game object's data.

To illustrate the latter, here's the minimal visual representation of the Asteroid composable. It receives asteroidData, which is the container for all information regarding the state of this particular game object:

@Composable
fun Asteroid(asteroidData: AsteroidData) {
    val asteroidSize = asteroidData.size.dp
    Box(
        Modifier
            .offset(asteroidData.xOffset, asteroidData.yOffset)
            .size(asteroidSize)
            .rotate(asteroidData.angle.toFloat())
            .clip(CircleShape)
            .background(Color(102, 102, 153))
    )
}

This code snippet is enough to describe the whole visual representation of an asteroid.

We start with a Box – one of Compose's most basic layout primitives, which allows us to have entities overlap (which is useful since we manually take care of placing the individual entities). We then use Jetpack Compose's Modifiers to specify the position of the asteroid in the form of an offset, its size, rotation angle, shape (by clipping a CircleShape), and background color.

Note that Compose offers quite high-level APIs even for these basic shapes – for example, we can use .rotate directly, without having to manually do geometry work to figure out how to get our entities facing the right way.

To keep this snippet as concise as possible, I've also introduced some extension functions on GameObject that make it possible to reuse the logic of computing the offset of a game object based on its position and size, called xOffset and yOffset, which I've snuck into the previous code snippet already. Their implementation is relatively straightforward:

val GameObject.xOffset: Dp get() = position.x.dp - (size.dp / 2)
val GameObject.yOffset: Dp get() = position.y.dp - (size.dp / 2)

A slightly more complicated composable would be the Ship component, which combines the shapes of a triangle and circle to create a minimalistic spaceship:

@Composable
fun Ship(shipData: ShipData) {
    val shipSize = shipData.size.dp
    Box(
        Modifier
            .offset(shipData.xOffset, shipData.yOffset)
            .size(shipSize)
            .rotate(shipData.visualAngle.toFloat())
            .clip(CircleShape)
            .background(Color.Black)
    ) {
        Canvas(modifier = Modifier.fillMaxSize(), onDraw = {
            drawPath(
                color = Color.White,
                path = Path().apply {
                    val size = shipSize.toPx()
                    moveTo(0f, 0f) // Top-left corner...
                    lineTo(size, size / 2f) // ...to right-center...
                    lineTo(0f, size) // ... to bottom-left corner.
                }
            )
        })
    }
}

The Box defining the ship is quite similar to the one we saw for an Asteroid, but we additionally add a Canvas to draw some additional shapes on top of our spaceship – in this case, a triangle path. In typical Compose fashion, we just add this Canvas in the lambda block following our Box, meaning the Canvas will inhert the coordinate system of its parent, including its offset and rotation.

These composables are then just rendered to a play surface – nothing more than a Box with a locked aspect ratio of 1.0f (to keep it quadratic). Of course, applying some artistic talent to these visual representations of the game is also possible, but we're keeping it minimal for now.

Continued in Part 2

There's still a bit more work to do until we can call our game done. In part 2 of this blog post series, we will look at additional rendering details, the geometry and linear algebra behind the game's simple physics simulation, as well as frame-independent movement. Read on and find out!

Why Learn Kotlin?

Sebastian Aigner — Wed, 28 Apr 2021 18:00:55 +0000

Ksenia Shneyveys, Kotlin Marketing Manager for Education, shares insights on how and why to get started with the Kotlin programming language.

We would love to hear about your own motivations for learning Kotlin by leaving your story on our official blog! If you share your story with us, you will also have the chance to win an “Atomic Kotlin” ebook!

Kotlin has always been and continues to be a modern language in the industry, addressing the real needs of real developers. It is increasingly being adopted in many trending fields, including mobile, web, server-side and cloud development, data science, and education.

This all gives a strong indication that there is already significant demand for Kotlin developers and there will continue to be in the future.

So where can you start learning this skill? “Atomic Kotlin”!

This book is suited for Kotlin learners of all levels, and it is available in print and as an ebook. It breaks Kotlin programming language concepts into atoms, and provides hands-on exercises inside IntelliJ IDEA.

To celebrate the recent release of “Atomic Kotlin”, we asked our Twitter community to share their motivation for learning Kotlin. You all did not disappoint! There were a lot of great responses from the community, and we saw a couple of patterns emerging. In this post, we are going to share with you some of the answers people gave about why Kotlin is a great language to learn.

Buy your copy

Modern Language

#atomickotlin
I love Kotlin because it is clean, logical, safe, fast, modern, flexible, intuitive, versatile, and sits on the shoulder of the Java giant.
— Jan Meww (@JanMeww) March 3, 2021

#atomickotlin I want to learn more about Kotlin because of the modern syntax and some new ideas (e.g. coroutines) it brings to the current Java Web Development. Have tried it for some weeks and has been a bliss!!!
— Fabio Salas (@fabiosalasm) March 2, 2021

Kotlin is a modern programming language that combines all the best bits of imperative, object-oriented, and functional programming. It is general-purpose and multi-paradigm, and conciseness and safety are some of the language's key features.

Easy to Learn

Why I want to learn Kotlin? well..You don’t come across a language that often which is as elegant and easy to learn like Kotlin. To put icing on the cake it can coexist with your Java ecosystems as well ..what more can you ask #atomickotlin
— Nirmal (@thooskoo) March 15, 2021

Kotlin’s syntax is simple to grasp for beginners, while at the same time, the language offers sophisticated powerful features for experienced programmers.

Kotlin can build on the learners’ previous programming experience. It is simple to grasp for those with a Java or Python background. Kotlin’s syntax is also easy to learn for iOS developers because it is based on the same modern concepts they are already familiar with.

Great Materials

I have gone through the sample of Atomic kotlin book, solved exercises from github repo. It's truely awesome & I want to learn the whole content. I want to use only one language to most of my application development to ship it to different platform - Kotlin #atomickotlin
— Kalaiselvan (@kalaiselvan369) March 5, 2021

There is a lot of up-to-date material available to help you learn Kotlin. One of our favorite resources for learning Kotlin from scratch is “Atomic Kotlin”. Readers can see their progress while solving the tasks that are checked automatically within IntelliJ IDEA. There are hints and solutions to help them out if they get stuck. All the examples in "Atomic Kotlin" are available in this GitHub repository. You can compile, run, and test the examples which are automatically extracted directly from the book.

Buy your copy

You can learn what the authors of "Atomic Kotlin", Bruce Eckel and Svetlana Isakova, think about learning the language in this episode of the JetBrains Connect series. In it, they discuss the question of why to learn Kotlin, with host Paul Everitt:

Multiplatform

#atomickotlin

Want to learn Kotlin since I want to be a multi platform developer & focus more on building the product in different platforms rather than on learning different languages. I believe Kotlin would give me feathers to fly between platforms & reach my true potential.
— jplus (@jplus62611121) March 15, 2021

I'm a Ruby developer but want to learn Kotlin as my next language because it have all signs of a modern and concise programming language and it reads well too, which is important. I can build various things with it starting from API to fully-fledged Android app. Fun #atomickotlin
— Костромицкий Денис (@mojobiri) March 4, 2021

As a language that supports multiplatform targets, you can run Kotlin on virtually any device these days, be it a PC or a Mac, and also as native code. This means you can build mobile, web frontend, and backend applications with Kotlin as well.

In addition to being the official language for Android development, Kotlin is a proven technology for building cross-platform mobile applications. It eliminates all the disadvantages of other leading approaches and lets you create mobile apps with native performance and UIs while sharing the business logic completely. Visit the Kotlin Multiplatform Mobile portal to learn more.

Progression from Java

I really like the JVM, and I already know some Kotlin, but I would definitely like to learn more and get better at it, and learn more about the JVM too. Kotlin combines a lot of features from languages that I really like. #atomickotlin
— Moiré (@Moire9_) March 4, 2021

Because I’m finishing my studies which were heavily focused on Java and I just started a job in which I use Kotlin every day. No reason not to exceed the expectations! #atomickotlin
— Bartosz Rogowski (@rogowskibart) March 13, 2021

#atomickotlin
Because I want to write Kotlin, not Java with Kotlin syntax. Currently coding together with Spring and I love it.
— MeSmash (@MeSmash3) March 2, 2021

Kotlin’s interoperability allows it to be introduced seamlessly anywhere you’re already using Java. Kotlin fixes some of the issues Java suffers from and has some features that are absent from Java.

Career prospects

As a self-learner, I started with a couple of Java courses in December. Since then I've been pursuing Kotlin and have been engaged in learning the Kotlin ecosystem to land my first dev job! Having material to build on my foundation of the language will be impactful! #atomickotlin
— Bryan (@Lidberg_B) March 13, 2021

Knowing, cOncise, pragmaTic, Legible, sImple, perfomiNg... #atomickotlin
My learning with these features will make my career grow and my university dream came true through this language.
— Miguel Paulista (@mikePaulista_13) March 9, 2021

Many great companies already use Kotlin to build their products, and Kotlin skills are increasingly in demand as more and more businesses are adopting the language. Indeed, the number of Kotlin job postings has skyrocketed by 1400% since 2017 (Source: Dice).

Community Driven

#atomickotlin Kotlin is a community driven language and they give great importance for the feedback from devs. I feel good to be invested in such Lang and I believe it has a great future!
— (@GopalAkshintala) March 3, 2021

Because I'm believing in Kotlin Team and their works(books, features, libraries and everything made by Kotlin)#atomickotlin
— Hamza GATTAL (@hmzgtl05) March 13, 2021

Since its very beginning in 2011, Kotlin has been developed as an open-source language.

Kotlin evolves with the help of our diverse community, which includes almost 200 Kotlin User Groups, 37K+ Kotlinlang Slack users, and countless other members spread across our forum, our sub-Reddit, YouTube, Twitter, and many other platforms. Over 450 contributors are working on Kotlin, including 90 JetBrains developers.

Your feedback provides the basis for our roadmap. Thank you for your continued support!

Why did you Learn Kotlin?

If you haven’t already taken up Kotlin, we hope that this post has left you feeling inspired to do so. We would love to hear about your own motivations for learning Kotlin by leaving your story on our official blog! If you share your story with us, you will also have the chance to win an “Atomic Kotlin” ebook!

We’ll select 5 winners from the commenters on blog.jetbrains.com on May 12.

Diving into Kotlin collections

Sebastian Aigner — Mon, 29 Mar 2021 18:00:41 +0000

This blog post accompanies a video from our YouTube series which you can find on our Kotlin YouTube channel, or watch here directly!

Kotlin Collections! You’ve heard of them, you’ve used them – so it makes sense to learn even more about them! Kotlin's standard library provides awesome tools to manage groups of items, and we’re going to take a closer look!

Let's see what types of collections the Kotlin standard library offers, and explore a common subset of operations that’s available for all of the collections you get in the standard library. Let’s get started.

In the Kotlin standard library, we have three big types of collections: Lists, Sets, and Maps. Just like many other parts of the standard library, these collections are available anywhere you can write Kotlin: on the JVM, but also in Kotlin/Native, Kotlin/JS, and common Kotlin code.

Lists

Let’s start with the most popular candidate of a collection in Kotlin: a List. To rehearse:

A list is a collection of ordered elements. That means that you can access the elements of a list using indices – so you can say “give me the element at position two”. There’s also no constraints on duplicate elements in our list. We can just put in whatever elements we’d like. So, very few constraints on content, and maximum versatility in how we access the elements!

val aList = listOf(
    "Apple",
    "Banana",
    "Cherry",
    "Apple"
)

aList // [Apple, Banana, Cherry, Apple]

aList[2] // Banana

Sets

Next up, we have the Set! Sets are groups of objects where we don’t care about the order of elements. Instead, we want to make sure that our collection never contains any duplicates.

That’s the key property of a set: all of its contents are unique.

That makes sets a bit more of a specialized data structure, but there’s a good chance you want to use them in everyday scenarios anyway.

What are typical things you might want to store in a set? Tags, for example. Or, maybe you’re building a social network, and you want to store the IDs of all the friends that a certain user has. In both cases, you don't want to have duplicates in these collections, and probably don't care about the order.

A set can help you enforce these constraints without having to really think about it, and without manual duplication checks.

val emotions = setOf(
    "Happy",
    "Curious",
    "Joyful",
    "Happy", // even if we try to add duplicates...
    "Joyful" // ...to our set...
)

println(emotions) // ...the elements in our set stay unique!
// [Happy, Curious, Joyful]

Sets are actually also a common mathematical abstraction. Typical mathematical concepts, like unions, intersections, or the set difference also translate neatly into Kotlin code.

Maps

Last, but certainly not least, we have Map. A map is a set of key-value pairs, where each key is unique. It’s also sometimes called a “dictionary” for that reason. You encounter maps whenever you’re associating data – storing a persons name and their favorite pizza topping, or associating a license plate with vehicle information.

val peopleToPizzaToppings = mapOf(
   "Ken" to "Pineapple",
   "Lou" to "Peperoni",
   "Ash" to "Ketchup"
)

println(peopleToPizzaToppings)
// {Ken=Pineapple, Lou=Peperoni, Ash=Ketchup}

println(toppings["Ash"])
// Ketchup

Key-value pairs are everywhere, and just like in many other languages, maps are the go-to way to manage them in Kotlin.

Collections can be mutable

By default, these collections in Kotlin are read-only. This is in the spirit of immutability which accompanies typical functional paradigms – instead of changing the contents of a collection, you create a new collection with the changes applied, which you can then safely pass around in your application, ensuring that the original collection stays unchanged.

But we also have mutable flavors of all of the collections in Kotlin: we have MutableList, MutableSet, and MutableMap. Those are modifiable, meaning you can comfortably add and remove elements. With data where you’re inherently expecting change, you’d probably use these mutable variants.

Collections are iterable

Kotlin collections being iterable means that the standard library provides a common, standardized set of typical operations for collections, for example, to retrieve their size, check if they contain a certain item, and more.

Lists and sets directly implement the Collection interface, which in turn implements the Iterable interface. Maps have an iterator() operator function, and provide iterable properties, like their set of keys, their list of values, as well as the entries of the map, so key-value pairs.

Let’s learn about some shared functionality of iterables. The following examples are going to use a list, but really, we can just assume that we’re just working with an Iterable here – the concrete implementation does not matter. Also, all the functions discussed leave the original collection unchanged.

Looping over collections

A core function of an Iterable, as its name suggests, is that it provides a mechanism to access the elements that our collection contains, one after the other – to iterate it.

The easiest way to go through all the elements in a collection is the basic Kotlin for loop. When we use the for loop with an Iterable, the in operator cleverly understands that we want to go over the iterator:

val fruits = listOf(
    "Apple",
    "Banana",
    "Cherry"
)

for(fruit in fruits) {
    println(fruit)
}

// Apple
// Banana
// Cherry

In a more functional style, we can also write this same snippet using the forEach function:

fruits.forEach { fruit ->
    println(fruit)
}

// Apple
// Banana
// Cherry

In this case, forEach takes every element from our collection, and calls a function (which we provide) with the element as its argument.

Transforming collections: map

Let's continue with a classic when it comes to transforming collections: the map function! (Don’t be confused! The map function has nothing to do with the Map collection type. You can treat them as two completely different things.)

Just like the forEach function, the map function is of higher order. So, it:

Takes each element from our collection,
applies a function to it, and
creates another collection, containing the return values of those function applications.

The result of the map function doesn’t have to be the same type as the one of our input collection, either.

This makes the map function very versatile – whether you want to parse a collection of strings into a collection of integers, or resolve a list of user names to a list of full user profiles –– if you’re transforming one collection into another, it’s probably a good first instinct to think map.

val fruits = listOf(
    "Apple",
    "Banana",
    "Cherry"
)

val stiurf = fruits.map {
    it.reversed()
}

However, you might have a transformation inside your map function where you can’t generate valid results for all input elements. In this case, we can use the mapNotNull function, and our resulting collection will only contain those function results that evaluated to an actual value. This also ensures that type of our resulting variable is non-nullable.

val strs = listOf(
    "1",
    "2",
    "three",
    "4",
    "V"
)

val nums: List<Int> = strs.mapNotNull {
    it.toIntOrNull()
}

println(nums)
// [1, 2, 4]

If we need to keep track of the index of the element which we’re currently transforming, we can use the mapIndexed function. It’s quite similar in how it works, but in this case, we get two parameters in our transformation function: the index and the value:

val rank = listOf(
    "Gold",
    "Silver",
    "Bronze"
)

val ranking = rank.mapIndexed { idx, m ->
    "$m ($idx)"
}

println(ranking)
[Gold (0), Silver (1), Bronze (2)]

Filtering collections: filter and partition

If we have a collection, but we’re only interested in elements that fulfil a certain condition, the filter function comes to the rescue!

Just like the previous examples, filter accepts another function as its parameter. This time, instead of defining a transformation, we’re defining what you can call a predicate here.

A predicate is a function that takes a collection element and returns a boolean value: true means that the given element matches the predicate, false means the opposite. So this predicate acts as the “doorman” – if the value is true, the collection item is let through to the result collection, otherwise, it is discarded.

open class Person(val name: String, val age: Int) {
    override fun toString() = name
}

class Cyborg(name: String) : Person(name, 99)

val people = listOf(
    Person("Joe", 15),
    Person("Agatha", 25),
    Person("Amber", 19),
    Cyborg("Rob")
)

val discoVisitors = people.filter {
    it.age >= 18
}

println(discoVisitors)
// [Agatha, Amber, Rob]

If you’re testing a negative condition, you can use the filterNot function instead, which behaves identically, but inverts the condition.

val students = people.filterNot {
    it.age >= 18
}

println(students)
// [Joe]

Note that both filter and filterNot discard elements where the condition doesn’t match. But maybe we don’t want to discard the “other half” of elements, and instead we want to put those into a separate list. This is where the partition function comes into play.

By using partition, we combine the powers of filter and filterNot. It returns a pair of lists, where the first list contains all the elements for which the predicate holds true, and the second contains all the elements that fail the test. So, in our doorman analogy, instead of sending people who fail the check away, we just send them to a different place. (Using parentheses, we can destructure this pair of lists directly into two independent variables.)

val (adults, children) = people.partition {
    it.age >= 18
}

println(adults)
// [Agatha, Amber, Rob]

println(children)
// [Joe]

If you’re bringing a collection of nullable items to the party, you can use the filterNotNull function which, as you may have guessed, automatically discards any elements that are null, and gives you a new collection with an adjusted, non-nullable type accordingly.

Speaking of adjusting types – if your collection contains multiple elements from a type hierarchy, but you’re only interested in elements of a specific type, you can use filterIsInstance, and specify the desired type as a generic parameter.

val people = listOf(
    Person("Joe", 15),
    null,
    Person("Agatha", 25),
    null,
    Person("Amber", 19),
    Cyborg("Rob"),
    null,
)

val actualPeople = people.filterNotNull()

println(actualPeople)
// [Joe, Agatha, Amber, Rob]

val cyborgs = people.filterIsInstance<Cyborg>()

println(cyborgs)
// [Rob]

Retrieve collection parts: take and drop

Filtering allowed us to apply a predicate function, and create a new collection containing items that match. But what about the even simpler cases? Sometimes, we just want to grab a few elements from our collection.

For that, we have the take and drop functions. You might already be able to guess what they do. take gives you a collection of the first n elements from your original collection. So take(2) is going to give you the first two elements. On the opposite hand, drop(3) is going to leave out the first three elements of your original collection, and only gives you everything that follows after those three elements. And you don’t have to be afraid to “overdrop” either – dropping more elements from a collection than it contains just gives you an empty list:

val objects = listOf("🌱", "🚀", "💡", "🐧", "⚙️")

val seedlingAndRocket = objects.take(2)

println(seedlingAndRocket)
// [🌱, 🚀]

val penguinAndGear = objects.drop(3)

println(penguinAndGear)
// [🐧, ⚙️]

val nothing = objects.drop(8)

println(nothing)
// []

println(objects) // remember, the original collection is not modified!
// [🌱, 🚀, 💡, 🐧, ⚙️]

One huge benefit of the functions we’ve seen so far is their composability: Because mapping, filtering, taking, dropping, and all their friends return a new collection, it’s easy to just take that result, and immediately use it as an argument for the next collection function, turning collection into collection into collection.

However, we should keep in mind that chaining a number of these functions together means we generate a bunch of intermediate collections. Now, this isn’t going to set your computer on fire immediately, but it is still something to be aware of, especially when you work with very large collections. For this case, Kotlin has a few aces up its sleeve as well, called sequences, but we will dive into those at a later point.

Aggregating collections: sums, averages, minimums, maximums, and counting

Once we’re done transforming our data, we might want to get a single result value out of it. If we have a collection of numerical values like integers or doubles, we get some nice functions called average and sum out of the box, which help us calculate those values.

val randomNumbers = listOf(3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 6)

println(randomNumbers.average())
// 4.09090909090909091

println(randomNumbers.sum())
// 45

In some situations (...or, we might say, sum situations...), we have a collection of more complex objects, and want to still add them up somehow, based on their properties. Of course, we could first use the map function to obtain a collection containing only numbers – but by using the sumOf function, we can do all of this in a single function call: we can pass a function that acts as a selector (so a function that gives us whatever number we want to associate with the element) and sumOf will use the result of that selector function to add up all our elements.

val randomNames = listOf("Dallas", "Kane", "Ripley", "Lambert")

val cumulativeLength = randomNames.sumOf { it.length }

println(cumulativeLength)
// 23

If we’re only interested in the greatest or smallest value contained in our collection of numbers, we can use the maxOrNull and minOrNull functions.

println(randomNumbers.minOrNull())
// 9

println(randomNumbers.maxOrNull())
// 1

And just like sumBy, we have the sibling functions maxOf and minOf, where we once again pass a selector function, which is going to be used to determine the maximum or minimum of a collection.

val longestName = randomNames.maxOf { it.length }

println(longestName)
// 7

val shortestName = randomNames.minOf { it.length }

println(shortestName)
// 4

If we just care about the number of elements contained in our collection, we can use the count function – either without any parameters, to just get the number of all elements, or using a predicate. So that’s like filtering the collection first, and then counting the elements. But again, all wrapped into one.

val digits = randomNumbers.count()

println(digits)
// 11

val bigDigits = randomNumbers.count { it > 5 }

println(bigDigits)
// 3

There’s also the powerful joinToString function, which allows us to turn all elements of our collection into a string, complete with a metric ton of customization options like separators, prefixes and postfixes, limits or a placeholder if you have more elements than what your specified limit allows. And even joinToString accepts a transformation function, once again, so you don’t need to do some kind of separate mapping beforehand, it’s all built in. Truly powerful stuff to create a string from a collection.

val str = randomNumbers.joinToString (
    separator = "-",
    prefix = "pi://",
    limit = 5
) {
    "[$it]"
}

println(str)
// pi://[3]-[1]-[4]-[1]-[5]-...

If you want to refresh what kind of magic we can do with Kotlin strings, watch the Kotlin Standard Library episode that takes us into the depth of strings!

More collection goodness, coming soon!

That concludes our overview of Kotlin collections!

Next, we’re going to step up our game even more, and will take a look at some advanced collection functionality. Some of the functions we’ve seen today actually have some additional variants to them, which are worth an introduction. There’s also the whole world of modifying collections. Plus, each type of collection we’ve seen, Lists, Sets, and Maps, all have their own specialized functionality as well, We’re in for a whole bunch more Kotlin collection content!

Make sure you don’t miss it! To get reminded when new content is released, follow us here on dev.to/kotlin, and make sure to follow me on Twitter @sebi_io.

Also, make sure to find the subscribe button and notification bell on our YouTube channel!

If you don’t want to wait until that episode comes out, there’s only one solution! It’s time to go and explore some more Kotlin!

Natural language processing with Kotlin: Part-of-speech tagging with Hidden Markov Model

Sebastian Aigner — Mon, 22 Mar 2021 12:52:12 +0000

Thanks to Maria Khalusova for agreeing to cross-posting her article here. For more articles like this, check out her blog.

In NLP, part-of-speech tagging is a process in which you mark words in a text (aka corpus) as corresponding parts of speech (e.g., noun, verb, etc.). This task is fundamental to natural language processing and has multiple applications, for instance, Named Entity Recognition, or Word Sense Disambiguation.

The task may seem straightforward at first glance, but it becomes difficult when you realize that some words can be ambiguous, and their part-of-speech tag depends on the context. For example, take the word "sleep". In a sentence "I plan to sleep until lunchtime tomorrow.", this word is a verb. However, in the sentence "How much sleep does one really need?", it is a noun.

How do you go about this problem? We can group the most commonly used approaches into two categories:

Probabilistic methods: In these methods, you assign a tag based on the probability of it occurring in a given sentence. One of such approaches uses Hidden Markov Models, and that's what this post covers.
Deep Learning methods: Getting state-of-the-art results requires complex recurrent neural networks, such as Meta-BiLSTM Model described in this paper.

In this post, I've chosen to cover Hidden Markov Models for a couple of reasons. First, these models are amazingly versatile, and part-of-speech tagging is only one of their applications. Others include signal processing, speech recognition, handwriting recognition, musical score following, and much more... Perhaps, once you know how Hidden Markov Model works, you'll find your own application for it! Second, the algorithm used for retrieving tag predictions, the Viterbi algorithm, didn't click with me at first, and I had to spend some time building the intuition for it. So if my explanation helps you understand it, I would have saved you some time :) Finally, if you really would rather read about Deep Learning, don't worry, I'll be digging into some version of LSTM in one of the latter articles.

So here's what you'll learn from this post:

What is Markov Model?
What is Hidden Markov Model?
How to prepare data
Calculating Transition and Emission Matrices
Motivation for Viterbi algorithm
Viterbi initialization
Viterbi forward pass
Viterbi backward pass

Let's start with what Markov Chains are.

Markov Model

A Markov chain is a mathematical system that transitions from one state to another according to certain probabilistic rules. An important property of this system (also called "Markov property") is that the probability of transitioning to any particular state depends solely on the current state.

Let's illustrate this with a toy example. Suppose, every day we observe the weather, and it is always one of the following: sunny, rainy, or cloudy (and for simplicity no other weather conditions exist). These are called states. Day after day we note it down, and as such, we create a sequence of observations. We can describe this situation with a Markov model:

Each day we are in one of the states - it's either sunny, rainy, or cloudy, and every day we transition into another state. If it's sunny two days in a row, it means that we transition from state "sunny" to itself, but the transition still happens. After collecting the weather observations for some time, we can calculate the probabilities of transitioning from one state to another (including into the same state). These are the numbers that you see on the transitions. For example, if it's sunny today, the probability that it will be cloudy tomorrow is 0.4. And, according to the Markov property the weather (aka state) tomorrow depends only on the weather (aka state) today.

We can write all the transition probabilities from this Markov model as a matrix A:

	s1 (Sunny)	s2 (Cloudy)	s3 (Rainy)
s1 (Sunny)	0.5	0.4	0.1
s2 (Cloudy)	0.2	0.2	0.6
s3 (Rainy)	0.7	0.15	0.15

In matrix A, a number in a cell $a_ij$ represents the probability of transitioning from state i to state j. Because a transition to some state has to happen (even if it is to itself), the numbers in each row must add up to 1.

Once we have modeled the weather like this, we can answer all sorts of questions. For example, if it has been sunny today, what is the probability that it will be sunny tomorrow? The answer to this you can simply look up in the matrix - 0.5 (50% chance). We can also answer more complex questions, like, if we have started with a sunny day, what is the probability to have the following sequence of days - [Sunny, Sunny, Rainy, Cloudy, Sunny]?
This is where we would use the Markov property.

P ([s 1, s 1, s 3, s 2, s 1] ∣ M o d e l) = P (s 1) * P (s 1∣ s 1) * P (s 3∣ s 1) * P (s 2∣ s 3) * P (s 1∣ s 2)

All these numbers except for the first one we have in the transition matrix A, and the first one we know from the problem
statement (P(s1) = 1 because we started with a sunny day):

P([s1, s1, s3, s2, s1] | Model ) = 1 * a_11 * a_13 * a_32 * a_21 = 1 * 0.5 * 0.1 * 0.15 * 0.2 = 0.0015

Now, this was a toy example to give you an intuition for the Markov model, its states, and transition probabilities. However, things are a little more complicated with Part of Speech tagging, and we will need a Hidden Markov Model.

Hidden Markov Model

In the previous examples, the states were types of weather, and we could directly observe them. The name "Hidden Markov Model" implies that we cannot directly observe the states themselves, but instead we can observe something else. Back to the Part of Speech tagging task, our hidden states will be parts of speech. So what is instead observed? The data that the algorithm will "see" is text - a sequence of words. These words are the actual observations from the data. How can we now model this situation? To simplify things let's pretend we only have three hidden states - noun (NN), verb (VB), and other (O).

Just like in the previous example, we're going to have a matrix with transition probabilities with a minor addition. Here, we're also adding the initial state $p i$ and probabilities to transition from the initial state to any of the hidden states. These probabilities tell us how likely it is that the text begins with one of the given parts of speech.

In addition to this, a Hidden Markov Model has Emission probabilities - the probabilities to observe a word given hidden state. For example, given that we are currently in a "verb" state, here are the probabilities to observe each of the actual words in the corpus:

Here, you can see that if our model is in the VB (verb) state, there's a 0.15 probability to observe the word "running". The emission probabilities can also be populated in a matrix (we'll call it B), where rows represent each of the possible hidden states (part of speech tags), and columns represent each of the possible observables (words). Thus, the size of this matrix would be number of tags (states) by number of possible words. Just like with the transition matrix A, the sum of the numbers in each row has to add up to exactly 1, as in each of the hidden states we have to observe one of the words.

This is how we can describe the corpus with words and POS tags with a Hidden Markov Model. Given a labeled corpus, "training" a Hidden Markov Model would mean populating the transition and emission matrices. Once that's done, we can use those matrices for calculating predictions using the Viterbi algorithm.

We'll get to the prediction part later in this article. First, let's see how we can populate these matrices for actual labeled data.

Data & Some Preprocessing

The standard dataset that is used for training POS taggers for the English language is the Penn Tree Bank Wall Street Journal dataset. The corpus in it looks like this:

In        IN
an        DT
Oct.      NNP
19        CD
review    NN
of        IN
''        ''
The       DT
Misanthrope NN
''        ''
at        IN
Chicago   NNP
's        POS
Goodman   NNP
Theatre   NNP

All words, punctuation marks, and other symbols are marked with corresponding tags, for example:

IN: conjunction, subordinating, or preposition. Examples: of, on, before, unless.
DT: determiner. Examples: the, a, these.
NNP: noun, proper singular. Examples: Germany, God, Alice
CD: cardinal number. Examples: five, three, 13%
...and so on, check the complete description of what all the tags mean here.

As is the case with any machine learning project, we will use part of that data for training, and reserve the rest for evaluating the model.

To "train" a Hidden Markov Model and populate the transition and emission matrices, we will need to know what hidden states we have, and what words are in the corpus - the vocabulary.

To create the vocabulary, we'll take the training chunk of the data, and record all the words that are encountered more than once. At the end of the words we will also add a list of special words: "--n--", "--unk--", "--unk_adj--", "--unk_adv--", "--unk_digit--", "--unk_noun--", "--unk_punct--", "--unk_upper--", "--unk_verb--".

The first one, "--n--" is the special word for the initial state, the rest ("--unk--", "--unk_adj--", "--unk_adv--", etc.) will replace the unknown words which we will inevitably encounter in both the training and the test corpus.

Here's one way you can create the vocabulary:

import java.io.File

fun createVocabulary(trainingDataFile: File): List<String> {
/*
    Creates vocabulary from the training data file.
    The vocabulary includes all the words that are encountered more than once.
    This is done so that unknown words would have some representation in the the training data as well.
    The vocabulary is also appended with the list of various types of "unknown", and special "word" indicating
    beginning of the sentence - "--n--"
*/

    val lineList = trainingDataFile.readLines().filter { it != "" }

    val words = lineList.map { getWord(it) }

    val wordFrequences = words.groupingBy { it }.eachCount()
    val vocabFromFile = wordFrequences.filter { (_, value) -> value > 1 }.keys.toList().sorted()
    val unknowns = listOf(
        "--n--", "--unk--", "--unk_adj--", "--unk_adv--", "--unk_digit--", "--unk_noun--",
        "--unk_punct--", "--unk_upper--", "--unk_verb--"
    )

    return (vocabFromFile + unknowns).sorted()
}

private fun getWord(line: String): String {
    var word = "--n--"
    if (line.split('\t').isNotEmpty()) {
        word = line.split('\t')[0]
    }
    return word
}

fun main() {
    val trainingData = File("src/data/WSJ_02-21.pos")
    val vocabulary = createVocabulary(trainingData)

    File("src/data/vocab.txt").writeText(vocabulary.joinToString("\n"))

}

Now we are almost ready to start populating the transition and emission matrices. We only need a few functions that'll help us go over the training data:

We'll need some heuristics to replace words that are not in the vocabulary with one of the "--unk_[]--" tokens
As we will be calculating transition probabilities we'll need to calculate how often each tag comes after another one, which means we'll need to go through the training corpus line by line extracting both the word and the tag.
Finally, when we'll want to test our model, we'll need to get just the words from the test corpus to generate part of speech predictions for them.

import java.io.File

object Preprocessor {

    //  heuristics for replacing words that are not in the vocabulary with unk-[...]
    private val nounSuffix = listOf(
        "action", "age", "ance", "cy", "dom", "ee", "ence", "er", "hood",
        "ion", "ism", "ist", "ity", "ling", "ment", "ness", "or", "ry", "scape", "ship", "ty"
    )
    private val verbSuffix = listOf("ate", "ify", "ise", "ize")
    private val adjSuffix = listOf(
        "able", "ese", "ful", "i", "ian", "ible", "ic", "ish", "ive",
        "less", "ly", "ous"
    )
    private val advSuffix = listOf("ward", "wards", "wise")


    fun assignUnk(word: String): String = when {
        word.contains(Regex("[0-9]")) -> "--unk_digit--"
        word.contains(Regex("[^A-Za-z0-9 ]")) -> "--unk_punct--"
        word.contains(Regex("[A-Z]")) -> "--unk_upper--"
        nounSuffix.any { word.endsWith(it) } -> "--unk_noun--"
        verbSuffix.any { word.endsWith(it) } -> "--unk_verb--"
        adjSuffix.any { word.endsWith(it) } -> "--unk_adj--"
        advSuffix.any { word.endsWith(it) } -> "--unk_adv--"
        else -> "--unk--"
    }


    fun getWordAndTagFromLine(line: String, vocab: Map<String, Int>): Pair<String, String> =
        if (line.split('\t').isNotEmpty() && line.split('\t').size == 2) {
            var word = line.split('\t')[0]
            val tag = line.split('\t')[1]
            if (!vocab.containsKey(word)) word = assignUnk(word)
            Pair(word, tag)
        } else Pair("--n--", "--s--")


    fun getTestWordsAndTags(vocab: Map<String, Int>, testDataFile: File): List<Pair<String, String>> {
        val wordsAndTags = mutableListOf<Pair<String, String>>()
        val lines = testDataFile.readLines()

        for (line in lines) {
            val (word, tag) = getWordAndTagFromLine(line, vocab)
            if (word == "") {
                wordsAndTags.add(Pair("--n--", "--s--"))
            } else if (!vocab.containsKey(word)) {
                wordsAndTags.add(Pair(assignUnk(word), tag))
            } else wordsAndTags.add(Pair(word, tag))
        }

        return wordsAndTags
    }

}

Calculating Transition and Emission Matrices

To populate these matrices, we will start with computing a few helper counts:

transitionCounts: a mutableMapOf<Pair<String, String>, Int>() that will contain the number of times a sequence of two tags (tag_i, tag_j) has occurred in that order. We will use that count to later calculate the probability of transitioning from said tag_i to tag_j.
emissionCounts a mutableMapOf<Pair<String, String>, Int>() that will contain the number of times the word_k was came with tag_l. We will use these counts to calculate the emission probability of a word_k given tag_l.
tagCounts: this map will simply store the number of times each tag occurred.

private fun calculateCounts() {
    val preprocessor = Preprocessor
    var previousTag = "--s--"
    for (line in trainingCorpus) {
        val (word, tag) = preprocessor.getWordAndTagFromLine(line, vocab)
        transitionCounts[Pair(previousTag, tag)] = transitionCounts.getOrDefault(Pair(previousTag, tag), 0) + 1
        emissionCounts[Pair(tag, word)] = emissionCounts.getOrDefault(Pair(tag, word), 0) + 1
        tagCounts[tag] = tagCounts.getOrDefault(tag, 0) + 1
        previousTag = tag
    }
}

Now that we have all these counts, we can calculate the matrices themselves. For transition matrix, the formula is as follows:

P(t_i | t_{i-1}) = \frac{C(t_{i-1}, t_{i}) + \alpha }{C(t_{i-1}) +\alpha * N}

where:

$C(t_{i-1}, t_{i})$ is the count of the pair (previous tag, current tag) in the transitionCounts.
$C(t_{i-1})$ is the count of the previous tag in the tagCounts.
$N$ is the total number of tags.
$α\alpha$ is a smoothing parameter. Smoothing is a technique that helps us to avoid shooting ourselves in the foot if some of those counts end up being 0. By adding $α\alpha$ in the numerator and $α∗N\alpha * N$ in the denominator, instead of 0, we'll end up with a very small number.

To work with matrices, I'm using multik, a multidimensional array library for Kotlin. It has been publicly released recently, and you can read more about it on the Kotlin blog.

private fun createTransitionMatrix(
    alpha: Double = 0.001,
) {
    val tags = tagCounts.keys.toList().sorted()

    transitionMatrix = mk.empty<Double, D2>(NUMBER_OF_TAGS, NUMBER_OF_TAGS)

//      Go through each row and column of the transition matrix
    for (i in 0 until NUMBER_OF_TAGS) for (j in 0 until NUMBER_OF_TAGS) {

//          Define the Pair (prev POS tag, current POS tag)
        val key = Pair(tags[i], tags[j])

//          If the (prev POS tag, current POS tag) exists in the transition counts dictionary, change the count
        val count = transitionCounts.getOrDefault(key, 0)

//          Get the count of the previous tag (index position i) from tag counts
        val countPrevTag = tagCounts[tags[i]]

//          Apply smoothing to avoid numeric underflow
        transitionMatrix[i, j] = (count + alpha) / (alpha * NUMBER_OF_TAGS + countPrevTag!!)
    }
}

Similarly, we calculate the probabilities for the emission matrix.

P(w_i | t_i) = \frac{C(t_i, word_i)+ \alpha}{C(t_{i}) +\alpha * N}

where:

$C(t_i, word_i)$ : the number of times $word_i$ was assosiacted with $tag_i$ in the training data (stored in emissionCounts).
$C(t_i)$ : the count of $tag_i$ in the tagCounts.
$N$ : total number of tags.
$α\alpha$ : smoothing parameter.

private fun createEmissionProbsMatrix(
    alpha: Double = 0.001
) {
    val tags = tagCounts.keys.toList().sorted()

    emissionProbsMatrix = mk.empty<Double, D2>(NUMBER_OF_TAGS, NUMBER_OF_WORDS)
    val reversedVocab = vocab.entries.associate { (k, v) -> v to k }

    for (i in 0 until NUMBER_OF_TAGS) for (j in 0 until NUMBER_OF_WORDS) {

        val key = Pair(tags[i], reversedVocab[j])
        val count = emissionCounts.getOrDefault(key, 0)
        val countTag = tagCounts[tags[i]]
        emissionProbsMatrix[i, j] = (count + alpha) / (alpha * NUMBER_OF_WORDS + countTag!!)
    }
}

Once we have computed the Transition and Emission matrices we have described our data with a Hidden Markov Model, so we can pat ourselves on the back on this occasion - well deserved!

Now how can we use these matrices to predict POS tags for a sentence?

Viterbi Algorithm: Motivation

First, let's define the problem we are trying to solve here. We have the Hidden Markov Model that contains information about the transition probabilities from each hidden state to another hidden state, and information about the emission probabilities from each hidden state to each word from the vocabulary.

Now we have a sentence (aka a sequence of words (= observations)), and we need to find the sequence of tags (hidden states) for this sentence that will yield the highest probability given this sequence of words. In other words, what sequence of tags matches this sentence the best?

The brute force method of answering this question would mean calculating probabilities for all the possible sequences of tags, and then choosing the one with the highest probability. However, ALL combinations of tags are possible (even if the probability for some is minuscule). So if we had a sentence with only 3 words, and there were only 4 possible tags - A, B, C, D, we would end up with 3^4 = 81 combinations to search through. In reality, we have 40+ tags, and sentences are typically much longer than 3 words. In general, for a sentence with N words, and T tags the search space of possible state sequences X is $O(N^T)$ . As you can imagine this quickly becomes way too large to brute force search on.

Clearly, a smarter alternative is needed, and that's where the Viterbi algorithm can help. The Viterbi algorithm is a Dynamic Programming algorithm that we will use for finding the most likely sequence of hidden states that results in a sequence of observations. Dynamic Programming is a method for solving a complex problem by breaking it down into a collection of simpler subproblems, solving each of those subproblems just once, and storing their solutions using a memory-based data structure. In this case, we'll be taking advantage of knowing that an optimal state sequence X1 . . . Xj . . . XT contains a subsequence X1 . . . Xj, which is also optimal. With that, we'll start building up the optimal sequence step by step, keeping record only of the best probabilities, and the path that got us there, not all the possible paths. This approach allows to bring the complexity down to $O(N^2 *T)$

As we will need to store intermediate results, we will need two additional matrices:

bestProbs: as we build up the optimal sequence, this is the matrix that will contain intermediate optimal probabilities.
bestPaths: this matrix will hold the indices of visited states, so that once we finally find the best probability for the full sentence, we could traverse back the bestPaths to get the actual sequence of states that got us to the highest probability.

The Viterbi algorithm consists of three steps:

Initialization. At this step we create the auxiliary matrices bestProbs and bestPaths and populate the first column of each matrix. Both matrices will have the size of NUMBER_OF_TAGS rows by sentence.size columns.
Forward pass: We'll be populating both matrices column by column, at each step finding the best probability and path for the current subsequence.
Backward pass: Finally, we'll use bestPaths to backtrace the path that yielded the highest probability which we'll get from the last column of bestPaths corresponding to the last word in the sentence.

Let's dive deeper in each of these steps.

Viterbi: Initialization

The first matrix we'll be initializing is the bestProbs matrix. It has NUMBER_OF_TAGS rows and as many columns as we have words in a sentence for which we need to predict the tags. At this step we need to populate the first column, and leave the rest to be zeros. The first column represents probabilities to emit the first word given each of the states when we start from the initial state. For example:

bestProbs[1,1] is the probability to start in the hidden state T1 ( which is the same as transition from the initial state to the hidden state T1) and observe the word W1;
bestProbs[2,1] is the probability to start in the hidden state T2 (aka transition from the initial state to hidden state T2) and observe the word W1;
and so on

In general, bestProbs[i, 1] can be calculated as product of multiplying the probability to transition from the initial state to hidden state $T_i$ , and the probability to emit the word W1 given we are in hidden state $T_i$ . One of these numbers we have in the transition matrix, and the other one we have in the emission matrix. To avoid the arithmetic underflow when dealing with extremely small probabilities, we'll add the logarithm.

The bestPaths matrix should contain indices of all previously visited states, so for the first column everything is 0. As a note, when creating an empty matrix with multik via mk.empty<Double, D2>(a,b) it already initializes to all zeros. This means we only need to create an empty matrix of the required size at this step.

private fun initializeViterbiMatrices(
    sentence: List<String>,
): Pair<D2Array<Double>, D2Array<Int>> {
    /*
    returns two matrices: bestProbs = best probabilities (num of states by num of words in sentence) and
    bestPaths = best paths (num of states by num of words in sentence)
    */

    val tags = tagCounts.keys.toList().sorted()
    val bestProbs = mk.empty<Double, D2>(NUMBER_OF_TAGS, sentence.size)
    val bestPaths = mk.empty<Int, D2>(NUMBER_OF_TAGS, sentence.size)

    val startIdx = tags.indexOf("--s--")

//      populating the first column of the bestProbs to initialize it
    for (i in 0 until NUMBER_OF_TAGS) {
        if (transitionMatrix[0, i] == 0.0) {
            bestProbs[i, 0] = Double.NEGATIVE_INFINITY
        } else {
            bestProbs[i, 0] = ln(transitionMatrix[startIdx, i]) + ln(emissionProbsMatrix[i, vocab[sentence[0]]!!])
        }
    }
    return Pair(bestProbs, bestPaths)
}

Once the matrices have been initialized, we can start populating them column by column in the forward pass. Remember, that each column represents the next word in the sentence. By moving from left to right column by column at each step we increase the subsequence in question by one word.

Viterbi: Forward Pass

In the forward pass we start populating the matrices column by column, storing the intermediate best probabilities in bestProbs, and the path that got us there in bestPaths. Here's how we calculate what should be in the cell [j,i] where $j$ is the $Tag_j$ and $i$ is the $word_i$ .

bestProbs[j,i] = \max_{k}bestProbs_{k, i-1} * transitionMatrix_{k,j} * emissionMatrix_{j, {index of word i}}

To simplify, suppose we are calculating bestProbs for the third column. Here's what we do here: We look at the previous (i.e. second) column and find the maximum value in it - this is a cell with the best probability for the previous subsequence. We take that number, and note the tag (k) at which we got the maximum value. That is the tag we will be transitioning from into our current, third column. So now for each tag in the third column we can multiply the best probability value we got for the previous subsequence by the probability to transition from previous best tag into this tag, multiplied by the probability to observe current column's word given this tag.

At the same time, when calculating bestProbs, we'll be saving the index k that maximizes the formula into bestPaths so that we not only know the best probability at every step but also keep track of what tag resulted in the best probability.

This is what this will look like in Kotlin:

private fun viterbiForward(
    sentence: List<String>,
    bestProbs: D2Array<Double>,
    bestPaths: D2Array<Int>
): Pair<D2Array<Double>, D2Array<Int>> {

    val updatedProbs = bestProbs
    val updatedPaths = bestPaths

    for (i in 1 until sentence.size) for (j in 0 until NUMBER_OF_TAGS) {

        var bestProbabilityToGetToWordIFromTagJ = Double.NEGATIVE_INFINITY
        var bestPathToWordI = 0

        for (k in 0 until NUMBER_OF_TAGS) {

            val temp_prob =
                updatedProbs[k, i - 1] + ln(transitionMatrix[k, j]) + ln(emissionProbsMatrix[j, vocab[sentence[i]]!!])

            if (temp_prob > bestProbabilityToGetToWordIFromTagJ) {
                bestProbabilityToGetToWordIFromTagJ = temp_prob
                bestPathToWordI = k
            }
        }
        updatedProbs[j, i] = bestProbabilityToGetToWordIFromTagJ
        updatedPaths[j, i] = bestPathToWordI
    }
    return Pair(updatedProbs, updatedPaths)
}

Once we've built the matrices in the forward pass, we can easily get the highest probability - it'll be the maximum value in the last column. However, we are interested not in the probability number itself but in the sequence of tags that yielded this number. In the Backward pass, we'll use the bestPaths matrix to traceback the tags sequence that have maximizes the probability.

Viterbi: Backward Pass

The backward pass is the final step of the Viterbi algorithm. Once we implement it, we'll be able to use this algorithm to generate predictions for the sequence of POS tags matching any sentence.

In the previous, forward pass, we have populated two matrices - bestProbs and bestPaths- that we will now traverse backwards to identify the sequence of tags that is the most likely for the given sequence of words.

First we need to find the highest value in the last column of the bestProbs, and get its row index K. This value is the probability of the most likely sequence of hidden states, matching the given sequence of words. The index K represents the hidden state that we were in when we calculated this probability. That's the POS tag for the last word in the sentence, so let's record K into an array. Now we need to start backtracing the whole sequence of tags, and we will switch to the bestPaths matrix. In the last column of the bestPaths we locate the cell with index K. That cell contains the unique ID of the POS tag of the previous word, so we jump there, and continue the same way thus "unrolling" the whole sequence. As we go through the bestPaths matrix from right to left, in each cell we will find "directions" on where to go next.

Here's what it could look like if we only had 5 words and 4 tags:

Let's implement it in Kotlin:

private fun viterbiBackward(
    sentence: List<String>,
    bestProbs: D2Array<Double>,
    bestPaths: D2Array<Int>
): List<String> {
    val m = sentence.size
    val z = IntArray(m)
    var bestProbForLastWord = Double.NEGATIVE_INFINITY
    val tags = tagCounts.keys.toList().sorted()

    val posPredictions = mutableListOf<String>()

    for (k in 0 until NUMBER_OF_TAGS) {

//          finding the index of the cell with the highest probability in the last column of the bestProbs
        if (bestProbs[k, m - 1] > bestProbForLastWord) {
            bestProbForLastWord = bestProbs[k, m - 1]
            z[m - 1] = k
        }
    }
    posPredictions.add(tags[z[m - 1]])

//      traversing the bestPaths backwards.
//      each current cell contains the row index of the cell to go to in the next column
    for (i in m - 1 downTo 1) {
        val tagForWordI = bestPaths[z[i], i]
        z[i - 1] = tagForWordI
        posPredictions.add(tags[tagForWordI])
    }
    return posPredictions.toList().reversed()
}

Finally, we can combine all three steps of the Viterbi algorithm into one method that we'll be able to use to predict a sequence of POS tags that best matches any given sentence according to the Hidden Markov Model that we have calculated based on our data.

    fun predictPOSSequence(sentence: List<String>): List<String> {
        val (initialBestProbs, initialBestPaths) = initializeViterbiMatrices(sentence)
        val (updatedBestProbs, updatedBestPaths) = viterbiForward(sentence, initialBestProbs, initialBestPaths)
        return viterbiBackward(sentence, updatedBestProbs, updatedBestPaths)
    }

There are different ways we could evaluate the performance of this model. For simplicity let's take per-word accuracy - how many words have been assigned correct tag out of all predictions that have been made.

    fun score(testWords: List<String>, testTags: List<String>): Double {
        require(testWords.size == testTags.size) { "The size of testWords list doesn't match the size of the testTags list" }

        val predictions = this.predictPOSSequence(testWords)
        val numberOfCorrectPredictions = predictions.zip(testTags).count { it.first == it.second }

        return numberOfCorrectPredictions.toDouble() / predictions.size
    }

To get a good measure of performance, we should always evaluate a model on the data that hasn't been used during the training phase. Let's see how this model does on the test data:

The model we've just build has 95% accuracy. Not too shabby! All this hard work has paid off, and you've reached the end of this lengthy article too. Congratulations!

PS: If you'd like to toy with the code from this article and build something of your own, feel free to check out this project on GitHub.

If you enjoyed this article, check out Maria Khalusova's blog.

Migrating our Kotlin/JS app to the new IR compiler

Sebastian Aigner — Tue, 09 Mar 2021 14:11:05 +0000

Together with some colleagues, I maintain a small full-stack web application called CodeQuiz, which we built during a 48-hour hackathon at JetBrains, and use at events to gamify learning about Kotlin. I recently migrated its frontend which you can see below (and which is using the kotlin-react wrappers) to the new Kotlin/JS IR compiler backend.

The new compiler made a bunch of issues in our code visible, so I wanted to share my experience of migrating a Kotlin/JS app, and provide some hints on where to look when your app behaves unexpectedly after moving to the IR compiler.

What's the Kotlin/JS IR compiler?

The Kotlin/JS IR compiler is currently in development (with alpha stability) and on its way to become the new default way of compiling Kotlin to JavaScript. It's a completely re-engineered infrastructure for all things Kotlin/JS. This switch comes with a number of benefits for Kotlin/JS applications!

// Detect dark theme var iframe = document.getElementById('tweet-1362402517976551424-382'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1362402517976551424&theme=dark" }

Using it allows you to already test drive a bunch of new features, including TypeScript declaration generation, and profit from new optimizations like stronger DCE (and, as a result, smaller generated artifacts).

But it also means that you have to embrace its more strict rules regarding interoperation between Kotlin and JavaScript. This might require some adjustment at first, but will help write more predictable code that interoperates between Kotlin and JavaScript.

Why doesn't my code just work? 😱

Especially with code at the "boundary" between Kotlin and JavaScript, the legacy compiler was quite lenient – for example how it exported all symbols (e.g. a data class) from Kotlin code to the JavaScript world.

Unfortunately, this means that it was easy to rely on compiler-specific internal behavior – some things just happened to work, even though the compiler gave no guarantees that these things were supposed to work.

When using the IR compiler these mistakes become visible – it enforces proper, explicit interoperation between the world of Kotlin and the world of JavaScript (we call this the "Closed World" model). This stricter and more explicit control will help the compiler optimize your code more aggressively.

But, due to the nature of JavaScript being a dynamic runtime environment, some of these changes in behavior only appear during execution time. In the case of CodeQuiz, a number of modifications were necessary to get everything working. We'll look at them in detail in the next sections.

Ultimately, it boiled down to running and testing the application (both in development and production mode), and keeping an eye on the following:

Helping Kotlin's DCE via @JsExport (e.g. React components)
Using external interface to define React properties (RProps) and state (RState) (instead of (data) classes) and other areas of interoperation
Creating plain JavaScript objects for interaction with external components
Fixing npm dependencies that use export default
Making sure our Kotlin dependencies support Kotlin/JS IR

Turning on IR

To use the IR compiler for our project, we make a small change to our build.gradle(.kts) file. In the kotlin configuration block, change js to js(IR), and enable the generation of JavaScript artifacts via binaries.executable():

js(IR) {
    binaries.executable()
    browser {
        commonWebpackConfig {
            cssSupport.enabled = true
        }
        testTask {
            useKarma {
                useChromeHeadless()
            }
        }
    }
}

(Alternatively, the compiler type can also be set in the gradle.properties file, with the key kotlin.js.compiler=ir, which might be easier if you have a more complex project.)

We can now cross our fingers and execute the browserDevelopmentRun Gradle task to start our application.

Let's look at some of the symptoms our CodeQuiz app exhibited when first running the application with IR, and let's correct the related code.

Make JS- and React-related classes external interfaces

The external modifier helps Kotlin understand that a certain declaration is pure JavaScript. This prevents problems like ClassCastExceptions that would arise from the false assumption that something is a Kotlin object (like a data class) – even though in reality, we are dealing with a plain JavaScript object.

When using react-kotlin, this can often be observed regarding definitions of RState and RProps – with React, state and properties are pure JavaScript objects managed by the framework for us.

Turn RState into an external interface

When running my application with IR enabled for the first time, I got the following ClassCastException in regards to some React components' init method:

codequiz-server.js?20e3:19131 Uncaught 
ClassCastException {message: undefined, cause: undefined, name: "ClassCastException", stack: "ClassCastException↵    at THROW_CCE (webpack-inter…s/react-dom/cjs/react-dom.development.js:4056:31)"}
cause: undefined
message: undefined
name: "ClassCastException"
stack: "ClassCastException↵    at THROW_CCE (webpack-internal:///./kotlin/codequiz-server.js:19101:11)↵    at App.init (webpack-internal:///./kotlin/codequiz-server.js:101164:69)↵    at RComponent_init_$Init$ (webpack-internal:///./kotlin/codequiz-server.js:31545:11)↵    at new App (webpack-internal:///./kotlin/codequiz-server.js:101148:5)↵    at constructClassInstance (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:12716:18)↵    at updateClassComponent (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:17425:5)↵    at beginWork (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:19073:16)↵    at HTMLUnknownElement.callCallback (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:3945:14)↵    at Object.invokeGuardedCallbackDev (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:3994:16)↵    at invokeGuardedCallback (webpack-internal:///../../node_modules/react-dom/cjs/react-dom.development.js:4056:31)"
__proto__: RuntimeException
THROW_CCE   @   codequiz-server.js?20e3:19131
App.init    @   codequiz-server.js?20e3:101224

The stack trace suggests the init method of my App component. Since here, only application state is initialized, it was quite easy to pinpoint the underlying problem.

The offending code for the application state looks like this:

interface AppState : RState {
    var isPresenter: Boolean
    var lastMessage: Content?
    var isConnected: Boolean
    var chosenName: String?
}

This code happened to work with the legacy compiler, but the IR compiler marks our problem: if our interface describes the exact shape of a JavaScript object, we need to mark the interface as external.

The refactored code looks like this:

external interface AppState : RState {
    // . . .

I made sure that all interfaces implementing RState in my application were annotated with external by using a structural search and replace. If you're using IntelliJ IDEA 2021.1, you can copy an SSR template I prepared into your clipboard. To use it, open SSR via File | Find | Find Structurally [or Replace Structurally], click on the wrench icon, and select "Import Template from Clipboard". You can then click "Find" and "Replace All" to annotate all interfaces properly.

Turn RProps into an external interface

RState isn't the only type that is affected by this change – similar problems appear when React properties (RProps) aren't marked as external:

codequiz-server.js?20e3:100446 Uncaught TypeError: $this$attrs._set_presenterStartGameHandler_ is not a function
    at _no_name_provided__346.invoke_547 (codequiz-server.js?20e3:100446)
    at eval (codequiz-server.js?20e3:101430)
    at RElementBuilder.attrs_0 (codequiz-server.js?20e3:31443)

Analogously, this results from the RProps definition being just a Kotlin interface:

interface LobbyProps : RProps {
    var isPresenter: Boolean
    var presenterStartGameHandler: () -> Unit
    var playerLoginHandler: (String) -> Unit
    var playerList: PlayerList?
    var isDisabled: Boolean
}

The IR-approved versions of this code uses an external interface:

external interface LobbyProps : RProps {
    // . . .

Once again, this change can just be repeated for all components defining RProps interfaces in the Kotlin/JS application. This is easily automated via structural search and replace, as described in the previous section. Here is a template for auto-annotating your RProps as external – instructions for using SSR can be found in the previous section.

Use external interfaces over data classes!

If you've been using Kotlin's class or data class to create your RProps or RStates, you will need to do a similar refactoring. Code like this is invalid when using Kotlin/JS IR:

data class CustomComponentState(
   var name: String
) : RState

Instead, use the following, refactored version.

external interface CustomComponentState: RState {
   var name: String
}

Address limitations of external interfaces

Compared to a Kotlin interface or class, there are a few limitations when using external interface.

If you want to instantiate the interface from Kotlin code, you will have to mark your properties as var (val will not work here). Also, certain Kotlin-specific constructs, such as function types with receivers, are prohibited in external declarations.

In our codebase, the latter showed up as a compile error in an interface called ButtonProps. Here, we define a property inside which takes an extension function on the StyledDOMBuilder type to define any components that should be rendered in the button:

external interface ButtonProps : RProps {
    var inside: StyledDOMBuilder<BUTTON>.() -> Unit
    // . . .
}

Since these functions with receivers are just syntactic sugar for a function with an (implicitly named) parameter of the same type, we can refactor the external interface and pass the StyledDOMBuilder explicitly, resolving this problem:

var inside: (StyledDOMBuilder<BUTTON>) -> Unit

As luck would have it, our callsite was already structured so that this slightly changed style of function definition just works, so no change was needed there:

styledButton {
    props.inside(this)

    attrs {
        // . . .
    }
}

Create plain JS objects for interoperability

Inside the definition of a React component, objects implementing RState and RProps already exist, and we simply modify their properties.

When we create these objects ourselves, we (currently still) need to be a bit careful. In CodeQuiz, we had the following problem passing values to an external react-minimal-pie-chart component:

PieChart.default {
    attrs {
        data = props.statistics.answers.mapIndexed { index, (_, answerCounts) ->
            object: PiePoint {
                override var title = "Number $index"
                override var value = answerCounts
                // . . .
            }
        }.toTypedArray()
    }
}

...and that even though PiePoint is correctly specified as an external interface. The specific issue here turned out to be a bit finicky:

As of now, properties on a Kotlin object implementing an external interface are accessible from JavaScript, but, for example, they are not enumberable. react-minimal-pie-chart internally uses Object.assign to copy around some of the data we pass as props. It loses our non-enumerable properties in the process, which leads to some unexpected undefineds at runtime.

Until this problem is resolved (see the corresponding YouTrack issue), the safe route right now is to generate plain JavaScript objects ourselves.

The kotlin-wrappers actually include a helper function called jsObject<T> which is useful for creating such objects. The same snippet using these plain JavaScript objects looks like this:

PieChart.default {
    attrs {
        data = props.statistics.answers.mapIndexed { index, (_, answerCounts) ->
            jsObject<PiePoint> {
                title = "Number $index"
                value = answerCounts
                // . . .
            }
        }.toTypedArray()
    }
}

Since in a plain JavaScript object, all properties are enumerable, our charting library now works properly.

Help the DCE via @JsExport!

Dead Code Elimination (DCE) is the part of the Kotlin/JS IR compiler that helps keep your compiled production artifacts small. It's responsible for analyzing the Kotlin code for any pieces of code that aren't being used anywhere, and subsequently deleting them.

When packaging our application for production (which is when DCE is executed, e.g. via browserProductionRun or jsBrowserDistribution), this can present a problem for our React components.

Consider the following Evaluation class from our project:

class Evaluation(l: EvaluationProps) : RComponent<EvaluationProps, RState>(l) {
    override fun RBuilder.render() {

The only way this class is ever referenced via its KClass, when we tell React to render this component:

child(Evaluation::class) {
    attrs {
            // . . .

As of now, the IR DCE tries to be a bit too clever for its own good. It removes the contents of our class practically entirely (from its perspective, none of it, besides the type itself, is being used after all!). This causes the (unfortunately quite cryptic) error TypeError: r.render is not a function (or something similar).

To turn this error message into something a bit more actionable, we can (temporarily!) enable webpack's development mode in our Gradle build file (build.gradle(.kts)), which turns off the name minification:

browser {
    commonWebpackConfig {
        // . . .
        mode = org.jetbrains.kotlin.gradle.targets.js.webpack.KotlinWebpackConfig.Mode.DEVELOPMENT
    }
}

For now, we need to make sure our component doesn't get removed, we can mark the class with @JsExport. Then, DCE will not touch it:

@JsExport
class Evaluation(l: EvaluationProps) : RComponent<EvaluationProps, RState>(l) {
    override fun RBuilder.render() {

(As a small sidenote: declarations marked as external, such as an external interface, are always treated as reachable by DCE, and don't need this treatment. Functional components are also not affected, because their usage site doesn't refer to the ::class, but to the variable holding the component directly.)

In the case of kotlin-react, there are still some rough edges, like the warning Exported declaration uses non-exportable super type: RComponent. Together with making this kind of "workaround" obsolete, these are topics that still need addressing before the IR compiler becomes the default choice.

You can find a Structural Search and Replace template for this change right here. Find instructions on how to apply this automated replacement to your project in one of the previous paragraphs.

This is definitely one of the trickier issues to find, because it only manifests in production artifacts (when DCE is actually executed). For this reason, it's important to test your production artifacts!

Fixing dependencies on default exports

Our app uses a few external React components which we get from npm, including react-minimal-pie-chart.

Module parse failed: Unexpected keyword 'default' (35:6)
File was processed with these loaders:
 * ../../node_modules/source-map-loader/dist/cjs.js
You may need an additional loader to handle the result of these loaders.
|   var render = $module$react_dom.render;
|   var createGlobalStyle = $module$styled_components.createGlobalStyle;
>   var default = $module$react_minimal_pie_chart.default;
|   var default = $module$react_player.default;
|   'use strict';

We wrote the following external declaration for the component provided by this package, which worked for our used version, 5.0.2, beforehand, but not with IR:

@file:JsModule("react-minimal-pie-chart")
@file:JsNonModule
external interface PieChartProps: RProps {
    // . . .
}

@JsName("default")
external val PieChart: RClass<PieChartProps>

Here, we actually hit a bug in the IR compiler! It currently does not treat default as a reserved identifier. This causes a conflict when the library uses this identifier for its exports:

import Chart from './Chart';
export default Chart;

An issue exists to turn default into a reserved identifier, and this point will hopefully be addressed soon. Until then, the workaround is to wrap the definition in an external object, like so:

external interface PieChartProps : RProps {
    // . . .
}

@JsModule("react-minimal-pie-chart")
@JsNonModule
external object PieChart {
    val default: RClass<PieChartProps>
}

At the usage site for the component, we now use the PieChart.default value instead of the PieChart value previously:

PieChart.default {
    attrs {
        // . . .
    }
}

Fix library code

After fixing all of the other problems, I noticed a special case where the app would throw the following error:

Uncaught Error: `props.dangerouslySetInnerHTML` must be in the form `{__html: ...}`. Please visit https://reactjs.org/link/dangerously-set-inner-html for more information.

It took me a while to find the culprit, but I remembered that there was a place where we explicitly allowed HTML-formatted rich text in our application, and are using unsafe:

val label: RBuilder.() -> Unit = {
    span {
        attrs.unsafe {
            +answerText
        }
    }
}

It turns out that kotlin-wrappers actually contained a small mistake in its own interoperation code: it used a class instead of an external interface for their InnerHTML object – which is used to implement attrs.unsafe.

Make InnerHTML external interface #416

SebastianAigner posted on Feb 18, 2021

Fixes IR problem where unsafe throws

`props.dangerouslySetInnerHTML` must be in the form `{__html: ...}`

https://youtrack.jetbrains.com/issue/KT-45021

View on GitHub

This was a great point to make a small open-source contribution in the form of a pull request (and get the code improved further just hours later)!

Petition library authors to support IR

Luckily, all the libraries we are using in the project (including Ktor Clients and kotlinx.serialization) already support the Kotlin/JS IR compiler, and they provide artifacts that work with both backends. And there is a number of other libraries that already offer artifacts compatible with the IR compiler, like fritz2, KorGE, Kodein-DI, and more.

If you're using a Kotlin/JS library that currently does not ship IR-compatible artifacts, it might be a good idea to catch the maintainer's attention, and maybe help out yourself to ensure that your favorite libraries work well with the new compiler infrastructure. To make sure libraries can support both legacy and IR backends at the same time, there is also a mechanism for authoring libraries with backwards compatibility.

If you're a library author, and want to learn more about supporting the Kotlin/JS IR backend, please do not hesitate to reach out on the Kotlinlang Slack. You can either contact me directly, or get input from the team and community in the #javascript channel.

Closing thoughts

The new IR compiler introduces some changes that might require action from you – especially in places where Kotlin code meets the JavaScript platform. I hope this post helps diagnose some of these behavior changes, so that you can experience all the exciting stuff the new compiler brings as soon as possible.

If you encounter issues during your migration to the IR backend, share them with the team. We're happy to help, and rely on your feedback to make sure we can iron out any remaining problems as soon as possible. The easiest way to do this is to log your problems in the official Kotlin issue tracker.

Give the Kotlin/JS IR compiler a try in your projects, and prepare yourself for the future!

Tips and tricks for your Kotlin code explorations

Sebastian Aigner — Wed, 03 Mar 2021 13:27:40 +0000

This blog post accompanies a video from our YouTube series which you can find on our Kotlin YouTube channel, or watch here directly!

We’re going to take a look at some handy tips and tricks to help you explore Kotlin code, no matter whether it's your own code or the code from the Kotlin standard library or any of your other dependencies. They will equip you to discover some of the goodies in the code you’re working with on your own.

Working with Kotlin means we have some powerful companions by our side - IntelliJ IDEA and Android Studio - that help us in every step of the development process. Let's get to know them a little better and become more productive!

Reading and writing code

So – how do I read and understand Kotlin code that I encounter and modify?

Annotating code with types and names: Inlay hints

A big helper I rely on every day is these little grey bobbles wherever variable names or types appear, called inlay hints, and honestly, I don’t want to read code without them. They give us information about types and parameter names, right where that information is most useful.

We enable those in our preferences under Editor > Inlay Hints > Kotlin. (I have all of them enabled, but toggles are available for fine-grained control.)

Inlay hints for chained operations

These inlay hints help me a lot! For example, when I’m chaining a bunch of operations, they give me a quick check whether I am creating the correct type at the end:

Inlay hints for lambdas and DSLs

Inlay hints are also awesome whenever we encounter lambdas, especially lambdas with receivers, because they give us information about variables that are implicitly in scope, or which receiver your lambda actually provides.

Especially if we are working on a project that makes heavy use of Kotlin’s DSL functionality, these hints can help demystify some of these APIs:

Inlay hints for the Kotlin Gradle DSL

Oh, and as a little hint – if we are using the Gradle Kotlin DSL (so if our build file ends in .kts), these inlay hints also work here!

For someone who tried to avoid touching Gradle for quite a while myself, I’d say that the Kotlin Gradle DSL, together with these inlay hints, have helped me understand how the Kotlin DSL for Gradle works, what functionality is available in what scope, and so on:

Try using inlay hints yourself! Maybe you'll fall in love just like I did.

Code completion (`⌃` `Space`)

In my opinion, code completion is the number one way in IntelliJ IDEA or Android Studio to discover a Kotlin API. We know the drill – we put a dot behind a symbol, and get some suggestions of what we might want to call:

But code completion doesn't always pop up naturally. For example, when you're inside a lambda function. Via inlay hints, we might be able to determine that there's an implicit this in scope – and we could type this. and get code completion. But there's also a nicer (and easier) way.

We just press ⌃ Space (Ctrl Space on Win/Linux) to manually invoke completion based on our current scope. And this works practically everywhere – whether we are inside a builder block or returning to an unfinished statement.

IntelliJ IDEA's productivity guide tells me I've used the feature over 2400 times, and it's one of the most important tools in the belt.

Parameter info: Type popup (`⌘` `P`)

When we first type out a function with parentheses, we get a little hint bubble that tells us parameters that the function accepts. However, this popup only appears the first time we write those parentheses – when we jump away with our cursor and return to our function later, we might want to see this information again.

To magically summon the parameter info popup again, we simply press ⌘ P (Ctrl P on Win/Linux):

The popup also shows default parameters, overloads & co. As such, it can also help us discover new functionality on already existing code – like additional parameters we might still want to add to adjust the behavior of any functions we call.

Quick Documentation (`F1`)

If we hit F1 (Ctrl Q on Win/Linux), IntelliJ IDEA will show us a little text blurb explaining what a method is supposed to do – some quick documentation!

We can actually pin this window to the side of our IDE via the three little dots ⋮ in the pop-over. From there, we can select the option “auto-update from source,” reachable by the small cogwheel icon ⚙️. With this enabled, we always get up-to-date information on the side of our IDE about the symbol the cursor is currently over.

Navigating through code

Your IDE also empowers you when it comes to moving between pieces of code, usages and declarations, and more. Let's take a closer look!

Go to declaration or usages (`⌘` `Click` / `B`)

The shortcut ⌘ Click / B (Ctrl Click / B on Win/Linux) is known by its formal title “Go to declaration or usages”. As you may have guessed, it’s a two-way shortcut:

When we click on the usage of a symbol – a variable, a place where we’re constructing a new object, invoking a function, or implementing an interface – this shortcut will send us to the definition of what we just clicked.

And vice-versa, when we click on a symbol declaration – so a place where we’re declaring a variable, an interface, a class, or a function, this is going to give us a list of all places where that symbol is being used in our project.

This shortcut is an easy way to reveal interesting information about our code and the code we depend on. I find stepping into a library especially useful:

We can look at the code we’re calling, learn something from its implementation and structure, and we can immerse ourselves even deeper.

Rendered documentation

IntelliJ IDEA and Android Studio actually have something called “Rendered documentation”. This makes it more comfortable to read documentation blocks in code by changing the font and making the contrast between code and documentation even stronger.

Structure tool window

IntelliJ and Android Studio allow us to look at our currently opened file's structure, which helps us discover interesting parts quicker. We can enable the structure tool window via View > Tool Windows > Structure. This pops open information about the current file, which we can use as a navigation helper.

Searching (and finding)

If we can’t find what we’re looking for, we can always go back to the text search function. In IntelliJ IDEA and Android Studio, we can reach “Find in Files” via ⌘ ⇧ F (Ctrl Shift F on Win/Linux).

By default, this function only searches for text occurrences in our own code, not the code of our dependencies. However, we can set our search scope to include "Projects and Libraries" to widen where to look for text snippets.

Structural search

There’s also the so-called structural search, which is a bit of a specialized tool when it comes to exploring. It allows us to find constructs matching a specific schema, for example finding functions that have exactly one parameter.

You might not use this one as often. But it’s always good to have heard about the fact that it exists – you never know when that knowledge might come in handy, and then you can look into the topic a bit more deeply.

Go and explore!

I hope you learned something new. If you have some more must-have settings, shortcuts, or whatever, feel free to share them with me, either here, on YouTube, or Twitter.

There are tons of other productivity shortcuts in your Kotlin IDE. If you’re interested in more of them, check out Trisha Gee's video covering the 15 IntelliJ IDEA shortcuts:

Alternatively (or additionally), grab yourself a virtual keymap overview, which has all the most-used shortcuts for the operating system of your choice!

Try integrating your new knowledge when you work with Kotlin code the next time. Have fun exploring!

Hosting Kotlin/JS on GitHub Pages via GitHub Actions

Sebastian Aigner — Fri, 05 Feb 2021 12:16:28 +0000

GitHub Pages is an easy way to host your Kotlin/JS application - and in combination with the continuous integration service GitHub Actions, you can easily set up a smooth development (and deployment!) experience for your projects. Read on to learn how you can build and test your application on GitHub's CI, and get your Kotlin/JS web apps published for free.

In this post, we are going to configure GitHub Actions and Pages so that new code pushed to our repository or added via pull request is automatically validated (by having the project built and our tests run on GitHub's servers), and code pushed to the main branch of our project is automatically made available to the public (by generating and publishing production artifacts of our app).

To achieve this, we are going to:

Set up a quick Kotlin/JS project
Share its code on GitHub (via IntelliJ IDEA's Git integration)
Set up GitHub Actions' workflow YAML
Set up GitHub pages to work with the result of our Actions workflow

If you already know the basics, and want to skip straight to the configuration section, click here – or find the sample project directly on GitHub

Setting up our sample Kotlin/JS application

To deploy a project, we must first have a project. For this demo, I will use an example generated via project wizard – if you already have a Kotlin/JS application you would like to deploy, feel free to skip ahead and use it instead.

A boilerplate Kotlin/JS project is quickly generated in IntelliJ IDEA by navigating to File | New | Project..., selecting "Kotlin" in the sidebar, and choosing a project template in the "Kotlin/JS" section (I'm choosing a React project). We just need to make sure that a Project JDK is selected. For everything else, the default settings are fine and don't need to be adjusted:

Now that we have a basic project, let's start by getting it on GitHub - creating a repository and pushing our code.

Putting our code on GitHub

We could, of course, use GitHub's web interface to set up our repository, and wire up our local repository manually – but IntelliJ IDEA's Version Control Integration makes it even smoother to move our project code into a new GitHub Repository. We simply select "Share Project on GitHub" in the "VCS" tab.

If this is your first time using this feature, IntelliJ IDEA might ask you to authenticate using your GitHub account. You then get to specify your repository name, its visibility (private/public), and can provide a short description that will show up on GitHub:

Once confirmed, IntelliJ IDEA will create a GitHub project for you, and show you the "Add Files For Initial Commit" window, which we can simply accept by pressing "Add":

As you might have noticed, IntelliJ IDEA makes our life a bit easier here by auto-generating a default set of .gitignore files, making sure that any files which shouldn't be checked into source control (such as build artifacts or local configurations) are correctly ignored.

Once the commit is finished, we see a small notification bubble containing a link to our GitHub repository:

Setting up GitHub Actions

Time to set up GitHub Actions! Actions will be responsible for building and testing our project – turning our Kotlin source files into .js and .html artifacts, running any unit tests we might have, and copying files into the right location for deployment on GitHub Pages (a branch called gh-pages by convention). We configure Actions using so-called workflows, which are YAML configuration files containing instructions on how to transform and build code.

GitHub looks for workflows in the .github/workflows directory, so let's create this directory structure at our project's root. Inside, let's create a new file called build-and-publish-kjs.yml, and add the following configuration to it:

name: Build and Publish
on: [ push, pull_request ]
jobs:
  build:
    name: Test and Build
    runs-on: ubuntu-latest
    steps:

      # Setup Java 1.8 environment for the next steps
      - name: Setup Java
        uses: actions/setup-java@v1
        with:
          java-version: 1.8

      # Check out current repository
      - name: Fetch Sources
        uses: actions/checkout@v2

      # Build application
      - name: Test and Build
        run: ./gradlew build

      # If main branch update, deploy to gh-pages
      - name: Deploy
        if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main'
        uses: JamesIves/github-pages-deploy-action@3.7.1
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          BRANCH: gh-pages # The branch the action should deploy to.
          FOLDER: build/distributions # The folder the action should deploy.
          CLEAN: true # Automatically remove deleted files from the deploy branch

Understanding our Action YAML

We could just use this configuration as is, and move on to the next step – but it's always good to try and understand what's going on in our project. And while the YAML configuration reads quite naturally, I still want to quickly look at this configuration file section by section (Additionally, feel free to consult the GitHub Actions documentation for more information):

We first give our workflow a name – in this case, Build and Publish.
Next, we define the triggers for this workflow – what events kick off this process. Because we want to make sure that pushed code always compiles, and that pull requests are also fine, we set the value to push and pull_request.
Next, we define a job, which groups the different steps we want to run in a given environment (in our case, ubuntu-latest).
Now, we define the different steps our build needs to go through:
- The Kotlin compiler needs Java to be present, so we use a predefined GitHub Action to install Java 1.8.
- We use another predefined Action to check out the code for our project.
- We run the ./gradlew build command. build is a standard lifecycle task for Kotlin/JS projects which installs all dependencies from npm and Gradle, compiles the application, and runs any tests included in the project.
- Lastly, we use another Action to deploy our project to GitHub Pages – but only if the workflow is running on the master or main branch (we don't want development branches to be deployed to the public!). We point this action to the build/distributions folder, which is where building a Kotlin/JS project creates the final .js and .html artifacts.

TL;DR: This workflow builds and tests all commits and pull requests – if the commit is on the master or main branch, the changes are prepared for publishing via Pages.

Committing the workflow file

To enable our newly created workflow in the GitHub project, we commit and push our new build-and-publish-kjs.yml.

If you do this via IntelliJ IDEA's "Commit and Push" functionality, please note that, depending on how your GitHub account is authenticated, you may encounter the following error when pushing a commit containing workflow files to your GitHub repository:

error: failed to push some refs to 'https://github.com/SebastianAigner/kotlin-js-on-github-pages.git'
To https://github.com/SebastianAigner/kotlin-js-on-github-pages.git
!   refs/heads/master:refs/heads/master [remote rejected] (refusing to allow an OAuth App to create or update workflow `.github/workflows/build-and-publish-kjs.yml` without `workflow` scope)
Done

This is because of an OAuth issue with GitHub requiring a specific scope to push workflow-modifying commits. You can find more info on it and vote on it here. Instead, you can commit the code via the terminal integrated in IntelliJ IDEA, using git push origin master:

Once the push has finished, we can watch the progress of our Kotlin/JS application being built and prepared for publishing in the "Actions" tab of our GitHub repository.

Setting up GitHub Pages

After it finished, the workflow we created and triggered in the previous section put our final .js and .html artifacts on the gh-pages branch of our repository, just as we planned:

It's time to enable the GitHub Pages feature of our repository, and point it to this branch. We can do this using the "Settings" tab.

Somewhere close to the bottom, we can find a section labelled "GitHub Pages". We select the gh-pages branch and / (root) (default) and hit "Save":

After saving our changes, scrolling back to this section reveals the URL under which our application is available. (I have previously set up a custom domain for GitHub pages, so this one is used by default.) If you don't have a custom configuration, the URL of your application usually takes the shape of <username>.github.io/<reponame>. Visiting this URL shows our Kotlin/JS application in all its glory, and for the world to see:

You can use this link to share your Kotlin/JS application with friends and colleagues, or post it on your favorite news aggregator and wait for the upvotes to roll in! 😉

(If your project is located in a private repository, you might want to change the GitHub Pages visibility. You can find more info on this in the GitHub docs.)

Take your project for a spin!

We're done! From now on, whenever you push code, your project will automatically be built on GitHub's CI servers. If a commit is faulty, you can see so on the web interface – and you will even be reminded via email!

Pushing to the main branch of the repository means your page (which is reachable at <username>.github.io/<reponame>) will automatically be updated.

And when somebody makes a pull request, build and test status is also shown directly in context:

Thank you!

I hope you found this post useful. GitHub Actions is a powerful, but also complex tool, and I hope that these instructions make it a bit easier to get started together with Kotlin/JS. You can find the sample project I used to create the screenshots for this tutorial on GitHub as well, and use it as a source of inspiration or template for a future project.

If you're interested in similar content, consider giving me a follow, here on dev.to or on Twitter @sebi_io to stay up to date.

If you have ideas for future tutorials, please share them in the comments below, or anywhere else you can get a hold of me!

Special thanks to Jakub Chrzanowski for sharing some of his CI wisdom with me for this blog post.!

Kotlin Standard Library Safari: Strings

Sebastian Aigner — Tue, 26 Jan 2021 12:12:30 +0000

This blog post accompanies the first episode of our YouTube series "Kotlin Standard Library Safari", which you can find on our official Kotlin YouTube channel, or watch here directly!

What's Kotlin Standard Library Safari?

In the "Kotlin Standard Library Safari" series, we’re going through the useful functionality the standard library in Kotlin has to offer, one subject at a time. In the process, we’re hopefully going to unearth some hidden gems together, which will come in handy the next time you write Kotlin code. Because if you know how to wield it, the Kotlin standard library is a powerful tool which can help you be more productive solving your problems, and be more expressive in your code.

This episode is all about strings – how we manipulate them, extract information, compare them, and much more. Let's get going!

Creating strings

Strings are one of the most prevalent data types that you’re probably familiar with. After all, all kinds of information get stored in the form of text strings, one way or another. Probably even in your first ever Kotlin program, you had a string literal saying “Hello, World”, or something similar:

println("Hello, World!") // Hello, World!

But, as you may know, even string literals can do a bit more in Kotlin than just statically storing some text.

With string interpolation (also called "string templates"), we can enrich our strings by referencing variables, calling functions, or even evaluating complex expressions:

val name = "Johnathan"
println("Hello, $name!") // Hello, Johnathan
println("Your name is ${name.count()} long!") // Your name is 9 long!

When our strings contain multiple lines of text or special characters that are usually reserved, like quotation marks or backslashes, we can use multiline strings, which are denoted by triple-quotes. Everything will still behave as expected:

val name = """
    Johnathan,
    The Great,
    The "Knowledgeable"
""".trimIndent()
println("Hello, $name!")
println("Your name is ${name.count()} long!")

/* Prints:
Hello, Johnathan,
The Great,
The "Knowledgeable"!
Your name is 41 long!
*/

When you type out these triple-quoted "raw" strings in IntelliJ IDEA, you can notice that a call to trimIndent() is automatically added to the string. This function removes the common indent of all input lines, and also removes the first and the last lines if they are blank. This means that if we have some XML or JSON stored in a string, for example, we can keep its nice formatting without having to worry that we might introduce additional characters. Note how, in this example, the printed output is not indented, and doesn't begin or end with an unnecessary empty line:

fun main() {
    val myJson = """
    {
      "name": "jane",
      "lastname": "doe",
      "age": 29
    }
    """.trimIndent()
    println(myJson)
}

/* Prints:
{
  "name": "jane",
  "lastname": "doe",
  "age": 29
}
*/

If you are worried about performance when invoking an extra function during string creation, fear not. For constant strings, this transformation is evaluated at compile time with no runtime overhead since Kotlin 1.3.40.

Of course, there are tons of other places where strings could come from. For example, Kotlin can read terminal input via the readLine() function. Or, we could read text from a file, to name just two examples:

val fromStdIn = readLine()
val fromFile = File("input.txt").readText()

Another neat way of creating Kotlin strings yourself is via the buildString function. We can give this function a code block which populates a StringBuilder, which offers us functions like append and appendLine. This is particularly useful if you are crafting large strings, and performance is a concern.

val name = "Jane"
val myString = buildString {
    repeat(10) {
        append("Hello, ")
        append(name)
        appendLine("!")
    }
}
println(myString)

/* Prints:
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
Hello, Jane!
*/

And there is a lot we can do with all these strings! Next to some functionality like upper- and lowercasing a string, Kotlin also comes with some functions that are particularly useful when we want to extract information out of them.

Extracting information from strings

Extracting information usually means removing anything that isn’t useful. For example, we might want to detect strings that don’t contain any real information at all – so empty strings, that have no characters in them, or blank strings, that only contain whitespace.

To check whether we have this kind of string, we can use the isBlank and isEmpty functions:

println("   ".isBlank()) // true
println("".isEmpty()) // true

We can also conveniently replace those empty and blank strings with default values via the ifBlank and ifEmpty functions. Here, we can add a block which generates a default value for our strings if the corresponding condition is met:

val neverBlankString = " ".ifBlank {
    "Never blank!"
}
println(neverBlankString) // Never blank!

When we have a string with meaningful content, but that information is surrounded by whitespace we can use the trim function to remove empty space from the beginning and end of our string, like so:

val input = "    valuable info "
println(input.trim()) // valuable info

If there are some other characters or maybe a common phrase which we don’t really care about on the sides of the text, there’s three more functions which are our allies: removePrefix, removeSuffix, and removeSurrounding. With those, we can get rid of characters in the front, the back, and enclosing the text respectively:

val input = "##placeholder##"
println(input.removePrefix("##")) // placeholder##
println(input.removeSuffix("##")) // ##placeholder
println(input.removeSurrounding("##")) // placeholder

Another big way of extracting information from strings is via regular expressions, or short regexes. But this is a bit of a bigger topic, so those will have to wait for a future episode.

Comparing strings

Now that we know some tricks of how to get strings containing only the information we really care about, we can do things like compare those strings with each other.

Equality checks are easy enough, with the double-equals sign checking whether two strings are identical:

val stringA = "astring"
if(stringA == "astring") {
    println("Everything cool!")
}

// Everything cool!

But did you know that you can also compare two strings based on their alphabetical order? This can be done by using the less-than and greater-than signs.

println("a" < "b") // true
println("c" < "a") // false

In situations where we want to compare two strings regardless of how their text is capitalized, we can use the compareTo and equals functions with the ignoreCase parameter set to true. Not only does this look nicer than calling toLowerCase on both strings separately it also has better performance:

val input = "QuICK brOWN fox"
println(input.equals("Quick Brown Fox", ignoreCase = true)) // true

Turning strings into collections

But often strings also contain multiple pieces of information which we would like to work with individually. To rip apart a string into pieces, there’s a number of options.

Most generally, the split function is our friend here: we can specify a character, word or other string which will be used as a splitter, and the function gives us a nice list of the individual string pieces cut up by our delimiter. Consider, for example, the following input string, which contains 5 letters, all separated by ;:

val input = "A; B; C; D; E"
println(input.split("; ")) // A, B, C, D, E

The output is a list with five elements.

And if we only want to make a limited number of cuts, we could also pass a limit to the split function. And then we’ll only get three pieces of text as a result, for example, with the last piece being the remainder of the string which wasn’t split any further. Let's consider the previous example once more, this time with a limit:

val input = "A; B; C; D; E"
println(input.split("; ", limit = 3)) // A, B, C; D; E

The output is a list with three elements: A, B, and C; D; E.

For the special case of splitting a string up by lines, there’s the very accurately named lines function which, as you may have guessed, gives you back your text line-by-line. The nice part here is that we don’t have to remember what the escape sequence for newlines are. Was it \n? \r\n? Who cares, we have the lines function!

val input = """
    Well this is crazy
    I'm a multiline string
    So split me maybe?
""".trimIndent()
println(input.lines())
// [Well this is crazy, I'm a multiline string, So split me maybe?]

And of course, we can do all sorts of fancy things once we have a collection.

Treating strings as collections of characters

And, to be exact, even an individual string behaves like a collection – since it's a collection of characters! On the one hand, this means that we can use the array operator to pick out characters at specific indexes, which can prove quite useful. But it also means that stuff like mapping, filtering, and friends all work directly on strings as well.

val input = "Hello, World"
println(input[1]) // e
println(input.filter { it.isUpperCase() }) // HW

We will tackle those collection operations in a future episode of "Kotlin Standard Library Safari" – so, if you don’t want to miss me exploring those in the future, make sure to follow @kotlin here on dev.to, and subscribe to our official YouTube channel to get updates on when a new episode is available!

Conclusion

That's all for this first segment on the Kotlin standard library. I hope that you enjoyed this brief overview of things to do with strings in Kotlin. There is, of course, still more stuff you can do. A great starting point for your own exploration of the string APIs available is the documentation on text!

If you know of some tips you'd like to see featured in a future episode, make sure to share them on Twitter and tag @sebi_io and @kotlin – and maybe we'll see your tip in a future part of this series!

I hope you learned something new, and that you will go and explore some more Kotlin!

DEV Community: Sebastian Aigner

Idiomatic Kotlin: Solving Advent of Code Puzzles, Passport Validation

Day 4. Passport processing

Solving Day 4, Part 1

Solving Day 4, Part 2

A refactoring excursion

Returning to solving part 2

Conclusion

Exploring Kotlin Lists in 2021

Lists

What’s a list?

Creating lists

Accessing list items

Slicing

Mutable Lists

Creating mutable lists

Add / Remove / Update

Fill and Clear

In-place modifications

Views on Lists

Outro

Advanced Kotlin Collection Functionality

Checking predicates: any, none and all

Predicates for empty collections

Collection parts: chunked and windowed

The chunked function

The windowed function

Un-nesting Collections: Flatten and Flatmap

Combining collections: zip and unzip

The zip function

The unzip function

The zipWithNext function

Custom aggregations: reduce and fold

The reduce function

The fold function

That's it!

Tips & tricks for building a game using Compose for Desktop

Rendering: Clipping and Coordinate Systems

Clipping

Device-Independent Pixels and Density

A Game of Geometry and Linear Algebra

Frame-Independent Movement With Delta Timing

Closing Thoughts

How I built an "Asteroids" game using Jetpack Compose for Desktop

The Game

The Building Blocks

The Game Loop

Game State Management

Game Objects

Rendering to the Screen

Continued in Part 2

Why Learn Kotlin?

Modern Language

Easy to Learn

Great Materials

Multiplatform

Progression from Java

Career prospects

Community Driven

Why did you Learn Kotlin?

Diving into Kotlin collections

Lists

Sets

Maps

Collections can be mutable

Collections are iterable

Looping over collections

Transforming collections: map

Filtering collections: filter and partition

Retrieve collection parts: take and drop

Aggregating collections: sums, averages, minimums, maximums, and counting

More collection goodness, coming soon!

Natural language processing with Kotlin: Part-of-speech tagging with Hidden Markov Model

Markov Model

Hidden Markov Model

Data & Some Preprocessing

Calculating Transition and Emission Matrices

Viterbi Algorithm: Motivation

Viterbi: Initialization

Viterbi: Forward Pass

Checking predicates: `any`, `none` and `all`

Collection parts: `chunked` and `windowed`

The `chunked` function

The `windowed` function

Combining collections: `zip` and `unzip`

The `zip` function

The `unzip` function

The `zipWithNext` function

Custom aggregations: `reduce` and `fold`

The `reduce` function

The `fold` function

Code completion (`⌃` `Space`)

Parameter info: Type popup (`⌘` `P`)

Quick Documentation (`F1`)

Go to declaration or usages (`⌘` `Click` / `B`)