DEV Community: Will BL

Can AI Deobfuscate Minecraft?

Will BL — Thu, 05 Jan 2023 19:05:19 +0000

OpenAI's ChatGPT, like many similar Large Language Models, has been used to perform many programming-related tasks. Writing obfuscation mappings seems to be a good fit for its abilities, as it involves reading and summarising text (code). I tested its capabilities.

Background

If you already understand what obfuscation mappings are, feel free to skip this part.

Minecraft: Java Edition is a game written in Java (quelle surprise). Java source code is compiled to JVM bytecode for distribution, a little like the relationship between C and machine code.

To modify the game, we want access to the Java source. However, there are two things in the way:

Firstly, we have to decompile the JAR, turning the bytecode back into Java. This is a very difficult job: a lot of work has been put into the tools to make this happen.

However, even then, we do not have readable code: Mojang, the developers of Minecraft, put their code through ProGuard, which shrinks the bytecode by doing things such as renaming all methods and classes to things like a, b, c, az, and so on. So we need a mapping from these obfuscated names to readable names!

For a while, the main mappings available were made by the MCP project, and had a restrictive license. These were used first by MCP itself, and later Minecraft Forge, a modloader and API.

When Fabric, another modloader/API project, came onto the scene in 2018, it had to make its own mappings, which it called Yarn. It made Yarn freely available (CC-0), and effected a 'cleanroom policy' - no MCP names were to be mentioned in any Fabric community spaces, in case a Yarn contributor was 'tainted' by the strictly-licensed MCP mappings.

And so it came to pass that the mappings were fractured; for there were two names for each class. "Lo," said Mojang, "we shall bestow unto them an Official Mappings set, exported from Proguard itself".

Forge quickly switched to the Official Mappings. Fabric decided that a non-public domain mappings set wouldn't do; and besides, these names were all a bit strange and unlike what they were used to anyway, so they would prefer to maintain and use their own mappings, thank you very much.

These days, there are four main mappings sets:

Official Mappings (aka. Mojmap) exactly like what Mojang sees but without parameter names.
Parchment, a project to create mappings for parameter names, and also including Javadoc, to be layered on top of the Official Mappings.
Yarn, still going with its cleanroom meant to protect against a mappings set that no longer exists.
Quilt Mappings. Quilt is a fork of Fabric made by many former-Fabric developers. Quilt Mappings are like Yarn, but without a cleanroom, and so may take inspiration from the Official Mappings. Its main purpose is to be like Yarn, but Mojmappier, or to be like Mojmap, but Yarnier - depending on who you ask.

First Tests

First, I set up a Quilt Mappings workspace. Why Quilt Mappings and not Yarn? Two reasons:

QM is missing mappings for more things, so it was easier to find obfuscated names for testing.
The QM workspace allows using the QuiltFlower decompiler, which can produce much nicer output than many other decompilers.

It took me a few tries to find a prompt that worked well. I settled on this:

I have this Java method in the class ClassName, but I don't know what to call it:
code snippet
Please suggest a name for this method and each of its parameters.

NativeImage#copyRect

I first tried this method in NativeImage:

    public void m_vmneozhc(NativeImage arg, int i, int j, int k, int l, int m, int n, boolean bl, boolean bl2) {
        for(int o = 0; o < n; ++o) {
            for(int p = 0; p < m; ++p) {
                int q = bl ? m - 1 - p : p;
                int r = bl2 ? n - 1 - o : o;
                int s = this.getPixelColor(i + p, j + o);
                arg.setPixelColor(k + q, l + r, s);
            }
        }
    }

It suggests I call the method copyImage(NativeImage sourceImage, int sourceX, int sourceY, int targetX, int targetY, int width, int height, boolean flipHorizontally, boolean flipVertically).

This is pretty good! Mojang calls the method copyRect, quite similar to ChatGPT's naming. It also names the parameters sensibly.

LocalPlayer#sendIsSprintingIfNeeded

Next, I tried this method, in the class which QM calls ClientPlayerEntity (but Mojang calls LocalPlayer):

    private void m_nfwipcth() {
        boolean bl = this.isSprinting();
        if (bl != this.lastSprinting) {
            ClientCommandC2SPacket.Mode lv = bl ? ClientCommandC2SPacket.Mode.START_SPRINTING : ClientCommandC2SPacket.Mode.STOP_SPRINTING;
            this.networkHandler.sendPacket(new ClientCommandC2SPacket(this, lv));
            this.lastSprinting = bl;
        }
    }

It suggests updateSprintingStatus().

This is not wrong, but it's a bit generic as a name - it doesn't capture that it involves telling the server over the network about the player's sprinting status.

MultiPlayerGameMode#performUseItemOn

Lets try a longer method. I'm not going to show you the whole method this time (Minecraft is closed-source, after all), but I'll show you a snippet of this method in the class QM calls ClientPlayerInteractionManager, and Mojang calls MultiPlayerGameMode:

    private ActionResult m_wiqrsdhj(ClientPlayerEntity arg, Hand arg2, BlockHitResult arg3) {
        BlockPos lv = arg3.getBlockPos();
        ItemStack lv2 = arg.getStackInHand(arg2);
        if (this.gameMode == GameMode.SPECTATOR) {
            return ActionResult.SUCCESS;
        } else {
// ...

This is alright. The method is run when right-clicking on a block - but only when holding an item. ChatGPT's name doesn't really capture this. Still, it's less ugly than Mojang's performUseItemOn.

Can it do more?

So ChatGPT can name methods, not perfectly, but alright. But can it map whole classes?

I tried it out with this prompt:

I have this Java class, but all the names are garbled:
code snippet
For context, <context of how the class is used>.

For each garbled name, suggest a proper name.

BlockStatePredictionHandler

I found a class which wasn't mapped in QM - Mojang calls it BlockStatePredictionHandler - and stuck it in the prompt. I added some context, too:

For context, the class ClientWorld has a private final field of this class. The methods of this class are called from the classes ClientWorld and ClientPlayerInteractionManager.

Placeholder	Mojmap	ChatGPT
`C_czisrdmd`	`BlockStatePredictionHandler`	`BlockStateTracker`
`f_irkdwisi`	`serverVerifiedStates`	`blockStateMap`
`f_hcvpxtyt`	`currentSequenceNr`	`tickCounter`
`f_bmpbngyn`	`isPredicting`	`isTracking`
`m_plwiurco`	`retainKnownServerState`	`addBlockState`
`m_mmtdqsga`	`updateKnownServerState`	`updateBlockState`
`m_vfahppjl`	`endPredictionsUpTo`	`rollbackBlockStates`
`m_rhbdpkkw`	`startPredicting`	`startTracking`
`close`	N/A	`stopTracking`
`m_gqtwgmpw`	`currentSequence`	`getTickCounter`
`m_nssimvch`	`isPredicting`	`isTracking`
`C_ivvmpfib`	`ServerVerifiedState`	`TrackedBlockState`
`f_bfinrexc`	`playerPos`	`playerPosition`
`f_ovpelcnt`	`sequence`	`trackedTick`
`state`	`blockState`	`blockState`
`m_twfpffxt`	`setSequence`	`updateTrackedTick`
`m_fxchmlnj`	`setBlockState`	`updateBlockState`

While ChatGPT's names are coherent, and certainly consistent (see the consistency between fields and their getters/setters), it has a few flaws. See, for example, how it tried to rename close, a non-obfuscated name inherited from Java's AutoCloseable.

I didn't know what this class was actually for, so I asked ChatGPT - perhaps this would also give us some insight into why it named the symbols in the way it did?

The last part of that, at least, is completely incorrect - block changes made by a player in creative mode are definitely not rolled back when the player leaves the game. This cast doubt on the rest of the explanation, so I searched through a few Discord servers to see if anyone else had an explanation.

Here's a (trimmed down) explanation by SizableShrimp.

Basically, clients keep track of a sequence counter in their BlockStatePredictionHandler. When client wants to perform any block modifying actions from MultiPlayerGameMode, it will gen a new sequence number by increasing the counter.
Now, while in prediction mode, any calls to ClientLevel#setBlock will not be actually set in-world! They will be stored in the prediction handler.
Now, after client finishes its clientside block-modifying action, this sequence number is sent to the server alongside the action the client is requesting to do. Server will then do the block-modifying action that the client requested (if possible).
Once the server is "done", it will send a packet back to the client that initiated the action to process all block sequences up to that sequence id.
They use this predictive action system thingy so that there aren't visual glitches on the client when what the client thinks should happen is different from what the server actually performs.
The client will now delay setting blocks during predictive actions, wait for a server response on what actually happened, and then update it on the client.

(Full explanation is available in this message on the Forge Discord)

Interesting - so ChatGPT correctly told us that the class tracks block changes and can roll them back, but it misinterprets the sequence number as a tick counter, and doesn't understand that the class is used to hold off on committing to clientside predictions until verified by the server.

It doesn't understand anything, in fact. This is one of the flaws of asking ChatGPT to explain something to you - it has no consistent internal model of the world. ChatGPT, like other Large Language Models, is able only to read some text and write more text that sounds like it should come after - as a result, it is very prone to giving completely incorrect answers about things while sounding very confident and authoritative.

MatrixUtil

Let's try one more class. This one has some methods for messing with matrices. It's unmapped in QM; Mojang calls it MatrixUtil. It's quite long (~200 lines) compared to our other tests.

I gave ChatGPT the class, with the context:

For context, m_xockxffa is called from the class ItemRenderer, and m_jeovkadj and m_ibrymiig are called from the class AffineTransformation.

Placeholder	Mojmap	ChatGPT
`C_bmzqkdhp`	`MatrixUtil`	`MatrixMathHelper`
`f_arxnkhow`	`G`	`RADIUS_FACTOR`
`f_ddnzxrmh`	`CS`	`COSINE_OF_45_DEGREES`
`f_kqccjreh`	`SS`	`SINE_OF_45_DEGREES`
`m_xockxffa`	`mulComponentWise`	`scaleMatrix`
`m_kcqrzkbi`	`approxGivensQuat`	`computeUnitVectors`
`m_kniviwwf`	`qrGivensQuat`	`computeUnitVectorsFromYawPitch`
`m_skxdryqk`	`stepJacobi`	`computeQuaternionFromMatrix`
`m_kkfnnzhk`	N/A	`computeQuaternionFromYawPitch`
`m_kjbxjxks`	N/A	`computeUnitVectorsFromQuaternion`
`m_pjmgwfwq`	N/A	`computeMatrixFromQuaternion`
`m_geumxflx`	N/A	`computeMatrixFromYawPitch`
`m_ibrymiig`	`svdDecompose`	N/A
`m_jeovkadj`	`toAffine`	N/A

Whoever named the constants must've been a mathematician.

ChatGPT has a good start: MatrixMathHelper is a solid name; SINE and COSINE _OF_45_DEGREES is descriptive (and impressive, seeing as it's computed in radians). Unfortunately, it goes downhill from there.

The rest of the names look descriptive - in many cases, they look at first glance to be better names than Mojang's. But once again, it's a case of ChatGPT being very good at sounding confident whilst being totally incorrect.

The so-called scaleMatrix method, taking a 4x4 Matrix and a float and returning a matrix with each element multiplied by the float, is named in a way that confuses matrix-scalar multiplication with the linear transformation of scaling.

computeQuaternionFromMatrix is a technically-correct name - the method takes a 3x3 matrix and returns a quaternion. But it fails to convey what the method actually does. stepJacobi at least points you in the right direction.

There's an even more damning problem, though: ChatGPT started hallucinating methods.

In the table, you'll see some methods for which there are no ChatGPT names (because it got distracted and didn't name them), and some without Mojang names (because they don't exist - ChatGPT made them up).

Oh dear.

Conclusion

ChatGPT is an interesting tool. There is no doubt that it is very good at text prediction and generation. This sometimes generalises to being able to paraphrase, summarise, and appear to understand the meaning behind text. This is not true - it cannot understand meaning.

For deobfuscation name mapping, Large Language models such as ChatGPT may be useful as assistive tools. They cannot be relied on to automate the naming process.

In the specific case of Minecraft, with the existence of Mojmap, there is limited value in maintaining alternate mappings. There is more value in augmenting Mojmap, e.g. with Parchment. Therefore it would be useful to see how the process of mapping parameter names and writing Javadoc can be made more efficient.

There is less room for error in parameter naming than class/member naming. Large Language Models could be used to help automate this process - however, specialised non-ML tools may be as or more useful.

Stay tuned - I will test ChatGPT's ability to write Javadoc for decompiled methods in another post. I don't suspect it will do very well.

Thanks to:

Shedaniel's Linkie, an enormously useful tool for translating between mappings.
The QM and Yarn contributors
OpenAI.
The Quiltflower contributors.

Top 5 IntelliJ IDEA Tips & Tricks!

Will BL — Sun, 26 Sep 2021 18:16:39 +0000

IntelliJ IDEA is an excellent all-rounder IDE by Jetbrains. It's most often used for Java and Kotlin, but can also be used to develop for JS, Rust, and many more languages. It's what I use for most of my programming, and so I thought I'd share what config tweaks, tricks, and plugins I've used that make it nicer.

Darcula? Make it darker!

Darcula is a very nice theme, but it always bugged me that it wasn't nearly as dark as the light theme is light. It felt like there was more potential there. Darcula Darker, by Vecheslav Druzhbin, darkens backgrounds, contrasting it more against text.

Rainbow Brackets

This plugin may sound silly at first, but rainbow brackets honestly vastly improves readability of nested brackets. It works by giving each pair of brackets its own colour, so that nested sets are visually distinct.

While we're on the rainbow topic, for those who work with CSV files often, Rainbow CSV may be useful.

Errors should be visible!

Sometimes, in very dense code, it can be difficult to find a single-character syntax error at a glance, as its signified by a tiny red squiggly line. How do you fix that, and make it stand out? Add a red highlight, too!

(This can be done by going into Settings -> Editor -> Color Scheme -> General -> Errors and Warnings -> Error and setting Background to #630202)

Comments are for humans

One thing I will never understand is that comments, despite being more for humans' benefit than the rest of the code, are always put in low-contrast greys, as if the machine, envious that it cannot read them, doesn't want me to be able to either.

So make your comments bright pink.

(This can be done by going into Settings -> Editor -> Color Scheme -> Language Defaults -> Block Comment + Line Comment and setting Foreground to #FC7CE1)

nya-nya-nya-nya-nya (or is it nyan-nyan-nyan?)

I don't have any reason for this other than that it makes me smile. But do I really need another reason?

More!

I have yet more opinions on how to get the most out of your editor - follow me to get notified when part two of this series is out! You'll get to learn:

How to get even more powerful git support
How you might be able to get IntelliJ IDEA Ultimate for free
Which side of the Emacs v. Vim editor war I'm on!

Let's write a tiny JSON parser in Kotlin! Part 4: Strings

Will BL — Thu, 20 May 2021 13:36:53 +0000

Today we're going to learn how to parse strings in JSON. This is more complicated than the datatypes we've done so far, but don't worry, it's not bad :)

Recap

Let's recall our string definition from part 0:

A string is a string of characters enclosed within double quotes ("). Any character can be put within a string, except for the following, which must be escaped:

double quotes (") [Escaped with \"]

backslash (\) [Escaped with \\]

backspace [Escaped with \b]

form feed [Escaped with \f]

line feed [Escaped with \n]

carriage return [Escaped with \r]

horizontal tab [Escaped with \t]

In addition, the escape code \/ resolves to a forward slash (/), and a backslash followed by u followed by four hex digits resolves to the character at the Unicode codepoint specified by said hex digits.

Writing it

First, let's set up the skeleton of our method:

fun readString(): String? {
    val oldCursor = cursor
    val result = StringBuilder()
}

The quotes

A string will always start with a quote, so if we don't see one at the start, we can fail:

    if (step() != '"') {
        cursor = oldCursor
        return null
    }

The loop

We'll next use a loop to iterate over each character, firstly making sure that there is a next character, and it isn't a double-quote (since that would mark the end of the string) - then, in the loop we'll store the current character in a variable so we can do our logic on it.

    while (hasNext() && peek() != '"') {
        val char = step()
    }

Escaped characters

First, we need to check for escaped characters. These will always start with a \ and will always have at least one other character afterwards - so let's check for a backslash character, and then begin pattern-matching on the next character:

        if (char == '\\') { // just a single backslash, written as a double backslash to escape it
            when (val it = step()) {

Now we can check for each valid following character, and add that to our string:

                '"' -> result.append('"')
                '\\' -> result.append('\\')
                '/' -> result.append('/')
                'b' -> result.append('\b')
                'f' -> result.append(0x0C.toChar())
                'n' -> result.append('\n')
                'r' -> result.append('\r')
                't' -> result.append('\t')
                'u' -> result.append(readHexChar())
                else -> return null

A few things of interest here:

Kotlin does not support \f for form feeds, so we have to use the raw ASCII value.
We've put the \u0000-reading logic into a new function, readHexChar, which we'll write in a second.
We have else -> return null as there are no other valid characters after a backslash. If you want to be slightly less spec-conforming, you could use else -> result.append(it)

`readHexChar()`

Let's make our readHexChar method:

private fun readHexChar(): Char? {
    val oldCursor = cursor
    return try {
        read(::isHexDigit).toInt(16).toChar()
    } catch (e: NumberFormatException) {
        cursor = oldCursor
        null
    }
}

This is fairly simple, using our read and isHexDigit functions, then parsing the resulting hex string, then converting it to a Char. Simple!

The other characters

The other branch of our if, for non-escaped characters, is nice and simple:

        } else {
            if (char >= 32.toChar()) {
                result.append(char)
            } else {
                cursor = oldCursor
                return null
            }
        }

The characters disallowed in JSON strings are the first 32 ASCII codepoints, the control characters. If we encounter one of those, we can fail. Otherwise, we add it to our string.

Leaving the loop

    }
    skip()
    return result.toString()

Finally, we skip the last character (since it's a double quote) and return our final string.

Conclusion

Here's our final code:

    fun readString(): String? {
        val oldCursor = cursor
        val result = StringBuilder()
        if (step() != '"') {
            cursor = oldCursor
            return null
        }

        while (hasNext() && peek() != '"') {
            val char = step()
            if (char == '\\') {
                when (val it = step()) {
                    '"' -> result.append('"')
                    '\\' -> result.append('\\')
                    '/' -> result.append('/')
                    'b' -> result.append('\b')
                    'f' -> result.append(0x0C.toChar())
                    'n' -> result.append('\n')
                    'r' -> result.append('\r')
                    't' -> result.append('\t')
                    'u' -> result.append(readHexChar())
                    else -> return null
                }
            } else {
                if (char >= 32.toChar()) {
                    result.append(char)
                } else {
                    cursor = oldCursor
                    return null
                }
            }
        }
        skip()
        return result.toString()
    }

Want to improve it? You could instead throw an exception when encountering invalid characters, instead of returning null.

Let's write a tiny JSON parser in Kotlin! Part 3: Bools & Nulls

Will BL — Mon, 10 May 2021 11:30:47 +0000

Lets learn how to read the two simplest elements: Booleans and Nulls. These don't get their own diagrams due to their simplicity, but they can be seen in the grammar diagram for values:

Booleans

A boolean is super simple, of course - true or false (no yeses and nos like YAML has!)

So we can use the read(String) method we defined earlier to read either true or false:

fun readBoolean(): Boolean? {
    val oldCursor = cursor
    if (read("true") == "true") {
        return true
    }
    cursor = oldCursor
    if (read("false") == "false") {
        return false
    }
    cursor = oldCursor
    return null
}

It will attempt to read true, if that's not there it'll backtrack and try to read false, and if that's not there it'll give up and give you null. Simple.

But not as simple as Null!

Null

Null is the simplest data type (is it even a data type (in JSON)?)

fun readNull(): Boolean {
    val oldCursor = cursor
    if (read("null") == "null") {
        return true
    }
    cursor = oldCursor
    return false
}

Now you can probably see how this works - try to read the symbol null: if it's there, return true ("yes, there is a null here"); if it's not there, put the cursor back to where it was before and return false ("nope, no nulls here")

Conclusion

Awesome, we've learned how to read our first two JSON data types! These, of course, were the simplest - next up we'll be doing Strings - definitely trickier!

Let's write a tiny JSON parser in Kotlin! Part 2: The Parser

Will BL — Thu, 06 May 2021 21:01:19 +0000

So now we have an understanding of JSON and the components that make it up, but we have to create something that will read it - this will be our parser.

Our parser will step through our JSON file character by character to read it, so it'll need to:

remember our JSON
remember where in the JSON it's currently looking
know if it's at the end
read the current character
step to the next character

So let's do it!
First, we'll create a class which will take in our JSON as a CharSequence (we could use a String, too - the functionality is the same, but this lets you do something weird like throw a StringBuilder in there), and will remember where it's currently reading:

class Parser(private val input: CharSequence) {
    var cursor: Int = 0
}

Now let's give it the ability to read and step through the input.

class Parser(private val input: CharSequence) {
    var cursor: Int = 0

    private fun step(): Char {
        return input[cursor++]
    }

    private fun peek(): Char {
        return input[cursor]
    }

    private fun skip() {
        cursor++
    }
}

peek() will read the current character, skip(), will move the cursor forwards by one, and step() and read and then move on.

However, trying to read past the end will throw an exception, so let's add a method to check if we're at the end:

    private fun hasNext(): Boolean {
        return cursor < input.length
    }

This is everything we'll need for moving along our JSON text. Next, let's add some functions to actually read it:

fun read(predicate: (Char) -> Boolean): String {
    val result = StringBuilder()
    while (predicate(peek())) {
        result.append(step())
    }
    return result.toString()
}

fun read(pattern: CharSequence): String {
    val result = StringBuilder()
    var patternCursor = 0
    while (peek() == pattern[patternCursor++]) {
        result.append(skip())
    }
    return result.toString()
}

These two functions are vital to how the parser will do its job:

They read the JSON one character at a time, and store what they've read in a buffer, until the character they read reaches a certain condition. For the first function, that condition is given via a predicate, and in the second, the condition is that the buffer's contents match a given string. Once the condition is met, the buffer's contents are returned.

These five methods, simple as they seem, are extremely powerful, and all we need as a foundation for our parser!

Making a Discord Vaccine Bot

Will BL — Fri, 19 Feb 2021 13:58:44 +0000

I recently made a Discord bot that tracks the UK vaccine rollout, using Java, JDA, and the coronavirus.data.gov.uk API.

First Ideas

I'd been browsing the excellent UK Coronavirus dashboard, and I saw the 'developers' guide' link.

I gave it a quick look over, saw that it didn't require registering for an API key or anything, and had an idea: make a Discord bot to spread vaccine positivity in my group chats :)

JDA and Java

I've done work on Discord bots for communities using JDA before, so I knew I'd use that for this project just because I had familiarity with it.

Regarding language, I had the option to either use Java or Kotlin - I went with Java just because it tends to be quicker for Gradle to set up a Java project (not having to download the latest Kotlin stdlib), but I do much prefer Kotlin and for most purposes will use that.

First attempts

My first attempt was very basic, and amounted to something similar to this:

private static final String API_URL = "[...]";
private static final Timer TIMER = new Timer();

private static JDA bot;

public static void main(String[] args) {
    try {
        bot = JDABuilder.createDefault(args[0])
                .setActivity(Activity.watching("the vaccine rollout"))
                .build();

        TIMER.scheduleAtFixedRate(new TimerTask() {
            @Override
            public void run() {
                checkStats(bot, args[1]);
            }
        }, 2000, 1000 * 60 * 60 * 24);

    } catch (LoginException e) { [...] }
}

private static void checkStats(JDA bot, String channelId) {
    long amount;
    try {
        amount = JsonParser.parseReader(new InputStreamReader(new GZIPInputStream(new URL(API_URL).openStream()))).getAsJsonObject().getAsJsonArray("data").get(0).getAsJsonObject().get("value").getAsLong();
    } catch (IOException e) { [...] }

    channel.sendMessage(":tada: **Good news!**\nThere have now been "+amount+" vaccinations in the UK!").queue();
}

This worked, but was a bit barebones.

It does three things:

sets up a bot with JDA, with the token given from the first command-line argument (JDABuilder.createDefault(args[0]))
sets the bot's activity as "watching the vaccine rollout" (setActivity(Activity.watching("the vaccine rollout")))
sets a timer, which every 24 hours (1000 * 60 * 60 * 24 milliseconds) checks the vaccination count and sends a message to a channel as given in the second argument (checkStats function)

I took me an hour or so to get to this point. Most of the time was spent trying to work out why the input I was receiving was garbled.

Because I hadn't read this:

RTFM, folks.

One GZIPInputStream later, everything was working! But what else could I do?

Extra features

The next thing I did was percentage calculation, just done by dividing by 680000 (roughly 0.01 times the UK population). There were decimal place mishaps at first:

But it eventually worked well.

I later added the count as words, which someone said they could parse more easily than the numerals:

There have now been 10971047 (that's ten million nine hundred seventy-one thousand forty-seven) vaccinations in the UK! That's 16% of the population!

I used a library called 'tradukisto' to do the work for me. It also supports other languages than English, which is cool.

I also later added a command, ,vaccinebot, to get an immediate count:

Conclusion

This was a fun quick project, and it's had an uplifting impact on my friends:

If anyone else wants to run an instance of it, the code is available.

I'm grateful that Public Health England have made the COVID data available via an accessible and documented API, and that the COVID dashboard website exists.

Let's write a tiny JSON parser in Kotlin! Part 1: Characters

Will BL — Thu, 18 Feb 2021 23:45:45 +0000

In the previous post, we looked at the elements that make up JSON. Now, lets start building something that can recognise them!

To start off with, we'll need to define three important building blocks of JSON: whitespace, digits, and hex digits.

Whitespace

You'll remember that whitespace is a string of zero or more spaces, line feeds (\n), carriage returns (\r), and horizontal tabs (\t).

To check if a character is a valid part of whitespace, we can define the following extension function:

fun Char.isWhitespace(): Boolean {
    return this == ' ' || this == '\r' || this == '\n' || this == '\t'
}

which checks if the character is one of the characters allowed in whitespace. Simple!

Digits

Digits are even simpler: the Arabic numerals 0..9. We can check if a character is a valid digit with this:

fun Char.isDigit(): Boolean {
    return this in '0'..'9'
}

This works because in ASCII (and therefore also in Unicode) the characters for the ten digits are all assigned consecutive numbers - so
this in '0'..'9' becomes '0' <= this && this <= '9', and '0' and '9' become their respective ASCII codepoints.

Hex Digits

fun Char.isHexDigit(): Boolean {
    return this in '0'..'9' || this in 'a'..'f' || this in 'A'..'F'
}

This is pretty much the same as isDigit, but with extra checking for digits A through F. Remember to check for both upper and lower case!

Conclusion

All of these functions could also be written as expression functions - give that a go!

In the next post, we'll start work on our parser.

Mixins, Kotlin, and Default Methods - why they don't work together

Will BL — Sun, 25 Oct 2020 19:01:51 +0000

A note for people using Mixins which implement interfaces which have default implementations of methods which are written in Kotlin.

Lets say you have an interface in Kotlin:

interface MyKotlinterface {
    fun doMyThing() {
      x.y.z.doThing(this)
    }
}

and a mixin:

@Mixin(PlayerEntity.class)
abstract class MyMixin implements MyKotlinterface {}

you might think that you can then do (player as MyKotlinterface).doMyThing(). However, you will get an AbstractMethodError. Why? because Kotlin doesn't compile your interface how you'd expect it to.

You'd think that it would compile like:

public interface MyKotlinterface {
     public default void doMyThing() {
          x.y.z.doThing(this)
     }
}

but that would only work on JVMs at or above version 8, when default implementations for methods in interfaces were added - so Kotlin compiles it like this instead, which works on all JVMs:

public interface TestInterface {
    /* All calls to the default impl of this method
     * are rewritten by the kotlin compiler to use
     * the static method
     */
    public void doMyThing();

    public static final class DefaultImpls {
        public static void doMyThing(TestInterface obj) {
            x.y.z.doThing(obj)
        }
    }
}

The side-effect of this is that when you add your interface to a class with that mixin, it bypasses the kotlin compiler's rewriting - and so, attempts to access the default impl of the method on the player will crash with an AbstractMethodError. There are different fixes for this, depending on which Kotlin version you're using:

For Kotlin <1.2, you'll have to just write your implementations in your mixin class.
For Kotlin 1.2<=version<1.4, annotate your method with @JvmDefault, and put this in your build.gradle:

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
    kotlinOptions {
        freeCompilerArgs += '-Xjvm-default'
    }
}

For Kotlin >=1.4, just put this in your build.gradle:

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
    kotlinOptions {
        freeCompilerArgs += '-Xjvm-default=all'
    }
}

and your default methods will be compiled as they should be!

with thanks to this blog post and this decompiler/bytecode viewer

Let's write a tiny JSON parser in Kotlin! Part 0: Understanding JSON

Will BL — Tue, 13 Oct 2020 13:46:40 +0000

Nearly every language has some support for JSON: many have official libraries for dealing with it, and for those that don't there are plenty of third-party libraries - you're often spoiled for choice.

With all these options, you'd almost never have to parse JSON yourself. But let's do that anyway!

What is JSON Made of?

JSON is built up of various elements - we can start with the simplest of these and work our way up until we can parse entire documents.

On the JSON website, there are diagrams that show how each element of the JSON specification is defined. These can really help you wrap your head around the grammar, so they'll be reproduced in this post.

Whitespace

Lets start with the simplest element - whitespace is a string of zero or more spaces, line feeds (\n), carriage returns (\r), and horizontal tabs (\t).

Strings

A string is a string of characters enclosed within double quotes ("). Any character can be put within a string, except for the following, which must be escaped:

double quotes (") [Escaped with \"]
backslash (\) [Escaped with \\]
backspace [Escaped with \b]
form feed [Escaped with \f]
line feed [Escaped with \n]
carriage return [Escaped with \r]
horizontal tab [Escaped with \t]

In addition, the escape code \/ resolves to a forward slash (/), and a backslash followed by u followed by four hex digits resolves to the character at the Unicode codepoint specified by said hex digits.

Numbers

Numbers are pretty complicated:

They consist of an optional - if the number is negative, followed by a digit. If the digit is nonzero, there can follow any amount of digits. If it is zero, there can be no more digits and we move on.

There can then be a point (.), after which there can be any amount of digits representing a decimal fraction.

Afterwards, there can be an e (or E), followed by an optional sign for the exponent (+/-). After this, there can be any amount of digits for the exponent, for scientific notation-style numbers.

😰 Wow, that's a lot. Numbers were definitely the hardest part of this parser to write for me. But don't worry if you don't understand it yet - you'll become well-acquainted with the grammar when writing the parser!

Values

In JSON, a value is whitespace, followed by one of:

an object
an array
a number
the boolean value true
the boolean value false
null

followed by whitespace.

Arrays

Arrays are nice and simple. A collection of zero or more values (or whitespace) enclosed in square brackets, with a comma after every value but the last.

Objects

An object is a collection of zero or more string : value pairs, surrounded by curly brackets, and with commas after all elements but the last.

Closing thoughts

A few elements of the grammar can seem complicated at first (looking at you, numbers...), but those diagrams can be really useful. You may also want to take a look at: