DEV Community: Martin Häusler

LSM4K 1.0.0-Alpha released

Martin Häusler — Sun, 08 Jun 2025 10:26:22 +0000

Hello everyone,

I'm proud to announce the 1.0.0-alpha release of LSM4K, my transactional Key-Value Store based on the Log Structured Merge Tree algorithm. I've been working on this project in my free time for well over a year now (on and off).

https://github.com/MartinHaeusler/LSM4K

Executive Summary:

Full LSM Tree implementation written in Kotlin, but usable by any JVM language
Leveled or Tiered Compaction, selectable globally and overridable on a per-store basis
ACID Transactions: Read-Only, Read-Write and Exclusive Transactions
WAL support based on redo-only logs
Compression out-of-the-box
Support for pluggable compression algorithms
Manifest support
Asynchronous prefetching support
Simple but powerful Cursor API
On-heap only
Optional in-memory mode intended for unit testing while maintaining same API
Highly configurable
Extensive support for reporting on statistics as well as internal store structure
Well-documented, clean and unit tested code to the best of my abilities

If you like the project, leave a star on github. If you find something you don't like, comment here or drop me an issue on github.

Error as Value vs. Exceptions

Martin Häusler — Fri, 26 Jul 2024 22:14:00 +0000

In the last ~20 years, exceptions have been the predominant way to handle errors in programming. Java, C#, C++, Python and many others offer this feature and the concept is pretty similar in all of them. With newer languages like Go and Rust entering the stage, we see the rise of a different way of error handling: errors as values. In this article, we're going to take a closer look at this new paradigm and I'll give you my thoughts on it. Please note that this piece of text is an opinion and not a universal truth, and it may change over time.

What is "Error as Value"?

The basic idea behind the Error as Value pattern is to return dedicated error values from a function whenever this function encounters an invalid state. Contrary to the exception workflow, Error as Value suggests to return the error as a result of the function. This way, the error becomes part of the function signature, which leads to two major effects:

Everyone who is calling the method will see at first glance that it can produce an error
Everyone who is calling the method will be forced to deal with this potential error situation in their code (languages vary in how strict this is being executed upon)

The implementation details of this pattern vary. For instance, Go functions often return pairs - the first entry of the pair is the actual result (if any) while the second entry is the error which has occurred (if any):

f, err := os.Open("filename.ext")
if err != nil {
    log.Fatal(err)
}
// do something with the open *File f

The Go compiler will not allow you to have unused variables, so you're forced to check the err value.

In Rust, things are handled a bit differntly, but ultimately to the same effect. Rust has a generic Result<T, E> enum which has two concrete implementations: Ok and Err. Any function which returns a Result<T,E> can thus produce an actual result or an error, and the caller has to differentiate between the two:

    let fileResult = File::open("hello.txt");

    let f = match fileResult  {
        Ok(file) => file,
        Err(error) => panic!("Problem opening the file: {error:?}"),
    };

Advantages of Error as Value

The proponents of this approach will often cite the following advantages:

Errors become parts of the function signature
The compiler forces the programmer to deal with the errors
Errors as values have superior performance to exceptions
Errors as values provide clearer control flow

My thoughts on Error as Value

Please note that everything that follows is my informed opinion as a server side developer and mine alone. It is a hot take, and you may disagree with it violently.

Errors should not happen.

The argument that Error as Value is better for performance, while technically true¹, has zero significance in practice to me. If our program ends up in an error state for some reason, we're already disqualified from the performance race, because our program went off the track. There's no reason to optimize for performance in an error state because they should occur rarely to begin with.

If your program uses errors for regular control flow, you were doing something wrong in the first place.

Real recoverable errors are a myth.

I've often heard, in different contexts, the distinction between a "recoverable error" and an "unrecoverable error". To me, this has always been weird and artificial. The philosophy in early Java was that checked exceptions are to be used for recoverable errors. By that definition, how is my API request going to recover from an SQLException which tells me that my commit cannot proceed because it violates a data constraint on the database server? It won't. The best thing we can do is to log it and send it to a central error collector to get the issue fixed by changing the source code. The program (i.e. the server as a whole) may "recover" (i.e. the error in the processing of the single request doesn't terminate the entire server) but the request cannot be salvaged at this point. Attempting to deal with an SQLException on the level of the method which called commit() is therefore futile, as it has no alternative way to continue.

Java was the first language to introduce "checked exceptions" which you're forced to:

either declare explicitly in your method signature
or are forced to catch and process within the method itself

To this day, Java is the only language I know which has this concept. C# didn't inherit it, and even JVM aliens like Groovy and Kotlin ditched it completely. The fact that any modern Java framework or library will also forego checked exceptions in favor of unchecked ones should be enough to tell you: checked exceptions (while nice in research papers and conference talks) turned out to be a really impractical idea.

For Rust and Go, Error as Value are recommended to be used for "recoverable errors" and panics (which we will discuss later) for unrecoverable errors. Do you see the pattern? It's checked exceptions all over again.

The unfailable function fallacy

The Values as Errors methodology encodes errors in function signatures. However, if you look at Go and Rust standard libraries, not all functions in this methodology have errors in their signatures. This implies that there are functions which cannot fail.

In a perfect world, we would be able to write functions which can never fail. However, that is not the case in practice. Consider these:

A simple integer addition a + b can "fail" if the values are sufficiently high enough to exceed the range of representable integers.
You're calling another function in your function. This additional call may exceed the maximum stack size of the platform you're running on. This crashes your program, but the function itself did nothing "wrong".
You're sending data over the network, but right in the middle of it, the connection is lost. Maybe you're on a mobile network and entered a zone with bad reception. Maybe somebody physically unplugged the ethernet cable. None of this is the programmer's fault, but it will cause a function in the program to fail.
You're calling malloc (every non-trivial program does, one way or another) and the underlying runtime has no memory left for you.

The key message here is: your program will fail, in arbitrary places, for arbitrary reasons, no matter how well-engineered it may be. Outside of academic boundaries, there's no way to protect us from that. How we deal with these situations is the key question.

Going back to Rust and Go, if we're serious about Error as Value, then every function would need to have an error output, from simple arithmetics to database commits over the network. Needless to say, this is enormously impractical (which is why Rust and Go "chicken out" with panics; we will talk about those in a moment).

The conclusions I draw from this are:

Every function can and will fail, no matter which task it attempts to perform.
We cannot conceivably enumerate all the different reasons why it failed, and it is futile to try.
The attempt to encode errors as part of function signatures, while valiant, is impractical and ultimately futile.

This eliminates the advantages of "clearer control flow" (it's not) and "including errors in the signature" (you can't, not exhaustively at least).

Don't panic

Rust and Go, aside from Error as Values, have another "failure mode", which is called "panic". When a panic occurs, the code exists the normal control flow, exits the current function, exists the calling function, and so on, until the program either terminates as a whole, or (and this is the fun part) it reaches an error boundary. In Rust, that's called catch_unwind, in Go it's called recover.

If this process of "exiting functions through a different control flow" seems familiar to you, it is because this is exactly how Exceptions behave.

Rust and Go have exceptions. They just don't tell you.

To me, that is the point where the entire house of cards with Error as Value collapses under its own weight. This is hypocrisy.

The merit of try-catch

Exceptions and try-catch blocks have a lot of merits which people tend to forget:

they offer a structured way of handling exceptions
they represent error boundaries which apply to everything that happens in the try block, no matter if it's in the same function or way down the call chain
they free you from dealing with each error in every place individually and centralize error handling to the places where errors can actually meaningfully be dealt with
they de-clutter control flow for all down-stream functions because they don't need to handle their own exceptions
exceptions automatically track the place they've occurred in

The last point is especially important. Sure, in Rust, you can use the ? operator on any function which returns a Result<T,E> and it will exit out of the current function if an error is present, otherwise it will give you the plain result. It's cool, but... if you eventually decide to look at an error, you'll have no clue where it came from. Tracking this down can be really hard. Yes, there are ways in Rust to manually attach tracking information to error objects, but at this point I feel Rust is just attempting to fix the problems it created for itself. In Go, to my knowledge there is no such thing. It's if err != nil galore all the way.

The true drawbacks of Exceptions

From my point of view, there are a couple of problems with exceptions:

Exceptions can be overlooked. The library function you're calling may be throwing an exception, and you're not aware of it as a programmer.
Nothing forces you to eventually deal with exceptions.
Exceptions and try-catch can be abused for regular control flow.

Sure enough, all three of those are tackled by Errors as Values. But to me, it creates more problems than it solves. The issues I've mentioned above are matters of programmer discipline and clean code. I realize that I may sound like a C programmer talking about malloc and free when the first garbage collectors were introduced. But Error as Value is so invasive to the code structure that I can't imagine working with it in any serious fashion.

Error as Value in Kotlin

Kotlin sits somewhere in the middle. It does have exceptions (inherited from Java) but it also has it's own way of doing Error as Value. Consider this method signature:

fun String.toIntOrNull(): Int?

The standard library in Kotlin defines a toIntOrNull() method for String. It will parse the string, and if it is parseable into an integer, the integer will be returned. Otherwise, null will be returned. As Kotlin is a null-safe language, the programmer is forced to handle the null case separately. This is in a very similar vein as Error as Value, but with the error being restricted to the null value. The downside is that the error case cannot carry any semantic information (why did the parsing fail? At which position?). The advantage is that it is syntactically very pleasant and very clear. As a developer calling this API, you cannot use it the wrong way (the compiler won't let you) and you cannot forget anything. I've grown rather fond of this approach. One caveat is that it only works for methods which have actual return values, and that null must not be a "regular" return value for the method (otherwise you can't distinguish between null as in "whoops that's an error" or null as a regular non-error return value).

Conclusion

That's all I've got to say today. I hope you've found it interesting, it's just a topic that's been on my mind for quite a while now. And even though I'm fully on the "team exceptions" at the moment, I'm not excluding the possibility that my mind will change in the future if the arguments are compelling enough. And I think we can at least all agree that both solutions are better than <errno.h> in C.

Exceptions are notoriously slow compared to regular control flow. The main reason for that is that most languages which utilize exceptions include a stack trace in the exception, and collecting this information is costly. ↩

So I tried Rust for the first time.

Martin Häusler — Tue, 04 Jun 2024 17:11:42 +0000

In my professional life, I'm at home on the Java Virtual Machine. I started out in Java over 10 years ago, but since around 4 years, I'm programming almost entirely in Kotlin. And while Kotlin serves a lot of different use cases and compilation targets, I've already felt like I need a "native tool" in my tool belt. I can deal with C if I have to, but it feels very dated at this point. And I heard all the buzz about Rust being the most loved programming language out there, so I thought I might take a look and see for myself.

Here's how that went.

Installation

The installation process was super smooth on my linux machine. I've installed rustup and let cargo init do its job; project setup done. I started my Visual Studio Code, got a few plugins, and was hacking away within minutes. I especially love the fact that cargo is also a built-in dependency management tool. Nice!

Advent of Code

Whenever I try a new language, the first thing I throw it at is usually the Advent of Code. It is a set of language-agnostic problems which start out very easy and gradually get more difficult; perfect for testing a new language!

I picked the "Day 1" example of 2023. We're given a textual input file; each line contains digits and characters. We need to find the first and the last digit in each row, convert them into a number (e.g. 8 and 3 turn into 83) and sum them up. After a little bit of tinkering, I ended up with this:

static INPUT: &'static str = include_str!("inputs/1.txt");

fn main() {
    let mut sum = 0;
    for line in INPUT.lines() {
        let numbers: Vec<u32> = line
            .chars()
            .filter(|c| c.is_numeric())
            .map(|c| c.to_digit(10).unwrap())
            .collect();
        let first = numbers.first().unwrap();
        let last = numbers.last().unwrap();
        sum += first * 10 + last;
    }
    println!("{}", sum);
}

This ended up producing the correct result. This may not be idiomatic rust (I have yet to figure out what that is for myself) but it gets the job done. I have a lot of things to say about this little program.

include_str!

We're already starting out with a macro on the first line. I'm REALLY not a fan of macros¹, but that's just how Rust rolls I guess. If we look beyond the problems of macros overall, the include_str! macro is really nice. It includes the text of an input file as a string in the program, and the compiler verifies that the file path is correct and the file exists. This should raise some eyebrows: this macro is doing more than just producing regular Rust code, it talks to the compiler. This pattern of macros opening sneaky paths to compiler intrinsics is going to repeat on our journey. It allows the compiler to provide better error messages, but macros are also really good at hiding what code is actually being executed. At the very least, in Rust all macros are clearly demarcated with a trailing ! in their name so you know where the "magic" is².

Give me a lifetime

Right in our first line we're also hit with the expression &'static. There are really two things going on here:

& means that the type that comes next (str) is actually a "borrowed" reference. We will dig into this later, for now know that a borrowed thing is immutable.
'static is a "lifetime specifier". All of these start with a single quote ('). Rust is the only language I've ever seen that uses single quotes in a non-pair-wise fashion. And good lord is it ugly! Looking past the syntax, 'static is a special lifetime that means "lives for the entire duration of the program", which makes sense for a constant like this.

main

The main function declaration is nothing special, aside from the nice fact that you're not forced to accept parameters, nor are you forced to return an integer (looking at you, Java!). fn is fine as a keyword, but I still prefer Kotlin's fun.

let mut

Variables are declared with let, types can optionally be annoated after the name with : SomeType. Fairly standard, all fine. The keyword mut allows us to mutate the variable after we've defined it; by default, all lets are immutable. Kotlin's var and val approach is a little more elegant in that regard, but it's fine.

Semicolon!! `;`

Gods why. That's all I have to say. Why does the darn thing refuse to die? Chalk up another language with mandatory semicolons, they just keep coming.

for

Next is our standard "foreach" loop. A noteworthy detail is that there are no round braces ( ... ), which is also true for if statements. Takes a little time to get used to, but works for me. lines() splits the text into individual lines for us.

Iterators

Next we see chars() which gives us an iterator over the characters in the current line. Rust actually doesn't differentiate between a lazy Sequence and a lazy Iterator, like Kotlin does. So we have all the functional good stuff directly on the Iterator. We filter for the numeric characters, we map them to integers, we collect the result in a Vec<u32> (which is roughly comparable to a List<Int> in kotlin).

Lambdas

The lambda syntax in rust is... iffy at best, and a monstrosity at worst. The examples in the filter and map calls are very basic ones, things can get much worse. Rust actually has lambdas with different semantics (kind of like kotlin's callsInPlace contract). Sure, these things are there to aid the borrow checker, but I really miss my it from kotlin. filter { it.isDigit() } is hard to beat in terms of readability.

Type Inference... sometimes.

Much like the Kotlin compiler, the Rust compiler is usually pretty clever at figuring out the types of local variables and such so you don't have to type them all the time. Except that my let numbers absolutely required a manual type annotation for some reason. It's not like collect() could produce anything else in this example, so I don't really get why.

Unwrap all the things!

You may have noticed the calls to unwrap(). The method unwrap() is defined on Rust's Result<T, E> type, which can be either an Ok<T> (success) or an Err<E> (error). unwrap() on Ok simply returns the value, unwrap() on Err causes a panic and terminates the whole program with an error message. And no, there is no try{ ... } catch { ... } in Rust. Once your program panics, there's no turning back. So unwrap() shouldn't be used in production code. For this simple toy example, I think it's good enough. Later I learned that if you're inside a method that returns a Result, you can also use ? to propagate the error immediately.³

print it!

Rust wouldn't be Rust if println was straight forward, because it really isn't. Right off the bat we see that println! has an exclamation mark, so it's a macro. And oh boy does it do a lot of magic. Rust doesn't really have string interpolation, so the standard way to print something is println!("foo {} bar", thing) which will first print foo, then the thing, then bar. That is, provided that thing is printable, but that's another story. You could also write println!("foo {thing} bar") which almost looks like string interpolation, but it's really just a feature of the println! macro. It breaks down quickly, for example when you try to access a field inside the curly brackets. This won't compile: println!("foo {thing.field} bar"). So while it may seem simple at first glance, it really isn't and it does a whole lot of magic. Kotlin string templates are not without issues (especially if you want a literal $ character in the string) but they are more universal because they allow for arbitrary logic in their placeholders.

The Pros and Cons

At this point, it's time for a first verdict.

Pro: Tooling is good
Pro: Compiles natively
Pro: Memory safe without manual memory management or garbage collector
Pro: small executables, excellent runtime performance
Pro: Feels "modern"
Con: Syntax. Good lord the syntax.
Con: Macros. No need to explain that.
Con: Error handling, or rather the absence thereof
Con: Enormously steep learning curve with borrowing and lifetimes
Con: Type inference sometimes works, sometimes doesn't

... and that's just for the first example. I've since performed more extensive experiments with the language, and I'll report about these in another article.

I can see why system level programmers enjoy Rust. It's certainly a big step up from C. But on a grand scale, compared to the effectiveness and productivity of Kotlin, I can't yet see where Rust's popularity is really coming from.

As programmers, we already have enough tools at our disposal to shoot ourselves into the foot; macros just add to that danger. ↩
In C, that's not the case. You can make a macro look like a totally normal function call, but it does arbitrary other things when compiled. ↩
The Go language should take notes here. if(err != nil) return err is only funny so many times before it gets really old really fast. ↩

Is your Kotlin compiler slow? Here's a potential fix

Martin Häusler — Mon, 31 Aug 2020 17:30:34 +0000

I'm working at a large Java/Kotlin project at my company. It's several 100k lines of server-side code, with millions more coming in through dependencies. The Kotlin compiler sure has its work cut out for it. But as of late, the team noticed that the compiler was getting slower and slower, even though the project source size wasn't growing nearly at the same rate. Also, builds were consuming an abundance of CPU resources - on all available cores. Build times for a full project build had increased from about 2 minutes to more than 10 minutes. Something was definitely wrong.

A couple of days ago, the whole bubble burst and it became blatantly clear what was going on:



java.lang.OutOfMemoryError: GC Overhead Limit Exceeded

... coming from a Kotlin compiler class. It turned out that it wasn't kotlinc itself which was burning our CPU - it was the JVM it was running on, struggling really hard to free memory resources.

The question obviously was: how do you assign more RAM to the kotlinc process? Easier said than done.

The Setup

Like many other folks out there, we use IntelliJ IDEA to write our source code, which is then handed to Gradle to build it. Nothing too complicated, right? Well, if we take a closer look, this is what actually happens...

Here's what each step (roughly) does:

IntelliJ: Our IDE. Basically just invokes ./gradlew compile and monitors its output.
gradlew: The gradle wrapper. A shell script which downloads the desired version of gradle (specified in the build), and passes on whatever arguments you gave it to this specific gradle version.
gradle: The actual gradle executable. Here, the compileKotlin call is evaluated, and the task gets passed to the gradle daemon (which is started if it isn't already running).
gradle daemon: A background worker created by gradle. Here the actual heavy lifting is performed.
kotlinc: The actual kotlin compiler.

There are two very important things to realize here:

Gradle, IntelliJ and kotlinc are all implemented on top of the JVM.
Each of the processes runs in its own JVM instance. From the perspective of the operating system, they are different processes - with different memory pools.

The conclusion here is that no matter how much memory you assign to IntelliJ IDEA, it won't change the amount of RAM available to kotlinc upon build. Also, no matter how much memory you give to gradle, since kotlinc is its own process, it won't benefit from it.

The Fix

With all the insights gathered so far, the question still remains how to assign memory to the kotlinc process. For us, the following worked:



org.gradle.jvmargs=-Xmx8g -Dkotlin.daemon.jvm.options=-Xmx6g

We put this line into a file named gradle.properties, which is located right next to our gradlew executable (so, top-level in the project). This will assign 8GB of RAM to the gradle process, as well as another 6GB dedicated to the kotlin compiler.

You can of course play with the numbers. It's an implementation detail of the kotlin gradle plugin whether it calls kotlinc as a standalone process or embedded, so I took the defensive option and assigned more RAM to gradle than to kotlinc, but other configurations may work as well.

Now, if you change those properties in the file and save it, your kotlin compiler may still be slow and/or run out of memory. That's because your gradle daemon may still be running in the background, using your previous settings. Also, IntelliJ has an abundance of caches. The surefire way to make sure your changes are actually taking effect is to:

Reboot your machine. I mean it. This will kill the gradle daemon, plus any in-memory caches used by IntelliJ.
Start IntelliJ.
Inside IntelliJ, do not click "build" just yet. Your gradle settings may be served from a persistent cache, which is clearly not what we want. Instead, refresh your gradle project first.
Once the refresh is completed, perform a full project rebuild from IntelliJ.

The results

Our builds are now a lot faster again! We went from over 10 minutes at 100% CPU back to less than 1 minute, with much more reasonable CPU loads. Because the change was done in the build setup, our CI/CD server builds are now faster as well.

Conclusion

If your Kotlin compiles take way longer than they should, and your CPU goes crazy on all cores while kotlin is compiling, consider increasing the maximum heap size of your kotlin compiler. It may be running low on memory.

Announcing the Chronos Project v1.0.0 Open-Source Release

Martin Häusler — Sun, 31 May 2020 11:43:40 +0000

Today I'm pleased to announce that Chronos, my long-term data persistence and versioning project for the JDK, has finally been released in its 1.0.0 version as an open-source project on Github.

What is Chronos?

The Chronos Project started as my PhD project, but eventually turned into a production-ready versioned database during my work at Txture. It offers:

ChronoDB: A versioned key-value store
ChronoGraph: A TinkerPop-compliant Graph Database with versioning support
ChronoSphere: An EMF-compliant model repository

Versioning

Versioning means that every change made to the database will be retained and managed indefinitely. This allows for the ability of time travel through your data and its history, similar to an AS OF Query in SQL. Chronos employs a smart data storage layout to ensure that the data is not needlessly duplicated on your hard drive, but you can still access it as if it were.

Here's a basic example where we fetch yesterday's data with ChronoGraph:

long yesterday = System.currentTimeMills() - TimeUnit.DAYS.toMillis(1);
try(ChronoGraph txGraph = graph.tx().createThreadedTx(yesterday)){
    // txGraph will contain the database state as it was yesterday
    Vertex johnDoe = txGraph.traversal().V().has("name", "John Doe").next();
    // johnDoe will also have the same state as it did yesterday
}

Branching

In the same way as Chronos offers version control, it also allows for branching in your data, just like you've come to expect it from e.g. Git or SVN:

try(ChronoGraph txGraph = graph.tx().createThreadedTx("my-branch")){
    // txGraph now points to branch "my-branch".
    // Everything you do here will only be visible on this branch.
}

... and much more!

There's a whole bunch of features included in Chronos, such as:

full ACID transactions
secondary indexing
full & incremental backups
dateback (a rebase-like API)

All in an open-source project. Free for everyone under the aGPL v3.0 license.

Final Words

This project has been very dear to me since its inception, and it will continue to be. I'm proud to release the 1.0.0 version finally as open-source. Visit our Github Repository and give it a star if you think it's interesting :)

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 3)

Martin Häusler — Tue, 11 Feb 2020 21:55:43 +0000

This is the third part of my "DI From Scratch" series. In the previous article, we built a basic DI container. Now, we want to take it yet another step further and automatically discover the available service classes.

DI Stage 7: Auto-detecting Services

Our current state represents (a strongly simplified, yet functional) version of libraries such as Google Guice. However, if you are familiar with Spring Boot, it goes one step further. Do you think that it's annoying that we have to specify the service classes explicitly in a set? Wouldn't it be nice if there was a way to auto-detect service classes? Let's find them!

public class ClassPathScanner {

    // this code is very much simplified; it works, but do not use it in production!
    public static Set<Class<?>> getAllClassesInPackage(String packageName) throws Exception {
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        String path = packageName.replace('.', '/');
        Enumeration<URL> resources = classLoader.getResources(path);
        List<File> dirs = new ArrayList<>();
        while (resources.hasMoreElements()) {
            URL resource = resources.nextElement();
            dirs.add(new File(resource.getFile()));
        }
        Set<Class<?>> classes = new HashSet<>();
        for (File directory : dirs) {
            classes.addAll(findClasses(directory, packageName));
        }
        return classes;
    }

    private static List<Class<?>> findClasses(File directory, String packageName) throws Exception {
        List<Class<?>> classes = new ArrayList<>();
        if (!directory.exists()) {
            return classes;
        }
        File[] files = directory.listFiles();
        for (File file : files) {
            if (file.isDirectory()) {
                classes.addAll(findClasses(file, packageName + "." + file.getName()));
            } else if (file.getName().endsWith(".class")) {
                classes.add(Class.forName(packageName + '.' + file.getName().substring(0, file.getName().length() - 6)));
            }
        }
        return classes;
    }

}

Here we have to deal with the ClassLoader API. This particular API is quite old and dates back to the earliest days of Java, but it still works. We start with a packageName to scan for. Each Thread in the JVM has a contextClassLoader assigned, from which the Class objects are being loaded. Since the classloader operates on files, we need to convert the package name into a file path (replacing '.' by '/'). Then, we ask the classloader for all resources in this path, and convert them into Files one by one. In practice, there will be only one resource here: our package, represented as a directory.

From there, we recursively iterate over the file tree of our directory package, loking for files ending in .class. We convert every class file we encounter into a class name (cutting off the trailing .class) to end up with our class name. Then, we finally call Class.forName(...) on it to retrieve the class.

So we have a way to retrieve all classes in our base package. How do we use it? Let's add a static factory method to our DIContext class that produces a DIContext for a given base package:

    public static DIContext createContextForPackage(String rootPackageName) throws Exception {
        Set<Class<?>> allClassesInPackage = ClassPathScanner.getAllClassesInPackage(rootPackageName);
        Set<Class<?>> serviceClasses = new HashSet<>();
        for(Class<?> aClass : allClassesInPackage){   
            serviceClasses.add(aClass);
        }
        return new DIContext(serviceClasses);
    }

Finally, we need to make use of this new factory method in our createContext() method:

    private static DIContext createContext() throws Exception {
        String rootPackageName = Main.class.getPackage().getName();
        return DIContext.createContextForPackage(rootPackageName);
    }

We retrieve the base package name from the Main class (the class I've used to contain my main() method).

But wait! We have a problem. Our classpath scanner will detect all classes, whether they are services or not. We need to tell the algorithm which ones we want with - you guessed it - an annotation:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface Service {

}

Let's annotate our services with it:

@Service
public class ServiceAImpl implements ServiceA { ... }

@Service
public class ServiceBImpl implements ServiceB { ... }

... and filter our classes accordingly:

    public static DIContext createContextForPackage(String rootPackageName) throws Exception {
        Set<Class<?>> allClassesInPackage = ClassPathScanner.getAllClassesInPackage(rootPackageName);
        Set<Class<?>> serviceClasses = new HashSet<>();
        for(Class<?> aClass : allClassesInPackage){
            if(aClass.isAnnotationPresent(Service.class)){
                serviceClasses.add(aClass);
            }
        }
        return new DIContext(serviceClasses);
    }

We are done... are we?

And there you have it - a minimalistic, poorly optimized, yet fully functional DI container. But hold on, why do you need several megabytes worth of library code if the core is so simple? Well...

Our classpath scanner is very wonky. It certainly doesn't cover all cases, nesting depths, inner classes etc. Libraries like Guava's ClassPath do a much better job at this.
A big advantage of DI is that we can hook into the lifecycle of a service. For example, we might want to do something once the service has been created (@PostConstruct). We might want to inject dependencies via setters, not fields. We might want to use constructor injection, as we did in the beginning. We might want to wrap our services in proxies to have code executed before and after each method (e.g. @Transactional). All of those "bells and whistles" are provided e.g. by Spring.
Our wiring algorithm doesn't respect base classes (and their fields) at all.
Our getServiceInstance(...) method is very poorly optimized, as it linearly scans for the matching instance every time.
You will certainly want to have different contexts for testing and production. If you are interested in that, have a look at Spring Profiles.
We only have one way of defining services; some might require additional configuration. See Springs @Configuration and @Bean annotations for details on that.
Many other small bits and pieces.

Summary

We have created a very simple DI container which:

encapsulates the creation of a service network
creates the services and wires them together
is capable of scanning the classpath for service classes
showcases the use of reflection and annotations

We also discussed the reasoning for our choices:

First, we replaced static references by objects and constructors.
Then, we introduced interfaces to further decouple the objects.
We discovered that cyclic dependencies are a problem, so we introduced setters.
We observed that calling all setters for building the service network manually is error-prone. We resorted to reflection to automate this process.
Finally, we added classpath scanning to auto-detect service classes.

If you came this far, thanks for reading along! I hope you enjoyed the read.

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 2)

Martin Häusler — Tue, 11 Feb 2020 21:52:40 +0000

This is the second part of my "DI From Scratch" series. In the previous article, we discussed our basic example, and the problems there are with the "manual" approach. Now, we want to automate the wiring of our service network.

DI Stage 4: Automating the wiring

Let's focus on the main() method, and try to find a more automatic way of creating the service network. Certainly, forgetting to call a setter is a real threat here, and the compiler won't even be able to warn you about it. But our services are structurally different, and there is no uniform way to access them... or is there? Java Reflection to the rescue!

All we essentially want to provide our setup with is a list of our service classes. From there, the setup should construct and wire the service network. In particular, our main() method is only really interested in ServiceA, it doesn't even need ServiceB. Let's rewrite it like so:

    public static void main(String[] args) throws Exception {
        Set<Class<?>> serviceClasses = new HashSet<>();
        serviceClasses.add(ServiceAImpl.class);
        serviceClasses.add(ServiceBImpl.class);

        ServiceA serviceA = createServiceA(serviceClasses);

        // call business logic
        System.out.println(serviceA.jobA());
    }

But how can we implement the "magic" createServiceA method? Turns out, it's not that hard...

    private static ServiceA createServiceA(Set<Class<?>> serviceClasses) throws Exception{
        // step 1: create an instance of each service class
        Set<Object> serviceInstances = new HashSet<>();
        for(Class<?> serviceClass : serviceClasses){
            Constructor<?> constructor = serviceClass.getConstructor();
            constructor.setAccessible(true);
            serviceInstances.add(constructor.newInstance());
        }
        // step 2: wire them together
        for(Object serviceInstance : serviceInstances){
            for(Field field : serviceInstance.getClass().getDeclaredFields()){
                Class<?> fieldType = field.getType();
                field.setAccessible(true);
                // find a suitable matching service instance
                for(Object matchPartner : serviceInstances){
                    if(fieldType.isInstance(matchPartner)){
                        field.set(serviceInstance, matchPartner);
                    }
                }
            }
        }
        // step 3: from all our service instances, find ServiceA
        for(Object serviceInstance : serviceInstances){
            if(serviceInstance instanceof ServiceA){
                return (ServiceA)serviceInstance;
            }
        }
        // we didn't find the requested service instance
        return null;
    }

Let's break it down. In Step 1 we iterate over our classes, and for each class, we attempt to get the default constructor (i.e. the constructor with no arguments). Since neither ServiceAImpl nor ServiceBImpl specifies any constructor (we deleted them when introducing the getters/setters), the Java compiler provides a public default constructor - so that will work fine. Then, we make this constructor accessible. That's just defensive programming to make sure that private constructors will work too. Finally, we call newInstance() on the constructor to create the instance of the class, and add it to our set of instances.

In Step 2 we want to wire together our individual service instances. To do so, we look at each service object one by one. We retrieve it's Java class via getClass(), and ask that class for all of its declaredFields (declared means that private fields will be returned too). Just like for the constructor, we make sure that the field is accessible, and then we check the Type of the field. This will provide us with the service class we need to put into the field. All that's left to do is to find a suitable matchParter, an object which is of the type specified by the field. Once we find one, we call field.set(...) and assign the match partner to the field. Note that the first parameter of the field.set(...) method is the object which will have its field value changed.

In Step 3, the network is already complete; all that's left to do is to find the instance of ServiceA. We can simply scan through or instances and check if we found the right one by using instanceof ServiceA.

This might be a little daunting, so maybe try to read this once more. Also, you might want to brush up on your knowledge of Java reflection basics if any of that seems weird to you.

So what did we gain?

Our services are wired together automatically.
We can no longer forget to call a setter (in fact, we don't need them anymore).
Our application will fail on startup if the wiring fails, not during the business logic.

The primary pain that we need to treat next is the fact that we do not want to repeat this whole procedure every time we want to get a hold of a service; we want to have the ability to access every service in the network, not just one.

DI Stage 5: Encapsulating the Context

The object which is responsible for holding the service network is called the Dependency Injection Container, or (in Spring terms) the Application Context. I'm going to use the "context" terminology, but the terms are really synonyms. The primary job of the context is to provide a getServiceInstance(...) method which accepts a service class as parameter, and returns the (finished and wired) service instance. So here we go:

public class DIContext {

    private final Set<Object> serviceInstances = new HashSet<>();

    public DIContext(Collection<Class<?>> serviceClasses) throws Exception {
        // create an instance of each service class
        for(Class<?> serviceClass : serviceClasses){
            Constructor<?> constructor = serviceClass.getConstructor();
            constructor.setAccessible(true);
            Object serviceInstance = constructor.newInstance();
            this.serviceInstances.add(serviceInstance);
        }
        // wire them together
        for(Object serviceInstance : this.serviceInstances){
            for(Field field : serviceInstance.getClass().getDeclaredFields()){
                Class<?> fieldType = field.getType();
                field.setAccessible(true);
                // find a suitable matching service instance
                for(Object matchPartner : this.serviceInstances){
                    if(fieldType.isInstance(matchPartner)){
                        field.set(serviceInstance, matchPartner);
                    }
                }
            }
        }
    }

    @SuppressWarnings("unchecked")
    public <T> T getServiceInstance(Class<T> serviceClass){
        for(Object serviceInstance : this.serviceInstances){
            if(serviceClass.isInstance(serviceInstance)){
                return (T)serviceInstance;
            }
        }
        return null;
    }

}

As you can see, the code didn't change much from the previous step, except that we now have an object to encapsulate the context (DIContext). Internally, it manages a set of serviceInstances which is created just like before from a collection of service classes. The Step 3 from above has moved into its own getServiceInstance method, which accepts the class to retrieve as a parameter. Since we cannot use instanceof anymore (it requires a hard-coded class, not a dynamic variable value), we have to fall back to serviceClass.isInstance(...) to do the same thing.

We can use this class in our new main():

    public static void main(String[] args) throws Exception {
        DIContext context = createContext();
        doBusinessLogic(context);
    }

    private static DIContext createContext() throws Exception {
        Set<Class<?>> serviceClasses = new HashSet<>();
        serviceClasses.add(ServiceAImpl.class);
        serviceClasses.add(ServiceBImpl.class);
        return new DIContext(serviceClasses);
    }

    private static void doBusinessLogic(DIContext context){
        ServiceA serviceA = context.getServiceInstance(ServiceA.class);
        ServiceB serviceB = context.getServiceInstance(ServiceB.class);
        System.out.println(serviceA.jobA());
        System.out.println(serviceB.jobB());
    }

As you can see, we can now easily pull out complete service instances from the context by calling getServiceInstance as often as we need to, with different input classes. Also note that the services itself can access each other simply by declaring a field of the proper type - they don't even have to know about the DIContext object.

There are still some problems though. For example, what if we want to have a field in our services which does not refer to another service (say, an int field)? We need a way to tell our algorithm which fields we want it to set - and which ones to leave alone.

DI Stage 6: Annotating fields

So how can we tell our algorithm which fields it needs to assign? We could introduce some fancy naming scheme and parse the field.getName(), but that's a very error prone solution. Instead, we will use an Annotation:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Inject {

}

@Target tells the compiler on which elements we can use this annotation - we want it to be applicable on fields. With @Retention we instruct the compiler to keep this annotation until runtime, and not to discard it during compilation.

Let's annotate our fields:

public class ServiceAImpl implements ServiceA {

    @Inject
    private ServiceB serviceB;

    // rest is the same as before

}

public class ServiceBImpl implements ServiceB {

    @Inject
    private ServiceA serviceA;

    // rest is the same as before

}

An annotation in and on itself does nothing. We need to actively read the annotation. So let's do it in the constructor of our DIContext:

       // wire them together
       for(Object serviceInstance : this.serviceInstances){
           for(Field field : serviceInstance.getClass().getDeclaredFields()){
               // check that the field is annotated
               if(!field.isAnnotationPresent(Inject.class)){
                   // this field is none of our business
                   continue;
               }
               // rest is the same as before

Run the main() method again; it should work just like before. However, now you are free to add more fields to your services, and the wiring algorithm won't break.

Closing words

So far, we have created a DI container which is very basic but functional. It relies on us providing it with the collection of service classes. In the next part, we will discuss how we can actually discover our service classes.

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 1)

Martin Häusler — Tue, 11 Feb 2020 21:48:15 +0000

Dependency Injection (DI) can be a very difficult topic to grasp, since there seems to be a lot of "magic" going on. Usually it involves a bunch of annotations scattered all over the place, with objects appearing seemingly out of nowhere. I certainly know that it took me a long while to really understand the concept. If you ever find it hard to understand what Spring and Java EE are doing behind the curtains (and why!), read on!

In this tutorial, we are going to build a very bare-bones, yet fully functional dependency injection container from scratch in Java. Here are some rules to keep things manageable:

Absolutely no libraries allowed, except for the JDK itself.
No pre-prepared code dumps. We will go through everything and reason about it.
No bells & whistles, just the basics.
Performance doesn't matter, no optimizations.
Executable main() methods at every step.

In contrast to other articles, I will not explain up-front what DI is or why it is useful. Instead, we'll start with a simple example and let it "evolve".

DI Stage 0: Basic example

We start with a very basic example. We want two classes, let's call them ServiceA and ServiceB, and ServiceA needs ServiceB to do its work. Well, you might implement it with static methods and be done with it! So here goes:

public class ServiceA {

    public static String jobA(){
        return "jobA(" + ServiceB.jobB() + ")";
    }

}

public class ServiceB {

    public static String jobB(){
        return "jobB()";
    }

}

public class Main {

    public static void main(String[] args) {
        System.out.println(ServiceA.jobA());
    }

}

If we run this code, it will print:

jobA(jobB())

Cool! So why even bother with more than this? Well...

The code is very poor in terms of OO principles. ServiceA and ServiceB should, at the very least, be objects.
The code is tightly coupled and very hard to test in isolation.
We have absolutely no chance of swapping neither ServiceA nor ServiceB with a different implementation. Imagine one of them is doing credit card billings; you don't want that to actually happen in your test suite.

DI Stage 1: Getting rid of static references

The main problem we identified in Stage 0 is that there are only static methods, resulting in extremely tight coupling. We would like our services to be objects talking to each other, such that we may replace them as needed. Now the question arises: how does ServiceA know which ServiceB to talk to? The most basic idea is to simply give our instance of ServiceB to the constructor of ServiceA:

public class ServiceA {

    private ServiceB serviceB;

    public ServiceA(ServiceB serviceB){
        this.serviceB = serviceB;
    }

    public String jobA(){
        return "jobA(" + this.serviceB.jobB() + ")";
    }

}

Service B didn't change much:

public class ServiceB {

    public String jobB() {
        return "jobB()";
    }

}

... and Main now needs to assemble the objects before it can call the jobA method:

   public static void main(String[] args) {
        ServiceB serviceB = new ServiceB();
        ServiceA serviceA = new ServiceA(serviceB);

        System.out.println(serviceA.jobA());
    }

Nothing fancy here, right? We certainly improved some things:

We can now replace the implementation of ServiceB which is used by ServiceA by providing another object, potentially even of another subclass.
We can test both services in isolation with a proper test mock for ServiceA.

So all cool? Not quite:

It's hard to create mocks, as we require a class.
It would be much nicer if we required interfaces instead. Also, this would further reduce the coupling.

DI Stage 2: Using interfaces

So let's use one interface for each of our services. They're about as simple as it gets (I renamed the actual classes to ServiceAImpl and ServiceBImpl, respectively):

public interface ServiceA {

    public String jobA();

}

public interface ServiceB {

    public String jobB();

}

Now, in ServiceAImpl, we can actually use the interface:

public class ServiceAImpl implements ServiceA {

    private final ServiceB serviceB;

    public ServiceAImpl(ServiceB serviceB){
        this.serviceB = serviceB;
    }

    // jobA() is the same as before
}

That also makes our main() method a bit nicer:

  public static void main(String[] args) {
        ServiceB serviceB = new ServiceBImpl();
        ServiceA serviceA = new ServiceAImpl(serviceB);
        System.out.println(serviceA.jobA());
    }

This would be perfectly acceptable for simple use cases from an OO perspective. If you can get away with this, by all means, stop here. However, as your project gets bigger...

it will become more and more complex to create the network of services inside your main() method.
you will encounter cycles in your service dependencies which cannot be resolved using constructors as shown in our example.

DI Stage 3: Breaking the cycle with setters

Let's assume that not only ServiceA needs ServiceB, but also the other way around - we have a cycle. Of course, we may still declare a parameter in the constructor of the *Impl classes, like so:


 // constructor examples
 public ServiceAImpl(ServiceB serviceB) { ... }
 public ServiceBImpl(ServiceA serviceA) { ... }

... but that will do us no good: we will be unable to create an actual instance of either of the two classes. To create an instance of ServiceAImpl, we would first require an instance of ServiceB, and to create an instance of ServiceBImpl we first require an instance of ServiceA. We're deadlocked.

So what do we do instead? Well, since we are dealing with cyclic dependencies, we need the ability to first create the services and then wire them together. We do this with setters:

public class ServiceAImpl implements ServiceA {

    private ServiceB serviceB;

    // no constructor anymore here!

    @Override // <- added getter to ServiceA interface
    public ServiceB getServiceB() { return serviceB; }

    @Override // <- added setter to ServiceA interface
    public void setServiceB(final ServiceB serviceB) { this.serviceB = serviceB; }

    // jobA() same as before
}

Our main() method looks like this:

    public static void main(String[] args) {
        // create instances
        ServiceB serviceB = new ServiceBImpl();
        ServiceA serviceA = new ServiceAImpl();

        // wire them together
        serviceA.setServiceB(serviceB);
        serviceB.setServiceA(serviceA);

        // call business logic
        System.out.println(serviceA.jobA());
    }

Can you see the pattern? First create the objects, then connect them to form the service graph (i.e. service network).

So why not stop here?

Doing the wiring part manually is error prone. You might forget to call a setter and then it's NullPointerException galore.
You might accidentally use a service instance that is still under construction, so it would be beneficial to encapsulate the network construction somehow.

Up next

In the next blog post, we will discuss how we can automate the wiring we are doing manually right now.

A Critical Look on Relational Databases and SQL

Martin Häusler — Fri, 25 Oct 2019 14:38:59 +0000

Relational Databases took the world by storm around 1990. Ever since, they've become a must-have in the toolbox of professional developers - nearly independent of the language you might be using, you will need SQL at some point. However, since around 2000, the NoSQL movement has regained some attention as well. In this article, I will provide some of my personal thoughts on databases, focusing on relational databases and the SQL language. As I've been working with both relational and non-relational systems, I hope that this article can provide some insights and interesting perspectives.

The Advantages of Relational Databases

The fact that Relational Database Management Systems (RDBMSs) have such a high market share has several reasons.

Relational Algebra

First and foremost, an RDBMS, regardless of its implementation, is always based on Relational Algebra which formally defines its operations and data structures. This is very convenient: all the formal foundations have already been taken care of, the system is known to be sound and complete, "all" database vendors have to do is go ahead and implement them. This is a big difference to NoSQL databases. For example, it is not obvious a priori whether or not you can formulate a query for every possible information demand in a Graph Query language such as Gremlin or Cipher. The authors of those languages had to think those theoretical questions through before even writing the first line of code.

A second reason why RDMBSs are so successful nowadays is the enormous amounts of money which were put into their development. There was a lot of hype surrounding RDBMSs in research, therefore investors were willing to put a lot of money into it. This process spiraled upwards to the point where "database" was used as a synonym to "RDBMS". In a way, RDBMSs became popular not necessarily by means of features or well-foundedness, but at least partially also due to a massive hype cycle which provided the financial resources to make all the research and optimizations happen.

Declarativeness and Optimizations

Another big advantage of RDBMSs is that they can operate in a highly optimized fashion, as SQL is declarative in nature. Operators can be shifted around in the operator tree, indices can be applied where appropriate, there's just a lot of things happening behind the scene before your query even hits the actual B-Trees. Again, there has been a tremendous amount of research efforts going into this topic, and people are still coming up with new ideas on how to make queries even faster. Try using the EXPLAIN statement in one of your more complex queries to see what I mean.

Fixed Schema for optimal Resource Usage

In most SQL databases out there, the schema has to be fixed up-front. And, aside from special LOB (Large OBject) types such as CLOB and TEXT, they pretty much all have a fixed length in bits. Even for a VARCHAR you have to specify a maximum length. Therefore, for any given row in your database, the system knows in advance how much memory the row will need - before it is actually loaded into RAM. This is a huge deal and helps immensely when implementing database systems in languages with manual memory management, such as C.

SQL is a language meant for humans

To many people, in particular in the more business-oriented offices, SQL is much easier to approach than a general-purpose programming language. SQL, as the primus inter pares of Domain Specific Languages, has been designed to be close to natural english. From a marketing perspective, this is a great boon: "Query your data without needing programmers! It's even standardized so you can use any database you want with it!" And I think it's safe to say that it worked, people fell for it.

Data Transformation

RDBMSs excel at providing your data exactly in the format you need. They can slice and dice the database content in just about any way you can possibly imagine. This is an impressive and often overlooked capability.

Constraints

Relational databases excel at specifying constraints. Whether it is about value constraints (minimum & maximum of a numeric value, null handling...), foreign key constraints or access rights, RDBMSs have you covered in all directions. This ensures that your data remains in a correct and consistent state.

The Problems and Disadvantages of RDBMSs

As the title implies, I firmly believe that there are a number of issues with RDBMSs out there.

SQL is a language for humans

Yes, you read that right - this caption appeared as an advantage. But SQL being a human-readable language also implies a lot of problems:

It is inherently non-systematic. The number of keywords needed is enormous compared to general-purpose programming languages. In my opinion, this makes it harder to learn, rather than easier, as advertised.
It suffers from poor composability. Unifying two arbitrary queries can be challenging. SQL engines often impose constraints which affect the whole query, rather than individual parts. This becomes particularly evident with GROUP BY and ORDER BY clauses.
It is a nightmare to parse. Read any mailing list on the development of relational databases you want, you will see the people responsible for the parser complain about it, and rightfully so.

In total, the caption of this section could also be written as "SQL is not a language for machines". Let's not forget that the vast majority of queries out there are not written manually - they are being generated. Whether they originate from Object-Relational Mapping systems, string templates or other mechanisms, most queries are not written by a human sitting in front of a text field. SQL as a language therefore focuses on the wrong audience, causing many issues in the process - the generators have a hard time producing it, and the SQL engines have a hard time parsing it. So, who benefits from this situation? Did we, as software development community, take a wrong turn somewhere?

The SQL Standard is a bad joke

... and everyone knows it. Even calling it a "Standard" is an insult to real standards. First of all, using only the elements found in the standard, it's already challenging to write a simple SELECT-FROM-WHERE query. Often times, you need vendor-specific extensions to get your work done (the SQL standard often speaks of "language opportunities"). For instance, you might think that there is a standardized way to do a paginated LIMIT / OFFSET query in SQL. Well, too bad, you're out of luck. While pretty much any RDBMS out there can do it, the SQL syntax is different for every one of them. Data types also often don't do the same thing across different vendors. And don't even get me started on stored procedures. There is a reason why companies look for experience with certain RDBMS products in their job announcements, rather than looking for experience with RDBMSs in general. If you see such an advert, you can be sure that this company is already neck deep in the vendor lock-in. In my experience, the idea of writing an application which can make effective use of more than one type of RDBMS as its backend is an illusion.

`UPSERT`, the unknown being

When dealing with even slightly more sophisticated applications, it is often the case that you load some data from the database, make some modifications to it, and then save it back. RDBMSs have a really bad habit here: they force you as the developer to distinguish between INSERT and UPDATE, all the time. Tracking this object state correctly throughout an entire application, only to decide at the final moment which of the two keywords to use, is a real pain. In the end, you put in a lot of effort only to make the SQL constraint checker happy. The UPSERT keyword has not yet made it into the big RDBMSs. There are ways to do it, in the form of INSERT ON CONFLICT (PostGreSQL) or ON DUPLICATE KEY UPDATE (MySQL), but overall the situation is still pretty bad. To add insult to injury, Object-Relational Mappers cannot rely on those features, as they have to talk to multiple different RDBMS backends. In comparison, do you know the primary way of inserting data into Neo4j? It's the MERGE keyword, which is just another way of saying UPSERT. Neo4j is not without issues, but they at least got that right from the get-go. There are certainly use cases where it actually makes a difference if you create a new object or update an existing one, but they're a small minority in my experience. In 99% of all cases, I just have an object I want to have saved to query it later.

Object-Relational Mappers are a Pain

I don't mean any offense against O/R mapping frameworks or their authors. In fact, I'm thankful they exist, because without them, the pain would be even bigger than it already is. Yes, RDBMSs are powerful and highly optimized, and thousands of hours went into their development to ensure they are correct and fast. We would be mad not to make use of them, right? Well, too bad the interface we have to use to talk to them (SQL) is horrible. I intend to write another article about this topic where I'll go into greater detail.

The Future of RDMBSs

RDBMSs are here to stay. Even if all modern applications were developed with NoSQL databases, there would still be enough legacy applications to keep all of us busy for centuries to come. However, change is happening. JSON and XML support is creeping into RDBMS engines, giving us at least some more flexibility. But what about SQL as a language? I personally think that Datalog would be a very nice, clean and effective replacement which is easily generated and easily parsed. I thought of it as a toy when I first came into contact with it in university, but the more time I spend with databases, the more I think that this could very well be worth a shot. Unfortunately, I've never seen any actual support for it in the big database systems.

Final thoughts

The relational model has many use cases and advantages, and has come a long way. RDMBSs are stable, mature and have a proven track record. However, SQL as a language is becoming a limiting factor. I think using an RDBMS as the "default" choice for a database is a good habit, but I would advise everyone to at least try a NoSQL solution as well if the opportunity presents itself.

Talk & Event Announcement: Kotlin/Everywhere

Martin Häusler — Mon, 16 Sep 2019 11:22:22 +0000

Hey all,

I just wanted to let you know that I will be a speaker at the Kotlin/Everywhere event.

What:
The event is organized by GDG Innsbruck (Google Developer Group) and hosted by Txture.

When:
October 10th 2019, 19:00

Where:
At the Txture headquarters (Grabenweg 68, SOHO 2.0), in the beautiful city of Innsbruck, Austria!

Audience:
The event is intended for all developers who are interested in the Kotlin language.

Speakers:
We will have at least two talks, including my own and one by a member of JetBrains, the developers of the Kotlin language!

Attendance Fee:
None! :)

How do I attend?
Just let us know that you'll be attending by subscribing to the event at the Meetup Page.

My Talk:
I will be talking about our journey with Kotlin at Txture. We use it extensively in production, so we have quite the treasure trove of tales to tell ;)

Hope to see some of you there!

Immutable Data Structures

Martin Häusler — Sun, 07 Jul 2019 15:37:41 +0000

Functional programming is currently on the rise due to its ability to prevent a lot of errors right out of the gate. The two corner stones of the functional paradigm are pure functions and immutable data structures. Today we will look at immutable collections - and why there's more to them than you might think.

Levels of Immutability

There are actually several different "levels" of immutability. It's difficult to keep them separated because the terminology is often used in the wrong way. I try to be as clear and consistent as possible.

Mutable Collections

Mutable collections are the ones we most commonly see in languages like Java, C# and JavaScript. They are usually built into the standard library. Their primary characteristic is that we can create an instance of those collections, and then afterwards modify it as we please.

List<String> list = new ArrayList<>();
list.add("Hello");
list.contains("Hello"); // -> true

Pros

Very high flexibility. There's no use case you cannot cover with mutable collections.
Fast insertion and modification.

Cons

Very often when you receive a collection as a method argument, you have to make a defensive copy. A full copy is O(n) in memory and time complexity.
If you do not create defensive copies, you might end up using a collection and sharing it with another piece of code. If either party makes a change, that change is visible to the other piece of code, potentially breaking it. Such bugs can be extremely subtle and hard to debug.
Mutable collections do not lend themselves well to concurrent programs. Concurrent modifications on shared mutable collections require proper external locking.

Use them for...

Local variables which are not shared.
Locally aggregating computation results.
Fields which get in touch with any reflection-based framework (most prominently, Object-Relational Mappers such as Hibernate).
Collections which are subject to frequent changes by a single thread.
As a fallback when all other types of collections don't work.

Unmodifiable Collections

Unmodifiable collections are merely views on underlying mutable collections. You cannot modify the view, but any modification on the underlying mutable collection will be visible in the view.

List<String> original = new ArrayList<>();
original.add("Test");
List<String> view = Collections.unmodifiableList(original);

// changing the view is forbidden
view.add("x"); // -> Runtime exception: the view is unmodifiable!

// reading from the view is fine
view.contains("Test"); // -> true

// changes in the original collection...
original.add("Foo");
// ... are visible in the view
view.contains("Foo"); // -> true

Pros

Useful if you have one object who "owns" the list, and merely lets others read from it.
If you create an unmodifiable view, and throw away all references to the original, you end up with a truly immutable collection (see below).

Cons

The interface of an unmodifiable view is the same as the original collection. Which means: view.add(...) will be valid according to the compiler. It will appear in your code completion. If you don't know that you are dealing with an unmodifiable view, you're in for a lot of runtime exceptions. Unmodifiable views can fool the user of your API into believing that they are working with mutable collections. Some languages, such as Kotlin, actually differentiate between a List and a MutableList interface for this very reason.
There is no efficient way to create a "changed copy" of an unmodifiable view which contains an additional change. The only way to do that is to copy the entire view content into a new (mutable) collection, and apply the change there. That's O(n) time and space complexity.
Massive Pitfall: If you receive a collection as an argument, and you are calling e.g. Collections.unmodifiableList on it, you are not safe from external modifications! Remember: you only create a view. The original collection you received may still be changed from the outside. This is therefore not a replacement for defensive copies!

Use them for...

Sharing your object state with the world in a read-only fashion. Don't forget to document in your API that the collections you return are unmodifiable views.

Persistent Collections

This category of (im)mutability is extremely important, but often overlooked or even mislabeled. Persistent collections have nothing to do with "persistence" as in "storing to disk". Persistent collections are completely immutable structures, except that they offer smart copy-transforms. The primary principle of a persistent data structure is:

Sounds familiar? That's exactly what ImmutableJS does. The name of this library itself is extremely misleading, because what it actually provides are persistent collections, which are much more useful than strictly immutable ones. For Java, there's for example the (aptly named) PCollections library:

PSet<String> original = HashTreePSet.empty(); 
PSet<String> mod1 = original.plus("Hello");
PSet<String> mod2 = mod1.plus("World");

// the original does not know what happened in mod1 and mod2
original.contains("Hello"); // -> false
original.contains("World"); // -> false
original.size(); // -> 0

// mod1 knows the original and itself, but does not know about mod2
mod1.contains("Hello"); // -> true
mod1.contains("World"); // -> false
mod1.size(); // -> 1

// mod2 knows original and mod1, as well as its own modification
mod2.contains("Hello"); // -> true
mod2.contains("World"); // -> true
mod2.size(); // -> 2

The easiest way to think about persistent collections is: make a change, get a copy that includes the change (copy being the operative word). You may think that this is extremely inefficient, as a copy would entail an O(n) cost in runtime and space complexity. While this naive "copy and change" approach would satisfy the interface, that is acutally not what happens in persistent collections. The trick is that all existing data doesn't change; therefore we are free to reuse it as often as we like. In the example above, mod2 will internally reuse the contents of mod1. There's no actual copying going on, but you will never realize it, because you cannot change mod1 anymore after creating mod2.

Pros

True, reliable immutability. No need for defensive copies, ever.
Still relatively easy to work with.
Safe under concurrent read/write, write/write concurrency can sometimes require additional synchronization.
All operations on persistent collections are pure functions. No side effects, no surprises.

Cons

Nesting of persistent collections can be tricky to pull off (hello, Redux!).
If you apply a large amount of modifications to a single persistent data structure, performance will suffer. It might be more efficient to create a mutable copy, apply all your modifications, and then create a persistent version of it (resulting in O(n) + O(n) space and time complexity).
If you are unaware that you are working with a persistent collection, you might forget to assign the return value of list.plus("test") to a variable, and then wonder why "test" is not in your list.
Some collection frameworks, such as PCollections, implement the regular (mutable) collection interfaces (e.g. PSet<T> extends Set<T>) for compatibility reasons. Thus, the persistent collections inherit the mutating methods, such as void add(E element) which throw runtime exceptions when called.

Use them for...

Representing shared state under read-write access.
Scenarios with a single producer and multiple concurrent consumers.
Scenarios where you have to compute multiple alternative solutions which are based on an initial solution.

(Truly) Immutable Collections

Truly immutable collections only offer read access through their API, and have to be filled at creation time. Afterwards, their contents are sealed and set in stone forever. You can think of truly immutable collections as "persistent collections without copy-on-write helpers".

To the best of my knowledge, there is no truly immutable collection interface in Java. In Kotlin, you can do this:

// "listOf" creates a truly immutable list in Kotlin.
val list = listOf("Test")
list.contains("Test") // -> true 
list.add("foo") // -> compile error: interface "List" has no method "add"

Pros

Easy to implement efficiently (all data has to be known in advance)
Iteration / analysis may be faster than with a persistent collection

Cons

There is no use case for truly immutable collections which would not work with persistent collections.
Very limited flexibility.

Use them for...

Defining constants.

Closing Thoughts

I hope you enjoyed this tour of (im)mutable collections. It is sometimes hard to properly communicate about this topic because the terminology is often misused. I hope that this blog post will help you in the future to distinguish the types of (im)mutable collection, their pros and cons, and when to use them.

Why I stopped using Coroutines in Kotlin

Martin Häusler — Sat, 22 Jun 2019 11:18:18 +0000

It's not a secret that I'm pretty enthusiastic about Kotlin as a programming language, despite a few shortcomings and strange design choices. I got the chance to work on a medium-sized project using Kotlin, Kotlin Coroutines and the coroutine-driven server framework KTOR. Those technologies offer a lot of merit, however I've found them difficult to work with in comparison to e.g. plain old Spring Boot.

Debugging

Consider the following piece of code:

suspend fun retrieveData(): SomeData {
    val request = createRequest()
    val response = remoteCall(request)
    return postProcess(response)
}

private suspend fun remoteCall(request: Request): Response {
   // do suspending REST call
}

Let us assume we want to debug the retrieveData function. We place a breakpoint in the first line. Then we start the debugger (in IntelliJ in my case), it stops at the breakpoint. Nice. Now we perform a Step Over (skipping the call createRequest). That works too. However, if we Step Over again, the program will just run. It will NOT stop after remoteCall.

Why is that? The JVM debugger is attached to a Thread object. For all means and purposes, this is a very reasonable choice. However, when Coroutines enter the mix, one thread no longer does one thing. Look closely: remoteCall(request) calls a suspending function - we don't see it in the syntax when we call it though. So what happens? We tell the debugger to "step over" the method call. The debugger runs the code for the remote call and waits.

This is where things get difficult: the current thread (to which our debugger is bound) is only an executor for our coroutine. What will happen when we call a suspending function is that at some point, the suspending function will yield. This means that a different Thread will continue the execution of our method. We effectively tricked the debugger.

The only workaround I've found is to place a breakpoint on the line I want to go to, rather than using Step Over. Needless to say, this is a major pain. And apparently it's not just me.

Furthermore, in general debugging it is very hard to pin down what a single coroutine is currently doing, as it jumps between threads. Sure, coroutines are named and you can enable the logging to print not only the thread but the coroutine name as well, but the mental effort required to debug coroutine-based code is a lot higher in my experience than thread-based code.

Binding data to the REST call

When working on a microservice, a common pattern is to receive a REST call with some form of authentication, and pass the same authentication along for all internal calls to other microservices. In the most trivial case, we at least want to keep the username of the caller.

However, what if those calls to other microservices are nested 10 levels deep in our call stacks? Surely we don't want to pass along an authentication object as a parameter in each and every function. We require some form of "context" which is implicitly present.

In traditional thread-based frameworks such as Spring, the solution to this problem is to use a ThreadLocal object. This allows us to bind any kind of data to the current thread. As long as one thread corresponds to the processing of one REST call (which you should always aim for), this is exactly what we need. A good example for this pattern is Spring's SecurityContextHolder.

With coroutines, the situation is different. A ThreadLocal will no longer do the trick, because your workload will jump from one thread to another; it is no longer the case that one thread will accompany a request during the entirety of its lifetime. In kotlin coroutines, there's the CoroutineContext. In essence, it is nothing more than a HashMap which is carried alongside the coroutine (no matter on which thread it mounts). It has a horribly over-engineered API and is cumbersome to use, but that is not the main issue here.

The real problem is that coroutines do not inherit the context automatically.

For example:

suspend fun sum(): Int {
    val jobs = mutableListOf<Deferred<Int>>()
    for(child in children){
        jobs += async {  // we lose our context here!
            child.evaluate() 
        }
    }
    return jobs.awaitAll().sum()
}

Whenever you call a coroutine builder, such as async, runBlocking or launch, you will - by default - lose your current coroutine context. You can avoid it by passing the context into the builder method explicitly, but god forbid you ever forget to do that (the compiler won't care!).

A child coroutine could start off with an empty context, and if a request for a context element comes in and nothing is found, the parent coroutine context could be asked for the element. However, that does not happen in Kotlin, the programmer is required to do that manually, every single time.

If you are interested in the details of this issue, I recommend having a look at this blog post.

`synchronized` will no longer do what you think it does

When working with locks or synchronized blocks in Java, the semantic I am thinking about is usually along the lines of "nobody else can enter while I'm in this block". The nobody else part of course implies that there is an "identity" of some sort, which in this case is the Thread. That should raise a big red warning sign in your head by now.

Let's consider the following example:

val lock = ReentrantLock()

suspend fun doWithLock(){
   lock.withLock {
       callSuspendingFunction()
   }
}

This is dangerous. Even if callSuspendingFunction() does nothing harmful, the code will not behave as you might think it does:

Enter the lock
Call the suspending function
The coroutine yields, the current thread still holds the lock
Another thread picks up our coroutine
We are the same coroutine, but we do not own the lock anymore!

The number of potentially conflicting, deadlocking or otherwise unsafe scenarios is staggering. You might argue that we "just" need to engineer our code to handle that. I would agree, however we are talking about the JVM. There is a vast ecosystem of libraries out there. And they are not prepared to handle it.

The upshot here is: the moment you start using coroutines, you forfeit the possibility to use a lot of Java libraries, simply because they expect a thread-based environment.

Throughput vs. Horizontal Scaling

A big advantage of coroutines for the server-side is that a single thread can handle a lot more requests; while one request waits for a database response, the same thread can happily serve another request. In particular for I/O bound tasks, this can increase the throughput.

However, as this blog post has hopefully demonstrated to you, there is a non-zero cost associated with using coroutines, on many levels.

The question which arises is: is the benefit worth that cost? And in my opinion the answer is no. In cloud- and microservice-environments, there should always be some scaling mechanism, whether it is Google AppEngine, AWS Beanstalk, or some form of Kubernetes. Those technologies will simply spawn new instances of your microservice on demand if the current load increases. Therefore, the throughput one individual instance can handle is much less important, considering all the hoops we would have to jump through for using coroutines. This reduces the value we get from using coroutines.

Coroutines have their place

Coroutines are still useful. When developing client-side UIs where there is only one UI thread, coroutines can help to improve your code structure while being compliant with the requirements of your UI framework. I've heard that this works pretty well on Android. Coroutines are an interesting topic, however for the server-side I feel that we are not quite there yet. The JVM developers are currently working on Fibers, which are in essence also coroutines, but they have the goal to play nicely with the JVM infrastructure. It will be interesting to see how this effort develops, and how Jetbrains will react to it with respect to their own coroutine solution. In the best possible scenario, Kotlin coroutines will just "map" to Fibers in the future, and debuggers will be smart enough to handle them properly.

DEV Community: Martin Häusler

LSM4K 1.0.0-Alpha released

Error as Value vs. Exceptions

What is "Error as Value"?

Advantages of Error as Value

My thoughts on Error as Value

Errors should not happen.

Real recoverable errors are a myth.

The unfailable function fallacy

Don't panic

The merit of try-catch

The true drawbacks of Exceptions

Error as Value in Kotlin

Conclusion

So I tried Rust for the first time.

Installation

Advent of Code

include_str!

Give me a lifetime

main

let mut

Semicolon!! ;

for

Iterators

Lambdas

Type Inference... sometimes.

Unwrap all the things!

print it!

The Pros and Cons

Is your Kotlin compiler slow? Here's a potential fix

The Setup

The Fix

The results

Conclusion

Announcing the Chronos Project v1.0.0 Open-Source Release

What is Chronos?

Versioning

Branching

... and much more!

Final Words

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 3)

DI Stage 7: Auto-detecting Services

We are done... are we?

Summary

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 2)

DI Stage 4: Automating the wiring

DI Stage 5: Encapsulating the Context

DI Stage 6: Annotating fields

Closing words

Understanding Dependency Injection by writing a DI Container - from scratch! (Part 1)

DI Stage 0: Basic example

DI Stage 1: Getting rid of static references

DI Stage 2: Using interfaces

DI Stage 3: Breaking the cycle with setters

Up next

A Critical Look on Relational Databases and SQL

The Advantages of Relational Databases

Relational Algebra

Declarativeness and Optimizations

Fixed Schema for optimal Resource Usage

SQL is a language meant for humans

Data Transformation

Constraints

The Problems and Disadvantages of RDBMSs

SQL is a language for humans

The SQL Standard is a bad joke

UPSERT, the unknown being

Object-Relational Mappers are a Pain

The Future of RDMBSs

Final thoughts

Talk & Event Announcement: Kotlin/Everywhere

Immutable Data Structures

Levels of Immutability

Mutable Collections

Pros

Cons

Use them for...

Unmodifiable Collections

Pros

Cons

Semicolon!! `;`

`UPSERT`, the unknown being

`synchronized` will no longer do what you think it does