DEV Community: Igor Artamonov

Performance of Lock in Coroutines

Igor Artamonov — Tue, 18 Mar 2025 19:03:27 +0000

Recently, while having a work conversation, I heard a thesis from a colleague that “In Kotlin coroutines you have to use Kotlin’s Mutex to synchronize modification of an object from different threads,” which seems to be a common misconception, and here I want to explain why it’s not always right. With the numbers.

So, basically, I see that it’s common to think: “if you use coroutines and inside those coroutines you have a shared object which you modify from multiple coroutines, then to synchronize the access to it you have to use Kotlin’s Mutex”

The docs imply that, and also they say that you should never have blocking code in your coroutines. But this “blocking code” is a very confusing term. It’s supposed to mean that this code wastes CPU on doing nothing (i.e. on waiting) for a time long enough to run something else during this time.

So a lock is definitely a blocking code, right? Sometimes. But it doesn’t automatically mean you should avoid using it. Because the key point is “long enough.” Which means that the waiting time is longer than switching the context to another operation like copying blocks of stack/memory, which in turn also assumes polluting the CPU caches, possibly switching to another core, etc.

All of those are comparably slow operations (from 1ns to access data in the current CPU L1 cache, to hundreds and thousands of nanoseconds to copy the memory to another place). Note, though, that it’s still a hundred times faster than an IO operation.

So the rule of thumb is to use coroutines for IO operations, and standard primitives for non-IO operations. Unless you have some very specific case. And a shared object access is nothing like that, because system locks are very fast and memory efficient.

And, btw, even to switch a coroutine context you would probably rely on a lock or two somewhere inside the coroutines library, so a JVM lock is unavoidable most likely.

Okay, enough theory and assumptions. Let’s benchmark it and see if it makes sense.

I have a very simple idea of a code that:

fully utilizes CPU while while executing many short and independent tasks
no IO
have an object to synchronize the state between threads

The “task” is calculating a SHA3 hash of the input which is the current task index and the thread index. Each one goes in its own thread. It’s a fully deterministic operation and does the exact same amount of work regardless of the order of execution. And as a synchronization step I compare it to the last known value and choose the lowest value.

The code for the task:

fun cpuTask(i: Int, rail: Int): String {
    val sha3 = SHA3.Digest256()
    return Hex.toHexString(sha3.digest("CPU/$rail/$i".toByteArray()))
}

Called from:

class State {
    var current: String? = null

    fun update(next: String) {
        if (current == null || current!! > next {
           current = next
        }
    }
}

val state = State()
val lock = ReentrantLock()

runBlocking {
    //
    // run 25K threads (coroutines)
    for (rail in 1..25_000) {
        launch(Dispatchers.Default) {
            var prev: String? = null

            //
            // Execute a CPU bound task 1K times in each thread
            // resulting in ~10 updates to the shared state
            for (i in 1..1000) {
                //
                // Run a task
                val next = cpuTask(i, rail)

                //
                // Check if it's better for the current thread.
                // Only in this example, to avoid too many locks. 
                // Version with "Always" suffix doesn't have this check
                if (prev == null || prev > next) {
                    prev = next

                    //
                    // Update the shared state
                    // Here it's a JVM Lock, but there is a version with Mutex
                    lock.withLock {       
                         state.update(next)
                    }

                }
            }
        }
    }
}

There are two ways to have the synchronization though. One is to always compare with the shared value (the benchmarks with the “Always” suffix). Another one is having a two-step process, first compare with the best value for the current thread, and only the best is compared to the “shared best.” This way dramatically decreases the amount of locks, and having both ways allows us to compare the effect of lock.

I implemented the same logic in different ways to compare the performance:

Coroutines + Java Lock
Coroutines + Coroutines Mutex
Plain Java Threads + Java Lock
Reactor with No Lock
Plain Java Threads with No Lock

The last one is a base number to compare the others, because it’s supposed to be the most optimal code.

I ran this code on the same machine with nothing running on it except the code, using Ubuntu 22.04 / Intel i7 8 cores / OpenJDK Java 21. I ran 25,000 tasks (in parallel limited to the CPU count) each doing 1,000 calculations. I ran each 12 times in random order, so the benchmark should not be significantly affected by some preexisting state of the memory, JVM warm-up, or similar.

The first part is the basic benchmarks:

Here we see the “Threads No Locks,” which is our main basis, with 5620 ms.

And the other standard JVM and Reactor-based implementations that are not very different from each other. They are +100ms to execute, which is lower than the Standard Deviation, so we can’t really say they are much slower, but they are clearly in the same bucket. It’s safe to say that on average all of them have the same performance.

The code above doesn't do much locking because it selects the best per tread before locking. Now what if we always lock the state and compare inside?

Examples with the “Always” prefix demonstrate the effect of the unnecessary locks (i.e., when each value is compared to the global value).  Compared to that, we see that the optimal code as in the previous screen is 25%+ faster.

And now the Coroutines Mutex instead of JVM Lock:

Now we see that the Coroutines Mutex doubles the time of the execution. It’s not just a minor performance issue.

Note, it’s an “optimal” implementation that doesn’t use Mutex too often. Like the very first example with Lock.

What if we always synchronize with the Mutex? Will it be the same 25% loss of performance like with the others?

Oh, no!!! It actually makes it 5 times slower than the previous! Or almost 10 times slower than the JVM Lock.

This is the cost of using a coroutine primitive for a non-IO operation.

The final table looks like this:

In summary:

Use coroutines and related, including Mutex, only for IO-bound tasks;
Use Mutex if you lock over other coroutines (i.e., you call a suspend function inside the lock), but if you do everything right you would not have non-IO coroutines anyway
Use standard JVM primitives, including Lock, for CPU/Memory-bound tasks with no coroutines inside the lock
And of course avoid having long tasks under a lock, regardless of the type of lock

Extending NUC with External U.2 SSD

Igor Artamonov — Wed, 25 Sep 2024 16:33:58 +0000

Recently, I upgraded an Intel NUC by adding external SSDs as a replacement for the internal M.2 SSDs. Before I started, I wasn’t fully sure if it could be done because there isn’t much information on this topic. So, I’m writing down my experience in the hope that it will help someone if they decide to do the same.

First of all, why and what

I have an Intel NUC 12th Enthusiast, which has 3 slots for internal M.2 SSDs. It gives me about 6TB of disk space when I use 3 x 2TB Samsung SSDs. I run an Ethereum node on this machine, which alone consumes more than 4TB, and it’s growing. I wanted to run a couple of different implementations, so I needed much more space for that.

Technically, I could upgrade those 2TB SSDs to 4TB versions, but larger SSDs are not so practical because, to my understanding, the quality/price ratio goes down, and a 2TB SSD is the sweet spot for M.2. Also, I have already worn out a couple of those SSDs, so I wanted to install larger and more durable ones, i.e., enterprise-grade SSDs with a 7.68TB capacity, which only come as external drives.

Here, I describe how I successfully connected two U.2 7.68TB SSDs to an Intel NUC, what the challenges were, and why it may not work for everyone.

Hardware

There are a few protocols for drives, and one of them is NVMe, which usually works on top of PCIe. That’s because PCIe is the most performant interface. There are a few ways to connect to PCIe. In addition to the more common large slot on the motherboard (which I don’t have with Intel NUC), there’s the M.2 slot. Yes, M.2 is a PCIe slot, just a very small one. So technically, it should be possible to connect any PCIe device through M.2.

Enterprise 2.5” NVMe drives may come with a U.2 connection, which is also a PCIe connection. (For reference, U.2 is called SFF-8639). So, it seems like it should work together because I’m connecting one PCIe device to the motherboard through a PCIe interface.

But how do you connect a U.2 drive to an M.2 slot?

Usually, you use a SAS cable to connect those drives. SAS is a protocol that can fully utilize NVMe without a performance loss. On the drive side, the cable connects to U.2 directly (SFF-8639) and then goes as a mini-SAS to the motherboard (SFF-8643).

How do you connect this mini-SAS to a NUC? It turns out you can buy an adapter from mini-SAS to M.2.

So first, you buy a SAS cable (U.2 to mini-SAS / SFF-8639 to SFF-8643). Any brand should work, I guess.

Then, you have to buy an adapter from mini-SAS to M.2. In my case, it was a StarTech M2E4SFF8643 because I didn’t trust no-name brands. But probably any other adapter would work, and there are plenty on Amazon.

One problem with this StarTech adapter is that it comes as an M.2 2260, and this “xx60” means it’s 60mm long. Usually, you have either 42mm or 80mm drives. Those are the most common sizes, and the NUC has mountings only for those two sizes. So, you may want to buy an extension or DIY one from a plastic piece to mount it in the standard 80mm slot.

A note about these M.2 adapters: there are apparently many types, not just for SAS. Another option is Oculink SFF-8612, which is commonly used to connect an external full-size PCIe 16x, usually a GPU. You could use it for drives as well, which I guess could be scaled to more drives, but that would be a more complex approach.

The third thing you need to buy is a Power Supply Unit (PSU). With SAS cables, you must have an external power source. Standard M.2 is powered directly from the motherboard, but with SAS, this option isn’t available.

What I forgot initially is that you cannot simply turn on most PSUs because they are designed to be turned on using the power button on the computer case. To turn it on manually, you connect two pins on its main cable. Google "PSU paper clip test" to understand it better. You could use a paper clip or a short wire, or, to make it more durable, you can buy a jumper clip that connects the correct pins. There are plenty of them on Amazon as well.

Software Part

Once I connected everything and turned it on, I noticed a weird issue: I didn’t see the drives in the BIOS, or, weirder, I saw them semi-randomly. On a power reset, I might see one of the drives, both, or none. Initially, I thought I had a hardware issue.

But when I tried to boot from a USB drive and checked the lsblk, I found both drives even though I hadn’t seen them in the BIOS. That’s weird, and I still haven’t figured out the reason. Maybe it’s a bug in my BIOS version, or maybe there are some incompatibilities between expected SATA and actual SAS. Maybe it's specific to Intel NUC. BTW, if you have any idea what the cause is, please let me know in the comments.

After a few experiments, I became convinced that they work perfectly fine in Linux. But it still doesn’t make much sense to install an OS on them because BIOS cannot boot from them as it doesn’t see the drives.

I guess a workaround is resetting the NUC until it sees any of the drives to boot. Inconvenient, but it might work. Or you could install and boot the OS from an external USB flash drive, mounting the main filesystem from those drives once it’s booted. Alternatively, you could do the same using a network boot.

Remember that I have 3 slots but only 2 drives? So, my solution was to put a standard M.2 SSD in the 3rd slot and install an OS on that drive. When it boots, it successfully mounts the other drives and uses them for larger databases.

It all works fine now, but in my case, it required 3 M.2 slots, as one was populated with a standard M.2 SSD to boot. Keep this in mind, as it may not work for everyone if you don’t have enough M.2 slots. But for this particular case, with Intel NUC 12, it’s definitely possible.

Summary

That’s all you need for the hardware part:

U.2 SSD drive(s)
SAS U.2 to mini-SAS cable(s)
Mini-SAS to M.2 adapter(s)
M.2 2260 to M.2 2280 extender(s)
PSU
24-pin jumper

You need a cable, adapter, and extender for each drive. And, obviously, a single PSU can power them all.

You may want to leave one M.2 slot for a standard M.2 SSD to install an OS on that drive.

Also, keep in mind that once you connect everything together, it will be a bunch of cables and hardware. You won’t be able to put the cover back on the NUC case, so it’s going to sit exposed somewhere on the shelf. And you might not realize how hot it gets without proper ventilation.

The Why Behind the Code

Igor Artamonov — Tue, 22 Aug 2023 20:25:24 +0000

Do you know why microservices architecture is so popular? Because it’s easier to rewrite the whole service rather than inspect the code to find the bug causing crashes in production.

That might be a joke, but I’m sure you'll agree that looking through the code to fix a bug or add a feature is something we sometimes prefer to avoid. Especially when we struggle to understand why this code even exists. I mean, it’s easy to understand what the code does, as it’s right there on the screen. But often, we don’t understand why it’s written that way. Is it a workaround for something? Does it fix some problem in a rare case? What happens if I delete this part of the code? And how can we decide?

People who've worked with me know my strong opinion on making Pull Requests, especially regarding code commit messages. In short, I believe that the commit message must answer the question “Why” the commit was made, with the understanding that the code answers the “What” it does. Most developers don’t share this view, and here I’m explaining my stance and trying to persuade you to adopt the same approach.

In fact, I wasn’t always so meticulous about commit messages. Many years ago, I found it perfectly acceptable to leave a message like just fixing bugs. And that’s what I typically see when I open someone else's code.

Here's a list of messages that represent what we usually see:

add CanUpdate to DbConnection interface
fix for when size is 0
fix use_react to support absolute paths
fix web_base.py
fix show history output
fix dependency

These messages, taken randomly from public GitHub repos, represent common types of commit descriptions. They seem like short summaries of what was done in the commit.

Such commit messages are like an email subjects, which for most people are simply recognizable label that help distinguish between emails in a list.

While these commit messages could be used as reference points for standup calls or weekly reports, they have no use after those meetings.

However, after years in software development, I've learned the hard way when commit messages truly come into play. They are a last-resort tool, something you refer to when worried that a code change might break something else, or perhaps after a change has unexpectedly caused a crash elsewhere. That's when you run a git blame to understand the reasons behind the existing code. Sometimes, you're the original committer, but you might not recall the reasons for writing the code that particular way.

Let’s accept it that a self-explanatory code is a unicorn and code comments are almost never enough to understand it. More to that, even if you in this rare situation with a clean code and good comments, none of this can explain a code change as a whole. It cannot explain a relation between different parts of the code. To link these parts and answer the questions, the commit message should be used. It's the only space suited for this purpose.

Think of the commit message as a medium to communicate with a future developer, or even a future version of yourself. It's a message that can explain multiple changes across several files, providing the rationale for those changes.

I hope my point is clear and resonates with experienced developers. But how should one write these commit messages?

For me, any code change is in response to a problem. Every commit (a “solution”) addresses something specific in mind (a “problem”).

Even if you're just updating a dependency, you're likely addressing concerns about outdated, incompatible, buggy, or slow code in the current version. Or you see it as a problem maintaining a code depending on an outdated library. Such an update solves the problem by upgrading to a newer version. If you're refactoring, you likely see issues with the current code's readability, maintainability, flexibility, etc.

So here's my suggestion: state both the problem and solution in the commit message. Like, Problem: what I wanted to solve; Solution: how I solved it.

I’m not an inventor of this approach, of course, but an advocate for it.

When you adopt this approach, you might find that it simplifies the commit message drafting process. You start recognizing that you always approached commits as answers to problems, but struggled with how to write both problem and solution as one phrase. It’s okay to use two distinct phrases, and it can be easier to write them this way.

Here is an example of what I suggest:

Problem: egress/ingress naming for log config is too confusing
Solution: rename to request/access logs

This approach is more informative than a vaguer message like:

renamed egress/ingress logs config to request/access logs

In the latter case, you might just shorten it to:

rename logs config

And it stops providing as much information as the original. Also it doesn't answer Why? anymore.

It's very tempting to put just a solution part describing what is inside the commit. I made this mistake many times, and see it very often others do. Developers, when they asked to write a commit messages as problem+solution start writing just solution: what's inside the commit. That's not helpful!

I agree, though, that some commits are straightforward enough that their intent is clear. In such cases, I think, it's acceptable to omit the obvious part. For example, when upgrading an outdated library, stating problem: netty 4.1 is outdated during a git blame would say enough about the solution.

Like in my commit above regarding config names, the “solution” part might have been redundant. However, almost 50 files were modified in that commit, so I chose to provide a comprehensive description to manage expectations.

What I suggest now is adopting this approach. Just think about the problem you solved, and what is the solution. Not necessary to push very hard and be formal, but always try to answer “Why” the commit was made, not what is inside it. Just put both in the commit message, without trying to save the characters. And you’ll notice how it improves your process and way of thinking. Plus makes it easier to maintain the code in future.

PS Please share your opinion: https://twitter.com/splix/status/1694084487221522583