DEV Community: John Rush

An unlikely experiment

John Rush — Sat, 22 Aug 2026 23:13:55 +0000

There are so many great places to learn about out there on the internet, and one of those great places is the legendary Linux Kernel. If you stumble around some of that codebase (even if you don't understand much) you might see an interesting pattern littered throughout:

if (unlikely(condition_that_does_not_happen_much)) {
    // do thing
} else {
    // do more common thing
}

What is this mysterious unlikely? Why do we need to tell the computer that something is unlikely, and what does that even do?

It turns out that this simple statement can end up having a huge benefit to performance (or detriment, if used improperly). Let's see if we can understand this a bit, so that we can in turn improve our own programs :)

What is `unlikely`?

The definition of unlikely is actually a simple macro:

#define unlikely(e) __builtin_expect(!!(e), 0)

It takes in an operand e which is passed to __builtin_expect along with 0. __builtin_expect is a special function that just annotates an expression to tell the compiler that it should "expect" the value of the expression to be equal to the second value which in this case is zero.

What we are effectively doing here is giving the compiler some apriori information about what branch of the program is likely or unlikely to be taken. With this information it can prime the CPU to start with the correct probabilities for branch prediction and improve performance.

Okay that was a lot, can you give an example?

Sure I can! Let's great a simple example where we have two identical loops, except one will use the unlikely macro and the other uses the opposing likely macro:

long unlikely_simple(long num_runs) {
    long total = 0;
    for (long i = 0; i < num_runs; i++) {
        if (unlikely(i % 1000 == 0)) {
            total += 2;
        } else {
            total += i + 1;
        }
    }

    return total;
}

long likely_simple(long num_runs) {
    long total = 0;
    for (long i = 0; i < num_runs; i++) {
        if (likely(i % 1000 == 0)) {
            total += 2;
        } else {
            total += i + 1;
        }
    }

    return total;
}

In these simple loops, we take the if branch 0.1% of the time and the else branch 99.9% of the time. Naturally the actually more "likely" path here is the else branch, but we can trick the compiler to thinking the opposite with our choice of macro.

Let's observe the performance of these two loops, and also modify that 1000 to be different values so we can see the effect of different percentages on the likely and unlikely loops:

method	percent chance to take first branch	average run time (micro seconds)
unlikely_simple	50%	48,899
unlikely_simple	10%	51,736
unlikely_simple	1%	58,496
unlikely_simple	.1%	53,637
likely_simple	50%	48,643
likely_simple	10%	75,979
likely_simple	1%	79,594
likely_simple	.1%	72,342

From the data we can see that as we decrease the percent of values that will take the first branch, the likely_simple compiler hint becomes less and less accurate. We hinted to the compiler that we should optimize towards the first branch, but real execution often goes to the second branch - leading to worse performance.

Okay so we've seen some data that shows our hint is having an effect on how the loop is processed. But why is that? The answer is in the assembly:

Assembly generated using the command: gcc -O2 unlikely.c -o bin/unlikely

Likely loop Assembly

0000000100000460 <_likely_simple>:
... 
# start of the loop
100000488: d343fd2d     lsr     x13, x9, #3
10000048c: 9bcb7dad     umulh   x13, x13, x11
100000490: d344fdad     lsr     x13, x13, #4
100000494: 9b0c29ad     madd    x13, x13, x12, x10
100000498: f10005bf     cmp     x13, #0x1

# branching check. if it's equal, go to else at "4b8"
10000049c: 540000e0     b.eq    0x1000004b8 <_likely_simple+0x58>

# the if branch. perform total += 2
1000004a0: 91000908     add     x8, x8, #0x2

# loop condition
1000004a4: 9100054a     add     x10, x10, #0x1
1000004a8: 91000529     add     x9, x9, #0x1
1000004ac: f1000400     subs    x0, x0, #0x1

# loop re-enter and loop exit checks here
1000004b0: 54fffec1     b.ne    0x100000488 <_likely_simple+0x28>
1000004b4: 14000004     b       0x1000004c4 <_likely_simple+0x64>

# the else branch. perform total += i
1000004b8: 8b0a0108     add     x8, x8, x10

# loop exit
1000004bc: 17fffffa     b       0x1000004a4 <_likely_simple+0x44>
1000004c0: d2800008     mov     x8, #0x0                ; =0
1000004c4: aa0803e0     mov     x0, x8
1000004c8: d65f03c0     ret

Unlikely loop Assembly

00000001000004cc <_unlikely_simple>:
...
# start of the loop
1000004f4: d343fd2d     lsr     x13, x9, #3
1000004f8: 9bcb7dad     umulh   x13, x13, x11
1000004fc: d344fdad     lsr     x13, x13, #4
100000500: 9b0c29ad     madd    x13, x13, x12, x10
100000504: f10005bf     cmp     x13, #0x1

# branching check. if it's NOT equal, go to if at "524"
100000508: 540000e1     b.ne    0x100000524 <_unlikely_simple+0x58>

# the else branch. perform total += i
10000050c: 8b0a0108     add     x8, x8, x10

# loop condition
100000510: 9100054a     add     x10, x10, #0x1
100000514: 91000529     add     x9, x9, #0x1
100000518: f1000400     subs    x0, x0, #0x1

# loop re-enter and loop exit checks here
10000051c: 54fffec1     b.ne    0x1000004f4 <_unlikely_simple+0x28>
100000520: 14000004     b       0x100000530 <_unlikely_simple+0x64>

# the if branch. perform total += 2
100000524: 91000908     add     x8, x8, #0x2
100000528: 17fffffa     b       0x100000510 <_unlikely_simple+0x44>
10000052c: d2800008     mov     x8, #0x0                ; =0
100000530: aa0803e0     mov     x0, x8
100000534: d65f03c0     ret

Lots of assembly, but the key part is the order of the branches in the likely loop versus the unlikely loop. In the unlikely loop, we have the total += i + 1 assembly immediately following the if condition check and a branch to the total += 2 assembly. The compiler uses the opposite ordering and branching in the likely loop, favoring to have the total += 2 assembly first and and the total += i + 1 second.

This ordering is important to the execution of the loop because the CPU will generally pipeline instructions, meaning that it will prefetch and pre-emptively execute several instructions after the currently executing one. The pre-fetch happens in the order of the instructions in memory, hence why the order matters. This explains why in our experiment we would see varying performance for the two functions. One of them would go against that pipelining and cause the CPU work to be wasted, and the other was aligned with it and lead to better performance.

Conclusion

There are lots of little gems like this in open source repos that we can all learn from. While unlikely and likely touch upon several low level concepts that likely won't matter for the average program, it is always good to have this knowledge in your backpocket. You never know when this type of code might 2X your performance!

What's an event loop anyways?

John Rush — Fri, 21 Aug 2026 00:09:32 +0000

Event loops are a paradigm for processing events different than your typical single-threaded or multi-threaded application. Your request gets broken down into async "events" that are executed in a loop to improve performance and minimize synchronization across threads. It is famously used by Node.js as the backbone of their event processing and also by several other technologies like Redis and Nginx.

In this article I'll explain the reason for why this paradigm was created and what it tries to optimize. By the end you'll come out a little wiser, and know more than just "don't block the event loop" :).

Motiviation - why event loops?

To understand why we need event loops we will explore a simple but key example. Take this straightforward HTTP request code, which sends a request and then tries to read the response from the socket:

def send_http_request_GET(domain: str, request: str) -> HttpResponse:
    socket_fd = get_socket_for_domain(domain)
    write_res = os.write(socket_fd, request)
    data = os.read(socket_fd, 1024)
    return HttpResponse(data)

We do two things in this call - write data out and read data in. Both of these actions will end up triggering syscalls through the kernel that write and fetch data. In terms of time spent on the CPU, this is relatively inexpensive; sending out packets takes very little time, and eventually reading the response will also take very little CPU time. The key time lost is from waiting on the server to respond to us. os.read will block this thread until the response is available, meaning that the thread cannot be used for any other processing during this time.

If our service is single-threaded, this means that we can't make any requests in parallel and are stuck waiting on any previous requests to finish. But of course, most services are not single-threaded, so this isn't a huge problem? Let's continue with the example code, imagining that instead we are processing these requests with multiple threads pulling from a request_queue.

class RequestQueue:
    queue: list[tuple[str, str]]
    lock: threading.Lock

def do_work(request_queue: RequestQueue):
    while len(request_queue.queue) > 0:
        domain = None
        request = None
        with request_queue.lock:
            if len(request_queue.queue) > 0:
                domain, request = request_queue.queue.popleft()

        if domain and request:
            send_http_request_GET(domain, request)

def send_http_request_GET(domain: str, request: str) -> HttpResponse:
    socket_fd = get_socket_for_domain(domain)
    write_res = os.write(socket_fd, request)
    data = os.read(socket_fd, 1024)
    return HttpResponse(data)

def process_requests(requests: list[tuple[str, str]]):
    queue = RequestQueue(queue=requests)
    threads = []
    for _ in range(10):
        thread = threading.Thread(target=do_work, args=(queue))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

Alright now we're cooking! We have multiple threads processing the list of requests, and we have 10X'ed our throughput, amazing. But we weren't here to just create yet another multi-threaded setup, so what is the problem here? Well, one major problem is that we are now managing 10 threads, which all have to be synchronized to get work from request_queue. Synchronization via locks can add a lot of overhead, slowing down our program (so we probably didn't actually get 10x gains here... unlucky). Another problem is that our throughput has a defined upper bound - 10 requests. We could solve that by bumping the number of threads. Let's say I want 10k requests as my throughput - I'll just increase it to 10k and things will be fine, right? Not quite. Aside from the increased sync overhead, we also have some additional issues that come up (as the saying goes, there is no such thing as a free lunch..):

Extra memory to manage each thread (linear scaling)
Coordination with every other thread -> latency (quadratic scaling)
More context switches for the CPU. These are not free and can be a drag on latency at high concurrency.
More threads == more CPU cache line thrashing!

We didn't have all these problems with our single-threaded program, so maybe we can tackle this problem in a different way. Let's see if we can get back that single-thread magic and start processing in parallel!

Enter the event loop

As you might have guessed, our event loop will be single threaded. So how are we going to avoid the blocking call that we saw earlier? Two things - a paradigm shift and a bit of help from the kernel.

Paradigm shift: working with Events

Now that we can't call read directly in our user code, we need to instead return something to express our intent to read as well as a way to continue executing our code. Let's call that "something" an Event. Concretely, an event describes the intent to do a syscall like read or write and a pointer to the code that should handle the result of that syscall. Putting that into code, the Event class would look something like this:

class Action(enum):
    READ = "read"
    WRITE = "write"

class Event:
    file_descriptor: int
    action: Action
    continuation: Callable[Event, Event]
    input: Any
    user_data: Any
    result: Any

Breaking it down, our event class contains:

A file_descriptor to say which fd we want to perform the action on
An action that we intend to execute
A continuation that describes what to call with the result of the event
input for the action
user_data to allow passing along extra data we need to continuation
result which will be read in the continuation

With this class we have created a bit of indirection between event processing and the start of processing. This enables us to defer execution of read until we are confident that there is actually data available to be read, which we will touch on in the next section. Before we get there though, we need to modify our code to actually talk in terms of events:

def on_read_finish(read_event: Event) -> Event:
    request_context = read_event.user_data
    return request_context.finish(HttpResponse(read_event.data))

def send_http_request_GET(domain: str, request: str, request_context: RequestContext) -> Event:
    socket_fd = get_socket_for_domain(domain)
    write_res = os.write(socket_fd, request)

    return create_read_event(socket_fd, 1024, continuation=on_read_finish, user_data=request_context)

Now we create an Event to express our intent to read, provide a pointer to our continuation, and some additional data we care about. Our code looks very different and has some extra boilerplate, but we are now in a position to execute the reads at the right time.

Waiting for data: the `select` syscall

Here's where the kernel comes in to help us build our event loop. The kernel provides a syscall called select which allows for asking to be notified when a set of file descriptors are ready for reading or writing. The key here is that select allows for waiting on multiple descriptors at the same time, enabling us to:

gather a list of events that we are waiting on data for (i.e. need to call read with)
start executing again when any of those events is ready for processing

This is the crux of event-based processing. We gather together as many pending events that we can, and then process them once they are ready. Rather than blocking for one single event to finish, we can continously make progress and add to the list of events that we are deferring until the data is available.

Putting it together

With our code returning Event objects and knowledge of the select syscall, we can now build our "async" event loop. The basic flow is this:

Gather all pending events from previous continuations
Call select on the list of associated file_descriptors
Execute the list of events associated with the list of file_descriptors returned by select (read/write)
Execute any continuations from the events in the previous step
Save any new events
Loop

And that flow in python would look something like this:

def execute_event(event: Event):
    result: int = -1
    if event.action == Action.READ:
        # execute read...
    elif event.action == Action.WRITE:
        # execute write...

    event.result = result

def event_loop():
    event_store = {}
    while True:
        pending_event_fds = [event.file_descriptor for event in event_store.values()]
        select_result = select.select(pending_event_fds, [], [])

        for fd in select_result[0]:
            event = event_store[fd]
            event_store.remove(fd)

            execute_event(event)
            if event.continuation:
                new_event = event.continuation(event)
                event_store[new_event.file_descriptor] = new_event

In an actual implementation we would need to add some initial events to get the loop started, for example by adding an event that periodically calls accept to accept new connections and start processing requests. After that we would just loop and execute the current events that we have.

A quick performance comparison

At this point we have two small implementations, one of multi-threaded event processing and another for event loop based processing. Let's put them to the test with a quick profile.

We will test both implementations with three separate workloads:

high concurrency IO requests
high concurrency high CPU requests
high concurrency contentious requests (set a key in a collection)

In these workloads we expect the event loop to generally perform better in IO / contention contexts, where a thread pool will usually do better when there is majority CPU work as it can utilize multiple CPU cores.

IO Experiment

throughput	latency

In the IO experiment we can see that the event loop matches the maximum request rate reached by the 256 thread thread pool. That is pretty amazing results for only using a single thread! In the chart on the right we can also see the P50, P95, and P99 latencies compared with each thread pool configuration. The latency improves for the thread pool as more threads are added, but generally it always underperforms the event loop (especially when tasks are longer).

CPU Experiment

throughput	latency

In the CPU experiment the event loop struggle a lot. In this case I can't event plot all of the throughput values for the different work lengths because the latency grows insanely fast. Thread pool definitely wins here by being able to chop through that work efficiently, and generally more threads is better up to ~256. It is probably better to stick to a lower number of threads when you start getting diminishing returns, as threads add memory overhead and context switch costs.

Contention Experiment (setting a key in a shared dictionary)

throughput	latency

The contention workload is another place where the event loop shines. This time around I just kept the # of threads static at 256, as modifying a single collection is a simple piece of work. We can see that as the request rate grows, lock contention becomes the dominating factor and the throughput of the thread pool stabilizes at ~25k requests/second. The event loop reaches 140k requests/second before leveling off. Naturally we see that the latency is much better with the event loop as well.

Benefits of event loops

Event loops have some unique performance characteristics that give them a unique advantage over the traditional thread pool for certain workflows. The primary advantages are:

Great performance on IO workloads
Single-threaded code means no need for synchronization and locking
Simpler programming model due to no synchronization

Drawbacks of event loops

The achilles heel of the event loop is obviously the ability to block the loop with heavy CPU work. There are also some other disadvantages:

Less clear programming model. Now we have to code in Events, which is less ergonomic than normal declarative programming.
Not all types of code can be converted to an Event, meaning it does not work well for all workloads (e.g. GPU interaction)

It is worth noting though that Event style programming can be made better. The commonly supported async model is another representation of Event style programming, where we use the preprocessor / compiler to generate the indirect code for us.

Conclusion

We explored how to create a simple event loop and did some profiling. Event loops are a great illustration of the tradeoffs that can be taken when it comes to different workloads, so keep it in mind for your next project!

Thanks for reading!

DEV Community: John Rush

An unlikely experiment

What is unlikely?

Okay that was a lot, can you give an example?

Conclusion

What's an event loop anyways?

Motiviation - why event loops?

Enter the event loop

Paradigm shift: working with Events

Waiting for data: the select syscall

Putting it together

A quick performance comparison

IO Experiment

CPU Experiment

Contention Experiment (setting a key in a shared dictionary)

Benefits of event loops

Drawbacks of event loops

Conclusion

What is `unlikely`?

Waiting for data: the `select` syscall