DEV Community: Serpent7776

Hive-index now shows operations stats

Serpent7776 — Tue, 24 Mar 2026 18:13:42 +0000

Previously I published initial version of hive-index, showing custom JSON operation stats.

This time I want to present an update: hive-index now also shows Hive operations stats page.

The custom JSON operation stats page has been moved to its own page at https://index.hive-adventures.com/custom_json_operations and index now only shows links to these two pages.

The top 2 operations take up 90% of all operations: custom_json_operation and vote_operation. The third one is comment_operation.

The project still lives at https://index.hive-adventures.com/

Enjoy, and if you have questions or comments, just ask.

Introducing Hive Index: A real-time custom json operations explorer for Hive

Serpent7776 — Thu, 05 Mar 2026 21:35:49 +0000

I'd like to share Hive Index, a web UI for exploring Hive blockchain activity with clear, live stats, built in Rust 🦀.

It runs as a single binary, continuously pulling Hive blocks and updating its internal database with metadata about the blocks and operations.

Currently, it aggregates custom_json operations and shows ranked stats for them.

If I read the data correctly, the top 10 is dominated by Splinterlands, but there's also follow operation at #7.

The data was synced starting at block 7,000,000 (it doesn't seem there was anything interesting before that). The current database weight is about 9GB.

If you're interested in trying it out, head to https://index.hive-adventures.com/

Let me know if it's useful to you and if you have any questions or suggestions!

Comparing images with AVX

Serpent7776 — Sun, 02 Nov 2025 17:11:05 +0000

This is a submission for the 2025 Hacktoberfest Writing Challenge

This is the first time I heard about hacktoberfest - a month-long initiative in October that encourages developers around the world to contribute to open source projects. This looked interesting, so I thought I'd give it a go.

Some time before that I started learning more about SIMD (AVX512 specifically) and was looking for a project where I can apply this in practice. One of the projects I knew was odiff, which I learned about at the FunOCaml conference I went to last year.

This blog post summarises my contribution to odiff, by rewriting odiff core algorithm into AVX512 assembly.

odiff

odiff is a very fast pixel-by-pixel image comparison tool. It is comparing images in the YIQ colour space rather than the RGB space to better represent visual differences. It's a really cool project. Go check it out.

It was originally written in OCaml and recently it was rewritten in zig for better SIMD support.

SIMD

Normally, CPU can only operate on a single memory location at a time (e.g. a byte, word, etc). This means that if we want to apply the same operation to large number of different memory locations, we need to do it in a loop, one element at a time. This is quite tedious and easy to get wrong. It also doesn't use all the power the CPU has.

There's a lot of workflows that could be optimised to do this is the hardware. Image processing, like in odiff, is one example. Other examples include audio and video processing. All of them essentially apply the same processing to a very large number of data samples.

Originally introduced as MMX extension for x86, but over time evolved to a large number of other extensions, with growing capabilities.
Each CPU has its own SIMD extensions: x86 has MMX, SSE and AVX, ARM has NEON, and RISC-V has RVV.

AVX512

AVX is a variant of x86 SIMD that operates on 256 bits of data at a time.
AVX512 is a evolution of AVX that operates on 512 bit of data. It's really awesome.

512 bits is 64 bytes, which for RGBA image data means we can operate on 16 pixels at a time in one instruction.

If you ever used array programming languages, AVX512 kind of feels similar (though of course it's definitely lower level). You always operate on whole arrays, not single elements. This requires a shift in the way you think about doing things, but it's very rewarding and can give you some nice performance boost.

vxdiff

This lead to the creation of vxdiff, a pure assembly AVX512 based reimplementation of the core odiff algorithm. It only implements the default set of odiff options (and odiff has some interesting options to tweak its behaviour), but it's still a capable tool. It's also easier to start simple rather than trying to implement all the possible options right away.

It was a really fun thing to do, and I learnt a lot about AVX

The algorithm

I will now explain the odiff core algorithm after translation to AVX. This might be a bit chaotic, because I was writing this in a bit of a hurry, so bear with me.

The full source is here, you can follow along.

First the input images are loaded into memory using RGBA8 format, which means that there's 3 colour channels (red, green, blue) and an alpha channel, each having 8 bits.
Once we've got the image data, the actual processing loop starts at .y_loop label, which iterates over image rows. Each row is handled by .x_loop label, which is a loop that loads 512bit of data at a time. rdi and rsi point to image data of the base and other images. After executing xmm1 and xmm2 will have 64 bytes of loaded data, which we can now operate on.

vmovdqu8 xmm1, [rdi]
vmovdqu8 xmm2, [rsi]

Now, we want to replace pixels having alpha equal to 0 with white. First we clear k6 and k7 mask registers with kxor. Then we check alpha channel to see which values are equal to 0 and store the result into k6 mask using vpcmpequb. At the indices where alpha was 0, mask will equal 1 and for all other indices it will be 0. Then we propagate that information to other channels (red, green and blue) using kshiftlb and kor. Then we store 255 at places where resulting mask is 1. We do this twice - once for the base image and second time for the image we compare against.

kxor k6, k6, k6              
kxor k7, k7, k7              
vpcmpequb k6 {k4}, xmm1, xmm0
kshiftlb k6, k6, 1           
kor k7, k7, k6               
kshiftlb k6, k6, 1           
kor k7, k7, k6               
kshiftlb k6, k6, 1           
kor k7, k7, k6               
vmovdqu8 xmm1 {k7}, xmm31

Next, we want to do some multiplications, so we'll convert bytes to floats. First we convert from bytes to dwords using vpmovzxbd and then from dwords to floats using vcvtudq2ps.

vpmovzxbd zmm1, xmm1  
vcvtudq2ps zmm1, zmm1 
vpmovzxbd zmm2, xmm2  
vcvtudq2ps zmm2, zmm2

Normalise alpha from initial 0-255 range to 0.0 - 1.0 range. k4 mask apply the vmulps operation (which does multiplication) only to alpha channel.

vmulps zmm1 {k4}, zmm1, zmm3
vmulps zmm2 {k4}, zmm2, zmm3

Do alpha-blending, so blend red, green and blue channels with white pixel using alpha channel. This essentially compute (RGB - 255.0) * A + 255.0. I'm not actually sure why it's computed this way, but this is what odiff used to do in the previous version. k5 apply the operations only to red, green and blue channels, leaving alpha channel unmodified.

vsubps zmm1 {k5}, zmm1, zmm30  
vshufps zmm10, zmm1, zmm1, 0xff
vmulps zmm1 {k5}, zmm1, zmm10  
vaddps zmm1 {k5}, zmm1, zmm30

Now we start RGB to YIQ conversion. We use precomputed matrix in zmm7, zmm8 and zmm9 to multiply to with RGB values to get YIQ values with vmulps. That will give us 3 separate vectors with Y, I and Q channels. Then we shuffle them around and add them up to get YIQ vector. We will use this to compute the difference between base image and the image we are comparing against.

vmulps zmm10, zmm1, zmm7 ; y
vmulps zmm11, zmm1, zmm8 ; i
vmulps zmm12, zmm1, zmm9 ; q

; yiq(R)                                    
vxorps zmm13, zmm13, zmm13                  
vshufps zmm13 {k1}, zmm10, zmm10, 0b00000000
vshufps zmm13 {k2}, zmm11, zmm11, 0b00000000
vshufps zmm13 {k3}, zmm12, zmm12, 0b00000000
; yiq(G)                                    
vxorps zmm14, zmm14, zmm14                  
vshufps zmm14 {k1}, zmm10, zmm10, 0b00000001
vshufps zmm14 {k2}, zmm11, zmm11, 0b00000100
vshufps zmm14 {k3}, zmm12, zmm12, 0b00010000
; yiq(B)                                    
vxorps zmm15, zmm15, zmm15                  
vshufps zmm15 {k1}, zmm10, zmm10, 0b00000010
vshufps zmm15 {k2}, zmm11, zmm11, 0b00001000
vshufps zmm15 {k3}, zmm12, zmm12, 0b00100000
; yiq                     
vaddps zmm16, zmm13, zmm14
vaddps zmm16, zmm16, zmm15

The result is that we have YIQ of base image in zmm16 and YIQ of the other image in zmm26. We take the difference of the two using vsubps, square the result and multiply by delta coefficient.

; YIQ diff                
vsubps zmm16, zmm16, zmm26

; YIQ*YIQ                 
vmulps zmm16, zmm16, zmm16
; YIQ*YIQ * delta coef    
vmulps zmm16, zmm16, zmm29

Now we have squares of differences of channels in zmm16 vector, so we need to sum those channels into a single one. This way even there's difference in both channel Y and I, it will be counted as a single difference.

vxorps zmm17, zmm17, zmm17                  
vxorps zmm18, zmm18, zmm18                  
vxorps zmm19, zmm19, zmm19                  
vshufps zmm17 {k1}, zmm16, zmm16, 0b10101010
vshufps zmm18 {k1}, zmm16, zmm16, 0b01010101
vshufps zmm19 {k1}, zmm16, zmm16, 0b00000000

; delta                                     
vaddps zmm16, zmm19, zmm18                  
vaddps zmm16, zmm16, zmm17

Finally, we count the number of different pixels. We compare the resulting values against maximum allowed delta in xmm28, which gives boolean mask in k6, which we then treat as an array with 1 or 0 values and sum it up. The sum is the number of pixels that differ when comparing base image against the other image.

vcompressps zmm16 {k1}, zmm16
vcmpgtps k6, xmm16, xmm28    
kmov eax, k6                 
popcnt eax, eax

Back to odiff

vxdiff is definitely an interesting project, but its usefulness is limited when compared to odiff, given all its options. Fortunately, vxdiff is now part of odiff. To activate it, you need to specify --enable-asm when invoking odiff.

That's all folks

Shout out to Dmitriy Kovalenko and the organisers of FunOCaml, because as strange as it may seem, this project wouldn't have happened if FunOCaml hadn't been organised.

Signal safety and why should you care

Serpent7776 — Wed, 23 Jul 2025 20:01:24 +0000

UPDATE 2025-07-25: Clarified async-signal safety in context of boost::asio::signal_set
UPDATE 2025-08-12: Mention sigaction as an alternative to signal. Thanks to Paul J. Lucas.

Calling a non-async-signal-safe function from a signal handler is essentially an undefined behaviour and can lead to all sorts of weird behaviours: deadlocks, race conditions and even memory corruption. You may even experience crashes originating from libc, such as the following:

malloc(): unsorted double linked list corrupted

Fatal glibc error: malloc.c:2599 (sysmalloc): assertion failed: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)

What are signals?

Signals are a way to do asynchronous messages on a POSIX system (such as BSD, Linux, macOS) to notify a process that a specific event happened. The key feature here is the asynchronous part. A signal can be delivered at almost any moment interrupting the application's normal flow. This might happen when the application is inside critical session or syscall. Your application can be in the middle of allocating or freeing memory, or performing any other standard C function call. Pretty scary.

Most signals are ignored by default, some automatically terminate the entire program. For most of them a custom signal handler can be attached to execute potentially any code when the signal is delivered. Potentially, because of all the restrictions described in the previous paragraph. Therefore the installed signal handler needs to be so-called async-signal-safe.

What is async‑signal safety?

A function is considered async-signal-safe if it can be safely called from within a signal handler without causing undefined behaviour. This is pretty circular definition. Basically there are two possibilities:

the function needs to be reentrant and only execute atomic operations
the function needs to block signals temporarily

A function is reentrant if it can be invoked again even if the previous invocation is still running. Basically it must be able to recurse into itself at any point. Such function cannot rely on global state or execute non-atomic operations.
An atomic operation is the one that completes as a single step that cannot be interrupted. It either happens completely or not at all - an intermediate state cannot be observed. Most operations are not atomic. Even seemingly simple operations might not be, e.g. incrementing a variable is a complex operation (unless it's of an atomic type).
Both properties drastically reduce the number of possible operations that can be safely executed from a handler.
The second option is to temporarily block signals while executing a function. This should not be overused as it may limit the program responsiveness.
The general advice is that a signal handlers should perform only the minimum required work.

Functions that are not async-signal-safe

Below are some examples of functions that should not be called from a signal handler. Note that this is not an exhaustive list.

Memory management: malloc, free, realloc, etc.
Standard C I/O: printf, fprintf, fopen, fclose, etc.
Thread synchronisation: pthread_mutex_lock, pthread_cond_wait, etc.
Dynamic symbol resolution: dlopen, dlsym, etc.
High‑level POSIX calls: system, getaddrinfo, etc.

It is not always obvious if a function is async-signal-safe or not. For example you might think that memset should be safe, but it turns out that memset is not async-signal-safe.

Functions that are guaranteed to be async-signal-safe

POSIX specifies a list of functions that are required to be async-signal-safe, here are a few examples:

I/O: read(2), write(2), creat(2), close(2), unlink(2)
Process control: kill(2), wait(2), abort(3)
Reading and writing to atomic flags (volatile sig_atomic_t)

To see the full list consult the POSIX specification or your operating system's documentation. For Linux its signal-safety(7)

Common practice

Use `volatile sig_atomic_t` flag

If any more complex work is required in response to a signal, the common practice is to set a volatile sig_atomic_t flag in the handler and return immediately. The bulk of the required work is then done elsewhere in the normal program loop after the flag has been checked. However, not every signal can be handled in this way (e.g. SIGSEGV)

volatile sig_atomic_t got_sigint = 0;

void handler(int sig) {
    got_sigint = 1;  // Safe atomic store
}

int main(void) {
    signal(SIGINT, handler);
    while (...) {
        // normal program loop
        do_work();
        if (got_sigint) {
            // handle signal
            handle_sigint();
            got_sigint = 0;
        }
    }
}

Note: this code uses signal, which is the traditional way of handling signals. There are newer APIs to do the same, which should be used instead, e.g. sigaction. The detailed comparison is out of scope here; it's a topic for a future post.

Block signals during critical sections

Use sigprocmask or pthread_sigmask to block relevant signals around non‑async-signal‑safe code and unblock immediately after.

Real world signals

Most of the code isn't likely to be invoked from a signal handler. However in some contexts this may happen. And it's not always clear what parts of code might be invoked from a signal handler.

Postgresql query timeout handlers

In Postgresql you can register a query timeout handler. It will be invoked automatically when a query exceeds the specified time limit. What might be surprising is that Postgresql uses SIGALRM signal to interrupt the query. This means that the timeout handler runs from a signal handler. You might want to produce a log that says that the query exceeded the specified time in the handler, but this could lead to all the weird behaviours mentioned above.

Handling Ctrl+C for safe exit

A common use of signals is to implement safe program termination by handling Ctrl-C key combination. This is done by handling the SIGINT signal. And again, you must be very careful what you do inside the handler. Typically you would simply set a flag, return from the handler and do the actual handling elsewhere in the program. Doing anything more complex directly in the handler is very risky.

boost::asio::signal_set

If you are using C++, boost::asio::signal_set automatically handles all of this for you. This is because it installs its own, small signal handler that does the minimal work and invokes your callback in the context of the thread running the asio IO service.

Conclusion

Be aware of the contexts in which the code might be called from a signal handler. It's very easy to introduce undefined behaviour in these places if you're not careful.

If you encounter weird memory corruption or other weird behaviours, signal handlers could be the first place to look for potential issues.

Snowflakes and other UUIDs

Serpent7776 — Wed, 09 Jul 2025 17:43:29 +0000

In software projects there's often a need to assign a unique identifier to a data record or entity so that it can be distinctly recognised. Aside from sequential numbering, there are two common methods of generating such IDs: UUIDs and snowflakes. This blog post describes both of them pointing out their strength and weaknesses.

UUID

UUIDs are 128-bit values that are meant to be unique for practical purposes.

An example UUID is a5de120b-fe73-4fbf-afbf-061551b299bb.

All UUIDs have the following structure: xxxxxxxx-xxxx-Vxxx-Rxxx-xxxxxxxxxxxx
The values in the V and R locations uniquely identify the UUID version and variant respectively. The x values are set depending on the version of UUID.

So we can see that the above example ID is a version 4 UUID.
89b910b8-59c1-11f0-94b8-325096b39f47 is a version 1 UUID.
0197db8e-0f63-74a6-9ee5-d402424d1cd4 is a version 7 UUID.

There's a few different UUID versions and I will describe them out-of-order, because it will be easier to explain:

Version 1 (timestamp + MAC address)

Concatenates a 60-bit timestamp with the 48-bit MAC address of the node that is generating the UUID. This sounds pretty neat and useful, until you find out that the timestamp is the number of 100-nanosecond intervals since midnight 15 October 1582 UTC - the date of Gregorian reform to the Christian calendar. This is quite a weird encoding. What's worse, timestamp bits are stored in the UUID from the lowest to the highest bits, which means these IDs cannot be trivially sorted by the date. The advantage of this version is that it is guaranteed to be a unique ID, unless generated on the same computer and at the exact same time. The issue is that it exposes the generating machine's MAC address.

Version 6 (timestamp + MAC address)

This is the same as version 1 except all timestamp bits are ordered from most significant to least significant. This fixes one issue with version 1 - IDs can now be trivially sorted by the date.

Version 4 (random)

All the bits are generated randomly. This means there's potentially a chance of duplication, but this is almost impossible.

Version 5 (namespace name-based)

This generates a unique ID from a SHA-1 hash of a concatenation of a namespace and a name. There is no temporal or random component, so the same input produces the same output every time. If you need to generate reproducible UUIDs from given names, you likely want this version.

Version 3 (namespace name-based)

The same as version 5, but uses MD5 instead of SHA-1. Unless you know you need it, you should use version 5 instead of this one.

Version 7 (timestamp and random)

Begins with a 48 bit big-endian Unix Epoch timestamp with approximately millisecond granularity. The following bits are random seeded counter

Version 2 (DCE Security version, with embedded POSIX UIDs)

Not very widely used.
Similar to version 1 - uses the current time, along with the MAC address, but replaces the low part of the time field with a local identifier e.g. the user ID or group ID of the local account that created the UUID. Compared to version 1, it has the advantage that you can know not only when and where the ID was created, but also who created the identifier. Replacing part of the timestamp with local ID has some important drawbacks though. It reduces the number of unique ID that are possible to generate and makes the timestamp lossy - they only have granularity of about 7 minutes.

Version 8 (custom)

There are one two requirements:

The variant bits have to be 10.
The version nibble has to be the value of 8.

The remaining bits are up to the vendor to customise.

The advantages of UUID are that they are standardised and there's many ready to use libraries. One disadvantage is that they are 128-bit values, which might be pretty big for some use cases. There might be other issues depending on the exact context.
For example if a deterministic ID is needed then only versions 3 and 5 are appropriate. If a timestamp-sortable ID is needed then only versions 6 or 7 are adequate. But what if both properties are needed? Versions 3 and 5 are not sortable by time, and versions 6 and 7 are not fully deterministic.

Snowflake

Snowflake were originally created at Twitter as a performant way to create a large amount of unique IDs in distributed context.

They are in spirit similar to UUID version 6. They contain timestamp, generating node's ID and a sequence number. Similarly to UUID version 6, snowflakes are trivially sortable by time. The big difference is that snowflakes are 64-bit compared to 128-bit UUID. For this reason they don't have any particular representation - they are presented simply as an integer.

Twitter's snowflake has the following fields widths:

sign bit - unused
41 bits for the timestamp
10 bits for node ID
12 bits for sequence number

Because it's just an integer without any structure, it's also very easy to customise. Simple bit manipulation operations are enough to create your own snowflake type in handful of lines. It's easy to change bit widths of the fields and also add or remove fields.

The main issue with snowflakes is that they are less well-known and there's less ready to be used libraries. It's also not standardised, which combined with the easy customisability cause that the existing libraries might not be compatible with each other.

With all this freedom you can create snowflake type that is both sortable by time and deterministic. It's just a matter of proper field interpretation.

pg-snowflake

I created a Postgresql extension that can create customisable snowflake IDs. You can create many distinct ID types, each having different bit allocations for its fields.

Here's the Github page for pg_snowflake extension.

Other snowflakes

There are a few other libraries creating snowflakes and UUIDs, you can find some here:

RFC 9562 - specifies UUIDs
Postgres UUID type
Snowflake ID
UUID
https://www.uuidtools.com/uuid-versions-explained
Twitter original snowflake implementation

LD_PRELOAD explained

Serpent7776 — Sun, 08 Jun 2025 21:18:04 +0000

EDIT 2025-06-09 Added libSegFault example as suggested by Jeremy from #include.

LD_PRELOAD is an environment variable on Unix-like operating systems that allows users to specify a shared library to be loaded before others when a program is executed. This can be used to override functions in existing libraries or to inject custom code into applications. This functionality is provided by system dynamic linker and works only for dynamically linked executables. Statically linked binaries skip this entirely.
This post will be focused on Linux and BSDs.

The dynamic linker

First, let's see what the dynamic linker is.

When a dynamically linked program is launched, before even calling the main function of the program a few things are done. One of the first things that the operating system does is that it reads the path of the dynamic linker from the executable image and then execute that as a program. This has to succeed or the whole program execution fails. When dynamic loader is executed, it loads the program image and all the dynamically loaded shared libraries that the executed program needs, and then starts the execution.
So basically, the dynamic linker is another program that handles execution of all other dynamically executed programs on the system.

On Linux the dynamic linker will be something like ld-linux.so, e.g. on my system it's /lib64/ld-linux-x86-64.so.2.

On FreeBSD it will be something like /libexec/ld-elf.so.1

LD_PRELOAD

So, how does LD_PRELOAD work?
It basically adds a list of shared libraries to be loaded by the dynamic linker after the program image is loaded, but before its shared object dependencies are loaded. This way, when a particular function symbol name is looked for, it will be found in preloaded shared object first. This is how function overriding is achieved.

Examples

Zlibc

Zlibc, also known as uncompress.so - facilitates transparent decompression when used through the LD_PRELOAD. It is therefore possible to read compressed (gzipped) files data as if they were uncompressed, essentially adding support for reading compressed files to any application.

LibSegFault

Another interesting example is LibSegFault, which is preloadable library that handles segfaults and produce nice stack trace on crashes.

Overriding malloc

The following is a simple example of a library that overrides malloc function.
It uses dlsym(RTLD_NEXT, "malloc") to look up the real malloc that is being overridden and counts the number of all allocations.

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

size_t malloc_count = 0;

void *malloc(size_t size) {
    static void *(*real_malloc)(size_t) = NULL;
    if (!real_malloc) {
        real_malloc = dlsym(RTLD_NEXT, "malloc");
    }
    void *p = real_malloc(size);
    malloc_count++;
    return p;
}

Note that you need to be careful when adding custom code to malloc, not to call itself recursively, which might be easier than you think.

This example is not very exciting, so let's take a look at something more interesting.

Overriding read and write

Another example would be a library that overrides read and write calls to add a sleep before doing the actual work:
This example comes from crawlio

This time the actual implementation of read and write is loaded in the constructor function. This is a function that is executed just after the library is loaded. This way the overloaded functions are loaded only once.

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <time.h>

typedef ssize_t (*orig_read_t)(int fd, void *buf, size_t count);
typedef ssize_t (*orig_write_t)(int fd, const void *buf, size_t count);

static orig_read_t original_read = NULL;
static orig_write_t original_write = NULL;

static void sleep_ms(unsigned int ms) {
        if (ms > 0) {
                struct timespec ts;
                ts.tv_sec = ms / 1000;
                ts.tv_nsec = (ms % 1000) * 1000000;
                nanosleep(&ts, NULL);
        }
}

__attribute__((constructor))
static void init() {
        original_read = (orig_read_t)dlsym(RTLD_NEXT, "read");
        if (!original_read) {
                fprintf(stderr, "Error loading original read function: %s\n", dlerror());
        }

        original_write = (orig_write_t)dlsym(RTLD_NEXT, "write");
        if (!original_write) {
                fprintf(stderr, "Error loading original write function: %s\n", dlerror());
        }
}

ssize_t read(int fd, void *buf, size_t count) {
        sleep_ms(200);
        return original_read(fd, buf, count);
}

ssize_t write(int fd, const void *buf, size_t count) {
        sleep_ms(200);
        return original_write(fd, buf, count);
}

First, compile the code into a shared library:

$ gcc -shared -fPIC -o my_readwrite.so my_readwrite.c -ldl

Then run a program with the library preloaded:

LD_PRELOAD=./my_readwrite.so find /tmp

You should notice that find prints results slower than normally.

Use Cases

testing - functions can be overridden to introduce arbitrary slow downs or random failures.
hot-fixes - shared library can be selectively overridden to change particular behaviour without rebuilding.
observability - this technique can be used to non-intrusively gather all sort of metrics and statistics.
alternative implementations - shared library can provide alternative implementations of overridden functions, e.g. malloc.
injecting custom code - preloaded shared libraries can inject arbitrary code on load time via constructor functions.

Other ways to preload a library (Linux only)

The LD_PRELOAD environment variable. What was described in this post above.
The --preload command-line option when invoking the dynamic linker directly. The dynamic linker is a normal executable file so it can executed manually like any other program. E.g. instead of executing ls one can do the same by executing dynamic linker and passing the executable as an argument: /lib64/ld-linux-x86-64.so.2 /bin/ls (full path to ls in required in this case). And this is what actually is being done by the system every time a dynamically linked executable is run. The dynamic linker accepts a few flags, one of them being --preload and it works the same as LD_PRELOAD environment variable.
The /etc/ld.so.preload file. This configuration file contains a list of white-space separated full paths to shared libraries that should be automatically preloaded by the dynamic linker. The big difference here is that this method preloads libraries for all processes whereas for the previous methods you could preload libraries only for selected processes.

Security implications

It's important to understand security implications of LD_PRELOAD. It can be used to override any system call, like open or execve and therefore can be used to hijack execution flow or hide malicious behaviour.

Linux provides some mitigations against that.
if the dynamic linker determines that a binary should be run in secure-execution mode (e.g. when executing a set-user-ID or set-group-ID program) preloading is very limited. E.g. pathnames containing slashes are ignored and shared objects are preloaded only from the standard search directories and only if they have set-user-ID mode bit enabled.

On FreeBSD LD_PRELOAD is ignored altogether for set-user-ID and set-group-ID programs.

Alternatives

Some of the uses described above might be achieved with the use of eBPF, but not al of them. eBPF is much more restricted, but also doesn't introduce the possible security issues of injecting arbitrary code.

Publish your first Hive Adventure

Serpent7776 — Mon, 03 Mar 2025 20:58:35 +0000

Hello fellow adventurers,

Today I'd like to announce an important update to the Hive Adventures platform, the place where anyone can create their own choose-your-own-adventure text-only game and publish it for other players.

Hive Keychain

Hive Keychain is a browser extension and mobile app that lets you access Hive powered websites, dApps, and services with your Hive account.

The website is now integrated with Hive Keychain. This allows you to publish games directly from your browser. All you need to do is to install and set up the extension.

Quality of life improvements

As well as this big change, you'll also find some important Quality of Life Improvements:

The site now remembers your game content across page loads. This makes it easier to iterate on your game design.
It is now easier to switch between writing mode and game testing mode.
An additional button has been added to the game test mode to reset the game to its initial state. This makes it easier to test all the paths in your game to make sure everything works as expected.

How to get started

Just go to https://try.hive-adventures.com/ and start creating. You can test your game freely locally in the browser. To publish, you'll need a Hive account and the Hive Keychain web browser extension.

Have fun playing and creating.

Play your first Hive adventure now

Serpent7776 — Tue, 07 Jan 2025 20:34:47 +0000

Hello fellow adventurers.

I'm building a hive-adventures, a platform that allows users to create, play, and share CYOA-style text-based games.

Last time I described the synchronisation method with the blockchain.

Today I'd like to announce an important next step: a place where you can play the games: https://hive-adventures.com/adventures

It's a simple website with a list of all stored adventures ready to be played.
Simply click on the game you want to play to start the adventure. The whole gaming experience takes place right in your browser, no further interaction with the server is needed.
The website has a simple and clean theme that helps you to immerse yourself in the adventure.

There are already some example games you can play.

The next big step will be to create an easy-to-use interface to publish games on the Hive blockchain.

Visit https://hive-adventures.com/adventures and start your adventure!

Have fun playing.

Synchronising hive-adventures with Hive blockchain

Serpent7776 — Fri, 15 Nov 2024 21:14:12 +0000

Hello fellow adventurers.

Today I'd like to get a bit more technical and describe how the hive-adventures will synchronise its data with the Hive blockchain.

The application will read the custom_json_operations of the following format:

{
  "type": "custom_json_operation",
  "value": {
    "id": "hive-adventure",
    "json": "...",
    "required_auths": [
      "anonymous"
    ],
    "required_posting_auths": []
  }
}

All these operations are stored locally and then presented to the players as HTML.

The id must be set to hive-adventure. The json field must contain the game text in the markdown format I described in the announcement blog post. Either required_auths or required_posting_auths must be set to the game's author. Currently only one author is supported.

On startup, the application looks at the largest block number stored in the local database. It then performs synchronisation in three steps:

First, the application enters massive sync mode.
Once started, the first step is to check the current head of the remote we're syncing with. If the remote's head is larger than ours, we synchronise with that remote, 1000 blocks at a time, until our head matches the remote's head.
When the massive sync is complete, the application will go into sync mode. It's similar to massive sync, but it synchronises one block at a time. After each block, it checks if the block time is less than 1 minute from the current time. If it is, the sync mode is finished.
After all this the application goes into live mode.
In live mode, the application synchronises with the remote one block at a time, just like in sync mode. This time, however, it sleeps between blocks to wait just long enough for the next block to be produced.
If a delay of 1,000 blocks or more is detected in live mode, the application will go back to massive sync mode. This shouldn't normally happen, but it can for example if the network goes down. This step is an important optimisation to quickly catch up with the current head in such situations.

The only place where an error will cause the application to quit is in the case of massive synchronisation. It's then up to systemd to restart the app after a while, in hope that the problem was temporary.

In all these modes, the application first tries to get the data from the hafah instance, and if that fails, from the witness APIs. Hafah is chosen as the primary data source because it allows to query only specific types of operations. In my case I'm only interested in custom_json_operations, so it serves as a slight network IO optimisation to fetch only those operations.

The synchronisation application is already running on my server as described above, but its data is not yet exposed to the public.

Please let me know if you have any thoughts on synchronising with Hive.

Try your first Hive Adventure now

Serpent7776 — Tue, 15 Oct 2024 20:19:21 +0000

Hello fellow adventurers!

I'm working on a choose-your-own-adventure (CYOA) text game creation platform built on Hive. If you haven't read it yet, here's the project announcement post.

Today I'd like to announce the first publicly available website at try.hive-adventures.com.

The goal

try.hive-adventures.com is a local testing environment where you can play and refine your adventure games right in your browser. Whether you're an experienced game designer or just starting to explore the world of interactive storytelling, this feature offers a safe space for you to experiment, iterate, and perfect your narratives. Think of it as a sandbox environment, where you can:

Experiment freely: take your time and explore various storylines, tweak dialogues, and change endings without the pressure of whether it's ready for an audience.
Preview and play your game as if you were the player, allowing you to catch plot holes, improve pacing, and enhance the overall flow of your adventure.
Catch any issues or errors to ensure that all paths, choices, and endings work exactly as intended.

All this without having to publish the game on the blockchain.

Whether you're building a sprawling fantasy epic with countless choices or a small, personal narrative, the ability to privately test your game is invaluable. By working iteratively, you can continuously refine your branching paths, dialogue options, and plot twists—without worrying about exposing unfinished work to your audience.

The current state

The current version is considered to be an alpha version and runs on a cheap VPS. It should be up and running though. Please report any errors.

The next steps

I'm currently working on the integration with Hive.
The game data will be read from the blockchain data, translated into an html page and this is what will be served to the user.
I have a rough version working on my local machine, but I need to work on some more integration details.

That's all for now. Head over to try.hive-adventures.com and create your first adventure today!

It's adventure time, but not any adventure, a Hive Adventure!

Serpent7776 — Tue, 03 Sep 2024 15:54:09 +0000

I'd like to announce the project I'm currently working on: a new choose-your-own-adventure (CYOA) text game creation platform built on Hive.

With this platform you can easily create and share your own interactive stories, where every decision the player makes affects the outcome of the game.
It is aimed at non-programmers, with an easy way to write stories that guide players through mazes of choices and thrilling outcomes.
It will allow you to create a web of possibilities that will keep the player engaged and eager to explore each path.

The game

The whole game is expressed as a markdown document. Here's how it looks like:

# Game title

## Start

Some text describing the situation.

- [choice A label](#choice-a-section)
- [choice B label](#choice-b-section)

## Choice A section

Some more text describing the effect user choice had on the world.

## Choice B section

Some other text describing some other effect user choice had on the world.

Heading contains the name of the game.
The sub-heading indicates the section of the game. Only one section is visible at a time.
Start is the first section of the game.

The choices are presented as a list of links to other sections. The choice label is what is presented to the user. The choice link is a slugified section label (this is a limitation of the markdown format - it cannot contain spaces). This basically means that all spaces are replaced by hyphens and the text is lowercased. The hash is optional and can be omitted. If the link is the same as the label, the link can be skipped and left blank. It's then automatically set to the slugified label.

That's it. You could probably start writing your first game straight away.

You can have as many choices as you like, though realistically you should limit it to 2-4 per section.
Currently the supported HTML tags are very limited. In particular, images, videos and audio are not available. External links are not supported either.
You can use any other markdown element you like, it doesn't have any special meaning.
There's no limit to the length of the game, but the standard Hive Blockchain rules apply.

Example game

# The Enchanted Forest

## Start

You find yourself at the entrance of a mysterious forest. The path splits in two directions.

- [Take the left path]()
- [Take the right path]()

## Take the left path

You choose the left path. It leads you to a clearing with a sparkling pond.

- [Drink from the pond](#drink)
- [Continue past the pond](#continue)

## Take the right path

You choose the right path. You come across an old, wooden cabin.

- [Knock on the cabin door](#knock)
- [Sneak around the cabin](#sneak)

## Drink

You drink the water and suddenly feel energized. You've gained magical powers!

**The End**

## Continue

As you walk past the pond, you discover a hidden treasure chest!

**The End**

## Knock

An old wizard opens the door and invites you in for tea and magic lessons.

**The End**

## Sneak

While sneaking, you trip on a root and fall into a portal, transporting you to another world!

**The End**

I hope this has sparked some interest. It's not available to the public yet. I have an initial version running locally on my machine, but it will take some time to get it ready for a wider audience.

Comparing C++ range libraries for filter+reverse case with non-trivial lambda

Serpent7776 — Mon, 03 Jun 2024 19:42:42 +0000

EDIT 2024-06-04: Fixed issue with flux implementation, thanks to tcbrindle for catching this.
EDIT 2024-06-24: Removed extra allocation from ranges libraries implementations. Thanks again to tcbrindle for suggesting this. This only affected the 'every other item' test.

C++ ranges are awesome as they often make the code simpler and easier to reason about. But how they compare against each other? To check that I did a micro-benchmark of c++ ranges for one particular use case I encountered in practice.

The scenario I tested was as follows:

I have an input data set that needs to be filtered based on non-trivial lambda.
Only a subset of this data is needed.
Then an index is added to the elements.
The last step is to reverse the sequence.

The repo with source code is here https://github.com/serpent7776/cpp-ranges-bench

It's basically filter+take+enumerate+reverse, but there are two tricky things here:

The non-triviality of the lambda being used for filtering. In the example code, such a lambda is often a trivial function. What happens when the cost of invoking such lambda is not not negligible?
Filter+reverse is a tricky combination. Filtering yields a result set of unknown size, which means that subsequent reverse cannot be done easily in the general case.

The ranges library needs to take both of these situations into account, otherwise it might introduce noticeable overhead.

Let's check it out.

The code uses catch2 to test for correctness and performance.
Catch2 is a unit testing framework for C++, but it also provides basic micro-benchmarking features.
Unfortunately, it doesn't provide any way to visualise the results, so I created a script and submitted PR adding plotting capabilities to Catch.

Tested implementations

I have reviewed the following implementations:

clike - a simple C-like implementation. This still uses c++ vector and its methods, but it's just a for loop with if statements. It collects the result in a vector and when it's done, it reverses it in-place.
algorithms - Manual implementation using STL algorithms. Instead of operating on the elements themselves, it allocates an extra vector with indices and operates on that. It only copies the elements in the last step.
boost_adaptors - Uses boost::adaptors. It provides very minimal feature set, but it's surprisingly capable. Due to the limitations of adaptors library, it allocates an extra vector of filtered elements and operates on that.
rangesv3 - Uses ranges-v3 - range library for C++14/17/20. Very similar to the implementation using std::ranges.
stdranges - Uses C++23 ranges. This is the shortest and cleanest implementation. The compiler I was using didn't have to implemented, so I used equivalent function from rangesv3 library. This may have affected the results.
fluxranges - Uses flux, a C++20 library for sequence-oriented programming. A bit more noisy compared to other range implementations.

I used the following versions of the libraries:

ranges-v3 0.12
fluxranges faf2e0f8b9f616001890a87b613f47313443f8e5
g++ stdlib, g++ (GCC) 13.2.1 20230801 All compiled in C++23 mode with the -O2 flag.

Test cases

I have generated some random data using the faker python library. The data is a set of usernames with an id and a set of connected names. I will search for usernames that match some criteria which involves looking through the connected names. This should be enough to consider the filter criteria as non-trivial.
To do the comparisons I have generated 10 000 rows of data.

The test cases I will check are:

all ismiths - select all items that have "ismith" as connection. There are 15 such entries in total.
5 ismiths - select at most 5 items that have "ismith" as connection.
empty result set - searches for an item that doesn't exist, yielding no results.
early single item - searches for all items that have "henry79" as connection. There's only one such item, early in the sequence.
late single item - search for all items that have "emilymclaughlin" as connection. There's only one such item, late in the sequence.
every other item - accept every other item, while also performing some operation on the input data.

Results

Flux and std::ranges are very similar with very good results. The only exceptions are 'early single item' and 'every other item' cases.
Ranges-v3 was generally on par with other solutions. There are two cases where it's slower: 'all ismiths' and 'early single item'.
boost adaptors are overall quite good. In 'all ismiths', 'empty result set' and 'late single item' are on par with fastest implementation and just slightly slower in other cases.
C-like solution was always on par or better than other solutions.

Some more details

Let's try to see some more details. I will look at the 'early single item' test. This case is interesting, because there are big differences between the implementations.

First, let's run the test case for a few more times. I'll make the test cases run 1000 times by adding this code to each implementation section.

auto _ = GENERATE(range(1, 1000));

This uses Catch2 data generator to execute the section multiple times, each time with a different value. In this case I'm not interested in the value itself, I'm only doing this to run the section again.

Now let's run perf stat on it. To collect the data I'll will only run this single test case and only one implementation at a time, ignoring the benchmarks. I do this by invoking the compiled binary with the test-case name, the -c option with a section name that specifies which implementation to use and the --skip-benchmarks option so that no benchmarks are run. You can read more about Catch2 command line usage.
perf stat is invoked with -o stats --append, which appends the perf data to stats file. Read more about perf stat usage.

: > stats  # truncate stats file
for t in clike algorithms boost_adaptors rangesv3 stdranges fluxranges; do
    perf stat -o stats --append ./read "early single item" -c $t --skip-benchmarks ;
done

I will then convert the stats file with a perf-stat-to-sc script I wrote. It will create a sc-im spreadsheet, which will make it easier to compare the results.

command	page_faults	cycles	instructions	insn_per_cycle	branches	branch_misses	backend_bound	bad_speculation	frontend_bound	retiring	real	user	sys
clike	1,417	9,763,699,969	27,310,955,366	27,310,955,366	6,611,235,612	84,103,168	7.2	22.2	21.2	49.4	3.573872222	3.437020000	0.119355000
rangesv3	1,399	11,634,561,888	29,211,691,985	29,211,691,985	7,103,602,542	123,617,995	8.1	22.9	22.8	46.2	4.132017782	4.001041000	0.116310000
boost_adaptors	1,394	10,715,252,768	28,262,420,566	28,262,420,566	6,857,638,043	104,769,454	7.0	24.6	21.7	46.7	3.909084680	3.807669000	0.083028000
fluxranges	1,394	10,775,098,223	28,262,194,528	28,262,194,528	6,857,615,031	104,014,045	7.6	25.1	21.4	46.0	3.965913088	3.814496000	0.132746000
stdranges	1,413	10,731,793,760	28,251,095,766	28,251,095,766	6,857,354,712	104,070,967	7.6	23.0	21.7	47.7	3.955925504	3.844894000	0.092972000
algorithms	7,623	10,706,979,451	28,341,604,071	28,341,604,071	6,867,444,605	104,582,547	6.9	25.5	21.5	46.1	3.911286798	3.801366000	0.089797000

The first thing to notice is the difference in the number of page faults. Most implementations had about 1400 faults. The exceptions are custom algorithms with ~7600 faults. The differences in cycles and instructions are quite small. It's a bit bigger for rangesv3 and smaller for C-like though. The number of branches is also similar, slightly higher for rangesv3 and slightly lower for C-like. There are bigger differences in branch misses. The C-like implementation has the least number of misses. Rangesv3 have the most misses.

But this collects stats from the entire program run. The issue is that most of the time is spent on reading and parsing the input file, which is the same in all cases. I want to collect stats only for the single implementation function. For this I can use valgrind's cachegrind with additional code instrumentation.
I will add CACHEGRIND_START_INSTRUMENTATION and CACHEGRIND_STOP_INSTRUMENTATION just before and after the implementation function I'm interested in and then run the binary with --instr-at-start=no which will disable instrumentation at startup and will wait for the first CACHEGRIND_START_INSTRUMENTATION to enable it.
Read more about cachegrind usage.

    SECTION("clike") {
        auto _ = GENERATE(range(1, 1000));
        CACHEGRIND_START_INSTRUMENTATION;
        const std::vector<Out> found = clike(data, accept, max_items);
        CACHEGRIND_STOP_INSTRUMENTATION;
        REQUIRE(found == expected);
    }

valgrind --tool=cachegrind --instr-at-start=no ./read 'early single item' -c fluxranges

This is the data I got:

implementation	instructions executed
clike	128 871
algorithms	274 757 967
boost_adaptors	274 757 967
rangesv3	549 464 985
stdranges	274 757 967
fluxranges	274 757 967

It's a bit strange that all the results except for clike and rangesv3 are the same.
But this confirms that C-like executes far fever instructions than other implementations and rangesv3, the most.

Conclusions

Ranges are really cool. They often make the code easier to read and reason about. But they do not always fit all the use cases. Sometimes, in some very specific cases, a for loop might give more control that is needed to get some better results. In general though, it's usually better to start with a ranges, as they will be good enough in most cases.