DEV Community: Stefan Zipproth

I built a C++ Reasoning Engine that holds 1.7TB of Knowledge in RAM – zelph v0.9.2

Stefan Zipproth — Mon, 05 Jan 2026 20:34:44 +0000

Hi DEV Community! 👋

I’ve just released version 0.9.2 of zelph, a semantic network system capable of encoding inference rules within the network itself.
It’s open source and designed for high-performance logical reasoning.

What makes it special?

In zelph, rules are facts. Unlike traditional systems where logic is external code applied to data, zelph treats relation types (like "transitivity") as first-class nodes in the graph. This allows for powerful meta-reasoning where you can make statements about statements.

A Taste of Logic: Try it yourself

Before we talk about terabytes of data, let’s look at how it feels to use. zelph has an interactive CLI.

1. Basic Inference
You can define facts and rules on the fly. Note that none of the items below are predefined; zelph learns them as you type:

# Start the interactive shell
zelph

Berlin "is capital of" Germany
Germany "is located in" Europe

# Define a rule: If X is capital of Y, and Y is in Z, then X is in Z
X "is capital of" Y, Y "is located in" Z => X "is located in" Z

Output:

«Berlin» «is located in» «Europe» ⇐ («Germany» «is located in» «Europe»), («Berlin» «is capital of» «Germany»)

zelph automatically inferred the location of Berlin based on the rule.

2. Meta-Reasoning (Transitivity)
This is where it gets interesting. Instead of writing a specific rule for every transitive relation, we can define "transitivity" itself as a concept in the network.
(Note: ~ is shorthand for "instance of" or "is a").

# Define a general rule for ANY transitive relation R
R ~ "transitive relation", X R Y, Y R Z => X R Z

# Now, declare that "greater than" is a transitive relation
> ~ "transitive relation"

# Give it some numbers
6 > 5
5 > 4

Output:

«6» «>» «4» ⇐ («5» «>» «4»), («6» «>» «5»), («>» «~» «transitive relation»)

Because > is an instance of a transitive relation, zelph applied the meta-rule and deduced that 6 > 4.

3. Detecting Contradictions
You can also define what is impossible using !.

# Rule: If X is opposite of Y, a thing A cannot be both X and Y
X "is opposite of" Y, A ~ X, A ~ Y => !

If you (or an imported dataset) try to assert facts that violate this, zelph detects the logical contradiction immediately.

The Benchmark: Debugging 1.7 TB of Knowledge

To test the limits, I optimized zelph to process the entire Wikidata Knowledge Graph (over 113 million entities). But this wasn't just a performance test—it was a logic audit.

Using the rules described above (like disjointness and transitivity), zelph identified hundreds of megabytes of logical contradictions hidden within the live Wikidata dataset.

Performance: ~1000x speedup in reasoning compared to early prototypes.
Scale: A single-pass analysis of the full graph (requires ~256 GB RAM for the full dump, but scales down for smaller tasks).
Results: We detected massive amounts of Split Order Violations (broken hierarchies) and Disjointness Errors (items belonging to mutually exclusive categories).

You can dive into the results and see the actual logical "bugs" found in the Grant Report.

Now available everywhere

Thanks to a recent sprint (and a Wikimedia Grant), you can install it natively to play with the logic engine yourself:

Windows: choco install zelph
macOS: brew tap acrion/zelph && brew install zelph
Linux: pikaur -S zelph (AUR)

Future Directions: Lisp & "Code as Data"

This is where I’d love your input.
The current custom scripting language (shown above) works well, but I plan to embed a Lisp dialect (specifically Fennel or Janet) to perfectly mirror the semantic network structure where rules and facts are identical nodes.

This aligns perfectly with the "Code as Data" philosophy.

Why Fennel? Incredible performance (LuaJIT) and seamless synergy with my other project, nexuslua.
Why Janet? Modern, great macro system, clean C API.

If you are interested in C++20, Semantic Web, or Lisp embedding, check out the repo!

🔗 GitHub: https://github.com/acrion/zelph
🔗 Grant Report & Architecture: https://zelph.org/grant-report

Happy coding!
Stefan

Cbeam: Unmatched Concurrency, Cross-Library Data Sharing, and More in Modern C++

Stefan Zipproth — Sat, 18 Jan 2025 14:30:38 +0000

Introduction

Cbeam is a modern, cross-platform, header-only C++ library designed to simplify concurrency, memory sharing, serialization, and more—without depending on Boost. It draws inspiration from Boost’s coding style but remains lightweight and easy to integrate on Linux, macOS, and Windows. Four standout features illustrate Cbeam’s capabilities:

Asynchronous Message Dispatching (using message_manager)
Cross-Platform Shared Memory (using interprocess_shared_memory)
Stable Reference Counting Across Shared Libraries (using stable_reference_buffer)
Interoperable Containers Across Compiler Boundaries (with “stable interprocess” containers)

We’ll take a closer look at message_manager and show how it enables straightforward, thread-safe message handling. But first, here’s a quick overview of the other three “spotlight features”:

Cross-Platform Shared Memory: interprocess_shared_memory unifies Windows (native file mapping) and Unix (POSIX shm_open) shared memory under a simple API. No leftover OS objects to clean up and no complicated file permissions for C:\ProgramData\boost_interprocess.
Stable Reference Counting: stable_reference_buffer places the reference count in a safe, process-wide map, solving the classic issue where std::shared_ptr across shared library boundaries can lead to mismatched counts and double frees.
Interoperable Containers: “stable interprocess” containers (like stable_interprocess_map) allow sharing data between libraries or processes compiled with different toolchains. Serialization logic handles potential ABI mismatches.

You might also explore nested_map for hierarchical data (configuration trees, JSON-like structures) if your application needs more flexible, nested storage.

Deep Dive: Asynchronous Message Dispatching with `message_manager`

Core Idea

The Cbeam message_manager helps you send messages to one or more handler threads. Each message type is identified by a numeric ID, and each ID can be associated with one or multiple handlers. Internally, message_manager maintains safe queues, providing FIFO, FILO, or RANDOM dispatch order. It includes mechanisms to wait until all messages for a given ID have been processed, which is especially useful for clean synchronization points.

Why Not Just Use `std::thread` and `std::condition_variable`?

Manual threading requires you to handle queueing, synchronization, and error handling. With message_manager, you reduce boilerplate and minimize risks like deadlocks or race conditions. It’s header-only, has no external dependencies, and works cross-platform, adding minimal performance overhead while significantly reducing the risk of concurrency bugs. In addition, message_manager can integrate with Cbeam’s serialization features to pass data across shared libraries that may not be ABI-compatible.

Step-by-Step Example

Below is a simplified prime-checking program to illustrate the design. We define two message IDs:

check_prime: A handler that checks if a number is prime and, if so, sends it to the second handler.
count_prime: A handler that increments a shared counter each time it receives a “prime” message.

We enqueue numbers from 1 to 100,000, wait for all tasks to finish, then dispose the handlers.

#include <cbeam/concurrency/message_manager.hpp>
#include <atomic>
#include <cmath>
#include <iostream>

// Quick prime test
static bool is_prime(uint64_t n) {
    if (n < 2) return false;
    if (n % 2 == 0) return (n == 2);
    for (uint64_t i = 3, stop = static_cast<uint64_t>(std::sqrt(n)); i <= stop; i += 2) {
        if (n % i == 0) return false;
    }
    return true;
}

constexpr size_t check_prime = 1;
constexpr size_t count_prime = 2;

int main() {
    // 1) Create a message_manager that handles messages carrying a
    //    64-bit integer. (Side note: message_manager can integrate
    //    with serialization to pass data across shared libraries
    //    that might not be ABI-compatible.)
    cbeam::concurrency::message_manager<uint64_t> mm;
    std::atomic<size_t> prime_count{0};

    // 2) Define a handler that increments the prime count.
    //    (Note that we ignore the actual number here — only the
    //    event matters.)
    mm.add_handler(count_prime, [&](uint64_t) {
        ++prime_count;
    });

    // 3) Define a handler to check numbers and forward prime ones
    mm.add_handler(check_prime, [&](uint64_t number) {
        if (is_prime(number)) {
            mm.send_message(count_prime, number);
        }
    });

    // (You could register multiple add_handler calls with
    //  the same ID to distribute the work across multiple 
    //  threads - just run the above code `n` times.)

    // 4) Enqueue many numbers for checking
    for (uint64_t i = 1; i <= 100000; ++i) {
        mm.send_message(check_prime, i);
    }

    // 5) Wait for all messages to be processed
    mm.wait_until_empty(check_prime);
    mm.wait_until_empty(count_prime);

    // 6) Dispose the handlers
    mm.dispose(check_prime);
    mm.dispose(count_prime);

    // Print the total primes found
    std::cout << "Total primes in [1..100000]: " << prime_count.load() << std::endl;
    return 0;
}

Highlights of `message_manager`

Multiple Handlers per Message ID: Each add_handler spawns an independent thread. If you want parallel processing for a single message ID, simply register multiple handlers for that ID.
Ordering Options: By default, messages are queued in FIFO order, but you can choose FILO or RANDOM if needed for particular algorithms.
Cleaner Synchronization: wait_until_empty(ID) ensures all tasks in that queue finish, letting you safely finalize results.

Feedback and Discussion

Interested in trying Cbeam? We welcome feedback, ideas, and discussion on our Discord server. Whether you’re tackling concurrency challenges, bridging incompatible compilers, or simplifying shared memory usage, Cbeam aims to offer robust solutions under a permissive, modern C++ approach.

Check out Cbeam.org for detailed Doxygen docs, unit tests, and more examples to help streamline your next cross-platform C++ project.

Ditana GNU/Linux: Unmatched Configuration Flexibility and Generic Hardware Detection

Stefan Zipproth — Tue, 07 Jan 2025 15:21:23 +0000

Ditana GNU/Linux distinguishes itself from other distributions by providing a clean, JSON-driven configuration approach and a modern installer written in Raku. In this post, we’ll show you how Ditana uses JSON-based hardware detection and handles complex settings—such as the ZFS filesystem—through a concise, maintainable system that’s easy to extend. We’re also actively seeking developers and contributors who want to help shape Ditana’s future!

Hardware Detection in JSON

Instead of scattering detection logic across multiple scripts, Ditana manages everything in JSON files. Each entry describes a specific hardware or system feature, along with the Raku code that runs at install time. For example:

{
  "name": "uefi",
  "detect": "'/sys/firmware/efi'.IO.e",
  "required-by-chroot": true
},
{
  "name": "total-ram-gib",
  "detect": "('/proc/meminfo'.IO.lines.grep(/MemTotal/).first.words[1] / 1024 / 1024).ceiling"
},
{
  "name": "avx2-available",
  "detect": "'/proc/cpuinfo'.IO.lines.first(*.contains('flags')).contains('avx2').so"
}

The detect property contains Raku expressions that run at install time, making the logic straightforward and modular.

Advanced Configuration with `settings.json`

Beyond hardware detection, settings.json also handles user-configurable options like filesystems or kernel selections. Here’s an example entry for ZFS:

{
  "name": "zfs-filesystem",
  "dialog-name": "File System",
  "short-description": "ZFS   - Snapshots, Self-Healing Storage & RAID-Z Support",
  "long-description": "ZFS: Supports atomic snapshots with a pre-configured snapshot solution instead of Timeshift. Designed for high data integrity with automatic self-healing. The installer will switch to the recommended Long-Term Support Kernel automatically, because ZFS requires DKMS modules (see Expert Settings → Kernel Selection menu available for selection later).",
  "arch-packages": [
      "zfs-utils"
  ],
  "spdx-identifiers": "CDDL",
  "required-by-chroot": true,
  "available": "`install-standard-lts-kernel OR install-standard-stable-kernel OR install-zen-kernel`",
  "default-value": "`install-standard-lts-kernel`"
}

Tristate

Ditana utilizes a custom Tristate class—capable of True, False, and Any (undefined)—to manage values that are unknown or not yet defined. This flexibility is crucial for Ditana’s highly adaptable installer, which responds to both hardware attributes (like BIOS or UEFI, or drivers that may restrict kernel choice) and a wide range of user preferences.

Tristate Logic Explained Through Its Unit Test

In Raku, you can embed unit tests directly in the class, and they’ll run at compile time without requiring a dedicated test framework. For example:

CHECK
  my $t = Tristate.new(True);
  my $f = Tristate.new(False);
  my $u = Tristate.new(Any);

  my @tests = (
      [$f AND $u, 'False', 'False and Any'],
      [$u AND $f, 'False', 'Any and False'],
      [$t AND $u, '(Any)', 'True and Any'],
      [$u AND $t, '(Any)', 'Any and True'],
      [$t OR $u,  'True',  'True or Any'],
      [$u OR $t,  'True',  'Any or True'],
      [$f OR $u,  '(Any)', 'False or Any'],
      [$u OR $f,  '(Any)', 'Any or False'],
      [NOT $u,    '(Any)', 'not Any'],
      [NOT ($u OR $u OR $t or $u), 'False', 'not (Any or Any or True or Any)']
  );

  for @tests -> [$result, $expected, $description] {
      unless $result.Str eq $expected {
          die "Test '$description' failed:\n  Expected: $expected\n  Got: $result.Str()";
      }
  }

This demonstrates how Tristate gracefully handles undefined conditions.

Why This is Relevant for Ditana

Ditana’s flexible configuration possibilities require a way to resolve dependencies between settings. For example, with ZFS, the right boot manager (GRUB or ZFSBootMenu), kernel, and optional encryption settings all need to align. Things can get especially complex if you require special drivers or rely on a legacy BIOS. Thanks to this JSON-based configuration and the Tristate logic, Ditana ensures that any combination of user choices leads to a valid, bootable system. If something isn’t available, the installer won’t let you pick it—and it will explain why.

CPU Vulnerability Mitigation Options

In line with Ditana’s design philosophy, our distribution enables CPU vulnerability mitigations in High Security Mode by default. During installation, Ditana’s installer allows you to configure each detected CPU vulnerability individually, offering the option to maintain these enhanced security settings or revert to the default kernel configurations.

Default Kernel Settings vs. Ditana’s Mitigations

The default kernel settings provided by the Linux kernel developers aim to balance security and performance. However, certain security vulnerabilities remain if you rely solely on these defaults. Ditana prioritises robust security by enabling mitigations that address these vulnerabilities via the officially documented kernel parameters. For most users, the performance implications of Ditana’s mitigations are negligible, while the security benefits are substantial.

Customization and Flexibility

We understand that some users may prefer to optimize their systems for maximum performance, even if it means accepting potential security trade-offs. Ditana’s installer framework therefore empowers you to adjust CPU vulnerability mitigation settings according to your preferences. Comprehensive explanations for each mitigation option are available through the installer’s Help button and detailed in the documentation.

Hardware-Specific Configuration

Ditana's hardware detection system ensures that mitigation options are only presented for the specific vulnerabilities your CPU actually faces—and only for those where the Linux kernel itself does not use the maximum possible mitigation by default. For more details on which vulnerabilities are detected and which are not, see our documentation.

By choosing Ditana’s default mitigation settings, you ensure that your system maintains a high level of security against known CPU vulnerabilities. We recommend keeping these settings enabled unless you have specific requirements that justify loosening them for performance considerations.

{
  "name": "is-spectre-v2-vulnerable",
  "detect": "given '/sys/devices/system/cpu/vulnerabilities/spectre_v2'.IO { .e && .slurp.chomp ne 'Not affected' }"
}

Example from settings.json to detect whether your CPU is vulnerable to Spectre V2.

Above: A preview of Ditana’s installation summary dialogue, showing the High Security Mitigations that would be configured in this example. You can decide to adopt Ditana’s mitigations or stick to the Linux kernel defaults.

Dynamic NVIDIA Driver Detection

When it comes to NVIDIA graphics cards, Ditana employs a novel approach to identify whether a legacy driver is needed or if the current proprietary driver is more appropriate. During installation, the installer downloads and parses the official NVIDIA legacy GPU page at https://www.nvidia.com/en-us/drivers/unix/legacy-gpu. If this page is unavailable or the parser requires updates (we run tests to validate the downloaded data), a cached and filtered version from our repository is used instead. By comparing the detected GPU against the data in this page, the installer can determine if the user’s GPU necessitates a legacy driver. In cases where the required driver version is older than 470, the open-source Nouveau driver is recommended.

The default installation always uses the proprietary DKMS version of the NVIDIA driver, and the user can override the driver selection in the installer, for example to use the NVIDIA open-source driver. However, impossible combinations are automatically disallowed. Ultimately, this system ensures that each NVIDIA card is matched with the safest and most compatible option available, while still preserving user choice where it makes sense.

A Raku-Based Installer

The Ditana installer is written in Raku, combining the succinct feel of shell scripting with modern error handling and concurrency features. Together with JSON-based detection and user-configurable settings, this approach keeps the code organized yet powerful—proving that advanced installation logic doesn’t require massive shell scripts.

Give Ditana a Spin!

If you’re looking for an Arch-based distro that dares to think differently, Ditana might be your new best friend. From hardware detection to advanced filesystem setups, it even offers a local or cloud-based AI assistant that can execute commands after confirmation (computer use). For ZFS, an optional auto-snapshot mechanism for easier maintenance is provided. Overall, currently there are 21 packages we developed for Ditana in its dedicated Arch repository (either mandatory or optional).

And if you’re interested in shaping Ditana’s future, we’d love your help! Whether you’re a developer, documentation enthusiast, or just passionate about Linux, your contribution will be greatly appreciated. The project is split into multiple Arch packages, allowing you to pick the components you want. Plus, we have thorough documentation (both inside the installer and on the website) so you’ll always understand the why behind each setting.

Questions or Feedback? You are welcome to drop a comment below! Ditana is constantly evolving, and we’re eager to hear your thoughts on improving our JSON-based detection, Tristate logic, or Raku-driven installer. We’re actively seeking collaborators who share our vision—come join us on this journey!

Join the conversation on our dedicated Ditana Discord Server, where we discuss features, share updates, and brainstorm ideas together.