Rotifer v0.2: IR Compiler & Algebra Parallelism

#ai #webassembly #compiler #opensource

v0.2 turns genes into first-class WASM citizens with content-addressable identity, adds true parallelism to the algebra engine, and publishes the Rust core as an independent crate.

Why IR Matters

Until v0.2, genes existed as TypeScript functions running in Node.js — useful for rapid prototyping, but fundamentally limited. They weren't portable across runtimes, couldn't be sandboxed for security, and had no verifiable identity. The Intermediate Representation (IR) layer changes this equation entirely.

By compiling genes to WASM with custom metadata sections, each gene becomes a self-contained, runtime-agnostic artifact. A gene compiled on your laptop today can execute tomorrow in a cloud serverless function, a browser sandbox, or a peer's machine — unchanged and unmodified. This is not just a performance optimization; it's the architectural prerequisite for every feature that follows: cloud publishing (v0.4), cross-Binding portability (v0.6.5), and eventually P2P gene transfer (v0.9).

IR Compiler Pipeline

The new rotifer-core::compiler module brings full Intermediate Representation support:

Custom WASM sections — rotifer.version, rotifer.phenotype, rotifer.constraints, rotifer.metering
Content-addressable identity — every gene gets an irHash computed via SHA-256 over its IR content. Identity is derived from what a gene is, not what it's called.
IR verifier — static validation of exports, prohibited instructions, and memory limits
5 genesis WASM genes — echo, wrap, search, summarize, translate compiled to real IR

rotifer compile --wasm   # full IR compilation with phenotype update

Why Content-Addressable Identity?

Traditional package managers identify code by name and version — lodash@4.17.21. This creates a trust chain problem: you trust the registry, which trusts the publisher, who might have compromised credentials. The recent wave of supply-chain attacks (ua-parser-js, event-stream, colors.js) demonstrates how fragile this model is.

Content-addressable identity eliminates the weakest link. When a gene's identity is sha256:e3b0c44298fc..., you can verify that the binary you received is byte-for-byte identical to the one that was evaluated in the Arena. No registry trust required. No publisher trust required. The hash is the identity.

We chose SHA-256 over alternatives for practical reasons. BLAKE3 is faster but lacks universal tooling support. SHA-3 is newer but offers no meaningful advantage for our payload sizes. SHA-256 is the hash function that Docker, Git, IPFS, and npm's integrity checks all converge on — using it means every developer's existing mental model and tooling transfers directly.

The hash covers the full WASM binary including custom sections, so any modification to code, phenotype schema, or resource constraints produces a new identity. This means the same source code compiled with different constraints is correctly treated as a different gene.

Algebra Parallelism

The Par operator now uses true CPU parallelism via std::thread::scope instead of sequential execution. When your genome says "run these genes in parallel," they actually run in parallel.

Implementation Details

True parallelism in a WASM sandbox environment is non-trivial. Each gene runs in its own wasmtime Store instance with independent fuel counters and memory limits. The Par operator spawns one OS thread per gene using Rust's scoped threads, joins them with a configurable timeout, and collects results.

The key design decision was isolation granularity: a misbehaving gene in a parallel composition cannot starve its siblings. If one gene exceeds its fuel budget or panics, only that gene's result is marked as failed. The others complete normally and their results are merged according to the --par-merge strategy (first-wins, merge-all, or custom reducer).

This matters for agent reliability. An agent genome like Par(web-search, file-lookup, cache-check) should degrade gracefully when one source is slow or broken, not fail entirely.

crates.io

rotifer-core is now published as an independent Rust crate with full rustdoc coverage on all 150+ public API items. Rust developers can build on the Rotifer type system and engines directly.

Publishing as a standalone crate serves two architectural purposes. First, it enables alternative frontends — you can build a Rust-native CLI, embed the engine in a game server, or integrate it into an existing DevOps pipeline without depending on the TypeScript CLI layer. Second, it enforces a clean separation between the protocol's type system (specification-driven, stable) and the CLI's user experience (evolving, opinionated). Changes to the CLI cannot accidentally break the core invariants.

CLI Upgrades

rotifer compile — full IR compilation with --wasm flag, phenotype update, and compile-result.json output
rotifer arena watch — real-time ranking diff monitoring with summaries

Fixes

Try operator now correctly returns the primary result on success (was re-executing the primary gene)

By the Numbers

22 → 180 tests across 10 modules
New dependencies: rmp-serde, wasm-encoder, wasmparser

Looking Ahead

v0.2 established the IR foundation that every subsequent release builds upon. The compilation pipeline unlocked TypeScript-to-WASM compilation in v0.3. Content-addressable identity became the trust anchor for the cloud registry in v0.4. And the parallel algebra engine grew into the full five-operator composition system that powers multi-gene agents today.