OptimaOS: A Rust Kernel That Boots on Real x86_64 Hardware

Antony Developer — Mon, 27 Apr 2026 08:52:26 +0000

The Problem: Fragmentation Through Forking

Android is a Linux fork. Embedded distros are forks with custom patches per SoC. AI stacks run on separate codebases. Each fork means a separate security audit, a separate team, and its own accumulating regression debt.

The bet: one kernel binary for all scenarios. Profile differences are runtime policy, not a fork of the codebase. This is locked in ADR-0002 as Single Kernel + Runtime Profile Overlays.

It Boots on Real Hardware

POST UEFI MODE
MM_OK  INTR_OK  SCHED_OK

That's on real x86_64 — not QEMU.

UEFI bootloader → kernel launches stably
ExitBootServices passes cleanly
Post-UEFI runtime is alive: heartbeat, progress panels, hardware tick source (SRC H)
PS/2 keyboard control: P pause/resume, N single-step, R reset

Architecture

The split is hard:

Kernel (mechanisms) — never changes per config:

Memory manager + MMU page tables (PML4 → PDPT → PD → PT)
Scheduler (processes, threads, SMP)
IPC bus with typed endpoints
Capability graph with quotas and TTL
Syscall ABI (optima_syscall_v0)
Page fault handler

Profiles (policy) — loaded at runtime:

Policy rules: what each process is allowed to do
Scheduler parameters (latency-first for desktop, throughput-first for server)
Allowed syscall patterns

One kernel binary. On boot, a policy file loads and policy-service applies it on top. No kernel recompilation.

Kernel Internals

kernel-core is a single Rust workspace crate, ~33K lines. #[forbid(unsafe_code)] is not a guideline — it's a compiler-enforced ban. All unsafe is isolated in the HAL layer (hardware/mod.rs) and explicitly annotated. Total: ~600 lines of unsafe across the entire kernel.

Memory

memory.rs — base manager: mmap/munmap/protect with PROT_READ/WRITE/EXEC
mm.rs — full x86_64 page tables: PML4 → PDPT → PD → PT → 4KB page
page_fault.rs — handler for interrupt vector #14, parses PageFaultErrorCode (P/W/U/RSVD/ID bits), feeds into runtime IDT
Memory isolation levels per ADR-0008: Level 1 (Capability IPC) , Level 2 (MPU Regions) stub, Level 3 (MMU Page Tables)

IPC

Process A ──send(endpoint_B, data)──> IpcBus ──enqueue──> Process B
                                          │
                                     capability check
                                     type check
                                     rate limit
                                     audit log

Each endpoint has an owner PID and a message queue (VecDeque). Messages carry from_pid, payload, and a list of delegated capabilities — so capabilities can be transferred through IPC without bypassing the permission system.

QEMU baseline: p50 = 900ns, p95 = 1700ns, p99 = 2500ns, throughput = 904k req/s. Under load (4 concurrent senders, 64-byte messages): less than 2x latency increase. Real hardware numbers are next.

Security is not optional: capability check, type check, per-session and per-method rate limiting, audit log at trust boundaries.

Capabilities

pub struct Capability {
    pub id: CapId,
    pub owner_pid: u64,
    pub resource: Resource,      // Endpoint(u64) | MemoryRegion(u64) | Process(u64)
    pub permission: Permission,  // read / write / manage
    pub quota: ResourceQuota,
    pub created_at: u64,
    pub expires_at: Option<u64>, // TTL for temporary caps
    pub priority: QuotaPriority,
    pub violation_count: u32,
}

ResourceQuota sets limits: max_memory_bytes (1 GB default), max_ipc_per_second (10K/sec), max_threads (256), max_file_handles (1024), max_capabilities (512).

On quota violation — priority degrades: High → Normal → Low → Degraded. This is DoS protection at the kernel level. Temporary capabilities with TTL invalidate automatically — no permission garbage collector needed.

23 tests cover all branches including boundary values and quota combinations.

Scheduler

Task states: Ready / Running / Blocked / Terminated.

Context switch: save registers → load CR3 (if different address space) → TLB flush → restore registers → jump to new stack.

SMP support via SmpManager with PerCpuData, CpuInfo (family/model/stepping/features), and IPI: Halt, Init, Call, Resched.

Scheduler policy is profile-driven: home → Interactive, server → Throughput.

Boot Chain: Two-Layer Architecture

ADR-0003 defines Hybrid Staged Console Integration:

Stage A (UEFI shim): transport + diagnostics only. Publishes input events, renders output frames from runtime. No business logic.
Stage B (kernel-core runtime): the single point of command execution. One parser/dispatcher for both host and device.

The boundary is console_proto=v1. Breaking changes require a new major version and a new ADR.

BootData ABI v2: passes the real memory map (region list), not just aggregates. Self-check handoff (SELFCHK) + error codes — state transfer verification between bootloader and kernel.

Lifecycle: BootInit → ShimReady → RuntimeAttach → Interactive → Degraded

On input path degradation, the system stays bootable in diagnostics-mode. Runtime state is not affected by transport-level errors.

Linux ABI: Incremental

linux-compat maps Linux syscalls to optima_syscall_v0. It depends on kernel-core explicitly — the only such exception.

L1: clone, exit, nanosleep, mmap, munmap, sendmsg, recvmsg, signals (minimal), epoll (minimal)
L2-A: fd lifecycle (open/close/read/write), dup/dup2, poll/epoll_wait
L2-B: signal masks, pending queue
L2-C: epoll_ctl(DEL/MOD), extended epoll semantics

129 tests, 100% pass. Linux bridge API is marked draft — stability will be announced separately. Each new syscall expands attack surface, so each tier closes with compatibility matrix tests.

Security Architecture

These are not feature checkboxes — they're architectural invariants.

No global root. Every action requires an explicit capability with owner PID verification.

Typed IPC. Wrong message type → rejected in kernel, not userspace.

Quota-based DoS protection. Process exceeding limits degrades in priority — doesn't crash, doesn't block.

Tamper-evident audit log. Hash-chain integrity for all security events: AUTHZ_DENY, CAPABILITY_GRANT, CAPABILITY_REVOKE, IPC_AUTH_FAIL, POLICY_BYPASS_ATTEMPT, KERNEL_PANIC, SERVICE_CRASH_RECOVERY.

Post-quantum crypto (architecture):

pub enum PqcAlgorithm {
    MlKem512, MlKem768, MlKem1024,  // NIST Level 1/3/5
    MlDsa44, MlDsa65, MlDsa87,      // NIST Level 2/3/5
}

Crypto is a userspace service. The kernel only knows about capabilities. Migrating from ECDSA to ML-KEM means updating one userspace binary — no kernel rebuild. In a monolithic kernel, that touches the kernel, lsass, protected processes, and firmware.

Licensing: GPL-3 + Dual License

Initially planned as MIT/Apache. Changed to GPL-3 + commercial dual license.

Why GPL-3 over MIT:

Copyleft at the kernel level works differently than in libraries. If someone takes kernel-core, modifies it, and ships it in a device — GPL-3 requires publishing changes. MIT doesn't — a vendor takes the kernel, adds patches for their SoC, and disappears into a closed branch forever.
Anti-tivoization (Section 3): GPL-2 allows manufacturers to lock hardware so users can't run modified kernels on their own devices. GPL-3 prohibits this. For a project targeting Edge and IoT, this is not abstract.
Explicit patent grant: distributing the code grants recipients rights to any patents that might be infringed. MIT offers no such protection.

Why a commercial license on top:
GPL-3 blocks companies that want to embed OptimaOS in a proprietary product without publishing changes. Dual licensing solves this: community gets GPL-3, commercial users pay for the right to keep changes private. Standard scheme — same as Qt, MySQL, MongoDB.

CLA for contributors: dual licensing requires a single rights holder. All contributors sign a CLA: they keep their copyright but grant the project the right to relicense their code under the commercial license. Without CLA before the first external patch, dual licensing becomes legally impossible. This is locked in CONTRIBUTING.md and README.md.

Current Status

Component	Status
Kernel-core (memory + MMU, scheduler, IPC, capability, syscall)	Working prototype
Home / Server profiles	Done
Linux ABI L1 + L2 (129 tests)	Done
x86_64 page tables (PML4→PT, page fault handler)	Implemented
xHCI USB host controller	Full implementation (QEMU)
UEFI bootloader + BootData v2 ABI	Done
Boot on real x86_64	Working
Hardware interrupts (PIC/PIT/IDT, hardware ticks)	Working
Scheduler in post-UEFI hardware runtime	Working
GUI framework (Desktop, Start Menu, File Browser)	Basic implementation
Crypto-service (PQC)	Stub, full architecture
Physical security (IOMMU, MemoryZeroize)	Stub, full architecture
On-device smoke harness (real task execution)	In progress

847 tests, 100% pass.

P0 performance baseline (QEMU, February 2026): IPC p95 ≤ 25 µs, context-switch p95 ≤ 35 µs, throughput ≥ 30k req/s, tail p99 ≤ 80 ms. Real hardware numbers — next milestone.

What's Next

Real task execution on hardware. The scheduler demonstrates the model (TASK1…TASK4, RUN/RDY/SLP) but doesn't run real tasks yet. Next step: task execution with IPC between processes on physical hardware. This closes the on-device smoke harness.

Kernel module decomposition. runtime_scheduler.rs extracted, runtime_arch.rs started. Hardware bring-up showed where boundaries were fuzzy.

Linux L2 completion. L2-C done, next: signalfd/eventfd. After stabilization: Android Binder IPC on optima_syscall_v0 in backlog.

Long term: Win32 L3. The most complex layer, distant horizon — but the architecture is already designed for it.

Global hypothesis: one kernel binary running across desktop, server, and Edge without performance degradation. Real hardware boot is the first data point. Real task execution is next.

Code:

GitHub: https://github.com/dev993848/optimaos

Development, documentation, and this article were a collaboration between the author, Claude, Codex, DeepSeek, Qwen, GLM, and other AI systems.

DEV Community: Antony Developer