Small language models (SLMs) have crossed an important threshold: models like SmolLM2-135M, Qwen 2.5 0.5B, and **Llama 3.2 1B **deliver competent local reasoning inside a 400MB to 1.5GB memory footprint.
Yet, trying to deploy these models on single-board computers (SBCs) like the Raspberry Pi 4 or 5 reveals an uncomfortable truth: the edge AI tooling stack is fundamentally broken.
In this post, I want to walk through the architecture of litrun, an open-source lightweight runtime engineered specifically for embedded Linux, and examine the systems engineering details that desktop-centric runners get wrong.
The 3 Traps of Edge AI Today
Trap 1: The Desktop Daemon Penalty
Desktop tools like Ollama were designed for Apple Silicon Macs and workstations. On an 8GB or 4GB Raspberry Pi, a background runtime that consumes 150MB+ before loading a single weight wastes critical memory that should be reserved for the KV cache.
Trap 2: Blind Allocations & Kernel OOM Panics
On Linux, memory exhaustion rarely leads to a clean error. When an LLM runtime attempts to allocate memory beyond available physical RAM, Linux starts aggressively swapping. On an SD card or slow eMMC, this causes total I/O thrashing, freezes the USB and network buses, and triggers an unrecoverable kernel OOM panic.
Trap 3: The Arcane Compilation Maze
Raw inference engines like llama.cpp are blazing fast, but expecting embedded engineers to maintain multi-arch cross-compilers or wait 45 minutes for on-device compilation creates friction that stops prototypes from reaching production.
Engineering litrun: Under the Hood
To solve this, we architected litrun around four core principles:
1. Hardware Fingerprinting in under 2 Milliseconds
Rather than invoking shell subprocesses or relying on heavy system profilers, litrun’s fingerprinter is written in pure C99 (core/fingerprint/fingerprint.c) with zero dependencies.
It parses /sys/firmware/devicetree/base/model to identify board hardware revision.
It reads /proc/cpuinfo to detect ARM part numbers (0xd08 for Cortex-A72, 0xd0b for Cortex-A76).
Crucially, it queries the Linux auxiliary vector (getauxval(AT_HWCAP)) to verify whether hardware Dot Product (sdot/udot) instructions are present. On Cortex-A76 (Raspberry Pi 5), this instruction delivers up to a 2.5x speedup for INT8 quantization math compared to generic NEON.
2. The Active Resource Governor
Before loading model weights into RAM, the governor computes the device's safe memory envelope:
RAM
needed
Model Size
+
(
Context Length
×
KV Cache per Token
)
+
250
MB
RAM
needed
=Model Size+(Context Length×KV Cache per Token)+250MB
By polling /proc/meminfo and cgroup v1/v2 memory bounds, litrun enforces a hard rule: if
RAM
needed
Available RAM
×
0.85
RAM
needed
Available RAM×0.85, execution is blocked or auto-downgraded to a smaller quantization.
3. Persistent RAM Caching (<50ms TTFT)
Spawning a fresh binary on every user message forces the operating system to reload hundreds of megabytes from slow flash storage. Litrun implements a persistent daemon pattern (PersistentLlamaServerEngine), keeping model weights locked in RAM across HTTP requests and multi-turn conversations.
4. Anti-Zombie Process Supervision
Process management on edge Linux must be resilient. Litrun registers signal handlers for SIGTERM and SIGINT along with atexit hooks. If an interactive session is terminated or an orchestrator sends a kill signal, the background server process is cleanly terminated within 2 seconds before escalating to SIGKILL, ensuring zero orphaned processes retain memory.
Results & Verification
On an HP ProBook test host and Raspberry Pi 5 fixtures:
Hardware probe: 1.87 ms
Model load & warmup: ~2.2s
Time-to-First-Token (TTFT): < 50 ms
Generation speed: 10–19 tokens/sec on compact SLMs
Test suite: 23/23 tests passing with 100% pass rate
Get Started in 30 Seconds
bash
# Install via pip
pip install litrun
# Or run the POSIX bootstrap
curl -sSL https://raw.githubusercontent.com/assassinaj602/litrun/main/scripts/install.sh | sh
# Inspect your hardware topology
litrun inspect
# Run a model
litrun run smollm2-135m --prompt "Explain edge AI"
# Start OpenAI REST server
litrun serve --port 8080
All source code, fixtures, and documentation are open source under the MIT License on GitHub: 👉 https://github.com/assassinaj602/litrun
Top comments (0)