Building an LLM System from Scratch in Pure Python & NumPy: Architecture, Invariants, and Clean Code

#ai #architecture #opensource #python

Hi everyone,

I wanted to share the design invariants and architecture of Draco AI, a full-stack, hardware-agnostic LLM system built completely from scratch in pure Python and NumPy. The core philosophy behind this project is strict code quality, mathematical correctness, and clean architecture—completely eliminating heavy framework dependencies like PyTorch or HuggingFace.

⚠️ 100% Fully Open-Source: To clarify, this is not a half-baked open-source release where only the inference wrapper is public. Draco AI is 100% open-source across its entire execution path, including BOTH deep decoupled layers: the execution mechanics of the Inference Engine (modeling/) and the cognitive/reasoning paths of the Thinking Engine (thinking_engine/).

🔗 Open Source & Repository

The baseline includes a robust pytest validation suite covering GQA, MLA, hybrid attention, Medusa heads, and speculative tree decoding execution paths.

GitHub Repository: https://github.com/DUCNGUYEN-creator/Draco_AI
Discord Community: https://discord.gg/dJXMbexdTh

🚀 Key Technical Highlights

Zero-AI-Dependency Execution Path: 100% pure Python/NumPy execution fallback. Hardware-agnostic architecture where Triton (GPU) and Numba (CPU JIT) act purely as lazy, optional accelerators.
Ternary Quantization (BitNet 1.58b): FFN experts support ternary weights, achieving up to 70-80% memory footprint reduction. The forward pass utilizes an addition-only matrix multiplication (Y = (X @ pos_mask.T - X @ neg_mask.T) * scale) without floating-point multiplications on the core execution path.
3-Tier Hierarchical EngramCache: Implements a coarse-to-fine routing mechanism: Tier 0 (Exact KV sliding window) $\rightarrow$ Tier 1 (Compressed block summaries) $\rightarrow$ Tier 2 (Table of Contents chapter vectors) for $O(1)$ fast lookups without linear scanning degradation.
12-Stage Self-Healing Cognitive Pipeline: Features a deep hallucination subsystem powered by 9 independent verifiers (retrieval, contradiction, numerical via SafeAST, symbolic, etc.) and 5 probability fusion methods (including Bayesian updates and Dempster-Shafer mass-on-uncertainty handling).
Mirostat v2 Core Correction: Implements token sampling using the true Basu (2020) negative-feedback dual-process loop ($\mu \leftarrow \mu - \eta * (\text{surprise} - \tau)$), dynamically narrowing the target cross-entropy window to maintain generation quality.
Real-Time Health & Precision Control: Features an active HealthMonitor tracking NaN/Inf propagation, saturation, and adversarial expert collapse. It signals a DynamicPrecisionManager for advisory dtype escalation/de-escalation based on overflow EMA while strictly respecting a hard VRAM budget.

🛠️ System Invariants & Determinism

To prevent silent corruptions common in speculative systems, Draco AI enforces strict execution invariants:

One-Way Data-Flow Graph: Structured strictly as runtime → layers → ops → kernels. No upward execution calls are permitted. device.py acts as the single source of truth for hardware capabilities.
RoPE Position Consistency: The rope_offset is captured exactly once per forward pass before any layer updates, preventing token drift across the 12-stage decoupled pipeline.
Deterministic Rollback Invariance: In speculative tree and Medusa multi-head draft decoding paths, the reject branch forces a strict nested rollback via a SnapshotStack. It restores both the KVCache slab and the EngramCache pointers to their exact states before the speculative proposal while enforcing add_noise=False on replay to eliminate stochastic routing drift.

I’d highly appreciate your insights regarding the mathematical fusion choices for uncertainty assessment, the decoupled architecture, or the pure-NumPy tensor layout. Let's discuss in the comments below!