Efficient memory management is one of the biggest bottlenecks when scaling Large Language Model (LLM) inference runtimes. Memory fragmentation and unpredictable allocation latencies can severely degrade token generation performance.
To tackle this problem, I built HexCore v2.2.0 — a zero-trust, ultra-low latency Paged KV Cache Allocator written in C++20 with CUDA kernels.
🔑 Key Features
- Paged KV Cache Allocator: Minimizes latency and memory fragmentation during autoregressive decoding.
- CUDA Accelerated Kernels: Custom GPU implementations for Paged Attention and quantized memory layouts.
- Zero-Trust Memory Safety: Fully guarded boundary checking verified with AddressSanitizer (ASan) and UndefinedBehaviorSanitizer (UBSan).
- Python/PyTorch Bindings: High-performance C++20 bindings for seamless integration into modern AI runtimes.
🛠 Tech Stack
- Core: C++20, CUDA
- Integration: Python, PyTorch (pybind11)
- Build System & CI: CMake, GitHub Actions CI/CD
🚀 Open Source & CI/CD Pipeline
The repository includes complete GitHub Actions CI/CD workflows testing host bindings, sanitizer safety, and build integrity across multiple platforms.
Check out the full source code and documentation on GitHub:
👉 GitHub Repository: password162156/hexcore-llm
I'd love to hear feedback, thoughts, or contributions from the C++ and AI systems engineering community! ⭐
Top comments (0)