While production serving stacks like vLLM @FlashInfer SGLang, and LMDEploy are powerhouse frameworks, they carry massive dependency trees and architectural overhead. My goal with HexCore was to build a clean, auditable, standalone C++/CUDA memory engine designed for maximum VRAM compression and minimal footprint.
I just completed full hardware verification on an NVIDIA Tesla T4 (sm_75) using compute-sanitizer:
✅ Numerical Correctness (21/21 checks passed):
FP16 Decode & Prefill max error vs. CPU reference: ~4.8e-4
INT4 & INT2 Quantized Decode/Prefill max error: < 1.0e-3
Sentinel block (-1) & sequence bounds guards verified on-device
✅ Zero Memory & Concurrency Bugs:
compute-sanitizer --tool memcheck: 0 errors
compute-sanitizer --tool racecheck: 0 hazards, 0 warnings
Host allocator & Prometheus exporter hardened with ASan + UBSan (-Wall -Wextra -Werror -pedantic)
📊 Current Baseline & Next Optimization (8k context, Tesla T4):
batch=1: 5.65 ms (5.9 GB/s)
batch=8: 6.34 ms (42.3 GB/s)
Currently implementing Split-K (Flash-Decoding) across thread blocks to saturate memory bandwidth on single-batch long contexts before public release.
⏳ Coming Soon to GitHub: Stay tuned — I am finishing the Split-K optimization pass and packaging the full repository, sanitizer logs, and reproducible T4 benchmarks for public release very soon!
Open to technical discussions, licensing, or acquisition inquiries with inference & edge-AI teams at Fireworks AI works AI, Together AI ther AI, Baseten, DeepInfra, Lepton AI, Modal, Hugging Face, Modular, a Qualcomm company, Qualcomm, and NVIDIA!
Top comments (0)