ShannonBase — The Next-Gen HTAP Database for the AI Era

ShannonBase (by Shannon Data AI) is a MySQL-compatible HTAP (Hybrid Transactional/Analytical Processing) database designed and optimized for modern big-data and AI workloads. Think of it as "MySQL for the AI era": it keeps familiar SQL and operational semantics while adding native embedding/vector support, built-in machine learning, a columnar in-memory engine, and a lightweight JavaScript runtime. The result is a unified platform where OLTP, OLAP, vector search and ML workflows can run together with minimal data movement.

https://github.com/Shannon-Data/ShannonBase

Key Design Principles

• Zero Data Movement: keep data, embeddings, models and inference as close to storage as possible - inside the database - to reduce latency, cost and operational complexity.
• Native ML & Vector Support: provide first-class vector types, embedding pipelines and in-DB ML training/inference primitives.
• HTAP with Intelligent Routing: combine row and column engines and route work to the best engine via cost-based and ML-driven decisions.
• SQL-First Developer Experience: enable data scientists and application engineers to use SQL (and optional JS stored procedures) rather than stitching many systems together.

Architecture Overview

• InnoDB + Rapid (IMCS Column Store) InnoDB (row store): handles transactional, write-heavy OLTP workloads and durability.
• Rapid (IMCS - In-Memory Column Store): a columnar, memory-optimized engine for analytics, aggregations and vector/semantic search.
• Intelligent Workload Routing: queries are dispatched to InnoDB or Rapid using a cost model and optional ML models that learn which engine yields better performance for a given query pattern.
• Version Linking & MVCC: Rapid supports MVCC via a version-linking mechanism so analytical reads observe consistent snapshots while writes occur in InnoDB.
• Redo-log Synchronization: InnoDB changes are applied to Rapid by replaying redo logs (synchronously), keeping both engines consistent without ETL.

Multimodal Data Types

• Structured data: classic SQL columns and indexing.
• JSON: efficient JSON storage and querying (MySQL-style semantics, extended where needed).
• GIS: spatial types and ST_* functions for location queries.
• VECTOR: native vector column type and helper functions (eg. Distance, etc.) for embeddings and similarity search.

Native Machine Learning

• Embedded model runtime: integrates LightGBM (and optionally XGBoost or other engines) so you can train and infer inside the DB. Stored procedures for ML: sys.ML_TRAIN, sys.ML_PREDICT_ROW, sys.ML_MODEL_LOAD, etc. - train models on DB tables and run predictions in place.
• Model imports: load pre-trained models to save training time and standardize inference.
• ONNX/ONNXRuntime support: run portable models (including small LLMs or local ONNX LLMs) where appropriate.

Retrieval-Augmented Generation (RAG) & Embeddings

• Embedding routines: built-in embedding APIs and stored procedures to create and manage embedding tables.
• Vector stores & RAG helpers: utilities to embed tables, perform ANN / nearest-neighbor searches, and assemble RAG inputs for LLMs (local or remote).
• LLM integration: run generation via integrated routines or attach an ONNX LLM runtime for on-prem inference.

Multilingual Stored Procedures: JerryScript

• JavaScript engine: JerryScript embedded for writing stored procedures and UDFs in JavaScript as an alternative to pure SQL, lowering the barrier for custom logic and preprocessing.

Benefits & Target Use Cases

• Benefits
  Unified platform for OLTP, OLAP, vector search and ML - reduces system sprawl.

• Low latency for inference and semantic search because embeddings and models live next to the data.
  Familiar SQL surface with optional JS for custom logic.
  HTAP efficiency via in-memory column engine for analytics and row engine for transactional integrity.

Target Use Cases

• Enterprise knowledge bases and RAG systems Real-time personalization and recommendation (online training & inference) Interactive analytics and BI over fresh transactional data Spatial analytics and location-aware services Simplified MLOps for small/medium in-DB models and feature stores

Conclusion
ShannonBase brings together a pragmatic set of features to address the needs of modern AI and analytics applications: an HTAP architecture with intelligent routing, native vector and embedding support, in-database ML training and inference, and a small JavaScript runtime for programmable extensions. For organizations that want to reduce data movement, simplify the analytics + ML stack, and deliver low-latency semantic search and in-place inference, ShannonBase offers a compelling, SQL-centric path forward.