VIDRAFT Releases Aether-7B-5Attn: A Fully Reproducible Open-Source LLM with Five Heterogeneous Attention Mechanisms
TL;DR: Korean Pre-AGI startup VIDRAFT has published
Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under Apache 2.0 — releasing not just weights but also training data recipes, full training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code. It combines five distinct attention mechanisms in a single model and is designed from the ground up to be independently reproducible by anyone with access to the public repository.
What it is
Aether-7B-5Attn is VIDRAFT's first fully open-source foundation LLM, published on Hugging Face on July 20, 2026. It is licensed under Apache 2.0, meaning it is free to use, modify, and redistribute for both research and commercial purposes.
What distinguishes this release from the majority of models currently marketed as "open-source" is its completeness. Most so-called open-source LLMs are, in practice, open-weight models: you can download the weights and run inference, but the training data composition, tokenization pipeline, code, configuration, and evaluation procedures remain opaque. VIDRAFT explicitly frames Aether-7B-5Attn as a counter-example to that norm:
- Model weights — base model and instruction-tuned variant
- Training data recipe — composition ratios and tokenization details
- Full training code and configuration
- Training logs and intermediate checkpoints
- Evaluation code
- Live demo — accessible via Hugging Face Spaces
The release is comparable in openness to projects like OLMo, Apertus, and LLM-jp, which have historically been driven by academic consortia or national research programs. VIDRAFT claims this is the first release of this transparency level from a single commercial startup.
How it works
Aether-7B-5Attn uses a Mixture-of-Experts (MoE) architecture with a total of 6.59 billion parameters, of which only approximately 2.98 billion are activated per token during inference — reducing compute cost relative to a dense model of equivalent total capacity.
The defining architectural feature is the integration of five heterogeneous attention mechanisms within a single model, interleaved across layers:
- Full Attention — standard scaled dot-product attention over the full context
- Differential Attention — cancels attention noise by computing the difference between two softmax attention maps
- Sliding Window Attention — restricts each token's attention to a local window, reducing quadratic complexity for long sequences
- NSA-family Sparse Attention — a native sparse attention variant designed for efficient long-context modeling
- Hybrid Attention — combines characteristics of the above to handle mixed context patterns
Rather than committing the entire model to a single attention strategy, this heterogeneous design allows different layers to specialize for different information-retrieval tasks (local syntactic patterns, long-range dependencies, noise reduction, etc.).
Training was conducted on approximately 144.2 billion tokens. The data mixture prioritizes mathematics (37.8%), Korean (21.6%), and English (21.6%), with the remaining share covering other domains. This makes Aether-7B-5Attn a bilingual Korean–English foundation model rather than an English-first model with Korean as an afterthought — a deliberate design choice aligned with VIDRAFT's sovereign AI positioning.
Benchmarks & results
The source article does not publish specific benchmark scores for Aether-7B-5Attn itself at this time. However, VIDRAFT reports the following public results from its broader model portfolio (Darwin model family), which provides context for the team's capabilities:
- GPQA Diamond: 90.9% — a graduate-level science reasoning benchmark
- Polaris Global Drug Discovery Leaderboard: 14 consecutive wins
- Metacognition Leaderboard: #1 ranking
- K-AI Leaderboard (Korean): #1 overall (Darwin family)
- Hugging Face cumulative downloads: over 1 million
Benchmark results specific to Aether-7B-5Attn should appear in the public repository's evaluation artifacts as the community runs evaluations.
How to try it
Both the base model and the instruction-tuned variant are publicly available on Hugging Face. A live demo is also available via Hugging Face Spaces. You can browse and download the model using standard Hugging Face tooling:
pip install huggingface_hub
huggingface-cli download VIDRAFT/Aether-7B-5Attn
⚠️ Verify the exact repository path (
VIDRAFT/Aether-7B-5Attn) on huggingface.co before downloading, as naming conventions may vary. Training code, configuration files, logs, and checkpoints are also hosted in the public repository.
FAQ
Q: Is this truly open-source, or just open-weights like most other "open" LLMs?
A: VIDRAFT explicitly distinguishes Aether-7B-5Attn from open-weight-only releases. The public repository includes training data recipes, full training code, hyperparameter configs, training logs, intermediate checkpoints, and evaluation code — making the full training pipeline independently reproducible, not just the final weights.
Q: Why combine five different attention mechanisms instead of picking one?
A: Different attention variants have different cost-quality tradeoffs depending on context length and information type. By heterogeneously interleaving them across layers, the model can apply the most appropriate attention strategy at each depth of the network rather than making a single global architectural compromise.
Q: What license is it under, and can I use it commercially?
A: Apache 2.0. This is a permissive license that allows commercial use, modification, and redistribution, with attribution required.
Q: How does this relate to VIDRAFT's Darwin models I may have seen on leaderboards?
A: Darwin is VIDRAFT's separate model family built using evolutionary model merging techniques, and it is what achieved the K-AI Leaderboard and other benchmark results mentioned above. Aether-7B-5Attn is a new, independently trained foundation model under the AETHER architecture — a different line focused on full reproducibility and architectural research.
Originally reported by 디지털데일리 (2026-07-20) — source article.
Top comments (0)