VIDRAFT Releases Fully Open-Source Aether-7B-5Attn: A Heterogeneous-Attention MoE Foundation Model Built for Korean and English Equally

TL;DR: Seoul-based AI startup VIDRAFT has released Aether-7B-5Attn on Hugging Face under the Apache 2.0 license — a 6.59B-total-parameter Mixture-of-Experts foundation model that activates only ~2.98B parameters per token and uniquely integrates five distinct attention mechanisms in a single architecture. The full release includes model weights, training data recipe, training code, and evaluation code, making it one of the most transparent open-source foundation model drops of 2026.

What It Is

Aether-7B-5Attn is a fully open-source foundation model developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI deep-tech company. The release package is unusually comprehensive even by open-source standards:

• Model weights — base model and instruction-tuned variant
• Training data recipe — documented composition and sourcing details
• Full training code — reproducible end-to-end pipeline
• Evaluation code — the same tooling used internally
• Training logs — step-by-step transparency
• Intermediate checkpoints — enabling research into training dynamics
• Live demo — accessible via Hugging Face Spaces

The license is Apache 2.0, meaning commercial use, modification, and redistribution are all permitted without royalty obligations.

VIDRAFT frames Aether-7B-5Attn as a sovereign foundation model — one designed not around English-first assumptions but with Korean and English treated as co-primary languages, alongside a heavy emphasis on mathematics.

How It Works

The model's architecture makes two notable structural bets that are worth understanding at a conceptual level.

Mixture-of-Experts with aggressive sparsity: The total parameter count is 6.59B, but only approximately 2.98B parameters are activated per token at inference time. This sparse-activation design means the model delivers reasoning quality closer to its full parameter budget while requiring substantially less compute per forward pass — a practical win for deployment cost and latency.

Five heterogeneous attention mechanisms in one model: Rather than using a single attention variant throughout all layers, Aether-7B-5Attn integrates five qualitatively different attention types:

1. Full (standard dense) Attention
2. Differential Attention
3. Sliding Window Attention
4. NSA-series Sparse Attention
5. Hybrid Attention

The 49 transformer layers are arranged according to a 7×7 Latin square pattern. In combinatorics, a Latin square ensures that each element (here, each attention type) appears exactly once in every row and column — meaning no single attention mechanism is clustered at any particular depth of the network. The practical intent is to distribute inductive biases evenly across the model's depth, avoiding the pattern where, say, local-context attention dominates early layers and global attention only appears late.

This architectural diversity is the "5Attn" in the model name — five attention strategies, systematically interleaved.

Training data composition (by token share):

• Mathematics: 37.8%
• Korean: 21.6%
• English: 21.6%
• (Remainder: other languages and domains)

The equal weighting of Korean and English — a deliberate design choice — directly reflects VIDRAFT's stated goal of building a non-English-centric model.

Benchmarks & Results

The CNMO source article does not report benchmark scores specific to Aether-7B-5Attn. For verified evaluation results, refer to the model card on Hugging Face, which includes the evaluation code and logged outputs.

Context from VIDRAFT's broader model portfolio: the company's Darwin model family previously ranked first on the K-AI leaderboard overall, achieved 90.9% on GPQA Diamond, took 14 top positions on the Polaris global drug-discovery leaderboard, and ranked first on a metacognition leaderboard. Darwin models have also crossed 1 million cumulative downloads on Hugging Face. Aether-7B-5Attn is a distinct architectural line from Darwin and targets different use cases, but this track record contextualizes the team's capability.

How to Try It

The base model, instruction-tuned model, and an interactive demo are all publicly available on Hugging Face. Search for VIDRAFT/Aether-7B-5Attn (verify the exact repository slug on the Hugging Face hub before use).

# Browse and download via the Hugging Face CLI
pip install huggingface_hub
huggingface-cli download VIDRAFT/Aether-7B-5Attn

For a quick browser-based test without any local setup, the live demo is accessible directly from the Hugging Face model page. Training code and evaluation scripts are linked from the same repository.

---

FAQ

Q: Why five different attention mechanisms instead of just picking the best one?
A: Each attention variant has different strengths: full attention captures long-range global context, sliding window keeps local context efficient, differential attention suppresses noise, and sparse variants reduce compute on long sequences. By interleaving all five in a Latin square layout, the model can handle diverse task types without the architectural biases that come from committing to any single strategy.

Q: With only 2.98B active parameters per token, how does this compare to dense 3B-class models?
A: Active-parameter count is not a direct quality equivalence — the MoE routing means the model still benefits from 6.59B total parameters for specialization across tokens and domains. In practice, MoE models at this sparsity ratio tend to outperform dense models of the same active parameter size, while costing roughly the same at inference. The training data recipe (especially 37.8% math) also shapes capability independently of raw parameter count.

Q: Can I fine-tune this for production use commercially?
A: Yes. The Apache 2.0 license permits commercial use, derivative works, and redistribution, provided the license and copyright notice are retained. Always review the full license text and the model card's usage notes before deploying.

---

Originally reported by CNMO 手机中国 (2026-07-20) — source article.