Aether-7B-5Attn: VIDRAFT's Fully Open-Source Foundation Model with Heterogeneous Attention MoE Architecture

TL;DR: Korean AI startup VIDRAFT has released Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model under Apache-2.0 on Hugging Face — with weights, training data recipe, training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code all public. It combines five distinct attention mechanisms in a single architecture and is designed for full reproducibility, not just open-weight inference.

What it is

Aether-7B-5Attn is an open-source foundation model developed independently by VIDRAFT, a Seoul-based AI deep-tech startup. Key characteristics from the release:

• License: Apache-2.0
• Architecture: Mixture-of-Experts (MoE) with 6.59B total parameters; only ~2.98B parameters are activated per token, keeping inference efficient
• Attention design: Five heterogeneous attention mechanisms combined in a single model — Full Attention, Differential Attention, Sliding Window, NSA-family sparse attention, and a Hybrid variant
• Layer structure: 49 layers arranged in a 7×7 Latin Square pattern, ensuring no single attention type is concentrated at any particular depth
• Training scale: ~144.2B tokens
• Data mix: Mathematics (37.8%), Korean (21.6%), English (21.6%), and additional sources — explicitly designed as a bilingual Korean-English foundation model rather than an English-first model
• Artifacts released: Base model weights, instruct model weights, training data recipe, full training code, hyperparameters, training logs, intermediate checkpoints, evaluation code, and a live demo

VIDRAFT positions Aether-7B-5Attn within its broader AETHER architecture tier — part of a four-layer AGI system that also includes the Darwin model family, the PROMETHEUS world model, and the HEPHAESTUS embodied AI layer.

The fully open approach is explicitly modeled after projects like Allen AI's OLMo, Apertus, and LLM-jp — making VIDRAFT one of the few single commercial startups (rather than national consortia or research labs) to attempt this level of transparency.

How it works

The core architectural bet in Aether-7B-5Attn is that a single model should not be locked into one attention strategy. Different token relationships and context lengths benefit from different inductive biases:

• Full Attention captures global context across the entire sequence.
• Differential Attention reduces attention noise by computing the difference between two softmax attention maps.
• Sliding Window Attention limits each token's attention scope to a local window, reducing compute on long sequences.
• NSA-family sparse attention selectively attends to the most relevant tokens without exhaustively computing all pairs.
• The Hybrid variant combines characteristics of multiple mechanisms.

Rather than stacking these naively (which would bias certain mechanisms to early or late layers), VIDRAFT uses a 7×7 Latin Square assignment: each attention type appears exactly once in each row and column of the 49-layer grid, distributing all five types uniformly across model depth.

The MoE structure further decouples total parameter count from per-token compute: the model has 6.59B parameters but routes each token through only ~2.98B of them, keeping inference costs closer to a sub-3B dense model while retaining the representational capacity of a larger one.

Benchmarks & results

VIDRAFT has disclosed the following publicly announced results from its Darwin model family (the broader product line that Aether-7B-5Attn feeds into architecturally):

• K-AI Leaderboard: Overall #1 ranking in Korea (2026)
• GPQA Diamond: 90.9%
• Polaris Global Drug Discovery Leaderboard: 14 consecutive first-place finishes
• Metacognition Leaderboard: #1 ranking
• Hugging Face cumulative downloads: Darwin family models have surpassed 1 million downloads

Note: The benchmarks above are attributed to the Darwin model family. Specific benchmark results for Aether-7B-5Attn as a standalone model were not separately enumerated in the source article. Readers should check the Hugging Face model card for the most up-to-date evaluation numbers.

How to try it

The base model, instruct model, and live demo are all publicly available on Hugging Face. Search for VIDRAFT or Aether-7B-5Attn on the Hub:

# Browse or download via Hugging Face CLI
pip install huggingface_hub
huggingface-cli download vidraft/Aether-7B-5Attn

(Replace the repository path above with the exact slug shown on the VIDRAFT Hugging Face page — confirm the canonical repo name before running.)

A live interactive demo is also available directly on the Hugging Face model page, so you can test the instruct model without any local setup.

---

FAQ

Q: Is this actually fully open-source, or just open-weight like most "open-source" LLMs?
A: VIDRAFT explicitly distinguishes between the two. The release targets full reproducibility: training data recipe, tokenization pipeline, training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code are all included — not just the final weights. The article explicitly calls this out as addressing a known gap in the ecosystem.

Q: Why does the "5Attn" suffix matter architecturally?
A: It signals the five heterogeneous attention mechanisms baked into the model. The 7×7 Latin Square layer assignment is specifically designed to prevent any one mechanism from dominating at a particular depth, which is a non-trivial architectural choice worth examining in the released training code.

Q: Can I use this commercially?
A: The model is released under Apache-2.0, which permits commercial use. Always verify the specific license terms on the Hugging Face repository page before deploying in production.

Q: Does VIDRAFT offer an API for this model?
A: The source article does not announce a public OpenAI-compatible API endpoint for Aether-7B-5Attn specifically. Check VIDRAFT's official channels for API availability updates.

---

Originally reported by ZDNet Korea (2026-07-20) — source article.