Open Weights Aren't Open Source: Aether-7B-5Attn, a From-Scratch Fully-Open Foundation Model (Apache-2.0)

#ai #machinelearning #opensource #llm

A from-scratch, fully-open foundation model (Apache-2.0): Aether-7B-5Attn

Most "open" LLMs ship one thing: a weights file. That's the compiled binary of an LLM. The training data, the training code, the hyperparameters, the logs — all closed. By the software definition of "open source," that's just a published binary.

VIDRAFT released Aether-7B-5Attn the other way. Under Apache-2.0, it opens the whole manufacturing process:

architecture source (attention + MoE + Latin square)
data recipe — reproducible byte-for-byte from public repos
tokenizer scripts
training code (FSDP launcher, loop, node scripts)
every hyperparameter
full training logs
evaluation code
intermediate checkpoints

You don't have to trust that the training corpus is clean. Run the published recipe (sources, configs, token counts, mixing ratios, tokenizer, EOS) and you regenerate the exact corpus. That's the difference between open weights and fully open — the line OLMo drew, and the club (OLMo/Apertus/LLM-jp/Viking) that until now was national labs and consortia. Aether joined it as a single startup.

Heterogeneous attention on a Latin square

The interesting engineering bit: instead of one attention mechanism everywhere, Aether mixes several — full, differential, sliding-window, an NSA-style native-sparse branch, and a hybrid.

Placement is the trap. Stack them arbitrarily and you can't separate the effect of the attention type from depth bias (a type clustering in shallow/deep layers). The fix is a 7×7 Latin square over 49 layers: each mechanism appears exactly once at every depth. It's an experimental control, not decoration — and since it's open, the hypothesis is falsifiable by anyone.

Total params      6.59B
Active / token    ~2.98B (MoE: 25 experts, top-7 + 1 shared)
Layers            49 (7×7 Latin square)
Context           4,096
Train tokens      144.2B
Hardware          NVIDIA B200 × 16
Data mix          math 37.8% · Korean 21.6% · English 21.6% · code 13.5%
Datasets          FineWeb-Edu, SmolLM-corpus, FineMath, OpenWebMath,
                  OpenCoder, HAERAE KOREAN-WEBTEXT / KOREAN-SyntheticText
License           Apache-2.0

⚠️ One gotcha: run inference at batch_size = 1

This is pinned in the model card and it matters. Some attention branches (the NSA-style path) do not consume the padding mask. Batch padded sequences together and queries can attend into pad tokens — silently corrupting outputs. No error, just quietly wrong numbers.

from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "FINAL-Bench/Aether-7B-5Attn-it"
tok = AutoTokenizer.from_pretrained(mid, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(mid, trust_remote_code=True,
                                             torch_dtype="auto", device_map="auto")

# ✅ one sequence at a time — do NOT left-pad and batch
msgs = [{"role": "user", "content": "Explain a Latin square in one line."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

If you port this to a batched server (or vLLM), handle the mask issue explicitly. The public demo backend enforces batch_size=1 for exactly this reason.

Why it's worth your attention

Not because a ~6.59B model tops the closed leaderboards — it doesn't, and that's not the claim. The claim is reproducibility: results are open, and so is the process behind them. If you care about sovereign / on-prem AI — models you can verify and rebuild without depending on a vendor's license staying put — this is a concrete, buildable reference.

Links