Aether-7B-5Attn: what a 'fully open' foundation model actually ships

#machinelearning #llm #opensource #ai

Aether-7B-5Attn: what a "fully open" foundation model actually ships

"Open source model" has quietly come to mean "here are the weights, trust us about the rest." You get a .safetensors file and a paragraph of prose about how it was made. You cannot reproduce it, cannot audit the data, cannot rerun the experiment, cannot disagree with it in any way that costs the publisher anything.

Aether-7B-5Attn is an attempt at the other definition. This post is an inventory of what that means concretely — file by file, number by number — and an honest look at what the results do and don't show.

One of three VIDRAFT releases covered by ETNews this week — and the only one published as open source rather than open weights. The scope comparison explains the difference.

The manifest

#	Artifact	Shipped
1	Weights (Apache-2.0)	✅ `model.safetensors`
2	Architecture source	✅ `aether_pkg/`
3	Training data recipe — source repos, configs, token counts, mixing weights, tokenizer, EOS	✅ card §5
4	Tokenization script — the exact file that was run	✅ `tokenize_one.py`
5	Training launcher + shell scripts	✅ `training/`
6	Every hyperparameter	✅ card §6
7	Complete training log — all 162,000 steps	✅ `logs/`
8	Intermediate checkpoints	✅ steps 110k / 115k / 162k
9	Eval code	✅ `eval/lmeval_run.py`

Items 3–8 are the ones that separate open weights from open process. They are also the ones that let you disagree with the authors using their own materials.

What it cost to make


Hardware	NVIDIA B200 × 16 (2-node FSDP)
Wall clock	~46 days total; final stage 30 d 11 h
Compute	~11,700 B200-hours
Steps	162,000
Tokens	144.2B
LR	cosine, peak 5e-5 → min 5e-6 (embedding LR ×0.1)
Parameters	6.59B total, ~2.98B active per token (MoE, 25 experts top-7 + 1 shared)
Layers	49 (7 × 7 Latin square)
Vocabulary	151,936 (Qwen-compatible tokenizer)
Context	4096

Data sources are public repos, listed with mixing weights: FineWeb-Edu, SmolLM-Corpus, FineMath, OpenWebMath, OPC FineWeb code corpus, plus HAERAE Korean WebText and Korean SyntheticText. Preprocessing writes a flat uint32 array with EOS 151645 at document boundaries — and tokenize_one.py is the script that did it, not a description of a script that did it.

The architecture, and how well the control actually holds

49 layers viewed as a 7×7 grid, with attention labels assigned by type = (row + col) mod 7. Every mechanism therefore appears exactly once per row and per column — the property that removes depth bias, the confound where a mechanism gets credit for the layer depth it happened to land in rather than for itself.

First correction to my own earlier post on this: the layout is not in config.json. There is no layer_types field. The assignment lives in the modeling module as a constant, so the audit reads:

import importlib
from collections import Counter

mod = importlib.import_module("aether_pkg.modeling_aether_v2_7way")
LAT, TYPES = mod.LATIN_SQUARE_7x7, mod.ATTN_TYPES
layers = [TYPES[LAT[L // 7][L % 7]] for L in range(49)]

print(all(len(set(r)) == 7 for r in LAT))          # rows  -> True
print(all(len(set(c)) == 7 for c in zip(*LAT)))    # cols  -> True
print(Counter(layers))                              # every type exactly 7×

Being able to make that correction from the artifact is the whole argument for shipping item 2 of the manifest.

Second, and more interesting — the control is exact in the first moment and only approximate in the second. Compute the mean layer index for each mechanism:

Mechanism	Layers	Mean depth	Std. dev.
`nsa`	0, 13, 19, 25, 31, 37, 43	24.0	13.64
`differential`	1, 7, 20, 26, 32, 38, 44	24.0	14.63
`full`	2, 8, 14, 27, 33, 39, 45	24.0	15.10
`linear`	3, 9, 15, 21, 34, 40, 46	24.0	15.10
`sliding`	4, 10, 16, 22, 28, 41, 47	24.0	14.63
`compress`	5, 11, 17, 23, 29, 35, 48	24.0	13.64
`hybrid`	6, 12, 18, 24, 30, 36, 42	24.0	12.00

Mean depth is identical to the last decimal across all seven — spread 0.0000. That is not luck; it falls out of the construction, since (t − r) mod 7 over the seven rows is always a permutation of 0–6. First-order depth bias is eliminated exactly.

The variance is not equalised: 12.00 to 15.10. hybrid sits more centrally than full or linear do. So the honest description of this control is "mean depth exactly balanced, depth dispersion not" — which is a meaningfully weaker and more accurate claim than "depth bias removed," and one you can only make because the layout is inspectable.

The most reusable table in the release

Each of the five mechanisms was measured as a standalone layer — prefill latency and peak memory. Cost is a property of the architecture, independent of trained weight values, so anyone can reproduce this:

Type	2K (ms / GB)	8K (ms / GB)	32K (ms / GB)
`full`	0.4 / 0.0	1.5 / 0.2	13.6 / 0.7
`differential`	0.5 / 0.1	3.7 / 0.3	46.5 / 1.1
`sliding`	0.6 / 0.1	1.9 / 0.2	7.6 / 0.8
`nsa`	1.0 / 0.1	3.6 / 0.3	26.8 / 3.5
`hybrid`	1.5 / 0.1	7.7 / 0.3	74.7 / 3.5

Read it and the theory becomes arithmetic:

At 2K, full is the fastest thing on the list. The quadratic term hasn't bitten yet.
At 32K, sliding (7.6 ms) is 1.8× faster than full (13.6 ms) — exactly how a locality-only design should behave as context grows.
hybrid runs both nsa and differential, and its cost lands at roughly their sum (26.8 + 46.5 ≈ 74.7). No surprises, which is itself a useful signal that the implementations are what they claim.
The full model handles 32K context in 3.46 GB.

To our knowledge no measured per-type cost profile for heterogeneous attention had been published before. It describes cost only — it says nothing about output quality, and the card is explicit about that.

The evaluation, without spin

lm-evaluation-harness 0.4.11, 0-shot, n=600, batch_size=1:

English	acc	Korean (KoBEST)	score
SciQ	73.7	HellaSwag (acc_norm)	44.6 ±2.2
PIQA	66.3	COPA	57.2 ±2.0
BoolQ	54.3	SentiNeg	55.7 ±2.5
ARC-Easy	52.5	WiC	48.8 ±2.0
WinoGrande	51.8	BoolQ	47.8 ±2.0
HellaSwag	37.2
ARC-Challenge	22.2

These are not frontier numbers, and nothing here pretends otherwise. This is an un-tuned base checkpoint trained on a 144.2B-token budget — roughly two orders of magnitude below what frontier bases consume. ARC-Challenge at 22.2 is near chance. A model card optimising for impressions would show SciQ and stop.

The language-modeling check is the more informative one:

Input	Perplexity
A natural Korean sentence	5.4
Same sentence, word order scrambled	41.3
Random tokens	2233.4

An 7.6× penalty for scrambling word order and a 400× penalty for randomness is what a model that actually learned grammatical and semantic structure looks like — as opposed to one that learned token frequencies. For a 144B-token budget, that is the honest headline.

What is deliberately not claimed

That heterogeneous attention beats a uniform stack.

There is no such result here. Establishing it requires a controlled ablation — identical data, identical token budget, identical compute, heterogeneous vs. homogeneous — and until someone runs it the status is unproven. The Latin square is the instrument built to make that ablation interpretable; it is not the answer.

Which is the actual argument for shipping items 3–8 of that manifest. A hypothesis nobody can test isn't a claim, it's advertising. Everything needed to run the experiment — recipe, code, hyperparameters, logs, checkpoints, eval harness — is in the repo, including everything needed to produce a result that contradicts the people who published it.

Using it

# batch_size = 1. Some branches (NSA family) do not consume a padding mask,
# so left-padded batches can silently corrupt results — no exception raised.
out = model.generate(**ids, max_new_tokens=128, do_sample=True, repetition_penalty=1.1)