Aether-6B-11Attn-base: publishing a research artifact while it is still training

#ai #machinelearning #opensource #llm

Aether-6B-11Attn-base: publishing a research artifact while it is still training

Most model releases are announcements of finished things. The training run completed, the benchmarks came back acceptable, the card was written afterwards, and the narrative was assembled once the outcome was already known.

Aether-6B-11Attn-base is not that. It is a mid-training checkpoint — step 174,000 — of an architecture experiment that is still running, published under Apache-2.0 while the outcome is genuinely unknown. That is an unusual thing to do, and it is the most interesting property of the release.

One of three VIDRAFT releases covered by ETNews this week. The scope comparison explains how the three differ.

What it is


Parameters	5.79B total (Mixture-of-Experts)
Layers	121
Sequence mixers	11 mechanisms across 4 families
Context	8192
Precision	bfloat16
Stage	mid-training, step 174,000 — no instruction tuning
Release scope	open weights
Licence	Apache-2.0 (weights)
Languages	Korean, English

The four families are the substance of it: alongside conventional attention, the stack carries state-space, convolutional and linear-time mixers. Published hybrids typically combine two of these — Mamba plus Transformer being the familiar one. To our knowledge this is the first model to train eleven heterogeneous mixers together in a single network.

That is a claim about reach, not about quality, and the distinction matters more than usual here.

Deep and narrow

121 layers at 5.79B parameters is an unusual shape. For comparison, the sibling Aether-7B-5Attn runs 49 layers at a slightly larger parameter count. This one is far deeper and correspondingly narrower per layer.

Depth is not free, and the practical consequences are blunt:

~10 s/token. 121 layers, no KV cache in this build. This is a research artifact, not a serving build, and nothing about it is optimised for throughput.
Greedy decoding degenerates into repetition. Expected for an un-tuned base checkpoint — use sampling with a repetition penalty.
batch_size = 1. Sparse branches in this family do not consume the padding mask; left-padded batches can attend into pad tokens with no exception raised.
Run the forward under torch.autocast(bfloat16) — several branches create fp32 intermediates.

If you were hoping to drop this into a serving stack, don't. If you want to study what an 11-way heterogeneous stack does to representations, that is what it is for.

Provenance, stated plainly

The card discloses something most cards would quietly omit: this model was reconstructed from a sharded FSDP training checkpoint, and the exact training code was not archived. The architecture code was rebuilt afterwards, then validated two ways:

Exact parameter match — all 11,047 checkpoint tensors load with 0 missing / 0 unexpected keys, and the parameter count matches the checkpoint exactly (5.7885B). Key mismatches are unforgiving; a wrong architecture does not load cleanly.
Perplexity sanity — the loaded model predicts real text sensibly (PPL 11.8 English / 6.5 Korean on sample sentences). A mis-reconstructed architecture would be orders of magnitude worse, not marginally worse.

Together those are strong evidence the rebuild is faithful. They are not equivalent to having the original code, and the card doesn't claim they are. Publishing that gap — rather than presenting the rebuild as the original — is the kind of disclosure that costs nothing to omit and everything to trust.

Open weights, and what that excludes

This is an open-weights release, not an open-source one. The weights are Apache-2.0 and free to use, modify and redistribute. The training data recipe, training code, logs and intermediate checkpoints are not published.

That is a deliberate scope choice, and worth being precise about, because "Apache-2.0" on a model page invites people to assume more than it says. A permissive licence on weights is a usage grant. It is not a reproducibility grant. You can build on this model; you cannot retrain it, audit its data, or independently verify how it was made.

Exactly one member of this family is published as open source — Aether-7B-5Attn, which ships the recipe, the training code, all 162,000 steps of logs, intermediate checkpoints and the eval harness. If you want to run experiments on this architecture family rather than run inference, that is the one to start from. I went through its manifest here.

The claim that is deliberately not made

Nothing here shows that eleven mechanisms beat one.

No such result exists — not for this model, not for its siblings. Establishing it requires a controlled ablation: identical data, identical token budget, identical compute, heterogeneous versus homogeneous. Until that is run, the honest status is unproven, and a mid-training checkpoint of a single configuration is not evidence either way.

What this artifact demonstrates is that eleven mixers from four families can be trained together in one stack at all, on a layout designed to keep an eventual comparison interpretable. That is a prerequisite for the experiment, not the experiment.

Publishing at step 174,000, before knowing whether the answer flatters anyone, is the part worth copying. The alternative — waiting for the run to finish and then writing the story backwards from the result — is how the field ended up with a benchmark table culture that nobody fully trusts.

Series

Links