5, 6 or 7? Auditing an architecture claim at the weights level

#ai #machinelearning #opensource #llm

5, 6 or 7? Auditing an architecture claim at the weights level

Here is a small scandal in model releases that nobody talks about: the numbers in model names are unaudited.

"8 experts." "128K context." "5 attention mechanisms." Nobody checks. The number goes in the name, the name goes in the headline, and the headline is what people remember. It is one of the few places in engineering where a marketing claim and a technical claim wear identical clothes.

So let's actually check one. Aether-7B-5Attn is a from-scratch foundation model published as open source — architecture code, data recipe, training code, logs and checkpoints, not just weights — which makes it one of the few models where this audit is even possible. ETNews covered the release this week.

The name says five. Let's see what the weights say.

Where the number is not

First surprise, and a correction to a common assumption: the layer layout is not in config.json. There is no layer_types field. If you go looking for it — as I did, and as an earlier version of this post wrongly told people to — you get None.

cfg = AutoConfig.from_pretrained("FINAL-Bench/Aether-7B-5Attn", trust_remote_code=True)
[k for k in cfg.to_dict() if "layer_type" in k or "attn_type" in k]   # -> []

The assignment lives in the modeling module as a constant. Which is fine — but it means the claim is only checkable because the architecture source was published. On a weights-only release you would be taking the name on faith.

import importlib
mod = importlib.import_module("aether_pkg.modeling_aether_v2_7way")
LAT, TYPES = mod.LATIN_SQUARE_7x7, mod.ATTN_TYPES
layers = [TYPES[LAT[L // 7][L % 7]] for L in range(49)]

That yields seven labels across 49 layers:

nsa · differential · full · linear · sliding · compress · hybrid

The repo ships a diagram of exactly this layout, and it is worth looking at closely — including its footnote:

Seven labels. Model named five. Neither number is wrong yet — but only one of them can be the count of distinct mechanisms, and prose alone won't settle which. Note the legend's parenthetical: "compress·linear = full-family → 5 distinct mechanisms." Hold onto that — the weights disagree with it.

Ask the weights instead

Labels are naming. Parameters are physics. So group the layers by label and compare the actual tensor names each one carries — readable straight from the safetensors header, no download of the 13 GB file required:

Label	Attention parameters	Distinctive keys
`nsa`	20	`compressed.block_compress_k/v`, `select_score`, `gate_logit`
`hybrid`	31	`diff.` + `nsa.` + `gate` + `merge_norm`
`differential`	10	`lambda_q1/k1/q2/k2`
`linear`	7	`gate`, `norm`
`compress`	5	`self_attn.full_attn.q/k/v/o_proj`
`full`	5	`self_attn.q/k/v/o_proj`
`sliding`	5	`self_attn.q/k/v/o_proj`

Every label is internally consistent — all seven layers carrying a given label have exactly the same parameter structure. Good sign; the layout is real, not decorative.

Now collapse identical signatures:

nsa | differential | linear | hybrid | compress | (full == sliding)

Six distinct parameter signatures for seven labels. Not five.

Why the honest answer is "it depends what you're counting"

Only one pair genuinely shares parameters: sliding is full with a window mask applied. Masking a path is a configuration of that path, not a separate mechanism. That collapse is real.

compress is the subtle one. It computes through a full-attention path — so at the level of what function it computes, it belongs with full, which is how you get to five. But in the weights it carries its own full_attn.* submodule with its own trained parameters. It is not sharing anything with the full layers; it is a separate module that happens to be functionally similar.

And linear is where the shipped diagram and the shipped weights part company. The legend groups it into the full-family to reach five. The weights say otherwise: linear layers carry gate and norm tensors that no full-attention layer has, which is the parameter signature of a kernelised linear-attention path, not of masked softmax attention.

So the release contains two different derivations of the number five:

Source	How it gets to 5
Model card prose	`sliding` and `compress` collapse into `full`
Diagram legend	`compress` and `linear` are "full-family"
Weights	only `sliding` collapses into `full` → 6

Both published derivations land on five; they disagree about which two labels collapse, and neither matches the parameter evidence. This is not a scandal — it is what happens when a number gets written in three places by hand. It is also precisely the class of discrepancy that stays invisible forever on a weights-only release.

So all three numbers are defensible, and they answer different questions:

Count	Question it answers
7	How many labels are placed on the grid?
6	How many distinct parameter signatures exist in the weights?
5	How many distinct functional mechanism classes are there?

The model is named for 5. The grid is built on 7. The weights say 6. None of that is dishonest — but you only get to know it because the artifact is inspectable, and the number of releases where you could run this audit at all is small.

The part this changes

This matters beyond pedantry. The whole point of mixing mechanisms is to ask "does heterogeneity help?" — and your effective diversity is an input to that question. If two of your seven slots are the same path wearing a mask, the honest denominator is smaller than the name suggests. Overstating it would overstate any eventual result.

Which leads to the claim that is not being made anywhere in this release: heterogeneous attention is not demonstrated to beat a uniform stack. There is no such result. It would take a controlled ablation — same data, same tokens, same compute, heterogeneous vs. homogeneous — and until someone runs it, the status is unproven. The layout is the instrument built to make that ablation interpretable, not the answer.

A release willing to publish "our headline hypothesis is unproven" has already forfeited the option of inflating its own numbers. The 5-vs-6-vs-7 question and the unproven disclosure come from the same discipline: publish the smaller number too.

Footnote for benchmarkers

There is a sibling release, AETHER-7B-7Attn-base, published as open weights only — 6.59B total / ~3B active, MoE, bfloat16, Korean and English, base stage, weights under Apache-2.0. Its architecture, layer layout, implementation and training details are not disclosed, so none of the audit above can be run against it.

One thing worth knowing if you benchmark both: they are different checkpoints. Same file size, same header, same tensor names and shapes — but different SHA-256, and every sampled tensor block differs.

Aether-7B-5Attn        sha256 8e77ad511a524347dd1c…
AETHER-7B-7Attn-base   sha256 9250f1f8a51ed6f3593a…

Treat them as separate models, not as two names for one file.

Practical note

# batch_size = 1 — the NSA-family branches do not consume a padding mask.
# Left-padded batches attend into pad tokens silently. No exception is raised.
out = model.generate(**ids, max_new_tokens=128, do_sample=True, repetition_penalty=1.1)