Don't Trust Architecture Claims — Verify Them (A Worked Example on an Open Model)

#ai #machinelearning #opensource #python

Don't trust architecture claims — verify them. A worked example on an open model

A model card says its attention layers are arranged as a Latin square. Nice claim. Can you check it?

With most "open" models: no. You get weights and a paragraph of prose. With a genuinely open one, the claim is a config file away — and that difference is the whole point of open-sourcing a foundation model.

Korean outlet ETNews covered the release of Aether-7B-5Attn this week — a from-scratch foundation model published under Apache-2.0 with the data recipe, training code, hyperparameters, logs, checkpoints and eval code. So let's do the thing open weights alone never let you do: audit the architecture claim.

The claim

49 layers (7×7). Attention labels placed as a Latin square, so each mechanism appears exactly once in every row and every column. type = (row + col) mod 7.

The reason this matters isn't aesthetics. If you mix attention mechanisms and stack them arbitrarily, any measured difference is confounded by depth bias — a mechanism that happens to land in early or late layers gets credit (or blame) for depth, not for itself. A Latin square removes that: every mechanism sits at every depth exactly once.

In other words, the architecture is an experimental control. Which is only meaningful if it's actually implemented that way.

Verify it

from transformers import AutoConfig
from collections import Counter

cfg = AutoConfig.from_pretrained("FINAL-Bench/Aether-7B-5Attn", trust_remote_code=True)
layers = getattr(cfg, "layer_types", None) or getattr(cfg, "attn_types", None)
print(len(layers), "layers")

N = 7
grid = [layers[r*N:(r+1)*N] for r in range(N)]

ok_rows = all(len(set(row)) == N for row in grid)
ok_cols = all(len(set(col)) == N for col in zip(*grid))
print("each type once per row :", ok_rows)
print("each type once per col :", ok_cols)
print("depth histogram        :", Counter(layers))

A real Latin square gives you True / True, and a histogram where every label appears exactly 7 times. If a model's config doesn't back its prose, you'll find out in ten lines.

You can do the same for the MoE claim (25 experts, top-7 + 1 shared):

print(cfg.num_experts, cfg.num_experts_per_tok, getattr(cfg, "shared_expert_intermediate_size", None))

What you'll notice: 7 labels, 5 mechanisms

The grid carries seven labels — nsa, differential, full, linear, sliding, compress, hybrid — but the model is named 5Attn. That's not a contradiction: linear and compress belong to the full-attention family, so seven placement labels collapse to five distinct mechanisms. The 7×7 grid is the placement structure; five is the mechanism count.

This is exactly the kind of detail that prose alone would let you hand-wave. The config doesn't.

One caveat that bites in production

# ✅ run inference one sequence at a time
out = model.generate(ids, max_new_tokens=256, do_sample=False)   # batch_size = 1

Some branches (the NSA-style sparse path) don't consume the padding mask. Left-pad a batch and queries can attend into pad tokens — no exception, just quietly wrong outputs. If you port this to a batched server or vLLM, handle the mask explicitly. The public demo backend pins batch_size=1 for this reason.

The honest part

Does heterogeneous attention actually beat a uniform stack? That is not settled, and the model card doesn't claim it is. The Latin square is the instrument built to answer that question without depth bias — not the answer itself. A controlled ablation is what would settle it.

Which is the real argument for full openness: a falsifiable claim plus the code to falsify it is worth more than a benchmark table you have to take on faith.

Links