We open-sourced a foundation model. For four days, nobody could load it.

We released Aether-7B-5Attn with everything: weights, the full architecture source, the training data recipe down to per-source token counts and mixing weights, the tokenization script, the FSDP training code, every hyperparameter, the complete 162,000-step log, the evaluation code, and intermediate checkpoints. Apache-2.0. Trained from scratch on 16 × B200 over 46 days.

Four days later it had 33 likes and 2 downloads.

Not 2,000. Two.

The instinct is to blame marketing. It wasn't marketing. AutoModelForCausalLM.from_pretrained did not work, and neither did the fix everyone reaches for first. This is what actually broke, why the obvious remedy also fails, and the one-line check that would have caught it before release.

Likes are a browsing signal. Downloads are a usage signal.

The gap between 33 and 2 is the whole diagnosis. Liking a model costs one click from a feed. Downloading it means someone opened a terminal and typed the thing that everybody types:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("FINAL-Bench/Aether-7B-5Attn")

That raised:

ValueError: The checkpoint you are trying to load has model type `aether_v2_7way`
but Transformers does not recognize this architecture.

Expected — it's a custom architecture. So you add the flag that exists precisely for this:

model = AutoModelForCausalLM.from_pretrained(
    "FINAL-Bench/Aether-7B-5Attn", trust_remote_code=True)   # same ValueError

trust_remote_code=True changed nothing. That is the part worth internalizing.

trust_remote_code is not a switch. It's a pointer dereference.

The flag does not mean "go look around the repo for something that looks like modeling code." It means "you are allowed to execute the code this config points at." If config.json has no auto_map, there is no pointer, and the flag has nothing to dereference. Transformers falls back to its built-in registry, doesn't find aether_v2_7way, and raises the identical error.

Our config.json had no auto_map. The modeling code was sitting right there in the repo under aether_pkg/, fully public, and completely unreachable through any standard entry point. The only working path was a 20-line ritual — snapshot_download, sys.path.insert, manual class construction, subclassing to restore generate, load_state_dict(..., strict=False) — which we had helpfully documented in the model card, right where it would deter everyone who read it.

The fix is four lines:

"auto_map": {
  "AutoConfig": "configuration_aether_v2_7way.AETHERV27wayConfig",
  "AutoModel": "modeling_aether_v2_7way.AETHERV27wayModel",
  "AutoModelForCausalLM": "modeling_aether_v2_7way.AETHERV27wayForCausalLM"
}

Except that alone still wouldn't have worked.

The trap underneath: nested relative imports silently don't resolve

Our code was laid out like a normal Python package:

aether_pkg/
  configuration_aether_v2_7way.py
  modeling_aether_v2_7way.py
  v2_attentions/
    nsa.py
    differential.py

with modeling_aether_v2_7way.py doing from .v2_attentions.nsa import NSAAttention. Idiomatic Python. It imports fine locally.

Remote code loading does not use Python's import machinery to find files. It scans your module with a regex, then builds filenames. From transformers/dynamic_module_utils.py:

relative_imports += re.findall(r"^\s*from\s+\.(\S+)\s+import", content, flags=re.MULTILINE)
...
new_import_files = [f"{str(module_path / m)}.py" for m in new_imports]

from .v2_attentions.nsa import captures v2_attentions.nsa, and the second line turns that into the literal filename v2_attentions.nsa.py — dots and all. No such file exists. The dot is never treated as a path separator.

So: files reachable by remote code loading must be flat at the repository root, with single-level relative imports only. from .nsa import NSAAttention, never from .pkg.nsa import.

Two more that bit us in the same pass:

• Absolute imports hide inside function bodies. One of our models had from aether_pkg.v2_attentions.mla import MLAAttention on line 368, inside a method. A regex sweep over the import block at the top of the file misses it entirely. Only actually instantiating the model surfaced it.
• GenerationMixin is no longer inherited by PreTrainedModel (transformers ≥ 4.50). Our class defined prepare_inputs_for_generation and had no .generate() at all. One base class in the signature.

Verifying you didn't silently break the weights, without downloading the weights

Our old loading snippet used load_state_dict(..., strict=False). That flag suppresses key mismatches. Combined with from_pretrained, a mismatch means missing tensors get randomly initialized in silence — the model loads, generates fluent-looking tokens, and is quietly garbage.

Before publishing any from_pretrained example, check that the checkpoint keys and the model keys agree. You don't need the 13 GB. A safetensors file starts with an 8-byte little-endian header length followed by a JSON header listing every tensor, so two HTTP Range requests are enough:

url = f"https://huggingface.co/{repo}/resolve/main/model.safetensors"
n = struct.unpack('<Q', requests.get(url, headers={'Range': 'bytes=0-7'}).content)[0]
hdr = json.loads(requests.get(url, headers={'Range': f'bytes=8-{7+n}'}).content)
ckpt_keys = set(hdr) - {'__metadata__'}

Then build the model from config alone — shrink hidden_size and vocab_size if you like, parameter names don't depend on dimensions — and diff:

missing  = model_keys - ckpt_keys   # will be randomly initialized. must be empty.
unexpect = ckpt_keys - model_keys   # will be silently dropped.

For us that read 4,630 : 4,630, missing 0, unexpected 0. Total download: a 558 KB header.

The check that would have prevented all of it

AutoConfig.from_pretrained(repo, trust_remote_code=True)

One line. Run it against the public repo, from a clean environment, after you push. It costs a few hundred kilobytes and it is the difference between a release and a rehearsal.

We swept all 240 of our public models with the rule "custom model_type AND no auto_map" and found three more repos in the same state, one of which had been sitting broken for weeks with nobody reporting it. Nobody reports it. They just close the tab.

It works now

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "FINAL-Bench/Aether-7B-5Attn"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()

ids = tok("대한민국의 수도는", return_tensors="pt").to("cuda")   # base model: give it a prefix, not an instruction
print(tok.decode(model.generate(**ids, max_new_tokens=32)[0], skip_special_tokens=True))

~14 GB VRAM in bfloat16. Run at batch_size=1 — the NSA branches don't consume a padding mask, so padded batches can corrupt results silently. There is no KV cache, so decoding is O(n²) and slow by construction: roughly 1.5 tok/s on a T4, 7 tok/s on a B200. It is a research base model with no safety tuning. Don't deploy it as-is.

What's inside, if you're curious: 6.59B total / ~2.98B active MoE, and its 49 layers place five distinct attention mechanisms on a 7×7 Latin square, so no mechanism concentrates at any depth. That last property is a control, not a performance claim — we haven't run the ablation that would isolate whether heterogeneous placement helps, and we say so on the card. The data recipe, training code and full logs are all in the repo if you want to check any of it yourself.

The lesson, stated plainly

"Open" is not a property of what you upload. It is a property of what someone else can actually run.

We published the training data, the code, the logs and the checkpoints — a genuinely reproducible release — and then shipped it behind a locked door because of four missing lines of JSON. The openness was real. The access wasn't. Those are different things, and only one of them shows up in your download counter.