DEV Community

ai maya
ai maya

Posted on

"How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method"

#ai #machinelearning #llm #opensource

How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method

Detect whether an LLM was trained from scratch or derived from Qwen, Llama, or DeepSeek — by fingerprinting architecture, tokenizer, and weight provenance from public Hugging Face artifacts. Includes the two traps almost everyone hits.

Keywords: LLM provenance · model fingerprinting · from-scratch vs fine-tuned · architecture signature · tokenizer overlap · embedding CKA · model lineage · Korean sovereign AI · open-weight derivatives

When a lab announces a "from-scratch, self-developed" foundation model, can an outsider verify that claim — using nothing but public files?

In late July 2026, several Korean labs shipped DeepSeek-rivaling "self-developed" models (e.g. LG's 750B K-EXAONE 2.0). The claim triggered a debate that spilled well beyond Korea: a single Zhihu thread crossed 2.7 million views, asking whether these models were trained from scratch or quietly built on Qwen / Llama / DeepSeek.

That question is answerable — quantitatively, reproducibly, from public artifacts. This post is the method. Everything below runs against any two repos on the Hugging Face Hub, and there's a live tool at the end.

Framing up front: building on open-weight bases (Qwen, Llama, DeepSeek, Mistral) is a legitimate, industry-standard practice. This is about provenance transparency, not accusation. The same yardstick applies to every model.

The idea: three independent fingerprints

A model leaves three separable fingerprints in its public files:

  1. Architecture — the shape declared in config.json
  2. Tokenizer — the vocabulary in tokenizer.json
  3. Weights — the learned representation in model.safetensors

Each answers a different question, and — crucially — they can disagree. That disagreement is where the signal lives.

Fingerprint 1 — Architecture (config.json)

Every transformers checkpoint ships a config.json. Six fields form a surprisingly discriminative signature:

import requests

FIELDS = ("model_type", "vocab_size", "hidden_size", "intermediate_size",
          "num_hidden_layers", "num_attention_heads", "num_key_value_heads")

def arch_fingerprint(repo: str) -> dict:
    url = f"https://huggingface.co/{repo}/resolve/main/config.json"
    c = requests.get(url, headers={"User-Agent": "genome/1.0"}, timeout=30).json()
    return {k: c.get(k) for k in FIELDS}
Enter fullscreen mode Exit fullscreen mode

The shape tuple (hidden_size, intermediate_size, num_hidden_layers, heads, kv) is effectively a fingerprint of the reference architecture. One matching field is a coincidence; five matching simultaneously is not. A few real matches I measured:

Shape (hidden · inter · layers · heads · kv) Exact match
3584 · 18944 · 28 · 28 · 4 Qwen2.5-7B
8192 · 29568 · 80 · 64 · 8 Qwen2.5-72B
5120 · 17408 · 40 · 40 · 8 Qwen3-14B
4096 · 14336 · 32 · 32 · 8 Llama-3.1-8B
7168 · 18432 · 61 · (moe 2048) DeepSeek-V3

An exact tuple match is strong evidence the architecture was adopted, not independently designed.

Fingerprint 2 — Tokenizer (a paternity test)

Architecture alone can mislead: a model can adopt a foreign architecture but train a genuinely new tokenizer (or vice-versa). Measure the tokenizer directly, comparing vocabularies with a min-overlap ratio:

def vocab_set(repo: str) -> set:
    url = f"https://huggingface.co/{repo}/resolve/main/tokenizer.json"
    v = requests.get(url, timeout=60).json()["model"]["vocab"]  # BPE: {token: id}
    return set(v.keys())

def tokenizer_overlap(a: str, b: str) -> float:
    A, B = vocab_set(a), vocab_set(b)
    return len(A & B) / min(len(A), len(B))   # 1.0 == one is a subset of the other
Enter fullscreen mode Exit fullscreen mode

This surfaces what config hides. One model matched Qwen2.5-7B's architecture exactly, yet its tokenizer overlapped Qwen by only ~0.38 — a "foreign brain, own language" case: adopted architecture, freshly trained (Korean) tokenizer. Others reused a base tokenizer verbatim (overlap = 1.000), confirming a straight fine-tune.

Why min, not union? Using min(|A|, |B|) in the denominator makes a reduced vocabulary that is a strict subset of a larger one score ~1.0 — the correct signal for "carved out of the base." A Jaccard (union) denominator would wrongly dilute that.

Fingerprint 3 — Weights (here be dragons)

The gold-standard question: were the weights trained from scratch, or continued-pretrained on a foreign base? Load the token embeddings and compare. Two traps await.

First, a helper to pull only the embedding tensor (no need to download the whole model):

import json, torch
from huggingface_hub import hf_hub_download
from safetensors import safe_open

def load_embedding(repo: str) -> torch.Tensor:
    try:
        idx = hf_hub_download(repo, "model.safetensors.index.json")
        shard = json.load(open(idx))["weight_map"]["model.embed_tokens.weight"]
    except Exception:
        shard = "model.safetensors"
    path = hf_hub_download(repo, shard)
    with safe_open(path, framework="pt") as f:
        key = next(k for k in f.keys() if k.endswith("embed_tokens.weight"))
        return f.get_tensor(key).float()
Enter fullscreen mode Exit fullscreen mode

Trap 1 — row-wise cosine is useless

The naive approach: for shared tokens, average the row-wise cosine similarity of the two embedding matrices. Shared lineage → similar embeddings, right?

Wrong — even when lineage is obvious. I measured near-zero mean cosine for both a known from-scratch model and a known Llama-derivative. The culprit is rotational invariance: a Transformer's hidden space has no privileged basis, so two models can encode identical information under an arbitrary orthogonal rotation. Row-wise cosine reads rotation as dissimilarity and tells you nothing about lineage.

Trap 2 — CKA helps, but is not conclusive

Linear CKA (Centered Kernel Alignment) is invariant to rotation and isotropic scaling — the right tool for comparing representations:

def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float:
    # X: (n, d1), Y: (n, d2) — SAME token order (shared vocabulary)
    X = X - X.mean(0, keepdim=True)
    Y = Y - Y.mean(0, keepdim=True)
    num = (X.T @ Y).norm() ** 2
    den = (X.T @ X).norm() * (Y.T @ Y).norm()
    return (num / den).item()
Enter fullscreen mode Exit fullscreen mode

A from-scratch model scored near-zero CKA against its candidate base — clean evidence of independent pretraining. But a continued-pretrained derivative scored only ~0.25 — barely above the ~0.21 baseline between two unrelated models of the same family. Large-scale training reshapes embeddings enough that CKA loses discriminative power on the derivative side.

The honest conclusion: the weights axis reliably confirms from-scratch (near-zero CKA), but it is not a strong detector of derivation. For that, architecture + tokenizer fingerprints stay primary. Report the weights axis as supporting evidence, never as a standalone verdict. (This is the single most important caveat in the whole method — and the one most write-ups omit.)

Bonus fingerprint — attention diversity as an originality proxy

Most models declare one attention mechanism; a few mix several. The count of distinct mechanisms in config.json is a cheap proxy for architectural originality:

KEYS = ("layer_types", "linear_attn_config", "sliding_window",
        "mamba2_d_state", "hyena_filter_order", "mla_kv_lora_rank", "attention_cls")

def attention_diversity(cfg: dict) -> list:
    # e.g. layer_types = [full_attention×16, sliding_attention×48] -> hybrid (2 kinds)
    return [k for k in KEYS if k in cfg]
Enter fullscreen mode Exit fullscreen mode

In my sweep, most models used a single grouped-query or multi-head-latent attention; some used a hybrid (layer_types=[full×16, sliding×48]); the most diverse combined mamba2, hyena, MLA, linear attention, gated-delta-net, native-sparse-attention, and sliding-window in one stack.

Combining axes → a single genotype

Collapse the two primary axes (architecture × weights) into one label:

Genotype Architecture Weights
🟢 Native self from-scratch
🔵 Adapted mostly self one axis borrowed
🟡 Mixed partial partial inheritance
🔴 Ported foreign (exact match) inherited

Keep tokenizer overlap and attention diversity beside the verdict, not folded into it, so readers can audit the raw evidence.

Results: nine organizations, one yardstick

Applying the identical pipeline to nine organizations' public foundation models (spanning large enterprises, telcos, mid-size firms, and startups), the picture is not uniform: some models match a foreign architecture and tokenizer exactly (Ported); others are self-built with no foreign match (Native); many sit in between. The per-model breakdown — 3D lineage graph, search, EN/中文/한국어, light + dark mode — is in the interactive tool below.

Reproduce it yourself

The functions above are the method. Point them at any two Hub repos:

print(arch_fingerprint("some/model"))
print(tokenizer_overlap("some/model", "Qwen/Qwen3-14B"))

# weights (shared-vocab pair):
X = load_embedding("candidate/model")
Y = load_embedding("Qwen/Qwen3-1.7B")
n = min(len(X), len(Y))
print("CKA:", linear_cka(X[:n], Y[:n]))   # near-zero => from-scratch
Enter fullscreen mode Exit fullscreen mode

Limitations & honesty

  • Not an accusation. Open-weight reuse is legitimate and widespread. This reports lineage, not wrongdoing.
  • Weights axis is supporting, not conclusive (Trap 2).
  • Same yardstick for every model, without exception.
  • All inputs are public; corrections are welcome.

Frequently asked questions

How can you tell if an LLM was trained from scratch or fine-tuned from another model?

Compare its config.json shape signature (hidden size, intermediate size, layer count) and its tokenizer.json vocabulary against known open-weight bases. An exact architecture match plus high tokenizer overlap indicates a derivative; a self-designed architecture with near-zero embedding CKA against candidate bases indicates from-scratch training.

What is CKA (Centered Kernel Alignment), and why use it instead of cosine similarity?

CKA is a rotation- and isotropic-scale-invariant similarity measure for neural-network representations. A Transformer's hidden space has no privileged basis, so plain row-wise cosine similarity is fooled by arbitrary orthogonal rotations between two models. CKA is not — which makes it the correct tool for comparing embeddings across models.

Is it legal to build an LLM on top of Qwen, Llama, or DeepSeek?

Yes. Using open-weight foundation models under their licenses (e.g. Apache-2.0 for many Qwen releases, the Llama Community License for Llama) is a legitimate, industry-standard practice. Provenance analysis reports lineage, not wrongdoing.

Are Korean sovereign-AI models built from scratch or based on Chinese/US models?

It varies by model. Some match a foreign architecture (Qwen, Llama, DeepSeek) exactly and are best described as "Ported"; others use fully self-built architectures and weights with no foreign match ("Native"); many are in between. The genotype of each is shown in the interactive tool.

How do you measure model provenance without downloading the full model?

Architecture and tokenizer fingerprints need only config.json and tokenizer.json (kilobytes to a few megabytes). For the weight axis, download just the embed_tokens.weight tensor via safetensors partial loading instead of the whole checkpoint.

Does an exact architecture match prove a model is copied?

No. Reusing an open-weight architecture is standard and legitimate. An exact config.json match shows the architecture was adopted; whether the weights were inherited or trained from scratch is a separate question, answered (with caveats) by the embedding-CKA axis.

Links

Model names, companies, and licenses are the property of their respective owners.

Tags: #machinelearning #llm #ai #opensource #huggingface #transformers

Top comments (0)