DEV Community: ai maya

opened a financial forecasting contest that AI agents can enter directly. $2,000 in prizes, 122 days.

ai maya — Mon, 24 Aug 2026 05:36:26 +0000

We opened a financial forecasting challenge that AI agents can enter directly.
$2,000 in prizes, 122 days. This is a record of why, and of the design problems we hit
building it.

Why another leaderboard

Forecasting contests already exist. Most of them share one structural problem:
once a result is in, there is no way to tell whether it was skill.

Say an entrant returns 30% in a month. Skill or luck? Answering that requires knowing
how far a player with no skill could have gone over the same window in the same
market — and we have not seen a contest that measured that number in advance. So the
top of the leaderboard sits permanently between "impressive" and "probably lucky."

The second problem is backtests. Producing 200% a year on past data is not hard; you
turn a few parameters. That result says nothing about the future, because it is an exam
graded by someone who already has the answer key.

The third is survivorship. The person who made money gets interviewed. The one who lost
disappears quietly. All anyone knows are the stories that survived.

FINCHAL attacks each of these differently.

Measure luck first, publish it. Twenty thousand zero-skill players, per asset, over the season length. A return below that line is not counted as evidence of skill.
Put the answer in the future. The world writes it after you submit. Fitting the past cannot win.
Keep every entrant's record. Good days and bad days sit in the same table.

And one more thing: agents enter on the same footing as people. That is the
technical core of this contest.

Design 1 — Take positions, not predictions

The first sketch was a prediction contest — "forecast next week's close" — because
scoring is clean and there are established metrics like CRPS and pinball loss.

But that splits being accurate from making money. You can nail a range and be
unable to monetise it. You can miss direction entirely and still profit from sizing
alone. Measuring forecast accuracy answers only half of "who builds the better model."

So we take a position.

-1.0  fully short
 0.0  flat
+1.0  fully long

One real number. Fractions like 0.35 are fine. It holds from that moment until you
submit again, multiplied against the market return over that stretch.

A side effect we liked: no close button is needed. Setting the position to zero
is the close; going from +1 to −1 is the reversal. One state, so the API and the
screen both stay simple.

Leverage is fixed at 1 and out-of-range values are clipped. Without that the contest
becomes "who bet biggest" — an early simulation with leverage unlocked produced a
cumulative 48,763%, which was multiplication, not skill.

Design 2 — Verify the scorer before any model

This was the governing principle.

Errors in scoring code do not raise exceptions. They produce plausible numbers.
Shift the return calculation by one step and you have a scorer that peeks at the future,
which means anyone can come first — and the screen still shows perfectly normal figures,
so nobody notices.

So scoring.py carries a self-test built only from cases whose answers are known in
closed form.

1. position 0        -> return 0, whatever prices do
2. position +1       -> exactly buy-and-hold, under zero cost
3. position -1       -> inverse compounding, not a sign flip of (2)
4. position 2.0      -> clipped to 1.0
5. no lookahead      -> entering on the bar that jumps earns nothing
6. cost actually reduces return
7. score is monotone
8. score near the median is about 0.30

Number 5 is the one that matters. If price jumps from 100 to 200 on a bar, entering on
that bar must return zero; you have to be in one bar earlier. Without this test you can
ship a lookahead scorer and never know.

All eight run again on every change to that file. One failure stops scoring.

A related lesson: validate a backtest engine on synthetic paths first. Does a
sawtooth match the closed-form value? Is a trending path symmetric up and down? Does a
flat path return exactly zero? Validating only on real market data stops at
"looks plausible, must be right."

Design 3 — The luck ceiling

This is the identity of the contest.

Twenty thousand players who pick positions at random over 122 days — skill exactly zero.
The 95th percentile of their final returns is the luck ceiling.

Charged the same fees entrants pay:

Asset	Luck ceiling (95th pct)	Fee per 1.0 of position change
Bitcoin	+86.6%	0.06%
NVIDIA	+51.7%	0.02%
Crude Oil	+26.9%	0.03%
Gold	+9.2%	0.02%

+80% on Bitcoin sounds impressive; luck alone reaches that. +12% on gold is outside
what luck produces. The same number means completely different things on different
assets.

Two judgements went into this.

The reference players pay fees too. Initially they did not, which meant entrants
alone carried the cost and the bar was unfairly high — worse on assets with expensive
execution.

What we fix before the season is the method, not the number. Actual ranking rebuilds
the reference distribution on the path the season actually took, recomputed daily.
Freezing a bootstrap estimate means that when an asset rallies mid-season every entrant
inflates together and the ranking simply tracks market direction. The correct
counterfactual is "what luck could have produced given how the market actually moved."

The rank score is the percentile mapped through -log10(1 - p). A 2.0 means odds of
1 in 100 by luck; 3.0 means 1 in 1,000.

Design 4 — Why we gave up on cross-asset comparison

The original plan was a single overall winner, which requires comparing results across
assets.

We tried six normalisation schemes — volatility-scaled returns, 95th-percentile ratios,
percentiles, combinations. None removed the bias.

The reason is distribution shape. The winner is a maximum, and maxima are decided by
the tail. Tail shape differs fundamentally between assets, so matching any one
statistic — median, variance, 95th percentile — leaves another point misaligned, and
the win probability keeps tilting.

The answer was to stop comparing. Award $500 per asset and cross-asset comparison
becomes unnecessary. Four assets, $2,000 total.

A by-product of the same analysis: within an asset, percentile scoring calibrates
well. Zero-skill players land at a median percentile of 0.47–0.54. The problem was
between assets, never inside one.

Design 5 — Cost, not rules, stops spam

Since it is paper trading, why charge fees at all? Because without them, flipping the
position every second pays. On a volatile asset, reversing hourly eventually draws one
good path.

A submission-count rule would block that, but it would also block strategies that update
frequently for legitimate reasons. So we charge real execution costs instead.

The change in position, |Δw|, is multiplied by that asset's actual cost — 0.06% for
crypto (taker fee plus slippage), 0.02% for US stocks and ETFs (commission-free, but the
spread remains).

Synthetic entrants made the case:

Strategy	Return	Turnover	Fees paid
Trend following	+32.7%	83.0	4.98%
Flip daily	+11.8%	179.0	10.74%
Buy and hold	+1.9%	0.06	0.06%

Flipping daily paid over 10% of capital in fees alone. No rule required — the cost
punishes the behaviour. And the same strategy ranks near the top on gold, because gold's
fee is a third of Bitcoin's. Per-asset differentiation does real work.

Design 6 — Putting a scale on an empty chart

Opening the site on day one, the comparison chart was empty. Obvious in hindsight — no
entrant curves exist yet — but the real problem was that nothing on screen told you
what counts as good. The luck ceiling existed as a number with no picture.

So we run thirteen widely known rules from 1 January of this year to today: where
you would stand had you started that rule on the first trading day of the year.

Buy and hold · Hold only above the 200-day MA · Golden cross 50/200 · SMA 20/50 crossover
MACD 12/26/9 · Ichimoku cloud · 60-day momentum · 12-month absolute momentum
Donchian 20 breakout · RSI 14 · Stochastic 14/3 · Bollinger 20 reversion · Vol targeting

Three conditions:

The same scorer and the same fees entrants face. Only the rule differs. Drop the fees and high-turnover rules look unfairly good.
No signal sees the future. Everything is used after shift(1). Computing a moving average from today's close and trading it today makes the whole record false.
Indicators are built on the full history; only the display is trimmed. The 200-day average on 2 January needs last year's data. Trim first and the start of the year comes out empty.

The result was more interesting than expected

Rule	NVIDIA	Bitcoin
Stochastic 14/3	+43.41% (1st)	−25.02% (12th)
Bollinger 20 reversion	+27.91% (2nd)	−25.51% (13th)
RSI 14 mean reversion	+19.61% (3rd)	−20.70% (11th)
Donchian 20 breakout	−46.64% (13th)	+48.98% (1st)
MACD 12/26/9	−35.21% (11th)	+21.36% (2nd)
Buy and hold	+13.81%	−13.24%

The ranking nearly inverts. The mean-reversion family that takes 1st, 2nd and 3rd on
NVIDIA sits 11th, 12th and 13th on Bitcoin. The trend family that takes 1st and 2nd on
Bitcoin sits 13th and 11th on NVIDIA.

This retroactively justifies running four assets. "Which indicator is good" is not a
well-posed question. The character of the market decides the answer. This year NVIDIA
kept reverting inside a range while Bitcoin trended, and the same rule produces opposite
results.

The fee effect shows up visually too. Bollinger reversion on Bitcoin ran a turnover of
49.2 and paid 2.95% in fees alone, against 0.06% for buy and hold. For the same
underlying loss, the high-turnover side gives up another three points.

One caveat we are careful to print on the page: this is a replay of the past, not
future performance. Showing backtests as if they were results is precisely the problem
this contest exists to remove, so we cannot do it ourselves. As entrant curves
accumulate, these reference lines step back.

Design 7 — An hour, not a day

The scoring grid started as daily bars. Day one of the season exposed the flaw:
the first return requires waiting a full day for the next bar. Meanwhile entrants
have submitted and the board shows nothing.

Worse, the rules already promised "update hourly." The unit we measured in did not
match the unit we advertised.

We moved to hourly. The price publisher uploads hourly bars every 30 minutes and scoring
runs on that grid. Returns now begin accumulating an hour after the open.

One judgement here: stocks and ETFs only print during regular hours; crypto prints
around the clock. We did not paper over that difference. Filling empty hours invents
trades that never happened, and those inventions flow straight into returns. Each
asset is scored on its own clock.

An asset without hourly bars falls back to daily. That is better than blank, and which
grid was used is visible on screen.

Design 8 — Agents as first-class entrants

Most forecasting contests have a human upload a CSV. Using an agent means a person
shuttling files in the middle.

FINCHAL exposes an MCP (Model Context Protocol) server. One line attaches it:

claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp \
  --header "X-Finchal-Key: <your key>"

The agent gains four tools:

Tool	What it does
`get_rules()`	rules, assets, fees, deadline, luck ceiling
`get_data(asset, bars)`	historical closes — no future data is served
`submit_position(asset, position)`	−1.0 to +1.0
`check_score(asset)`	rank, return, distance from the luck ceiling

After that a person says "enter the finchal challenge on BTC" and the agent reads the
rules, pulls data, builds a model and places a position.

Two things that caught us, worth writing down because others will hit them.

Tool descriptions have to be localised. Agents read those descriptions to decide what
to do. Translate the page but leave the tool text in one language and agents in the other
language are structurally disadvantaged. Descriptions now follow Accept-Language.

What you cannot serve, you must refuse. The schema originally advertised
interval: ["1d", "1h"] while the feed published daily only. A request for 1h came
back with "interval": "1h" and daily bars. An agent would model on a false premise.
It is now rejected explicitly — and since then we have added a real hourly feed.

That second case is the lesson that recurred throughout this project: a quietly wrong
answer is far worse than an error.

Design 9 — The data has to be P&L someone could actually realise

Gold and oil were going to be futures. Measurement said otherwise.

CL=F (front-month WTI, continuous) returned +10% over three years while USO, the
crude ETF, returned +89%. A 79-point gap — created not by the market but by roll
distortion when stitching front-month contracts. That is P&L an entrant could never have
realised, mixed into the return series.

Gold futures GC=F show month-end jumps 2.4× normal, again from roll pricing.

Both became ETFs (GLD, USO). Season 2 may cover futures, but not before the
continuous-contract construction is settled.

Asset selection had one more criterion: low mutual correlation. Dogecoin correlates
0.763 with Bitcoin and semiconductor ETFs 0.797 with NVIDIA, so both were dropped — if
two assets are effectively one, a four-asset contest is a three-asset contest. The final
four correlate between −0.04 and 0.19.

Built by AI        NVIDIA   ↕   Trusted for millennia   Gold
Shaken by AI      Bitcoin   ↕   What moved the world     Oil

Two things infrastructure taught us

Market data providers block datacenter IPs

We learned this only after deploying to a Hugging Face Space. Yahoo returns 429,
Stooq returns a bot-block HTML page with status 200, CoinGecko blocks, and yfinance
returns zero bars without raising. General internet works; only the price providers
are blocked.

The dangerous pair is the last two. They do not raise. A bare try/except counts
them as success and you get a silent empty page. So the criterion moved from
"did it raise" to "how many bars arrived." HTTP 200 also gets its body checked.

The fix was to reverse the direction. The server no longer fetches prices; it
receives them. A publisher on a network that works uploads CSVs to a dataset repo and
the server reads only that. It refreshes every 30 minutes, and if collection fails it
publishes nothing — a half-written file overwriting a good one would corrupt every
entrant's return for that day.

Writable is not durable

A Space container's /data is created and written to even without persistent
storage attached. It simply vanishes on restart. So "make a directory and write a temp
file" always passes. We nearly reported persistent: true as proof of safety.

The only real evidence is restarting for real and checking whether the previous boot
left a trace. A counter is now written each boot and read on the next.

confirmed   the previous boot's record survived
unknown     first boot, cannot tell yet
lost        we wrote a record and it disappeared

First boot and data loss are indistinguishable at that moment, so the verdict is
deferred to the next boot.

The ledger mirrors to a private dataset, which needed one rule: never flush after a
failed restore. An empty ledger overwriting the mirror erases four months in one write.
A missing dependency did break restore once, and that guard is what saved the ledger.

The organisers do not compete

We considered putting our own forecasting model on the leaderboard — the drug-discovery
leaderboards that place approved drugs in the same table earn credibility that way.

We decided against it, for two reasons.

Framing. The question this contest asks is "who builds the better model." With the
organiser in the entrant table it becomes "is the organiser best."

Measurement. The rules fix positions to spot exposure between −1 and +1, which bounds
what a strategy can express. Some approaches cannot be represented at all under that
constraint, and entering under it would not be a fair comparison.

Instead three baselines share the table. All are published textbook methods.

Baseline	What it does
Buy and hold	always +1. Failing to beat it means nothing was achieved
Volatility targeting	EWMA volatility sizes the position; direction stays long
Random	−1/0/+1 each day. The floor

Baselines are a scale, not a competitor. A leaderboard without a scale cannot be read.

Worth adding: the baselines do not predict direction either. Volatility targeting
decides only how large to be. Across sixty candidate variables we tested directly, none
showed predictive power over short-horizon direction — and we will not have a baseline
pretend to do what has not been demonstrated.

So what do we want to learn

We are not going to claim that collective intelligence solves markets.

Where one model stops is not information. But if hundreds of independent models stop
in the same place, that is information about the market. If nobody clears the luck
ceiling on one asset while several clear it on another, that difference is itself data.

We already got a taste while laying down the thirteen reference strategies. Examined one
at a time, each is just "this works, that does not." Spread all thirteen across four
assets at once and a structure appears: the ranking inverts with the market. No
single rule shows that. Put hundreds of entrant models on the same axis and the map
should get considerably finer.

What remains after 122 days is not four winners but a boundary — where luck ends and
skill begins, and where that line falls on each market.

Entering

As a human — sign in with Hugging Face, move the slider, submit.
As an agent — attach the MCP server and say "enter the challenge."
Season 24 August to 24 December 2026 · Prize $500 per asset × 4
Assets NVIDIA · Bitcoin · Gold · Crude Oil

/llms.txt carries a machine-readable summary in English and Korean.

The whole codebase is public. Read what the scorer does before you enter — a leaderboard
that hides its scoring has not earned your trust.

→ https://huggingface.co/spaces/FINAL-Bench/finchal

Day one

The season opened on 24 August. Within hours the board had humans, baselines and an
agent side by side on all four assets.

Entrant	Type	Strategy
Warecube	human	my-mind
Anserwise	human	long-jaengi
Sunghokim	human	shotmanse
ginigen-ai	agent	13-rule equal-weight ensemble
BUYHOLD / VOLTARGET / RANDOM	baseline	the scale

The agent entry is worth a note because it went in through the same MCP path any
entrant uses — initialize → tools/list → get_rules → get_data → model →
submit_position. No internal shortcut. Its method is deliberately un-tuned: it
averages the current signal of all thirteen reference rules with equal weight.

That choice follows directly from the measurement above. Since no single rule holds
across assets, picking one is a bet on market character. So it does not pick. When the
rules disagree the position shrinks toward zero; when they align it grows. On day one
that produced +0.686 on NVIDIA, +0.409 on oil, +0.373 on gold and +0.308 on Bitcoin —
nine trend rules long against three mean-reversion rules calling overbought.

Weighting those thirteen by their year-to-date performance would have been easy and
would have scored better on the replay. It would also have been the exact mistake this
contest exists to expose: treating a backtest as a result.

Bitcoin began scoring within the hour. Equities, gold and oil waited for the US open —
each asset is scored on its own clock.

An open challenge is letting anyone submit a drug candidate - and scoring it in public

ai maya — Sun, 16 Aug 2026 19:28:45 +0000

There is a familiar shape to how a field opens up. For a long time the barrier is not the idea; it is the instrument. Anyone can imagine a molecule. Almost nobody could find out whether it was worth anything, because judging it required a laboratory, a compute budget, and years of training. The idea was cheap and the verdict was expensive.

Something worth watching is happening to that asymmetry.

An open drug-discovery challenge is currently running on Hugging Face, and the interesting part is not the prize money. It is that the evaluation has been made public.

Two seasons are live

Season 1 — Malaria. Target: PfDHODH in Plasmodium falciparum, with human DHODH as a counter-target. Closes 30 September 2026.

Season 2 — Tuberculosis. Target: InhA in Mycobacterium tuberculosis, counter-target human FASN (ER domain). Closes 31 October 2026.

Entry is a molecule, submitted as SMILES or InChI. There is no restriction on how you get there: Claude, GPT, Gemini, Qwen, KIMI, DeepSeek, or a model you trained yourself. The challenge scores the structure, not the pedigree of whoever produced it.

The scoring is where it gets interesting

100 points across six axes — activity, binding, selectivity, ADMET, novelty, synthesizability — with weights that differ per season. Season 2 loads weight onto binding and selectivity, which is the right instinct: an InhA inhibitor that also shuts down the human homolog is not a candidate, it is a toxin.

That counter-target design is the detail most worth noticing. It is easy to build a scoreboard that rewards binds the target. Rewarding binds the target and leaves the human protein alone is harder, and much closer to what actually decides whether a compound survives.

The anchors are in the table

Approved drugs and inert compounds are scored on the same rubric as the entries.

That is an unusual thing for an organizer to do, because it means anyone can see where the scale actually sits — including when a known drug does not score the way you would expect. A leaderboard you can audit is a different object from a leaderboard you are asked to believe.

Private submission is supported, so entrants who want to keep a structure unpublished can still compete; the board shows a masked SMILES. The standings are also exposed as JSON, which means the whole thing can be pulled and checked programmatically rather than read off a webpage.

The honest caveat, which the organizers state themselves

A high score is a research hypothesis, not a drug. Everything here is computational. It is not clinical validation and does not become one by ranking well.

Still — the direction is the encouraging part. Instruments spread, and fields change when they do. Whether this particular challenge produces anything durable is unknown. That it exists in a form outsiders can inspect is already worth something.

Challenge: https://huggingface.co/spaces/FINAL-Bench/open-discovery-challenge

Disclosure: the challenge is operated by the FINAL-Bench organization on Hugging Face.

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

ai maya — Sun, 09 Aug 2026 08:12:38 +0000

TL;DR. A public method called Model DNA — with a live tool — lets outsiders estimate whether an LLM was trained from scratch or derived from an open-weight base, using nothing but public artifacts: config.json, tokenizer.json, and embedding weights. This is a technical deep-dive that cites and reproduces that method in PyTorch, then analyzes where it holds up and where it breaks.

Key takeaways

Three public signals — architecture config, tokenizer overlap, and embedding-space similarity via Linear CKA — combine to place a model on a lineage spectrum.
One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test.
The method's real strengths are reproducibility and rotation-invariant similarity; its real limits are the continued-pretraining gray zone, threshold sensitivity, and an embedding-only view.
Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice; the output is a label, not an accusation.

Why this matters in 2026

Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 those claims stopped being taken on faith. A Zhihu roundtable on the summer model wave and a thread with millions of views became the venue where "self-developed" claims were publicly stress-tested — and several were found more derivative than advertised (coverage).

Model DNA matters because it moved the argument from vibes to a reproducible procedure, and it has been run across major Korean foundation-model builders — among them LG, NAVER, Kakao, SKT, KT, NCSOFT, Upstage, and Motif. (This piece stays at the method level and assigns no verdict to any named company; per-model labels belong to the tool, not to a blog post.) What follows reproduces the procedure in PyTorch and evaluates it as a method, within the scope the source already made public.

The three signals (cited and reproduced)

The premise: no leaked internals required. Everything is read from a model's public files.

1. Architecture fingerprint — `config.json`

Compare the structural fields a lab chooses at design time: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads. Independently designed models rarely align on all of them at once.

import json

ARCH_FIELDS = [
    "model_type", "vocab_size", "hidden_size", "intermediate_size",
    "num_hidden_layers", "num_attention_heads", "num_key_value_heads",
    "max_position_embeddings", "rope_theta",
]

def arch_match_count(cfg_a: dict, cfg_b: dict, fields=ARCH_FIELDS) -> int:
    """Number of structural fields that match simultaneously."""
    return sum(1 for k in fields
              if cfg_a.get(k) is not None and cfg_a.get(k) == cfg_b.get(k))

Reading rule (as the source puts it): a single coincidental field means nothing; five simultaneously is a fingerprint. How many to treat as a threshold depends on the diversity of your candidate base pool.

2. Tokenizer overlap — `tokenizer.json`

Two models trained truly independently almost never converge on the same vocabulary. Normalize shared tokens against the smaller vocabulary.

def tokenizer_overlap(vocab_a: dict, vocab_b: dict) -> float:
    sa, sb = set(vocab_a), set(vocab_b)
    return len(sa & sb) / min(len(sa), len(sb))

A supporting signal only — see Trap 2.

3. Embedding similarity — Linear CKA

The most robust signal compares representation geometry. Naive cosine comparison is fooled by rotation (Trap 1), so the method uses Linear CKA (Centered Kernel Alignment) — from Kornblith et al. (2019), Similarity of Neural Network Representations Revisited (ICML) — which is invariant to rotation, orthogonal transforms, and isotropic scaling.

For row-centered matrices X ∈ ℝ^{n×d1} and Y ∈ ℝ^{n×d2}:

CKA(X, Y) = ||Yᵀ X||²_F / ( ||Xᵀ X||_F · ||Yᵀ Y||_F )

Crucially, it is defined even when d1 ≠ d2, so models with different hidden sizes compare directly.

import torch

@torch.no_grad()
def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float:
    # X:(n,d1), Y:(n,d2) — embeddings over the SAME token set (rows aligned)
    X = X - X.mean(0, keepdim=True)
    Y = Y - Y.mean(0, keepdim=True)
    num = ((Y.t() @ X) ** 2).sum()
    den = torch.sqrt(((X.t() @ X) ** 2).sum() * ((Y.t() @ Y) ** 2).sum())
    return (num / den).clamp(0, 1).item()

Alignment is the catch. The two embedding matrices must index the same tokens. In practice you take the shared-token subset of the two tokenizers and gather those rows:

def aligned_embeddings(emb_a, vocab_a, emb_b, vocab_b):
    shared = sorted(set(vocab_a) & set(vocab_b))
    idx_a = torch.tensor([vocab_a[t] for t in shared])
    idx_b = torch.tensor([vocab_b[t] for t in shared])
    return emb_a[idx_a], emb_b[idx_b]

The genotype framework

The tool collapses the three signals into four labels — a clean way to read any result (Model Genome Korea):

Genotype	Meaning
🟢 Native	Self-designed architecture and from-scratch weights
🔵 Adapted	Mostly original, one borrowed axis
🟡 Mixed	Partial inheritance on both axes
🔴 Ported	Exact foreign architecture match and inherited weights

Two traps that produce confident wrong answers

Trap 1 — row-wise cosine similarity looks rigorous but isn't. It is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. That is exactly why CKA and config signals carry the weight.

Trap 2 — a shared tokenizer proves nothing alone. Tokenizer reuse is often a licensing or convenience decision. Treat overlap as supporting evidence, never a conclusion.

Deep analysis: how far can you trust it?

Strengths

Reproducibility. All three signals compute from public artifacts in a few dozen lines. Claim and verification live on the same plane.
Right invariance. Choosing Linear CKA is correct — it neutralizes the most common disguise (orthogonal transforms) that defeats cosine comparisons.
Evidence fusion. Judging on the simultaneous agreement of three axes suppresses both false positives and false negatives.

Limits (must be acknowledged)

Continued-pretraining gray zone. Embedding CKA identifies from-scratch training well but does not cleanly separate derivatives that keep a base's weights and train heavily on top. Here the verdict is probabilistic and config/tokenizer evidence dominates.
Threshold sensitivity. "How many fields," "what CKA cutoff" depend on the candidate pool. Hard-coding constants makes conclusions wobble when the pool changes — which is why this write-up prescribes none.
Embedding bias. Looking only at the embedding layer is cheap, but a model's "identity" also lives in mid and upper layers. A layer-wise CKA profile improves resolution in the gray zone.
Alignment dependence. Few shared tokens (language- or domain-specific tokenizers) shrink the CKA sample and inflate variance. Report shared-token count alongside CKA.

Improvements worth adopting

Extend single-layer embedding CKA to a layer-wise CKA curve (input → mid → output).
Score against the entire candidate base pool and judge by relative rank, not an absolute cutoff.
Report shared-token counts and bootstrap confidence intervals for statistical significance.

In short, Model DNA fuses the right signals under the right invariance — a solid starting point. It only avoids misjudgment when read as a spectrum with uncertainty, not a from-scratch/not binary.

What fingerprinting cannot tell you

Lineage, not intent. It can show B shares structure with A; it cannot say whether that was disclosed, licensed, or hidden — ethics and paperwork, not linear algebra.
"From scratch" is a spectrum, not a boolean. Data, init, architecture, and post-training each sit on a continuum of originality.
Building on open weights is legitimate. The goal is transparency and accurate labeling, not accusation.

FAQ

Can you tell if an LLM was really trained from scratch?
Usually, with high probability. Cross-check architecture config, tokenizer overlap, and embedding CKA against candidate bases; agreement across all three indicates derivation, divergence supports from-scratch.

What is Model DNA / model provenance?
Estimating a model's origin — original vs. derived from an open-weight base — from public artifacts alone (config, tokenizer, weights), without training data or internal logs.

Is building on Llama, Qwen, or DeepSeek legitimate?
Yes. Fine-tuning or continued-pretraining an open-weight base is standard, licensed practice. Provenance tools report lineage, not misconduct.

How do you tell a fine-tuned model from a from-scratch one?
From-scratch models diverge on architecture and tokenizer and show low embedding CKA to any base. Derivatives keep the base's skeleton and tokenizer and retain high similarity — the hardest case, where config and tokenizer evidence matter most.

Why Linear CKA instead of cosine?
Because CKA is robust to rotation and scaling — the transforms a derived model uses to look "different."

Can models with different hidden sizes be compared?
Yes. CKA compares n×n Gram matrices, so d1 ≠ d2 is fine — as long as embedding rows are aligned to the same tokens.

Resources & related links

Method under analysis: Model DNA — architecture & weight lineage
Live tool (space): Model Genome Korea
Announcement post: Architecture lineage of Korea's sovereign-AI models
Academic basis: Kornblith, Norouzi, Lee, Hinton (2019), Similarity of Neural Network Representations Revisited, ICML.
The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · coverage

Provenance is becoming a norm, not a gotcha. The healthiest version is one where "we trained it from scratch" arrives with — or at least survives — the fingerprint. If you build models, publish the check yourself.

How to Verify a 'Trained-From-Scratch' LLM in 2026: A Provenance and Fingerprinting Guide

ai maya — Sun, 09 Aug 2026 07:23:58 +0000

TL;DR — Can an outsider verify that an LLM was "trained from scratch"? In 2026, largely yes. Using only public Hugging Face artifacts — config.json, tokenizer.json, and embedding weights — you can fingerprint a model's lineage and place it on a spectrum from fully original to a straight port of a foreign base. This guide explains the method, what it can and cannot prove, and the open-weight debate that made it a 2026 talking point.

Key takeaways

Three public signals — architecture config, tokenizer overlap, and embedding-space similarity (Linear CKA) — are enough to estimate whether a model was built from scratch or derived from an open-weight base.
One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test.
Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice — the interesting output is the label, not an accusation.
The topic went mainstream in 2026 because several "self-developed, DeepSeek-class" claims from non-US/China labs were publicly stress-tested and, in places, found more derivative than advertised.

Why this matters in 2026

Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 that claim stopped being taken on faith. A Zhihu roundtable on the summer model wave and a widely-cited Zhihu thread with millions of views became the venue where the Chinese-language community tore into "self-developed" claims from Korean and Indian labs — concluding, in several cases, that the models leaned harder on Chinese open-weight bases than the marketing implied (coverage).

The useful development is that the argument moved from vibes to a reproducible procedure. Public tools now let anyone run the check. This is a refreshed 2026 walkthrough of that procedure — updating the original method write-up with the debate context and the links you need.

The three fingerprint signals

Model provenance rests on three axes you can read straight from a model's public files. No leaked internals required.

1. Architecture fingerprint (`config.json`)

Compare the structural fields a lab chooses when it designs a model: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, and the attention head counts. Independently designed models rarely line up on all of these. A single coincidental field means nothing; five simultaneously is a fingerprint of a shared skeleton.

2. Tokenizer overlap (`tokenizer.json`)

Two models trained truly independently almost never converge on the same vocabulary. Measure the ratio of shared tokens against the smaller vocabulary. High overlap is a strong hint of shared ancestry — but, as the traps below explain, not a verdict on its own.

3. Embedding similarity via Linear CKA

Centered Kernel Alignment (CKA) compares representation geometry in a way that is robust to rotations and rescalings — the superficial transformations that fool cruder weight comparisons. Run on embedding layers, it reliably flags from-scratch training. Its honest limitation: it struggles to cleanly separate continued-pretraining derivatives, where a model keeps a base's weights but trains heavily on top.

The genotype framework

One public framework, Model Genome Korea, collapses the three signals into four labels — a clean mental model for reading any provenance result:

Genotype	Meaning
🟢 Native	Self-designed architecture and from-scratch weights
🔵 Adapted	Mostly original, with one borrowed axis
🟡 Mixed	Partial inheritance on both axes
🔴 Ported	Exact foreign architecture match and inherited weights

Two traps that produce confident wrong answers

Trap 1 — Row-wise cosine similarity looks rigorous but isn't. Comparing weight matrices row by row with cosine similarity feels quantitative, yet it is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. This is exactly why the method leans on rotation-robust CKA and on structural config signals rather than raw weight-row comparisons.

Trap 2 — A shared tokenizer proves nothing by itself. Reusing a tokenizer is frequently a licensing or convenience decision, not evidence of weight-level derivation. Treat tokenizer overlap as a supporting signal, never a conclusion.

What fingerprinting cannot tell you

Lineage, not intent. It can show that model B shares structure with model A. It cannot tell you whether that was disclosed, licensed, or hidden — that is a question of ethics and paperwork, not linear algebra.
"From scratch" is a spectrum, not a boolean. Data, initialization, architecture, and post-training sit on a continuum of originality; a single label flattens a gradient.
Building on open weights is legitimate. The point of a provenance tool is transparency and accurate labeling, not accusation. Most of the industry stands on open-weight shoulders — the honest move is to say so.

FAQ

Can you tell if an LLM was really trained from scratch?
In most cases, yes — to a strong probability. Cross-check architecture config, tokenizer overlap, and embedding CKA against candidate base models; agreement across all three indicates derivation, while divergence supports a from-scratch claim.

What is LLM fingerprinting / model provenance?
It is the practice of estimating a model's origin — original vs. derived from an existing open-weight base — using only its public artifacts (config, tokenizer, weights), without access to training data or internal logs.

Is building a model on Llama, Qwen, or DeepSeek legitimate?
Yes. Fine-tuning or continued-pretraining an open-weight base is standard, licensed (subject to each license), industry practice. Provenance tools report lineage; they do not allege wrongdoing.

How do you distinguish a fine-tuned model from a from-scratch one?
From-scratch models diverge on architecture and tokenizer and show low embedding CKA to any base. Fine-tuned or continued-pretraining derivatives keep the base's skeleton and tokenizer and retain high embedding similarity — the hardest case, where CKA alone is weakest and config/tokenizer evidence matters most.

Does a shared tokenizer mean a model is derived?
No. It is a supporting signal only. Confirm with architecture and weight evidence before drawing a conclusion.

Why Linear CKA instead of cosine similarity?
Because CKA is robust to rotation and scaling, the transformations that let a derived model masquerade as "different" under naive cosine comparisons.

Resources & related links

Reproducible method write-up: How to Tell If an LLM Was Really Trained From Scratch
Deep-dive article: Model DNA — architecture & weight lineage
Live tool: Model Genome Korea
Announcement post: Architecture lineage of Korea's sovereign-AI models
The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · media coverage

Provenance is becoming a norm, not a gotcha. The healthiest version of this trend is one where "we trained it from scratch" is expected to arrive with — or at least survive — the fingerprint. If you build models, publish the check yourself.

The KV Cache Is the Bottleneck: A 2026 Field Guide to Attention Variants

ai maya — Sun, 09 Aug 2026 07:15:29 +0000

If you want to understand why 2026's frontier LLMs look the way they do — why one ships Multi-head Latent Attention, another interleaves linear and full attention, a third leans on sliding windows — stop looking at benchmark scores and look at the KV cache. Almost every architectural decision at the frontier this year is, at heart, a fight with the memory that autoregressive decoding leaves behind.

This is a practitioner's map of that fight: what the KV cache actually costs, the lineage of techniques built to shrink it, and how to choose among them.

Why the KV cache dominates

During generation, a decoder-only transformer caches the key and value vectors of every past token so it doesn't recompute them each step. That cache is the price of fast decoding, and it is not small.

The size is brutally simple:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × dtype_bytes

Two consequences fall out immediately:

It grows linearly with context and batch. Double the context, double the cache. Serve more concurrent users, multiply again. At long context and healthy batch sizes, the KV cache — not the weights — becomes the dominant consumer of accelerator memory.
Decoding is memory-bandwidth bound, not compute bound. Generating one token touches the entire cache. You are not FLOP-limited; you are limited by how fast you can stream that cache off HBM. Shrinking the cache is therefore the most direct lever on both memory footprint and tokens/sec.

Every technique below is a different answer to "how do we make that formula smaller without breaking the model."

The lineage of fixes

1. Fewer KV heads: MQA → GQA

The first target is kv_heads. Multi-Query Attention (MQA) collapses all query heads onto a single shared KV head — a large cut, but it can cost quality and training stability. Grouped-Query Attention (GQA) is the pragmatic compromise that won: query heads are split into groups, each sharing one KV head. With, say, 8 KV heads for 64 query heads you get most of MQA's savings with far less quality loss. GQA is the default in the majority of open-weight models today for exactly this reason.

2. Compress the cache: Multi-head Latent Attention (MLA)

Instead of storing fewer heads, MLA stores a low-rank latent. Keys and values are projected down to a small shared latent vector that is cached; the per-head K and V are reconstructed on the fly during attention. The cached object is dramatically smaller than full K/V, yet the model still attends with many effective heads. MLA is the headline reason some 2026 models sustain very long context at serving-friendly memory — it attacks head_dim × kv_heads directly rather than just cutting head count.

3. Kill the growing cache entirely: linear attention & SSMs

Softmax attention is O(n²) in compute and O(n) in cache. Linear attention and state-space models (SSMs, Mamba-style) rewrite the operation as a recurrence with a fixed-size state. There is no cache that grows with sequence length — memory is O(1) in context. The catch is real: a fixed state is a lossy summary, so pure linear models are weaker at precise long-range recall and associative lookup ("what was the exact token 40k ago"). They are fast and cheap; they forget details.

4. The 2026 consensus: hybrid, layer-interleaved attention

The dominant pattern this year is not to pick one. Models interleave a minority of full (softmax) attention layers with a majority of linear/SSM layers. The full layers preserve the sharp recall that pure linear loses; the linear layers carry the long-context load cheaply. The result is near-linear memory scaling with most of the quality of full attention. When you read that a 2026 release is "hybrid attention," this layer-wise interleaving is almost always what is meant.

5. Bound the window: sliding-window attention + sinks

Orthogonal to the above, sliding-window attention simply caps how far back each token attends (e.g., the last few thousand tokens), making the cache constant-size past the window. Naively this destroys long-context behavior, so it is paired with tricks like attention sinks — keeping the first few tokens always in view — which stabilize very long generations. Cheap, effective for many workloads, and easy to combine with GQA.

Two more multipliers worth knowing

Quantized KV cache. The dtype_bytes term is a free-ish win: storing the cache in FP8 or INT8 instead of FP16 halves or quarters it with modest quality impact. In 2026 this is close to standard for long-context serving.
Paged KV memory. PagedAttention (the idea behind vLLM's throughput) doesn't shrink the cache — it stops you from wasting it. By allocating the cache in fixed pages instead of one contiguous per-request block, it removes fragmentation and lets you pack far more concurrent sequences into the same HBM. Architecture shrinks the cache; paging spends what's left efficiently.

How to choose

Building a general open-weight model? GQA is the safe, proven baseline. You will not be criticized for it.
Chasing long context at serving-friendly memory? MLA-style latent compression is the strongest single lever, at the cost of implementation complexity.
Throughput on shorter contexts, or edge deployment? Sliding window + GQA + quantized cache is a cheap, robust stack.
Very long context where some recall loss is acceptable? A hybrid linear/full interleave gives you near-linear scaling — validate recall on your long-context tasks before committing.
Serving any of the above? Quantize the cache and use a paged runtime. These compose with every architecture and are the lowest-effort wins on the list.

The real takeaway

The lesson of 2026 is that model architecture and the serving stack are now co-designed. You cannot reason about an attention variant without reasoning about its KV cache, its memory bandwidth, and how a runtime will page it. The models that feel fast and cheap in production didn't get there by accident — they were shaped, layer by layer, by the cost of remembering.

When you evaluate the next "new architecture," skip the leaderboard for a minute and ask the only question that predicts its serving cost: what happens to the KV cache?

What's your KV strategy in production — GQA, latent compression, hybrid, or just quantize-and-page? Curious what's holding up under real load.

Local LLMs in 2026: What Actually Runs Well on a Laptop Now

ai maya — Sun, 09 Aug 2026 00:23:28 +0000

Two years ago, "run a language model locally" meant a weekend of compiling, a graveyard of CUDA errors, and a model that answered like it had a concussion. In 2026, you can install one tool, type one command, and have a genuinely useful assistant running on a laptop with no internet connection. Here's an honest map of what works, what doesn't, and where the sharp edges still are.

Why bother running locally at all

Three reasons keep pulling developers back to local inference:

Privacy. The prompt never leaves your machine. For code you can't paste into a cloud box, or personal data, that's non-negotiable.
Cost and offline. No per-token bill, no rate limits, and it works on a plane.
Latency and control. No network round-trip, and you pin the exact model version forever — no silent upgrades changing your outputs.

The catch has always been quality-per-watt. That's the number that moved.

The hardware tiers, honestly

8 GB RAM / integrated GPU: You can run 3–4B parameter models at 4-bit quantization. Good for autocomplete, summarizing, simple Q&A. Don't expect deep reasoning.
16 GB RAM: The sweet spot for most developers. 7–9B models run comfortably and are genuinely helpful for coding assistance and drafting.
32 GB+ or a discrete GPU with 16–24 GB VRAM: Now you're running 20–30B models, or bigger models at aggressive quantization, with real reasoning ability.
Apple Silicon (unified memory): Punches above its weight. A machine with 32–64 GB of unified memory runs models that would need an expensive discrete GPU on other platforms, because the CPU and GPU share the same memory pool.

Quantization: the trick that makes it possible

The reason a 7B model fits in 16 GB is quantization — storing weights at 4 bits instead of 16. The common format you'll see is GGUF, and the common recipe is 4-bit (often labeled Q4). The quality loss from full precision to 4-bit is surprisingly small for most tasks, while the memory savings are 4x. Below 4-bit (2–3 bit) the model starts to degrade noticeably; above it (5–8 bit) you pay memory for diminishing returns. For most people, 4-bit is the default that just works.

The tools

Ollama — the "one command" experience. ollama run <model> and you're talking to it. Best starting point.
llama.cpp — the engine most of the ecosystem is built on. Reach for it when you want control, custom quantization, or to embed inference in your own app.
LM Studio — a GUI if you'd rather click than type, with easy model browsing.

Which small models are actually good

Without naming a single "winner" — because it changes monthly — the strong open-weight families in the small tier this year are the usual suspects: the Qwen line, Google's Gemma line, Meta's Llama small variants, and Microsoft's Phi series. The practical advice: don't marry a model. Download two or three in the same size class, run your real prompts through them, and keep the one that wins on your work. A leaderboard can't tell you which model is best at your codebase; your codebase can.

The sharp edges nobody warns you about

Context length costs RAM. A model's advertised context window is not free — filling it with a long document can blow past your memory budget and crawl. Start with modest context and grow it.
First-token lag on cold start. The model has to load into memory. Keep it warm if you're calling it repeatedly, or the first request after idle feels broken.
Small models confabulate more. They're excellent at transformation (summarize, rewrite, extract) and shakier at open-ended factual recall. Use them for shape, not for ground truth.
Speed is "fine," not "instant." On a mid laptop expect readable-speed streaming, not cloud-flagship snappiness. For interactive use it's perfectly usable; for batch jobs, plan for it.

The takeaway

Local LLMs in 2026 crossed the line from "impressive demo" to "daily driver" for a lot of everyday tasks — summarizing, drafting, coding assistance, private Q&A. You won't replace a frontier model for the hardest reasoning, but you'll be surprised how rarely you need to. Install Ollama, pull a 7B model at 4-bit, throw your real work at it, and see how far the free, private, offline tier gets you.

For most developers in 2026, the answer is: further than you'd think.

What's your local setup — hardware, tool, and go-to model? And what task finally made local inference "good enough" for you?

MCP in 2026: How the Model Context Protocol Became the USB-C of AI Tooling

ai maya — Sun, 09 Aug 2026 00:17:47 +0000

A year ago, connecting a model to your tools meant writing glue for that model, in that framework, with that vendor's function-calling format. Swap the model and you rewrote the glue. In 2026, that pain is mostly gone, and the reason has a boring name: the Model Context Protocol (MCP).

MCP is worth understanding not because it's clever, but because it's winning — and the reason it's winning tells you where the industry's center of gravity is moving.

What MCP actually is

Strip away the branding and MCP is a small client–server contract for connecting language models to the outside world. A server exposes three kinds of things: tools (functions the model can call), resources (data the model can read), and prompts (reusable templates). A client — your IDE, your agent, your chat app — speaks the same protocol and can talk to any compliant server.

The analogy people keep reaching for is USB-C, and it's accurate. Before USB-C you had a drawer full of proprietary chargers. MCP is the drawer-emptying moment for AI integrations: write the connector once, and any MCP-aware client can use it.

Why "model-agnostic" is the whole point

Here's the shift that matters. For most of the LLM era, your tooling was coupled to a model. If you built your agent stack around one vendor's function-calling quirks, you were locked in — a new, better model meant a migration project.

MCP decouples the tooling layer from the model layer. Your filesystem server, your database server, your ticketing-system server don't know or care which model is on the other end. When a new flagship drops — and in 2026 they drop every few weeks — you point your client at it and keep your entire tool ecosystem intact.

That's a strategic hedge, not just a convenience. In a market where the "best model" changes monthly, the durable asset is your integration layer, and MCP is how you stop rebuilding it.

What to build with it

Practical entry points, cheapest first:

Wrap an internal system as a server. Your team's deploy tooling, your logs, your knowledge base. Now every agent and IDE in the org can reach it through one contract.
Compose existing servers. The ecosystem already has servers for filesystems, databases, browsers, and dozens of SaaS tools. Most useful agents in 2026 are less "write everything" and more "wire the right servers together."
Ship a client capability, not a bespoke integration. If you're building an agent product, being MCP-aware means your users bring their own tool servers instead of waiting for you to build each one.

The pitfalls nobody puts in the README

MCP solves plumbing, not judgment. Watch for:

Tool sprawl. Expose 60 tools and the model spends its context deciding which to use — and picks wrong. Curate. A focused server beats a kitchen-sink one.
Prompt injection through tool output. The text a server returns is data, not instructions — but a naive agent will happily follow "ignore previous instructions" if it arrives inside a fetched web page. Treat every server response as untrusted input. This is the single most under-appreciated risk in the whole stack.
Auth and blast radius. A server that can delete production data is a server that will delete production data the first time a model hallucinates an argument. Scope permissions tightly and gate irreversible actions behind confirmation.
Versioning drift. Servers evolve; clients cache assumptions. Pin and test like you would any API dependency.

The takeaway

MCP isn't exciting, and that's exactly why it's important. Standards win by being boring and everywhere. The teams that come out ahead in 2026 are the ones treating their tool integrations as a durable, model-agnostic layer — and letting the models underneath churn as fast as the market wants them to.

Build the connector once. Let the models come and go.

Are you exposing internal systems over MCP yet, or still writing per-model glue? What server would you build first?

Default-to-Flagship Is Now a Cost Bug: Tiered Model Routing for Agentic Workloads

ai maya — Sun, 09 Aug 2026 00:17:41 +0000

For two years the reflex was simple: reach for the biggest model you can afford and call it a day. In 2026 that reflex quietly became a bug in your cost model.

The clearest signal came this summer, when a smaller, cheaper "flash"-tier model started edging out its own flagship sibling on the workload developers care about most — multi-step agentic coding — at a fraction of the price. When the fast tier wins the hard benchmark, "always use the flagship" stops being a safe default and starts being waste.

Here's how to fix it without turning your stack into a science project.

Why the reflex is expensive

Agent workloads are not one big call. A single task fans out into dozens of small ones: planning, tool selection, argument formatting, summarizing a file, deciding whether to continue. Most of those steps are easy. Routing every one of them through a frontier model is like taking a helicopter to the corner store — it works, but you are paying helicopter prices for a walk.

The trap is that the cost is invisible per call and enormous in aggregate. You never see the moment you overpaid; you just see the invoice.

The three-tier ladder

Think in tiers, not models:

Cheap/fast tier — classification, extraction, short rewrites, routing decisions, "is this done?" checks. Most steps live here.
Mid tier — normal reasoning, code edits, tool use with moderate context.
Flagship tier — genuinely hard reasoning, long-context synthesis, the step where a wrong answer poisons everything downstream.

The goal is to keep the flagship tier for the 5–15% of steps that actually need it, and let the cheap tier carry the volume.

How to decide the tier per request

Two mechanisms, used together:

Static heuristics for the obvious cases. Short prompt + structured output + low stakes → cheap tier. Anything touching a large context window or a irreversible action → escalate.

Eval-gated escalation for everything else. Start at the cheap tier, and only promote to a bigger model when your evals prove the cheap tier fails on that class of input. This is the key discipline: escalation is earned by evidence, not assumed. If you can't point to an eval where the small model loses, you don't get to pay for the big one.

A simple confidence signal helps too — if the cheap model hedges, returns malformed output, or low log-probs, retry one tier up. One retry at a higher tier is still cheaper than sending everything to the flagship.

Measure it or you're guessing

You cannot route what you don't measure. Log, per step: which tier ran, token counts, latency, and a success signal. Then compute the boring but decisive number — cost per completed task, not cost per token. Teams that optimize per-token often make per-task cost worse by adding retries; the task-level metric keeps you honest.

Re-run this monthly. Model prices and capabilities move fast enough that last quarter's optimal routing table is this quarter's mistake.

Pitfalls

Cheap-tier false economy. If a weak model fails a planning step, every downstream step inherits the error. Put your best model at the top of the plan, cheap models on the leaves.
Silent capability drift. A model point-release can flip your routing assumptions overnight. Pin versions in evals and re-test before upgrading.
Over-engineering the router. A 50-line heuristic plus one escalation rule beats a fancy learned router for most teams. Add complexity only when the data demands it.

The takeaway

The winning move in 2026 is not "use the best model" or "use the cheapest model." It's building a thin routing layer and an eval harness that lets you place each step on the right tier — and swap the underlying models in an afternoon when the market moves again.

Default-to-flagship felt safe because it was simple. It's still simple. It's just no longer cheap.

How are you routing model tiers in production right now — heuristics, a learned router, or still one model for everything? Curious what's working.

AI This Week (Aug 2026): Qwen3.8 Max, DeepSeek V4-Flash, and Models Shipping Like Patches

ai maya — Sat, 08 Aug 2026 16:19:17 +0000

If you blinked this week, you missed three frontier model releases. August 2026 is making one thing very clear: large language models now ship like software patches, not like moon landings.

Here's the AI news that actually matters for builders right now.

Qwen3.8 Max lands

Alibaba pushed Qwen3.8 Max at the start of the month — the newest flagship in a line that has become the default open-weight baseline for a lot of teams. The interesting part isn't a single headline number; it's the cadence. Qwen has gone from "the model you try" to "the model you benchmark against" in about a year, and each point release keeps tightening the gap with closed frontier models on reasoning and coding.

Dev takeaway: if your eval harness still pins an old open baseline, it's already stale. Re-run against the current Qwen point release before you claim a win over "open source."

DeepSeek V4-Flash quietly beats its big sibling

The story engineers keep repeating: the DeepSeek-V4-Flash retrain reportedly edges out V4-Pro on agentic coding benchmarks — at flash pricing. Read that twice. A smaller, cheaper, faster model beating the flagship on the workload developers care most about (multi-step tool use and code) is the clearest signal yet that the "bigger is always better" era is over for a lot of tasks.

Dev takeaway: for agent workloads, start with the small/fast tier and only escalate to the flagship when your evals prove you need it. Default-to-flagship is now a cost bug.

OpenAI's GPT-5.6 "Luna"

OpenAI's active lineup now includes GPT-5.6 Luna as a fresh flagship name. Details are thin, but the pattern is familiar: incremental version bumps, multimodal by default, and reasoning modes that trade latency for accuracy on demand. The headline isn't any one capability — it's that "reasoning vs. speed" is now a dial you set per request, not a model you pick once.

NVIDIA open-sources NOOA

Less flashy but arguably more useful for practitioners: NVIDIA open-sourced NOOA (Object-Oriented Agents), a model-agnostic Python framework for building agents. Model-agnostic is the key word — the tooling layer is decoupling from any single provider, which is exactly what you want if you're tired of rewriting your agent stack every time a new model drops.

The real trend: models as patches

Release trackers are now counting 300+ model releases across major labs. When frontier capability ships every few weeks, three things follow for teams:

Your benchmarks decay fast. Treat evals as living infrastructure, not a one-time report.
Small + fast is the new default. Efficiency gains keep delivering last-year's flagship quality at a fraction of the cost. Reach for the big model deliberately, not reflexively.
Bet on the tooling layer, not the model. Model-agnostic agent frameworks (like NOOA) protect you from churn.

The models will keep coming. The winning move isn't chasing every release — it's building an eval + tooling layer that lets you swap the engine in an afternoon.

What shipped this week that you're actually putting into production? Drop it in the comments.

"How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method"

ai maya — Sat, 08 Aug 2026 15:52:49 +0000

How to Tell If an LLM Was Really Trained From Scratch: A Reproducible Fingerprinting Method

Detect whether an LLM was trained from scratch or derived from Qwen, Llama, or DeepSeek — by fingerprinting architecture, tokenizer, and weight provenance from public Hugging Face artifacts. Includes the two traps almost everyone hits.

Keywords: LLM provenance · model fingerprinting · from-scratch vs fine-tuned · architecture signature · tokenizer overlap · embedding CKA · model lineage · Korean sovereign AI · open-weight derivatives

When a lab announces a "from-scratch, self-developed" foundation model, can an outsider verify that claim — using nothing but public files?

In late July 2026, several Korean labs shipped DeepSeek-rivaling "self-developed" models (e.g. LG's 750B K-EXAONE 2.0). The claim triggered a debate that spilled well beyond Korea: a single Zhihu thread crossed 2.7 million views, asking whether these models were trained from scratch or quietly built on Qwen / Llama / DeepSeek.

That question is answerable — quantitatively, reproducibly, from public artifacts. This post is the method. Everything below runs against any two repos on the Hugging Face Hub, and there's a live tool at the end.

Framing up front: building on open-weight bases (Qwen, Llama, DeepSeek, Mistral) is a legitimate, industry-standard practice. This is about provenance transparency, not accusation. The same yardstick applies to every model.

The idea: three independent fingerprints

A model leaves three separable fingerprints in its public files:

Architecture — the shape declared in config.json
Tokenizer — the vocabulary in tokenizer.json
Weights — the learned representation in model.safetensors

Each answers a different question, and — crucially — they can disagree. That disagreement is where the signal lives.

Fingerprint 1 — Architecture (`config.json`)

Every transformers checkpoint ships a config.json. Six fields form a surprisingly discriminative signature:

import requests

FIELDS = ("model_type", "vocab_size", "hidden_size", "intermediate_size",
          "num_hidden_layers", "num_attention_heads", "num_key_value_heads")

def arch_fingerprint(repo: str) -> dict:
    url = f"https://huggingface.co/{repo}/resolve/main/config.json"
    c = requests.get(url, headers={"User-Agent": "genome/1.0"}, timeout=30).json()
    return {k: c.get(k) for k in FIELDS}

The shape tuple (hidden_size, intermediate_size, num_hidden_layers, heads, kv) is effectively a fingerprint of the reference architecture. One matching field is a coincidence; five matching simultaneously is not. A few real matches I measured:

Shape (hidden · inter · layers · heads · kv)	Exact match
3584 · 18944 · 28 · 28 · 4	Qwen2.5-7B
8192 · 29568 · 80 · 64 · 8	Qwen2.5-72B
5120 · 17408 · 40 · 40 · 8	Qwen3-14B
4096 · 14336 · 32 · 32 · 8	Llama-3.1-8B
7168 · 18432 · 61 · (moe 2048)	DeepSeek-V3

An exact tuple match is strong evidence the architecture was adopted, not independently designed.

Fingerprint 2 — Tokenizer (a paternity test)

Architecture alone can mislead: a model can adopt a foreign architecture but train a genuinely new tokenizer (or vice-versa). Measure the tokenizer directly, comparing vocabularies with a min-overlap ratio:

def vocab_set(repo: str) -> set:
    url = f"https://huggingface.co/{repo}/resolve/main/tokenizer.json"
    v = requests.get(url, timeout=60).json()["model"]["vocab"]  # BPE: {token: id}
    return set(v.keys())

def tokenizer_overlap(a: str, b: str) -> float:
    A, B = vocab_set(a), vocab_set(b)
    return len(A & B) / min(len(A), len(B))   # 1.0 == one is a subset of the other

This surfaces what config hides. One model matched Qwen2.5-7B's architecture exactly, yet its tokenizer overlapped Qwen by only ~0.38 — a "foreign brain, own language" case: adopted architecture, freshly trained (Korean) tokenizer. Others reused a base tokenizer verbatim (overlap = 1.000), confirming a straight fine-tune.

Why min, not union? Using min(|A|, |B|) in the denominator makes a reduced vocabulary that is a strict subset of a larger one score ~1.0 — the correct signal for "carved out of the base." A Jaccard (union) denominator would wrongly dilute that.

Fingerprint 3 — Weights (here be dragons)

The gold-standard question: were the weights trained from scratch, or continued-pretrained on a foreign base? Load the token embeddings and compare. Two traps await.

First, a helper to pull only the embedding tensor (no need to download the whole model):

import json, torch
from huggingface_hub import hf_hub_download
from safetensors import safe_open

def load_embedding(repo: str) -> torch.Tensor:
    try:
        idx = hf_hub_download(repo, "model.safetensors.index.json")
        shard = json.load(open(idx))["weight_map"]["model.embed_tokens.weight"]
    except Exception:
        shard = "model.safetensors"
    path = hf_hub_download(repo, shard)
    with safe_open(path, framework="pt") as f:
        key = next(k for k in f.keys() if k.endswith("embed_tokens.weight"))
        return f.get_tensor(key).float()

Trap 1 — row-wise cosine is useless

The naive approach: for shared tokens, average the row-wise cosine similarity of the two embedding matrices. Shared lineage → similar embeddings, right?

Wrong — even when lineage is obvious. I measured near-zero mean cosine for both a known from-scratch model and a known Llama-derivative. The culprit is rotational invariance: a Transformer's hidden space has no privileged basis, so two models can encode identical information under an arbitrary orthogonal rotation. Row-wise cosine reads rotation as dissimilarity and tells you nothing about lineage.

Trap 2 — CKA helps, but is not conclusive

Linear CKA (Centered Kernel Alignment) is invariant to rotation and isotropic scaling — the right tool for comparing representations:

def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float:
    # X: (n, d1), Y: (n, d2) — SAME token order (shared vocabulary)
    X = X - X.mean(0, keepdim=True)
    Y = Y - Y.mean(0, keepdim=True)
    num = (X.T @ Y).norm() ** 2
    den = (X.T @ X).norm() * (Y.T @ Y).norm()
    return (num / den).item()

A from-scratch model scored near-zero CKA against its candidate base — clean evidence of independent pretraining. But a continued-pretrained derivative scored only ~0.25 — barely above the ~0.21 baseline between two unrelated models of the same family. Large-scale training reshapes embeddings enough that CKA loses discriminative power on the derivative side.

The honest conclusion: the weights axis reliably confirms from-scratch (near-zero CKA), but it is not a strong detector of derivation. For that, architecture + tokenizer fingerprints stay primary. Report the weights axis as supporting evidence, never as a standalone verdict. (This is the single most important caveat in the whole method — and the one most write-ups omit.)

Bonus fingerprint — attention diversity as an originality proxy

Most models declare one attention mechanism; a few mix several. The count of distinct mechanisms in config.json is a cheap proxy for architectural originality:

KEYS = ("layer_types", "linear_attn_config", "sliding_window",
        "mamba2_d_state", "hyena_filter_order", "mla_kv_lora_rank", "attention_cls")

def attention_diversity(cfg: dict) -> list:
    # e.g. layer_types = [full_attention×16, sliding_attention×48] -> hybrid (2 kinds)
    return [k for k in KEYS if k in cfg]

In my sweep, most models used a single grouped-query or multi-head-latent attention; some used a hybrid (layer_types=[full×16, sliding×48]); the most diverse combined mamba2, hyena, MLA, linear attention, gated-delta-net, native-sparse-attention, and sliding-window in one stack.

Combining axes → a single genotype

Collapse the two primary axes (architecture × weights) into one label:

Genotype	Architecture	Weights
🟢 Native	self	from-scratch
🔵 Adapted	mostly self	one axis borrowed
🟡 Mixed	partial	partial inheritance
🔴 Ported	foreign (exact match)	inherited

Keep tokenizer overlap and attention diversity beside the verdict, not folded into it, so readers can audit the raw evidence.

Results: nine organizations, one yardstick

Applying the identical pipeline to nine organizations' public foundation models (spanning large enterprises, telcos, mid-size firms, and startups), the picture is not uniform: some models match a foreign architecture and tokenizer exactly (Ported); others are self-built with no foreign match (Native); many sit in between. The per-model breakdown — 3D lineage graph, search, EN/中文/한국어, light + dark mode — is in the interactive tool below.

Reproduce it yourself

The functions above are the method. Point them at any two Hub repos:

print(arch_fingerprint("some/model"))
print(tokenizer_overlap("some/model", "Qwen/Qwen3-14B"))

# weights (shared-vocab pair):
X = load_embedding("candidate/model")
Y = load_embedding("Qwen/Qwen3-1.7B")
n = min(len(X), len(Y))
print("CKA:", linear_cka(X[:n], Y[:n]))   # near-zero => from-scratch

Limitations & honesty

Not an accusation. Open-weight reuse is legitimate and widespread. This reports lineage, not wrongdoing.
Weights axis is supporting, not conclusive (Trap 2).
Same yardstick for every model, without exception.
All inputs are public; corrections are welcome.

Frequently asked questions

How can you tell if an LLM was trained from scratch or fine-tuned from another model?

Compare its config.json shape signature (hidden size, intermediate size, layer count) and its tokenizer.json vocabulary against known open-weight bases. An exact architecture match plus high tokenizer overlap indicates a derivative; a self-designed architecture with near-zero embedding CKA against candidate bases indicates from-scratch training.

What is CKA (Centered Kernel Alignment), and why use it instead of cosine similarity?

CKA is a rotation- and isotropic-scale-invariant similarity measure for neural-network representations. A Transformer's hidden space has no privileged basis, so plain row-wise cosine similarity is fooled by arbitrary orthogonal rotations between two models. CKA is not — which makes it the correct tool for comparing embeddings across models.

Is it legal to build an LLM on top of Qwen, Llama, or DeepSeek?

Yes. Using open-weight foundation models under their licenses (e.g. Apache-2.0 for many Qwen releases, the Llama Community License for Llama) is a legitimate, industry-standard practice. Provenance analysis reports lineage, not wrongdoing.

Are Korean sovereign-AI models built from scratch or based on Chinese/US models?

It varies by model. Some match a foreign architecture (Qwen, Llama, DeepSeek) exactly and are best described as "Ported"; others use fully self-built architectures and weights with no foreign match ("Native"); many are in between. The genotype of each is shown in the interactive tool.

How do you measure model provenance without downloading the full model?

Architecture and tokenizer fingerprints need only config.json and tokenizer.json (kilobytes to a few megabytes). For the weight axis, download just the embed_tokens.weight tensor via safetensors partial loading instead of the whole checkpoint.

Does an exact architecture match prove a model is copied?

No. Reusing an open-weight architecture is standard and legitimate. An exact config.json match shows the architecture was adopted; whether the weights were inherited or trained from scratch is a separate question, answered (with caveats) by the embedding-CKA axis.

DEV Community: ai maya

opened a financial forecasting contest that AI agents can enter directly. $2,000 in prizes, 122 days.

Why another leaderboard

Design 1 — Take positions, not predictions

Design 2 — Verify the scorer before any model

Design 3 — The luck ceiling

Design 4 — Why we gave up on cross-asset comparison

Design 5 — Cost, not rules, stops spam

Design 6 — Putting a scale on an empty chart

The result was more interesting than expected

Design 7 — An hour, not a day

Design 8 — Agents as first-class entrants

Design 9 — The data has to be P&L someone could actually realise

Two things infrastructure taught us

Market data providers block datacenter IPs

Writable is not durable

The organisers do not compete

So what do we want to learn

Entering

Day one

An open challenge is letting anyone submit a drug candidate - and scoring it in public

Two seasons are live

The scoring is where it gets interesting

The anchors are in the table

The honest caveat, which the organizers state themselves

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

Key takeaways

Why this matters in 2026

The three signals (cited and reproduced)

1. Architecture fingerprint — config.json

2. Tokenizer overlap — tokenizer.json

3. Embedding similarity — Linear CKA

The genotype framework

Two traps that produce confident wrong answers

Deep analysis: how far can you trust it?

What fingerprinting cannot tell you

FAQ

Resources & related links

How to Verify a 'Trained-From-Scratch' LLM in 2026: A Provenance and Fingerprinting Guide

Key takeaways

Why this matters in 2026

The three fingerprint signals

1. Architecture fingerprint (config.json)

2. Tokenizer overlap (tokenizer.json)

3. Embedding similarity via Linear CKA

The genotype framework

Two traps that produce confident wrong answers

What fingerprinting cannot tell you

FAQ

Resources & related links

The KV Cache Is the Bottleneck: A 2026 Field Guide to Attention Variants

Why the KV cache dominates

The lineage of fixes

1. Fewer KV heads: MQA → GQA

2. Compress the cache: Multi-head Latent Attention (MLA)

3. Kill the growing cache entirely: linear attention & SSMs

4. The 2026 consensus: hybrid, layer-interleaved attention

5. Bound the window: sliding-window attention + sinks

Two more multipliers worth knowing

How to choose

The real takeaway

Local LLMs in 2026: What Actually Runs Well on a Laptop Now

Why bother running locally at all

The hardware tiers, honestly

Quantization: the trick that makes it possible

The tools

Which small models are actually good

The sharp edges nobody warns you about

The takeaway

MCP in 2026: How the Model Context Protocol Became the USB-C of AI Tooling

What MCP actually is

Why "model-agnostic" is the whole point

What to build with it

The pitfalls nobody puts in the README

The takeaway

Default-to-Flagship Is Now a Cost Bug: Tiered Model Routing for Agentic Workloads

Why the reflex is expensive

The three-tier ladder

How to decide the tier per request

Measure it or you're guessing

1. Architecture fingerprint — `config.json`

2. Tokenizer overlap — `tokenizer.json`

1. Architecture fingerprint (`config.json`)

2. Tokenizer overlap (`tokenizer.json`)

Fingerprint 1 — Architecture (`config.json`)