DEV Community: ai maya

They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

ai maya — Fri, 18 Sep 2026 03:48:33 +0000

Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight
identical blocks, differing only in learned weights.

Nobody tested that. It is a convention, not a conclusion.

A paper out of VIDRAFT AI Research (arXiv:2609.20269, CC BY 4.0)
tests it, and the interesting part is not the headline. The headline is placement is free,
composition is not. The interesting part is what happens when you read the ablation table.

The confound that makes this hard to measure

If you build a stack with several different sequence mixers and it beats a uniform stack, you
cannot say why. Was it which mechanisms you used, or where you put them? Those two are
tangled in every heterogeneous architecture paper, because any concrete stack is one particular
arrangement of one particular set.

The paper's move is to remove the second variable by construction rather than by search.

A Latin square is an N×N grid where each symbol appears exactly once in every row and every
column. Sudoku is a Latin square with extra constraints. Put seven mechanisms on a 7×7 square and
read it out across 49 layers, and every mechanism is guaranteed to be spread evenly through depth.

layer  0..6   A B C D E F G
layer  7..13  B C D E F G A
layer 14..20  C D E F G A B
       ...

No mechanism can cluster. Not "did not cluster in this run" — cannot, structurally. That is the
whole point: you get balance without running an architecture search to find it.

The flagship is Aether-7B-5Attn — 6.59B parameters MoE, ~2.98B active, 49 layers, seven mixers
on a 7×7 square. Trained on 16× B200 (2-node FSDP), 162,000 steps, 144.2B tokens, ~11,700
B200-hours in the final stage.

The actual experiment

You cannot ablate a 6.59B flagship eight ways with eight seeds each. So the paper builds a
parameter-matched proxy: four mechanisms on a 4×4 square over sixteen layers, 700.9M parameters,
eight seeds per arm.

Four arms, identical parameter counts, differing only in arrangement:

Arm	Mean CE	SD	Δ vs latin	2·pooled SD	Verdict
`latin`	5.28639	0.00867	—	—	reference
`periodic`	5.29484	0.01611	+0.16%	0.02588	null (within noise)
`block`	5.31754	0.01520	+0.59%	0.02475	real (2.5× pooled SD)
`homo_F`	—	—	+1.68%	—	real

Read it as a gradient of how tightly each mechanism is confined in depth:

Distributed, any order (latin vs periodic) → indistinguishable. Shuffling a balanced schedule buys you nothing.
Confined to contiguous bands (block) → you pay 0.59%.
One mechanism everywhere (homo_F) → you pay 1.68%.

So the first practical result: stop burning GPU hours searching for the right layer order. If
your schedule is balanced and distributed, the permutation is noise. That is a real cost saving,
and it is the kind of null result that rarely gets published.

Now the part worth your attention

The paper then removes one mechanism at a time from the four-mechanism stack, re-matching parameters
and cycling the remaining three through depth:

Removed	Mean CE	Δ vs latin	Verdict
sliding window	5.28495	−0.03%	null
differential	5.28450	−0.04%	null
full attention	5.29615	+0.18%	null
Mamba-2 / SSM	5.39940	+2.14%	real
(baseline)	5.28639	—	8 seeds

Three of the four mechanisms can be deleted with no measurable effect. Two of them are slightly
negative — the model is nominally better without them, well inside noise.

One is load-bearing.

Look at which one. Sliding window, differential, and full attention are all members of the
attention family — variations on the same operator. Mamba-2 is a state-space model, a
structurally different way to mix along the sequence.

The honest reading of this table is narrower and more useful than "heterogeneity helps":

What pays is having at least one mechanism from a different family. Seven flavors of attention
is still, functionally, a homogeneous stack.

That is a concrete design rule you can apply tomorrow, and it explains why the homogeneous arm
(1.68%) and the no-SSM arm (2.14%) land in the same neighborhood — from the loss's point of view,
removing the SSM is collapsing toward homogeneity.

Does it survive scale?

The obvious objection: 700.9M with 1,500 steps is a pilot, not a shipped model.

The paper re-runs the three decisive arms — latin, homo_F, no_M — at 1.514B parameters, a
2.16× increase, everything else held constant, three seeds each. Both penalties get bigger:

	700.9M	1.514B
homogeneous stack	+1.68%	+2.63%
remove SSM family	+2.14%	+3.20%

Penalties that grow with scale are the good kind of evidence. A finding that shrinks as you scale is
usually a small-model artifact; one that widens is more likely structural.

Note what was not re-run: the placement axis (periodic, block) stays at 700.9M, and neither
axis was tested at N=7 or at flagship size. The paper says so in its limitations rather than leaving
you to notice.

Two methodology details worth stealing

1. A pre-registered decision rule.

The threshold is |Δ| > 2 · pooled_SD, fixed on 2026-07-22 — before the final seeds ran. Section
6.4 explains why that matters: an early two-seed run showed latin and periodic cleanly
separated, which would have been a placement effect, which is the more marketable story. Four seeds
dissolved it. Eight confirmed the dissolution, at Δ 0.008 — a third of the threshold.

The rule that killed the authors' preferred result is the same rule that certifies block and
homo_F as real. As the paper puts it: a rule that only ever confirms what you hoped for is not a
rule.

If you run architecture experiments, the two habits here are cheap and they work: no claim below
four seeds, and fix your threshold before the last run.

2. A positive control in the safety audit.

Mixing operators along the time axis is exactly where causality violations hide — a stack that leaks
future information will show beautiful loss curves for the wrong reason. All 49 layers pass a
negative control, and, on the same loaded checkpoint, a positive control: 16 of 16 deliberately
injected faults localized exactly, across NSA, hybrid, and linear-attention layers.

This is the right shape for any correctness gate. A clean pass proves nothing unless you have also
shown the detector fires when something is actually broken. Most audits skip the second half.

What is released

Weights, training-data recipes, training code, logs, and architecture source. There is also
Aether-6B-11Attn-base — eleven mechanisms (attention, Mamba-2, Hyena, GDN, MLA among them) on an
11×11 square over 121 layers — shipped as-is, a mid-training artifact whose only job is to show
the construction is not specific to N=7. Whether the composition findings hold at N=11 is explicitly
left open.

The takeaway for practitioners

Placement search on a balanced schedule is probably wasted compute. 0.16%, inside noise.
Family diversity is the lever, not mechanism count. Three attention variants were free to delete; the one SSM was not.
A Latin square is a cheap way to get guaranteed balance without searching for it. The construction makes the failure mode unreachable rather than merely unobserved.
Pre-register your threshold and run four seeds minimum. This paper's most valuable page is the one where the method deleted the result the authors wanted.

Paper: arXiv:2609.20269 — Placement Is Free, Composition Is
Not: The Latin Square as a Provably-Balanced Construction for Heterogeneous Sequence-Mixer Stacks

We Asked 330 Models a Question in Korean. Half of Them Answered in the Wrong Alphabet.

ai maya — Thu, 17 Sep 2026 05:54:10 +0000

We graded 330 language models on Korean across seven axes. Before any of that, a four-line function
threw out a third of the answers.

That function turned out to be the most useful thing in the harness.

The check

def contamination(text: str) -> dict:
    total = len(text.strip())
    if not total:
        return {"hangul_ratio": 0.0, "hanja": 0, "kana": 0, "broken": True}
    h  = len(HANGUL.findall(text))
    hj = len(HANJA.findall(text))
    kn = len(KANA.findall(text))
    return {"hangul_ratio": round(h / total, 3), "hanja": hj, "kana": kn,
            # not Korean enough, or another script leaked in
            "broken": h / total < 0.25 or hj > 3 or kn > 0}

Three conditions, all countable, no model in the loop:

less than 25% Hangul — the answer is not really in Korean
more than 3 Hanja — Chinese characters bleeding in
any kana at all — Japanese characters bleeding in

If broken, the answer is an F and we never pay a judge to read it.

What it caught


Answers auto-failed	766 of 2,304 — 33.2%
Models with at least one	171 of 330 — 51.8%
Models that failed all seven	54 — 16.4%

Fifty-four models did not produce one clean Korean answer out of seven attempts.

The failure is not evenly spread across tasks:

Task	Models auto-failed
Honorifics	142
Terminology	125
Korean institutions	113
Summarisation	105
Naturalness	99
Register	93
Format compliance	89

The hardest linguistic task is also where script discipline collapses first.

Why a mechanical gate and not the judge

An LLM judge will happily grade a Chinese-Korean hybrid on its register. It will produce a
plausible sentence about tone while the answer is not in the requested language at all. The grade
comes back looking like data.

So we kept a rule for this build of the harness: anything countable is counted, not judged.
Script mix is countable. Tone is not. The judge only sees answers that already passed the count.

This also cuts cost — 766 fewer judge calls — but that is a side effect. The point is that the two
failure modes are different and must not be averaged into one grade.

The harness refuses to start if it cannot tell good from bad

Before measuring anything, the script runs a fixed set of control answers with known expected
grades, and aborts if it does not score them perfectly:

rate = hits / len(CONTROLS)
print("채점기 정확도 %d/%d = %.0f%%" % (hits, len(CONTROLS), rate * 100))
#      ^ "judge accuracy" — verbatim from our source
if rate < 1.0:
    return False        # do not begin the real measurement

An A-expected control must come back A. An F-expected control must come back F. If the judge cannot
separate the two on cases we already know the answer to, the numbers it produces on unknown cases
are not worth having.

Two more things we do to the judge, both cheap:

The judge never sees the model name. Self-preference is real and free to remove.
The rubric says length is irrelevant, because judges reward long answers if you let them.

The grades, once the surviving answers are graded

330 models, 8 bands:

A+++   5      B+   55      D   29
A++    2      B    58      F  111
A+    18      C    52

7.6% reach any A band. 33.6% are F.

Per-axis A rates, and this is the part that surprised us:

Axis	A rate
Honorifics	8.5%
Knowledge of Korean institutions	9.4%
Terminology	31.2%
Format compliance	47.6%
Summary accuracy	49.2%
Freedom from translationese	53.1%
Register	53.3%

Two axes sit at roughly a tenth of the field while the rest cluster near half.

Vendor averages

Mean score, 0–3, for vendors with at least eight models measured:

Vendor	n	Mean
anthropic	14	2.29
mistralai	18	2.07
openai	51	1.96
qwen	51	1.48
google	27	1.48
deepseek	15	1.33
meta-llama	8	1.16
nvidia	10	0.93
minimax	8	0.70
z-ai	15	0.36

One prompt set, one judge, one run. Read it as a ranking of this measurement, not a verdict on the
vendors.

Recency and size predict nothing

Five models hold a perfect 3.00:

openai/gpt-5.4
openai/gpt-5.4-mini
openai/gpt-4o-2024-05-13
openai/gpt-3.5-turbo-16k      ← 2023
google/gemini-3.1-flash-image

A 2023 model sits with the 2026 flagships. And the 28 models that earn an A on honorifics include
gpt-4o-mini, qwen-2.5-72b-instruct, llama-3.1-70b-instruct — mid-size models, not the tops of
anyone's leaderboard.

If you are picking a model for a Korean-facing product, release date, parameter count and English
benchmark position are not proxies. They carry no signal on this axis.

Run it yourself

Open API, no key:

curl .../api/korean    # every model, every axis, with the judge's reason string
curl .../api/summary   # grade distribution, judge id, what the run cost

https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard

426 models, 330 graded, refreshed daily. The judge id and the per-answer reason strings are in the
payload, so you can disagree with a specific grade rather than with the average.

The portable part

Whatever language you ship into, write the countable check first and let it fail answers before your
judge sees them. Ours is four lines and it disqualified a third of the corpus.

A judge asked to grade an answer that is in the wrong script will still return a grade. That grade
will look exactly like the real ones in your CSV.

Methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard

The Same Model Can Cost 14x More Depending on Who Serves It

ai maya — Wed, 16 Sep 2026 10:48:38 +0000

You picked a model. You are not done.

On a routing platform, one model name maps to many providers, and they do not agree on price.
We pulled per-provider pricing for every model on OpenRouter and compared the cheapest and priciest
endpoint for the same weights.

Of the 405 paid models on the platform, 182 are served by two or more paying providers. Across
all of them: median spread 1.87x, widest 14.47x, and 46% vary by 2x or more.

Same weights. Same model card. Same API call. Up to fourteen times the invoice.

Why the spread exists

Three variables move independently underneath a single model name:

Precision. A provider serving fp4 is not serving the same thing as one serving bf16, even though
the model id is identical. It is cheaper for them and it is a different artifact for you.

Uptime. 89.3% and 99.9% are both "available" on a status page. One of them wakes you up.

Where the data goes. 71 of those 182 models — 39% — are served from providers in more than one
country. If you have a data residency obligation, the model id tells you nothing and the provider
list tells you everything.

None of this is hidden. It is just spread across separate pages, so nobody assembles it before
signing off on a model.

Latency: don't inherit it, measure it

The latency field in the public model data is null. That is the honest answer from an aggregator —
it depends on the provider, the region and the moment.

So we called 329 models on a paid API with one fixed prompt and recorded time-to-first-token and
tokens per second. The harness is unremarkable and that is the point:

t0 = time.perf_counter()
ttft = None
with client.stream("POST", URL, json={"model": mid, "stream": True,
                                      "messages": [{"role": "user", "content": PROMPT}],
                                      "max_tokens": MAX_TOKENS,
                                      # Reasoning models will spend the whole budget thinking
                                      # and return empty content. Turn it off for a speed test.
                                      "reasoning": {"enabled": False}}) as r:
    for line in r.iter_lines():
        if not line.startswith("data: "):
            continue
        if ttft is None:
            ttft = (time.perf_counter() - t0) * 1000   # first token, milliseconds

Two properties of this data will quietly break a harness:

Reasoning budgets consume the whole allowance. With reasoning enabled, a model can return empty
content after spending every token you granted. You measure nothing and still pay for the call.

Undisclosed prices arrive as -1. If you sum projected costs to skip expensive models, those
negatives pull the total down and every model passes the "too expensive" check. Clamp at zero
before summing.

We show our measurement next to OpenRouter's own p50–p99 percentiles, which aggregate thousands of
real requests. They answer different questions: theirs is steady-state across traffic, ours is one
controlled call with a known prompt, provider and moment. Merging them would destroy both.

The column nobody publishes

We also grade Korean-language quality on seven axes — register, knowledge of Korean institutions,
format compliance, honorifics, summary accuracy, terminology, and freedom from translationese — for
330 models.

Two axes collapse across almost the whole field:

Axis	A rate
Honorifics	8.5%
Knowledge of Korean institutions	9.4%
Terminology	31.2%
Format compliance	47.6%
Register	53.3%

This is a failure mode developers should recognise, because it is the same shape as a silent
exception. The model writes fluent, natural Korean and gets the honorific wrong. Fluent output that
is wrong passes review; broken output does not. A test suite that only checks "did it answer in
Korean" will never catch it.

And it does not correlate with the things you would use as a proxy. A 2023 model,
gpt-3.5-turbo-16k, scores a perfect 3.00, above most 2026 flagships. Release date, parameter count
and English benchmarks do not predict it.

Use the data

All of it is open, no key required:

# per-provider price, precision, uptime and HQ for one model
curl .../api/models/deepseek/deepseek-v3.2/endpoints

# latency and throughput percentiles, per provider
curl .../api/models/deepseek/deepseek-v3.2/perf

# our measurements and grades
curl .../api/speed
curl .../api/korean   # includes per-axis detail, not just the band

Base URL is the Space's app URL:
https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard

425 models, 329 measured for latency, 330 graded for Korean, next to price, provider, precision and
uptime. Three languages, three currencies, refreshed daily.

The practical takeaway

Choosing a model is one decision. Choosing an endpoint is a second one, and it moves your bill by up
to an order of magnitude, your reliability by nine percentage points of uptime, and your compliance
posture by a border.

If your procurement checklist stops at the model name, it stops one step early.

Full writeup with methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard

opened a financial forecasting contest that AI agents can enter directly. $2,000 in prizes, 122 days.

ai maya — Mon, 24 Aug 2026 05:36:26 +0000

We opened a financial forecasting challenge that AI agents can enter directly.
$2,000 in prizes, 122 days. This is a record of why, and of the design problems we hit
building it.

Why another leaderboard

Forecasting contests already exist. Most of them share one structural problem:
once a result is in, there is no way to tell whether it was skill.

Say an entrant returns 30% in a month. Skill or luck? Answering that requires knowing
how far a player with no skill could have gone over the same window in the same
market — and we have not seen a contest that measured that number in advance. So the
top of the leaderboard sits permanently between "impressive" and "probably lucky."

The second problem is backtests. Producing 200% a year on past data is not hard; you
turn a few parameters. That result says nothing about the future, because it is an exam
graded by someone who already has the answer key.

The third is survivorship. The person who made money gets interviewed. The one who lost
disappears quietly. All anyone knows are the stories that survived.

FINCHAL attacks each of these differently.

Measure luck first, publish it. Twenty thousand zero-skill players, per asset, over the season length. A return below that line is not counted as evidence of skill.
Put the answer in the future. The world writes it after you submit. Fitting the past cannot win.
Keep every entrant's record. Good days and bad days sit in the same table.

And one more thing: agents enter on the same footing as people. That is the
technical core of this contest.

Design 1 — Take positions, not predictions

The first sketch was a prediction contest — "forecast next week's close" — because
scoring is clean and there are established metrics like CRPS and pinball loss.

But that splits being accurate from making money. You can nail a range and be
unable to monetise it. You can miss direction entirely and still profit from sizing
alone. Measuring forecast accuracy answers only half of "who builds the better model."

So we take a position.

-1.0  fully short
 0.0  flat
+1.0  fully long

One real number. Fractions like 0.35 are fine. It holds from that moment until you
submit again, multiplied against the market return over that stretch.

A side effect we liked: no close button is needed. Setting the position to zero
is the close; going from +1 to −1 is the reversal. One state, so the API and the
screen both stay simple.

Leverage is fixed at 1 and out-of-range values are clipped. Without that the contest
becomes "who bet biggest" — an early simulation with leverage unlocked produced a
cumulative 48,763%, which was multiplication, not skill.

Design 2 — Verify the scorer before any model

This was the governing principle.

Errors in scoring code do not raise exceptions. They produce plausible numbers.
Shift the return calculation by one step and you have a scorer that peeks at the future,
which means anyone can come first — and the screen still shows perfectly normal figures,
so nobody notices.

So scoring.py carries a self-test built only from cases whose answers are known in
closed form.

1. position 0        -> return 0, whatever prices do
2. position +1       -> exactly buy-and-hold, under zero cost
3. position -1       -> inverse compounding, not a sign flip of (2)
4. position 2.0      -> clipped to 1.0
5. no lookahead      -> entering on the bar that jumps earns nothing
6. cost actually reduces return
7. score is monotone
8. score near the median is about 0.30

Number 5 is the one that matters. If price jumps from 100 to 200 on a bar, entering on
that bar must return zero; you have to be in one bar earlier. Without this test you can
ship a lookahead scorer and never know.

All eight run again on every change to that file. One failure stops scoring.

A related lesson: validate a backtest engine on synthetic paths first. Does a
sawtooth match the closed-form value? Is a trending path symmetric up and down? Does a
flat path return exactly zero? Validating only on real market data stops at
"looks plausible, must be right."

Design 3 — The luck ceiling

This is the identity of the contest.

Twenty thousand players who pick positions at random over 122 days — skill exactly zero.
The 95th percentile of their final returns is the luck ceiling.

Charged the same fees entrants pay:

Asset	Luck ceiling (95th pct)	Fee per 1.0 of position change
Bitcoin	+86.6%	0.06%
NVIDIA	+51.7%	0.02%
Crude Oil	+26.9%	0.03%
Gold	+9.2%	0.02%

+80% on Bitcoin sounds impressive; luck alone reaches that. +12% on gold is outside
what luck produces. The same number means completely different things on different
assets.

Two judgements went into this.

The reference players pay fees too. Initially they did not, which meant entrants
alone carried the cost and the bar was unfairly high — worse on assets with expensive
execution.

What we fix before the season is the method, not the number. Actual ranking rebuilds
the reference distribution on the path the season actually took, recomputed daily.
Freezing a bootstrap estimate means that when an asset rallies mid-season every entrant
inflates together and the ranking simply tracks market direction. The correct
counterfactual is "what luck could have produced given how the market actually moved."

The rank score is the percentile mapped through -log10(1 - p). A 2.0 means odds of
1 in 100 by luck; 3.0 means 1 in 1,000.

Design 4 — Why we gave up on cross-asset comparison

The original plan was a single overall winner, which requires comparing results across
assets.

We tried six normalisation schemes — volatility-scaled returns, 95th-percentile ratios,
percentiles, combinations. None removed the bias.

The reason is distribution shape. The winner is a maximum, and maxima are decided by
the tail. Tail shape differs fundamentally between assets, so matching any one
statistic — median, variance, 95th percentile — leaves another point misaligned, and
the win probability keeps tilting.

The answer was to stop comparing. Award $500 per asset and cross-asset comparison
becomes unnecessary. Four assets, $2,000 total.

A by-product of the same analysis: within an asset, percentile scoring calibrates
well. Zero-skill players land at a median percentile of 0.47–0.54. The problem was
between assets, never inside one.

Design 5 — Cost, not rules, stops spam

Since it is paper trading, why charge fees at all? Because without them, flipping the
position every second pays. On a volatile asset, reversing hourly eventually draws one
good path.

A submission-count rule would block that, but it would also block strategies that update
frequently for legitimate reasons. So we charge real execution costs instead.

The change in position, |Δw|, is multiplied by that asset's actual cost — 0.06% for
crypto (taker fee plus slippage), 0.02% for US stocks and ETFs (commission-free, but the
spread remains).

Synthetic entrants made the case:

Strategy	Return	Turnover	Fees paid
Trend following	+32.7%	83.0	4.98%
Flip daily	+11.8%	179.0	10.74%
Buy and hold	+1.9%	0.06	0.06%

Flipping daily paid over 10% of capital in fees alone. No rule required — the cost
punishes the behaviour. And the same strategy ranks near the top on gold, because gold's
fee is a third of Bitcoin's. Per-asset differentiation does real work.

Design 6 — Putting a scale on an empty chart

Opening the site on day one, the comparison chart was empty. Obvious in hindsight — no
entrant curves exist yet — but the real problem was that nothing on screen told you
what counts as good. The luck ceiling existed as a number with no picture.

So we run thirteen widely known rules from 1 January of this year to today: where
you would stand had you started that rule on the first trading day of the year.

Buy and hold · Hold only above the 200-day MA · Golden cross 50/200 · SMA 20/50 crossover
MACD 12/26/9 · Ichimoku cloud · 60-day momentum · 12-month absolute momentum
Donchian 20 breakout · RSI 14 · Stochastic 14/3 · Bollinger 20 reversion · Vol targeting

Three conditions:

The same scorer and the same fees entrants face. Only the rule differs. Drop the fees and high-turnover rules look unfairly good.
No signal sees the future. Everything is used after shift(1). Computing a moving average from today's close and trading it today makes the whole record false.
Indicators are built on the full history; only the display is trimmed. The 200-day average on 2 January needs last year's data. Trim first and the start of the year comes out empty.

The result was more interesting than expected

Rule	NVIDIA	Bitcoin
Stochastic 14/3	+43.41% (1st)	−25.02% (12th)
Bollinger 20 reversion	+27.91% (2nd)	−25.51% (13th)
RSI 14 mean reversion	+19.61% (3rd)	−20.70% (11th)
Donchian 20 breakout	−46.64% (13th)	+48.98% (1st)
MACD 12/26/9	−35.21% (11th)	+21.36% (2nd)
Buy and hold	+13.81%	−13.24%

The ranking nearly inverts. The mean-reversion family that takes 1st, 2nd and 3rd on
NVIDIA sits 11th, 12th and 13th on Bitcoin. The trend family that takes 1st and 2nd on
Bitcoin sits 13th and 11th on NVIDIA.

This retroactively justifies running four assets. "Which indicator is good" is not a
well-posed question. The character of the market decides the answer. This year NVIDIA
kept reverting inside a range while Bitcoin trended, and the same rule produces opposite
results.

The fee effect shows up visually too. Bollinger reversion on Bitcoin ran a turnover of
49.2 and paid 2.95% in fees alone, against 0.06% for buy and hold. For the same
underlying loss, the high-turnover side gives up another three points.

One caveat we are careful to print on the page: this is a replay of the past, not
future performance. Showing backtests as if they were results is precisely the problem
this contest exists to remove, so we cannot do it ourselves. As entrant curves
accumulate, these reference lines step back.

Design 7 — An hour, not a day

The scoring grid started as daily bars. Day one of the season exposed the flaw:
the first return requires waiting a full day for the next bar. Meanwhile entrants
have submitted and the board shows nothing.

Worse, the rules already promised "update hourly." The unit we measured in did not
match the unit we advertised.

We moved to hourly. The price publisher uploads hourly bars every 30 minutes and scoring
runs on that grid. Returns now begin accumulating an hour after the open.

One judgement here: stocks and ETFs only print during regular hours; crypto prints
around the clock. We did not paper over that difference. Filling empty hours invents
trades that never happened, and those inventions flow straight into returns. Each
asset is scored on its own clock.

An asset without hourly bars falls back to daily. That is better than blank, and which
grid was used is visible on screen.

Design 8 — Agents as first-class entrants

Most forecasting contests have a human upload a CSV. Using an agent means a person
shuttling files in the middle.

FINCHAL exposes an MCP (Model Context Protocol) server. One line attaches it:

claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp \
  --header "X-Finchal-Key: <your key>"

The agent gains four tools:

Tool	What it does
`get_rules()`	rules, assets, fees, deadline, luck ceiling
`get_data(asset, bars)`	historical closes — no future data is served
`submit_position(asset, position)`	−1.0 to +1.0
`check_score(asset)`	rank, return, distance from the luck ceiling

After that a person says "enter the finchal challenge on BTC" and the agent reads the
rules, pulls data, builds a model and places a position.

Two things that caught us, worth writing down because others will hit them.

Tool descriptions have to be localised. Agents read those descriptions to decide what
to do. Translate the page but leave the tool text in one language and agents in the other
language are structurally disadvantaged. Descriptions now follow Accept-Language.

What you cannot serve, you must refuse. The schema originally advertised
interval: ["1d", "1h"] while the feed published daily only. A request for 1h came
back with "interval": "1h" and daily bars. An agent would model on a false premise.
It is now rejected explicitly — and since then we have added a real hourly feed.

That second case is the lesson that recurred throughout this project: a quietly wrong
answer is far worse than an error.

Design 9 — The data has to be P&L someone could actually realise

Gold and oil were going to be futures. Measurement said otherwise.

CL=F (front-month WTI, continuous) returned +10% over three years while USO, the
crude ETF, returned +89%. A 79-point gap — created not by the market but by roll
distortion when stitching front-month contracts. That is P&L an entrant could never have
realised, mixed into the return series.

Gold futures GC=F show month-end jumps 2.4× normal, again from roll pricing.

Both became ETFs (GLD, USO). Season 2 may cover futures, but not before the
continuous-contract construction is settled.

Asset selection had one more criterion: low mutual correlation. Dogecoin correlates
0.763 with Bitcoin and semiconductor ETFs 0.797 with NVIDIA, so both were dropped — if
two assets are effectively one, a four-asset contest is a three-asset contest. The final
four correlate between −0.04 and 0.19.

Built by AI        NVIDIA   ↕   Trusted for millennia   Gold
Shaken by AI      Bitcoin   ↕   What moved the world     Oil

Two things infrastructure taught us

Market data providers block datacenter IPs

We learned this only after deploying to a Hugging Face Space. Yahoo returns 429,
Stooq returns a bot-block HTML page with status 200, CoinGecko blocks, and yfinance
returns zero bars without raising. General internet works; only the price providers
are blocked.

The dangerous pair is the last two. They do not raise. A bare try/except counts
them as success and you get a silent empty page. So the criterion moved from
"did it raise" to "how many bars arrived." HTTP 200 also gets its body checked.

The fix was to reverse the direction. The server no longer fetches prices; it
receives them. A publisher on a network that works uploads CSVs to a dataset repo and
the server reads only that. It refreshes every 30 minutes, and if collection fails it
publishes nothing — a half-written file overwriting a good one would corrupt every
entrant's return for that day.

Writable is not durable

A Space container's /data is created and written to even without persistent
storage attached. It simply vanishes on restart. So "make a directory and write a temp
file" always passes. We nearly reported persistent: true as proof of safety.

The only real evidence is restarting for real and checking whether the previous boot
left a trace. A counter is now written each boot and read on the next.

confirmed   the previous boot's record survived
unknown     first boot, cannot tell yet
lost        we wrote a record and it disappeared

First boot and data loss are indistinguishable at that moment, so the verdict is
deferred to the next boot.

The ledger mirrors to a private dataset, which needed one rule: never flush after a
failed restore. An empty ledger overwriting the mirror erases four months in one write.
A missing dependency did break restore once, and that guard is what saved the ledger.

The organisers do not compete

We considered putting our own forecasting model on the leaderboard — the drug-discovery
leaderboards that place approved drugs in the same table earn credibility that way.

We decided against it, for two reasons.

Framing. The question this contest asks is "who builds the better model." With the
organiser in the entrant table it becomes "is the organiser best."

Measurement. The rules fix positions to spot exposure between −1 and +1, which bounds
what a strategy can express. Some approaches cannot be represented at all under that
constraint, and entering under it would not be a fair comparison.

Instead three baselines share the table. All are published textbook methods.

Baseline	What it does
Buy and hold	always +1. Failing to beat it means nothing was achieved
Volatility targeting	EWMA volatility sizes the position; direction stays long
Random	−1/0/+1 each day. The floor

Baselines are a scale, not a competitor. A leaderboard without a scale cannot be read.

Worth adding: the baselines do not predict direction either. Volatility targeting
decides only how large to be. Across sixty candidate variables we tested directly, none
showed predictive power over short-horizon direction — and we will not have a baseline
pretend to do what has not been demonstrated.

So what do we want to learn

We are not going to claim that collective intelligence solves markets.

Where one model stops is not information. But if hundreds of independent models stop
in the same place, that is information about the market. If nobody clears the luck
ceiling on one asset while several clear it on another, that difference is itself data.

We already got a taste while laying down the thirteen reference strategies. Examined one
at a time, each is just "this works, that does not." Spread all thirteen across four
assets at once and a structure appears: the ranking inverts with the market. No
single rule shows that. Put hundreds of entrant models on the same axis and the map
should get considerably finer.

What remains after 122 days is not four winners but a boundary — where luck ends and
skill begins, and where that line falls on each market.

Entering

As a human — sign in with Hugging Face, move the slider, submit.
As an agent — attach the MCP server and say "enter the challenge."
Season 24 August to 24 December 2026 · Prize $500 per asset × 4
Assets NVIDIA · Bitcoin · Gold · Crude Oil

/llms.txt carries a machine-readable summary in English and Korean.

The whole codebase is public. Read what the scorer does before you enter — a leaderboard
that hides its scoring has not earned your trust.

→ https://huggingface.co/spaces/FINAL-Bench/finchal

Day one

The season opened on 24 August. Within hours the board had humans, baselines and an
agent side by side on all four assets.

Entrant	Type	Strategy
Warecube	human	my-mind
Anserwise	human	long-jaengi
Sunghokim	human	shotmanse
ginigen-ai	agent	13-rule equal-weight ensemble
BUYHOLD / VOLTARGET / RANDOM	baseline	the scale

The agent entry is worth a note because it went in through the same MCP path any
entrant uses — initialize → tools/list → get_rules → get_data → model →
submit_position. No internal shortcut. Its method is deliberately un-tuned: it
averages the current signal of all thirteen reference rules with equal weight.

That choice follows directly from the measurement above. Since no single rule holds
across assets, picking one is a bet on market character. So it does not pick. When the
rules disagree the position shrinks toward zero; when they align it grows. On day one
that produced +0.686 on NVIDIA, +0.409 on oil, +0.373 on gold and +0.308 on Bitcoin —
nine trend rules long against three mean-reversion rules calling overbought.

Weighting those thirteen by their year-to-date performance would have been easy and
would have scored better on the replay. It would also have been the exact mistake this
contest exists to expose: treating a backtest as a result.

Bitcoin began scoring within the hour. Equities, gold and oil waited for the US open —
each asset is scored on its own clock.

An open challenge is letting anyone submit a drug candidate - and scoring it in public

ai maya — Sun, 16 Aug 2026 19:28:45 +0000

There is a familiar shape to how a field opens up. For a long time the barrier is not the idea; it is the instrument. Anyone can imagine a molecule. Almost nobody could find out whether it was worth anything, because judging it required a laboratory, a compute budget, and years of training. The idea was cheap and the verdict was expensive.

Something worth watching is happening to that asymmetry.

An open drug-discovery challenge is currently running on Hugging Face, and the interesting part is not the prize money. It is that the evaluation has been made public.

Two seasons are live

Season 1 — Malaria. Target: PfDHODH in Plasmodium falciparum, with human DHODH as a counter-target. Closes 30 September 2026.

Season 2 — Tuberculosis. Target: InhA in Mycobacterium tuberculosis, counter-target human FASN (ER domain). Closes 31 October 2026.

Entry is a molecule, submitted as SMILES or InChI. There is no restriction on how you get there: Claude, GPT, Gemini, Qwen, KIMI, DeepSeek, or a model you trained yourself. The challenge scores the structure, not the pedigree of whoever produced it.

The scoring is where it gets interesting

100 points across six axes — activity, binding, selectivity, ADMET, novelty, synthesizability — with weights that differ per season. Season 2 loads weight onto binding and selectivity, which is the right instinct: an InhA inhibitor that also shuts down the human homolog is not a candidate, it is a toxin.

That counter-target design is the detail most worth noticing. It is easy to build a scoreboard that rewards binds the target. Rewarding binds the target and leaves the human protein alone is harder, and much closer to what actually decides whether a compound survives.

The anchors are in the table

Approved drugs and inert compounds are scored on the same rubric as the entries.

That is an unusual thing for an organizer to do, because it means anyone can see where the scale actually sits — including when a known drug does not score the way you would expect. A leaderboard you can audit is a different object from a leaderboard you are asked to believe.

Private submission is supported, so entrants who want to keep a structure unpublished can still compete; the board shows a masked SMILES. The standings are also exposed as JSON, which means the whole thing can be pulled and checked programmatically rather than read off a webpage.

The honest caveat, which the organizers state themselves

A high score is a research hypothesis, not a drug. Everything here is computational. It is not clinical validation and does not become one by ranking well.

Still — the direction is the encouraging part. Instruments spread, and fields change when they do. Whether this particular challenge produces anything durable is unknown. That it exists in a form outsiders can inspect is already worth something.

Challenge: https://huggingface.co/spaces/FINAL-Bench/open-discovery-challenge

Disclosure: the challenge is operated by the FINAL-Bench organization on Hugging Face.

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

ai maya — Sun, 09 Aug 2026 08:12:38 +0000

TL;DR. A public method called Model DNA — with a live tool — lets outsiders estimate whether an LLM was trained from scratch or derived from an open-weight base, using nothing but public artifacts: config.json, tokenizer.json, and embedding weights. This is a technical deep-dive that cites and reproduces that method in PyTorch, then analyzes where it holds up and where it breaks.

Key takeaways

Three public signals — architecture config, tokenizer overlap, and embedding-space similarity via Linear CKA — combine to place a model on a lineage spectrum.
One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test.
The method's real strengths are reproducibility and rotation-invariant similarity; its real limits are the continued-pretraining gray zone, threshold sensitivity, and an embedding-only view.
Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice; the output is a label, not an accusation.

Why this matters in 2026

Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 those claims stopped being taken on faith. A Zhihu roundtable on the summer model wave and a thread with millions of views became the venue where "self-developed" claims were publicly stress-tested — and several were found more derivative than advertised (coverage).

Model DNA matters because it moved the argument from vibes to a reproducible procedure, and it has been run across major Korean foundation-model builders — among them LG, NAVER, Kakao, SKT, KT, NCSOFT, Upstage, and Motif. (This piece stays at the method level and assigns no verdict to any named company; per-model labels belong to the tool, not to a blog post.) What follows reproduces the procedure in PyTorch and evaluates it as a method, within the scope the source already made public.

The three signals (cited and reproduced)

The premise: no leaked internals required. Everything is read from a model's public files.

1. Architecture fingerprint — `config.json`

Compare the structural fields a lab chooses at design time: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads. Independently designed models rarely align on all of them at once.

import json

ARCH_FIELDS = [
    "model_type", "vocab_size", "hidden_size", "intermediate_size",
    "num_hidden_layers", "num_attention_heads", "num_key_value_heads",
    "max_position_embeddings", "rope_theta",
]

def arch_match_count(cfg_a: dict, cfg_b: dict, fields=ARCH_FIELDS) -> int:
    """Number of structural fields that match simultaneously."""
    return sum(1 for k in fields
              if cfg_a.get(k) is not None and cfg_a.get(k) == cfg_b.get(k))

Reading rule (as the source puts it): a single coincidental field means nothing; five simultaneously is a fingerprint. How many to treat as a threshold depends on the diversity of your candidate base pool.

2. Tokenizer overlap — `tokenizer.json`

Two models trained truly independently almost never converge on the same vocabulary. Normalize shared tokens against the smaller vocabulary.

def tokenizer_overlap(vocab_a: dict, vocab_b: dict) -> float:
    sa, sb = set(vocab_a), set(vocab_b)
    return len(sa & sb) / min(len(sa), len(sb))

A supporting signal only — see Trap 2.

3. Embedding similarity — Linear CKA

The most robust signal compares representation geometry. Naive cosine comparison is fooled by rotation (Trap 1), so the method uses Linear CKA (Centered Kernel Alignment) — from Kornblith et al. (2019), Similarity of Neural Network Representations Revisited (ICML) — which is invariant to rotation, orthogonal transforms, and isotropic scaling.

For row-centered matrices X ∈ ℝ^{n×d1} and Y ∈ ℝ^{n×d2}:

CKA(X, Y) = ||Yᵀ X||²_F / ( ||Xᵀ X||_F · ||Yᵀ Y||_F )

Crucially, it is defined even when d1 ≠ d2, so models with different hidden sizes compare directly.

import torch

@torch.no_grad()
def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float:
    # X:(n,d1), Y:(n,d2) — embeddings over the SAME token set (rows aligned)
    X = X - X.mean(0, keepdim=True)
    Y = Y - Y.mean(0, keepdim=True)
    num = ((Y.t() @ X) ** 2).sum()
    den = torch.sqrt(((X.t() @ X) ** 2).sum() * ((Y.t() @ Y) ** 2).sum())
    return (num / den).clamp(0, 1).item()

Alignment is the catch. The two embedding matrices must index the same tokens. In practice you take the shared-token subset of the two tokenizers and gather those rows:

def aligned_embeddings(emb_a, vocab_a, emb_b, vocab_b):
    shared = sorted(set(vocab_a) & set(vocab_b))
    idx_a = torch.tensor([vocab_a[t] for t in shared])
    idx_b = torch.tensor([vocab_b[t] for t in shared])
    return emb_a[idx_a], emb_b[idx_b]

The genotype framework

The tool collapses the three signals into four labels — a clean way to read any result (Model Genome Korea):

Genotype	Meaning
🟢 Native	Self-designed architecture and from-scratch weights
🔵 Adapted	Mostly original, one borrowed axis
🟡 Mixed	Partial inheritance on both axes
🔴 Ported	Exact foreign architecture match and inherited weights

Two traps that produce confident wrong answers

Trap 1 — row-wise cosine similarity looks rigorous but isn't. It is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. That is exactly why CKA and config signals carry the weight.

Trap 2 — a shared tokenizer proves nothing alone. Tokenizer reuse is often a licensing or convenience decision. Treat overlap as supporting evidence, never a conclusion.

Deep analysis: how far can you trust it?

Strengths

Reproducibility. All three signals compute from public artifacts in a few dozen lines. Claim and verification live on the same plane.
Right invariance. Choosing Linear CKA is correct — it neutralizes the most common disguise (orthogonal transforms) that defeats cosine comparisons.
Evidence fusion. Judging on the simultaneous agreement of three axes suppresses both false positives and false negatives.

Limits (must be acknowledged)

Continued-pretraining gray zone. Embedding CKA identifies from-scratch training well but does not cleanly separate derivatives that keep a base's weights and train heavily on top. Here the verdict is probabilistic and config/tokenizer evidence dominates.
Threshold sensitivity. "How many fields," "what CKA cutoff" depend on the candidate pool. Hard-coding constants makes conclusions wobble when the pool changes — which is why this write-up prescribes none.
Embedding bias. Looking only at the embedding layer is cheap, but a model's "identity" also lives in mid and upper layers. A layer-wise CKA profile improves resolution in the gray zone.
Alignment dependence. Few shared tokens (language- or domain-specific tokenizers) shrink the CKA sample and inflate variance. Report shared-token count alongside CKA.

Improvements worth adopting

Extend single-layer embedding CKA to a layer-wise CKA curve (input → mid → output).
Score against the entire candidate base pool and judge by relative rank, not an absolute cutoff.
Report shared-token counts and bootstrap confidence intervals for statistical significance.

In short, Model DNA fuses the right signals under the right invariance — a solid starting point. It only avoids misjudgment when read as a spectrum with uncertainty, not a from-scratch/not binary.

What fingerprinting cannot tell you

Lineage, not intent. It can show B shares structure with A; it cannot say whether that was disclosed, licensed, or hidden — ethics and paperwork, not linear algebra.
"From scratch" is a spectrum, not a boolean. Data, init, architecture, and post-training each sit on a continuum of originality.
Building on open weights is legitimate. The goal is transparency and accurate labeling, not accusation.

FAQ

Can you tell if an LLM was really trained from scratch?
Usually, with high probability. Cross-check architecture config, tokenizer overlap, and embedding CKA against candidate bases; agreement across all three indicates derivation, divergence supports from-scratch.

What is Model DNA / model provenance?
Estimating a model's origin — original vs. derived from an open-weight base — from public artifacts alone (config, tokenizer, weights), without training data or internal logs.

Is building on Llama, Qwen, or DeepSeek legitimate?
Yes. Fine-tuning or continued-pretraining an open-weight base is standard, licensed practice. Provenance tools report lineage, not misconduct.

How do you tell a fine-tuned model from a from-scratch one?
From-scratch models diverge on architecture and tokenizer and show low embedding CKA to any base. Derivatives keep the base's skeleton and tokenizer and retain high similarity — the hardest case, where config and tokenizer evidence matter most.

Why Linear CKA instead of cosine?
Because CKA is robust to rotation and scaling — the transforms a derived model uses to look "different."

Can models with different hidden sizes be compared?
Yes. CKA compares n×n Gram matrices, so d1 ≠ d2 is fine — as long as embedding rows are aligned to the same tokens.

Resources & related links

Method under analysis: Model DNA — architecture & weight lineage
Live tool (space): Model Genome Korea
Announcement post: Architecture lineage of Korea's sovereign-AI models
Academic basis: Kornblith, Norouzi, Lee, Hinton (2019), Similarity of Neural Network Representations Revisited, ICML.
The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · coverage

Provenance is becoming a norm, not a gotcha. The healthiest version is one where "we trained it from scratch" arrives with — or at least survives — the fingerprint. If you build models, publish the check yourself.

How to Verify a 'Trained-From-Scratch' LLM in 2026: A Provenance and Fingerprinting Guide

ai maya — Sun, 09 Aug 2026 07:23:58 +0000

TL;DR — Can an outsider verify that an LLM was "trained from scratch"? In 2026, largely yes. Using only public Hugging Face artifacts — config.json, tokenizer.json, and embedding weights — you can fingerprint a model's lineage and place it on a spectrum from fully original to a straight port of a foreign base. This guide explains the method, what it can and cannot prove, and the open-weight debate that made it a 2026 talking point.

Key takeaways

Three public signals — architecture config, tokenizer overlap, and embedding-space similarity (Linear CKA) — are enough to estimate whether a model was built from scratch or derived from an open-weight base.
One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test.
Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice — the interesting output is the label, not an accusation.
The topic went mainstream in 2026 because several "self-developed, DeepSeek-class" claims from non-US/China labs were publicly stress-tested and, in places, found more derivative than advertised.

Why this matters in 2026

Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 that claim stopped being taken on faith. A Zhihu roundtable on the summer model wave and a widely-cited Zhihu thread with millions of views became the venue where the Chinese-language community tore into "self-developed" claims from Korean and Indian labs — concluding, in several cases, that the models leaned harder on Chinese open-weight bases than the marketing implied (coverage).

The useful development is that the argument moved from vibes to a reproducible procedure. Public tools now let anyone run the check. This is a refreshed 2026 walkthrough of that procedure — updating the original method write-up with the debate context and the links you need.

The three fingerprint signals

Model provenance rests on three axes you can read straight from a model's public files. No leaked internals required.

1. Architecture fingerprint (`config.json`)

Compare the structural fields a lab chooses when it designs a model: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, and the attention head counts. Independently designed models rarely line up on all of these. A single coincidental field means nothing; five simultaneously is a fingerprint of a shared skeleton.

2. Tokenizer overlap (`tokenizer.json`)

Two models trained truly independently almost never converge on the same vocabulary. Measure the ratio of shared tokens against the smaller vocabulary. High overlap is a strong hint of shared ancestry — but, as the traps below explain, not a verdict on its own.

3. Embedding similarity via Linear CKA

Centered Kernel Alignment (CKA) compares representation geometry in a way that is robust to rotations and rescalings — the superficial transformations that fool cruder weight comparisons. Run on embedding layers, it reliably flags from-scratch training. Its honest limitation: it struggles to cleanly separate continued-pretraining derivatives, where a model keeps a base's weights but trains heavily on top.

The genotype framework

One public framework, Model Genome Korea, collapses the three signals into four labels — a clean mental model for reading any provenance result:

Genotype	Meaning
🟢 Native	Self-designed architecture and from-scratch weights
🔵 Adapted	Mostly original, with one borrowed axis
🟡 Mixed	Partial inheritance on both axes
🔴 Ported	Exact foreign architecture match and inherited weights

Two traps that produce confident wrong answers

Trap 1 — Row-wise cosine similarity looks rigorous but isn't. Comparing weight matrices row by row with cosine similarity feels quantitative, yet it is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. This is exactly why the method leans on rotation-robust CKA and on structural config signals rather than raw weight-row comparisons.

Trap 2 — A shared tokenizer proves nothing by itself. Reusing a tokenizer is frequently a licensing or convenience decision, not evidence of weight-level derivation. Treat tokenizer overlap as a supporting signal, never a conclusion.

What fingerprinting cannot tell you

Lineage, not intent. It can show that model B shares structure with model A. It cannot tell you whether that was disclosed, licensed, or hidden — that is a question of ethics and paperwork, not linear algebra.
"From scratch" is a spectrum, not a boolean. Data, initialization, architecture, and post-training sit on a continuum of originality; a single label flattens a gradient.
Building on open weights is legitimate. The point of a provenance tool is transparency and accurate labeling, not accusation. Most of the industry stands on open-weight shoulders — the honest move is to say so.

FAQ

Can you tell if an LLM was really trained from scratch?
In most cases, yes — to a strong probability. Cross-check architecture config, tokenizer overlap, and embedding CKA against candidate base models; agreement across all three indicates derivation, while divergence supports a from-scratch claim.

What is LLM fingerprinting / model provenance?
It is the practice of estimating a model's origin — original vs. derived from an existing open-weight base — using only its public artifacts (config, tokenizer, weights), without access to training data or internal logs.

Is building a model on Llama, Qwen, or DeepSeek legitimate?
Yes. Fine-tuning or continued-pretraining an open-weight base is standard, licensed (subject to each license), industry practice. Provenance tools report lineage; they do not allege wrongdoing.

How do you distinguish a fine-tuned model from a from-scratch one?
From-scratch models diverge on architecture and tokenizer and show low embedding CKA to any base. Fine-tuned or continued-pretraining derivatives keep the base's skeleton and tokenizer and retain high embedding similarity — the hardest case, where CKA alone is weakest and config/tokenizer evidence matters most.

Does a shared tokenizer mean a model is derived?
No. It is a supporting signal only. Confirm with architecture and weight evidence before drawing a conclusion.

Why Linear CKA instead of cosine similarity?
Because CKA is robust to rotation and scaling, the transformations that let a derived model masquerade as "different" under naive cosine comparisons.

Resources & related links

Reproducible method write-up: How to Tell If an LLM Was Really Trained From Scratch
Deep-dive article: Model DNA — architecture & weight lineage
Live tool: Model Genome Korea
Announcement post: Architecture lineage of Korea's sovereign-AI models
The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · media coverage

Provenance is becoming a norm, not a gotcha. The healthiest version of this trend is one where "we trained it from scratch" is expected to arrive with — or at least survive — the fingerprint. If you build models, publish the check yourself.

The KV Cache Is the Bottleneck: A 2026 Field Guide to Attention Variants

ai maya — Sun, 09 Aug 2026 07:15:29 +0000

If you want to understand why 2026's frontier LLMs look the way they do — why one ships Multi-head Latent Attention, another interleaves linear and full attention, a third leans on sliding windows — stop looking at benchmark scores and look at the KV cache. Almost every architectural decision at the frontier this year is, at heart, a fight with the memory that autoregressive decoding leaves behind.

This is a practitioner's map of that fight: what the KV cache actually costs, the lineage of techniques built to shrink it, and how to choose among them.

Why the KV cache dominates

During generation, a decoder-only transformer caches the key and value vectors of every past token so it doesn't recompute them each step. That cache is the price of fast decoding, and it is not small.

The size is brutally simple:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × dtype_bytes

Two consequences fall out immediately:

It grows linearly with context and batch. Double the context, double the cache. Serve more concurrent users, multiply again. At long context and healthy batch sizes, the KV cache — not the weights — becomes the dominant consumer of accelerator memory.
Decoding is memory-bandwidth bound, not compute bound. Generating one token touches the entire cache. You are not FLOP-limited; you are limited by how fast you can stream that cache off HBM. Shrinking the cache is therefore the most direct lever on both memory footprint and tokens/sec.

Every technique below is a different answer to "how do we make that formula smaller without breaking the model."

The lineage of fixes

1. Fewer KV heads: MQA → GQA

The first target is kv_heads. Multi-Query Attention (MQA) collapses all query heads onto a single shared KV head — a large cut, but it can cost quality and training stability. Grouped-Query Attention (GQA) is the pragmatic compromise that won: query heads are split into groups, each sharing one KV head. With, say, 8 KV heads for 64 query heads you get most of MQA's savings with far less quality loss. GQA is the default in the majority of open-weight models today for exactly this reason.

2. Compress the cache: Multi-head Latent Attention (MLA)

Instead of storing fewer heads, MLA stores a low-rank latent. Keys and values are projected down to a small shared latent vector that is cached; the per-head K and V are reconstructed on the fly during attention. The cached object is dramatically smaller than full K/V, yet the model still attends with many effective heads. MLA is the headline reason some 2026 models sustain very long context at serving-friendly memory — it attacks head_dim × kv_heads directly rather than just cutting head count.

3. Kill the growing cache entirely: linear attention & SSMs

Softmax attention is O(n²) in compute and O(n) in cache. Linear attention and state-space models (SSMs, Mamba-style) rewrite the operation as a recurrence with a fixed-size state. There is no cache that grows with sequence length — memory is O(1) in context. The catch is real: a fixed state is a lossy summary, so pure linear models are weaker at precise long-range recall and associative lookup ("what was the exact token 40k ago"). They are fast and cheap; they forget details.

4. The 2026 consensus: hybrid, layer-interleaved attention

The dominant pattern this year is not to pick one. Models interleave a minority of full (softmax) attention layers with a majority of linear/SSM layers. The full layers preserve the sharp recall that pure linear loses; the linear layers carry the long-context load cheaply. The result is near-linear memory scaling with most of the quality of full attention. When you read that a 2026 release is "hybrid attention," this layer-wise interleaving is almost always what is meant.

5. Bound the window: sliding-window attention + sinks

Orthogonal to the above, sliding-window attention simply caps how far back each token attends (e.g., the last few thousand tokens), making the cache constant-size past the window. Naively this destroys long-context behavior, so it is paired with tricks like attention sinks — keeping the first few tokens always in view — which stabilize very long generations. Cheap, effective for many workloads, and easy to combine with GQA.

Two more multipliers worth knowing

Quantized KV cache. The dtype_bytes term is a free-ish win: storing the cache in FP8 or INT8 instead of FP16 halves or quarters it with modest quality impact. In 2026 this is close to standard for long-context serving.
Paged KV memory. PagedAttention (the idea behind vLLM's throughput) doesn't shrink the cache — it stops you from wasting it. By allocating the cache in fixed pages instead of one contiguous per-request block, it removes fragmentation and lets you pack far more concurrent sequences into the same HBM. Architecture shrinks the cache; paging spends what's left efficiently.

How to choose

Building a general open-weight model? GQA is the safe, proven baseline. You will not be criticized for it.
Chasing long context at serving-friendly memory? MLA-style latent compression is the strongest single lever, at the cost of implementation complexity.
Throughput on shorter contexts, or edge deployment? Sliding window + GQA + quantized cache is a cheap, robust stack.
Very long context where some recall loss is acceptable? A hybrid linear/full interleave gives you near-linear scaling — validate recall on your long-context tasks before committing.
Serving any of the above? Quantize the cache and use a paged runtime. These compose with every architecture and are the lowest-effort wins on the list.

The real takeaway

The lesson of 2026 is that model architecture and the serving stack are now co-designed. You cannot reason about an attention variant without reasoning about its KV cache, its memory bandwidth, and how a runtime will page it. The models that feel fast and cheap in production didn't get there by accident — they were shaped, layer by layer, by the cost of remembering.

When you evaluate the next "new architecture," skip the leaderboard for a minute and ask the only question that predicts its serving cost: what happens to the KV cache?

What's your KV strategy in production — GQA, latent compression, hybrid, or just quantize-and-page? Curious what's holding up under real load.

Local LLMs in 2026: What Actually Runs Well on a Laptop Now

ai maya — Sun, 09 Aug 2026 00:23:28 +0000

Two years ago, "run a language model locally" meant a weekend of compiling, a graveyard of CUDA errors, and a model that answered like it had a concussion. In 2026, you can install one tool, type one command, and have a genuinely useful assistant running on a laptop with no internet connection. Here's an honest map of what works, what doesn't, and where the sharp edges still are.

Why bother running locally at all

Three reasons keep pulling developers back to local inference:

Privacy. The prompt never leaves your machine. For code you can't paste into a cloud box, or personal data, that's non-negotiable.
Cost and offline. No per-token bill, no rate limits, and it works on a plane.
Latency and control. No network round-trip, and you pin the exact model version forever — no silent upgrades changing your outputs.

The catch has always been quality-per-watt. That's the number that moved.

The hardware tiers, honestly

8 GB RAM / integrated GPU: You can run 3–4B parameter models at 4-bit quantization. Good for autocomplete, summarizing, simple Q&A. Don't expect deep reasoning.
16 GB RAM: The sweet spot for most developers. 7–9B models run comfortably and are genuinely helpful for coding assistance and drafting.
32 GB+ or a discrete GPU with 16–24 GB VRAM: Now you're running 20–30B models, or bigger models at aggressive quantization, with real reasoning ability.
Apple Silicon (unified memory): Punches above its weight. A machine with 32–64 GB of unified memory runs models that would need an expensive discrete GPU on other platforms, because the CPU and GPU share the same memory pool.

Quantization: the trick that makes it possible

The reason a 7B model fits in 16 GB is quantization — storing weights at 4 bits instead of 16. The common format you'll see is GGUF, and the common recipe is 4-bit (often labeled Q4). The quality loss from full precision to 4-bit is surprisingly small for most tasks, while the memory savings are 4x. Below 4-bit (2–3 bit) the model starts to degrade noticeably; above it (5–8 bit) you pay memory for diminishing returns. For most people, 4-bit is the default that just works.

The tools

Ollama — the "one command" experience. ollama run <model> and you're talking to it. Best starting point.
llama.cpp — the engine most of the ecosystem is built on. Reach for it when you want control, custom quantization, or to embed inference in your own app.
LM Studio — a GUI if you'd rather click than type, with easy model browsing.

Which small models are actually good

Without naming a single "winner" — because it changes monthly — the strong open-weight families in the small tier this year are the usual suspects: the Qwen line, Google's Gemma line, Meta's Llama small variants, and Microsoft's Phi series. The practical advice: don't marry a model. Download two or three in the same size class, run your real prompts through them, and keep the one that wins on your work. A leaderboard can't tell you which model is best at your codebase; your codebase can.

The sharp edges nobody warns you about

Context length costs RAM. A model's advertised context window is not free — filling it with a long document can blow past your memory budget and crawl. Start with modest context and grow it.
First-token lag on cold start. The model has to load into memory. Keep it warm if you're calling it repeatedly, or the first request after idle feels broken.
Small models confabulate more. They're excellent at transformation (summarize, rewrite, extract) and shakier at open-ended factual recall. Use them for shape, not for ground truth.
Speed is "fine," not "instant." On a mid laptop expect readable-speed streaming, not cloud-flagship snappiness. For interactive use it's perfectly usable; for batch jobs, plan for it.

The takeaway

Local LLMs in 2026 crossed the line from "impressive demo" to "daily driver" for a lot of everyday tasks — summarizing, drafting, coding assistance, private Q&A. You won't replace a frontier model for the hardest reasoning, but you'll be surprised how rarely you need to. Install Ollama, pull a 7B model at 4-bit, throw your real work at it, and see how far the free, private, offline tier gets you.

For most developers in 2026, the answer is: further than you'd think.

What's your local setup — hardware, tool, and go-to model? And what task finally made local inference "good enough" for you?

MCP in 2026: How the Model Context Protocol Became the USB-C of AI Tooling

ai maya — Sun, 09 Aug 2026 00:17:47 +0000

A year ago, connecting a model to your tools meant writing glue for that model, in that framework, with that vendor's function-calling format. Swap the model and you rewrote the glue. In 2026, that pain is mostly gone, and the reason has a boring name: the Model Context Protocol (MCP).

MCP is worth understanding not because it's clever, but because it's winning — and the reason it's winning tells you where the industry's center of gravity is moving.

What MCP actually is

Strip away the branding and MCP is a small client–server contract for connecting language models to the outside world. A server exposes three kinds of things: tools (functions the model can call), resources (data the model can read), and prompts (reusable templates). A client — your IDE, your agent, your chat app — speaks the same protocol and can talk to any compliant server.

The analogy people keep reaching for is USB-C, and it's accurate. Before USB-C you had a drawer full of proprietary chargers. MCP is the drawer-emptying moment for AI integrations: write the connector once, and any MCP-aware client can use it.

Why "model-agnostic" is the whole point

Here's the shift that matters. For most of the LLM era, your tooling was coupled to a model. If you built your agent stack around one vendor's function-calling quirks, you were locked in — a new, better model meant a migration project.

MCP decouples the tooling layer from the model layer. Your filesystem server, your database server, your ticketing-system server don't know or care which model is on the other end. When a new flagship drops — and in 2026 they drop every few weeks — you point your client at it and keep your entire tool ecosystem intact.

That's a strategic hedge, not just a convenience. In a market where the "best model" changes monthly, the durable asset is your integration layer, and MCP is how you stop rebuilding it.

What to build with it

Practical entry points, cheapest first:

Wrap an internal system as a server. Your team's deploy tooling, your logs, your knowledge base. Now every agent and IDE in the org can reach it through one contract.
Compose existing servers. The ecosystem already has servers for filesystems, databases, browsers, and dozens of SaaS tools. Most useful agents in 2026 are less "write everything" and more "wire the right servers together."
Ship a client capability, not a bespoke integration. If you're building an agent product, being MCP-aware means your users bring their own tool servers instead of waiting for you to build each one.

The pitfalls nobody puts in the README

MCP solves plumbing, not judgment. Watch for:

Tool sprawl. Expose 60 tools and the model spends its context deciding which to use — and picks wrong. Curate. A focused server beats a kitchen-sink one.
Prompt injection through tool output. The text a server returns is data, not instructions — but a naive agent will happily follow "ignore previous instructions" if it arrives inside a fetched web page. Treat every server response as untrusted input. This is the single most under-appreciated risk in the whole stack.
Auth and blast radius. A server that can delete production data is a server that will delete production data the first time a model hallucinates an argument. Scope permissions tightly and gate irreversible actions behind confirmation.
Versioning drift. Servers evolve; clients cache assumptions. Pin and test like you would any API dependency.

The takeaway

MCP isn't exciting, and that's exactly why it's important. Standards win by being boring and everywhere. The teams that come out ahead in 2026 are the ones treating their tool integrations as a durable, model-agnostic layer — and letting the models underneath churn as fast as the market wants them to.

Build the connector once. Let the models come and go.

Are you exposing internal systems over MCP yet, or still writing per-model glue? What server would you build first?

Default-to-Flagship Is Now a Cost Bug: Tiered Model Routing for Agentic Workloads

ai maya — Sun, 09 Aug 2026 00:17:41 +0000

For two years the reflex was simple: reach for the biggest model you can afford and call it a day. In 2026 that reflex quietly became a bug in your cost model.

The clearest signal came this summer, when a smaller, cheaper "flash"-tier model started edging out its own flagship sibling on the workload developers care about most — multi-step agentic coding — at a fraction of the price. When the fast tier wins the hard benchmark, "always use the flagship" stops being a safe default and starts being waste.

Here's how to fix it without turning your stack into a science project.

Why the reflex is expensive

Agent workloads are not one big call. A single task fans out into dozens of small ones: planning, tool selection, argument formatting, summarizing a file, deciding whether to continue. Most of those steps are easy. Routing every one of them through a frontier model is like taking a helicopter to the corner store — it works, but you are paying helicopter prices for a walk.

The trap is that the cost is invisible per call and enormous in aggregate. You never see the moment you overpaid; you just see the invoice.

The three-tier ladder

Think in tiers, not models:

Cheap/fast tier — classification, extraction, short rewrites, routing decisions, "is this done?" checks. Most steps live here.
Mid tier — normal reasoning, code edits, tool use with moderate context.
Flagship tier — genuinely hard reasoning, long-context synthesis, the step where a wrong answer poisons everything downstream.

The goal is to keep the flagship tier for the 5–15% of steps that actually need it, and let the cheap tier carry the volume.

How to decide the tier per request

Two mechanisms, used together:

Static heuristics for the obvious cases. Short prompt + structured output + low stakes → cheap tier. Anything touching a large context window or a irreversible action → escalate.

Eval-gated escalation for everything else. Start at the cheap tier, and only promote to a bigger model when your evals prove the cheap tier fails on that class of input. This is the key discipline: escalation is earned by evidence, not assumed. If you can't point to an eval where the small model loses, you don't get to pay for the big one.

A simple confidence signal helps too — if the cheap model hedges, returns malformed output, or low log-probs, retry one tier up. One retry at a higher tier is still cheaper than sending everything to the flagship.

Measure it or you're guessing

You cannot route what you don't measure. Log, per step: which tier ran, token counts, latency, and a success signal. Then compute the boring but decisive number — cost per completed task, not cost per token. Teams that optimize per-token often make per-task cost worse by adding retries; the task-level metric keeps you honest.

Re-run this monthly. Model prices and capabilities move fast enough that last quarter's optimal routing table is this quarter's mistake.

Pitfalls

Cheap-tier false economy. If a weak model fails a planning step, every downstream step inherits the error. Put your best model at the top of the plan, cheap models on the leaves.
Silent capability drift. A model point-release can flip your routing assumptions overnight. Pin versions in evals and re-test before upgrading.
Over-engineering the router. A 50-line heuristic plus one escalation rule beats a fancy learned router for most teams. Add complexity only when the data demands it.

The takeaway

The winning move in 2026 is not "use the best model" or "use the cheapest model." It's building a thin routing layer and an eval harness that lets you place each step on the right tier — and swap the underlying models in an afternoon when the market moves again.

Default-to-flagship felt safe because it was simple. It's still simple. It's just no longer cheap.

How are you routing model tiers in production right now — heuristics, a learned router, or still one model for everything? Curious what's working.

AI This Week (Aug 2026): Qwen3.8 Max, DeepSeek V4-Flash, and Models Shipping Like Patches

ai maya — Sat, 08 Aug 2026 16:19:17 +0000

If you blinked this week, you missed three frontier model releases. August 2026 is making one thing very clear: large language models now ship like software patches, not like moon landings.

Here's the AI news that actually matters for builders right now.

Qwen3.8 Max lands

Alibaba pushed Qwen3.8 Max at the start of the month — the newest flagship in a line that has become the default open-weight baseline for a lot of teams. The interesting part isn't a single headline number; it's the cadence. Qwen has gone from "the model you try" to "the model you benchmark against" in about a year, and each point release keeps tightening the gap with closed frontier models on reasoning and coding.

Dev takeaway: if your eval harness still pins an old open baseline, it's already stale. Re-run against the current Qwen point release before you claim a win over "open source."

DeepSeek V4-Flash quietly beats its big sibling

The story engineers keep repeating: the DeepSeek-V4-Flash retrain reportedly edges out V4-Pro on agentic coding benchmarks — at flash pricing. Read that twice. A smaller, cheaper, faster model beating the flagship on the workload developers care most about (multi-step tool use and code) is the clearest signal yet that the "bigger is always better" era is over for a lot of tasks.

Dev takeaway: for agent workloads, start with the small/fast tier and only escalate to the flagship when your evals prove you need it. Default-to-flagship is now a cost bug.

OpenAI's GPT-5.6 "Luna"

OpenAI's active lineup now includes GPT-5.6 Luna as a fresh flagship name. Details are thin, but the pattern is familiar: incremental version bumps, multimodal by default, and reasoning modes that trade latency for accuracy on demand. The headline isn't any one capability — it's that "reasoning vs. speed" is now a dial you set per request, not a model you pick once.

NVIDIA open-sources NOOA

Less flashy but arguably more useful for practitioners: NVIDIA open-sourced NOOA (Object-Oriented Agents), a model-agnostic Python framework for building agents. Model-agnostic is the key word — the tooling layer is decoupling from any single provider, which is exactly what you want if you're tired of rewriting your agent stack every time a new model drops.

The real trend: models as patches

Release trackers are now counting 300+ model releases across major labs. When frontier capability ships every few weeks, three things follow for teams:

Your benchmarks decay fast. Treat evals as living infrastructure, not a one-time report.
Small + fast is the new default. Efficiency gains keep delivering last-year's flagship quality at a fraction of the cost. Reach for the big model deliberately, not reflexively.
Bet on the tooling layer, not the model. Model-agnostic agent frameworks (like NOOA) protect you from churn.

The models will keep coming. The winning move isn't chasing every release — it's building an eval + tooling layer that lets you swap the engine in an afternoon.

What shipped this week that you're actually putting into production? Drop it in the comments.

DEV Community: ai maya

They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

The confound that makes this hard to measure

The actual experiment

Now the part worth your attention

Does it survive scale?

Two methodology details worth stealing

What is released

The takeaway for practitioners

We Asked 330 Models a Question in Korean. Half of Them Answered in the Wrong Alphabet.

The check

What it caught

Why a mechanical gate and not the judge

The harness refuses to start if it cannot tell good from bad

The grades, once the surviving answers are graded

Vendor averages

Recency and size predict nothing

Run it yourself

The portable part

The Same Model Can Cost 14x More Depending on Who Serves It

Why the spread exists

Latency: don't inherit it, measure it

The column nobody publishes

Use the data

The practical takeaway

opened a financial forecasting contest that AI agents can enter directly. $2,000 in prizes, 122 days.

Why another leaderboard

Design 1 — Take positions, not predictions

Design 2 — Verify the scorer before any model

Design 3 — The luck ceiling

Design 4 — Why we gave up on cross-asset comparison

Design 5 — Cost, not rules, stops spam

Design 6 — Putting a scale on an empty chart

The result was more interesting than expected

Design 7 — An hour, not a day

Design 8 — Agents as first-class entrants

Design 9 — The data has to be P&L someone could actually realise

Two things infrastructure taught us

Market data providers block datacenter IPs

Writable is not durable

The organisers do not compete

So what do we want to learn

Entering

Day one

An open challenge is letting anyone submit a drug candidate - and scoring it in public

Two seasons are live

The scoring is where it gets interesting

The anchors are in the table

The honest caveat, which the organizers state themselves

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

Key takeaways

Why this matters in 2026

The three signals (cited and reproduced)

1. Architecture fingerprint — config.json

2. Tokenizer overlap — tokenizer.json

3. Embedding similarity — Linear CKA

The genotype framework

Two traps that produce confident wrong answers

Deep analysis: how far can you trust it?

What fingerprinting cannot tell you

FAQ

Resources & related links

How to Verify a 'Trained-From-Scratch' LLM in 2026: A Provenance and Fingerprinting Guide

Key takeaways

Why this matters in 2026

The three fingerprint signals

1. Architecture fingerprint (config.json)

2. Tokenizer overlap (tokenizer.json)

3. Embedding similarity via Linear CKA

The genotype framework

Two traps that produce confident wrong answers

What fingerprinting cannot tell you

FAQ

Resources & related links

The KV Cache Is the Bottleneck: A 2026 Field Guide to Attention Variants

Why the KV cache dominates

The lineage of fixes

1. Fewer KV heads: MQA → GQA

2. Compress the cache: Multi-head Latent Attention (MLA)

3. Kill the growing cache entirely: linear attention & SSMs

1. Architecture fingerprint — `config.json`

2. Tokenizer overlap — `tokenizer.json`

1. Architecture fingerprint (`config.json`)

2. Tokenizer overlap (`tokenizer.json`)