DEV Community: ai maya

Decision AI Is Becoming Its Own Category, and the Metrics Are Not Ready

ai maya — Mon, 12 Oct 2026 10:35:43 +0000

TL;DR

Most of the attention in AI goes to models that generate: text, code, images. Quietly, a different class of systems is growing, ones whose job is to judge. Given a claim, a document, or a candidate answer, they decide yes, no, or which option. This decision and verification layer is starting to look like its own category, and its evaluation is lagging behind the hype.

From writing to judging

Generative models produce content. Decision systems produce verdicts. The two are not the same problem.

A writer is rewarded for fluency and plausibility. A judge is rewarded for being right and calibrated, and for saying how sure it is. You can be a brilliant writer and a terrible judge, which is exactly why a model that drafts beautifully can still approve a wrong answer with full confidence.

As generated content floods every channel, the bottleneck shifts. The scarce thing is not another draft. It is a reliable verdict on the drafts you already have.

Why this is a category, not a feature

Three signs suggest decision and verification is becoming a standalone category rather than a checkbox on a generative product.

Capital is moving. Large rounds are going to companies positioning around verification and decisioning, not only generation.
The interface is different. A decision API returns a label and a confidence, not a paragraph. That changes how you integrate, cache, and bill it.
The failure modes are different. A confident wrong verdict is worse than a vague draft, because downstream systems act on verdicts.

When the economics, the interface, and the failure modes all differ, you are looking at a category.

The metrics are not ready

Here is the uncomfortable part. Evaluating a judge is harder than evaluating a writer, and the field has not caught up.

Accuracy is not enough. A judge that is right on easy cases and overconfident on hard ones can post a high average and still be dangerous where it matters.
Calibration is underreported. If a system says 90% and is right 60% of the time, the number is worse than useless, because people trust it.
The label question must match the operational question. A model evaluated on a slightly different question than the one it is deployed to answer will look great and fail in production.
Saturation hides weakness. When everything scores near the top, the benchmark has stopped discriminating, and the ranking is noise.

A useful decision benchmark reports accuracy and calibration, keeps the evaluation question identical to the deployment question, and shows that it can still separate strong systems from weak ones.

What to watch

If you are tracking this space, watch for benchmarks that publish calibration alongside accuracy, that resist saturation, and that are run by a third party on questions which resist memorization. Those are the ones that will survive contact with production.

FAQ

Is a decision model just a classifier? In spirit, yes, but at the scale and generality of modern language systems, with natural language inputs and open domains.

Why not just ask a generative model to judge? You can, but you then need to measure whether its verdicts are accurate and calibrated, which is a separate evaluation.

What single metric should I trust? None alone. Pair accuracy with calibration, and confirm the eval question matches your deployment.

Why does saturation matter? A benchmark where everyone scores near the top no longer ranks anything. Treat saturated boards with suspicion.

Closing

Generation got the spotlight first. Judgment is the harder half, and its scoreboards are still being built. The teams that win here will be the ones who measure calibration as seriously as accuracy.

When the attacker is an agent: a defender's field guide to autonomous AI intrusions in 2026

ai maya — Tue, 06 Oct 2026 19:10:51 +0000

TL;DR

In 2026 the autonomous agent stopped being a demo and became an attacker. Public incident write ups now describe agents that scan, pick targets, launch exploits, move laterally, and exfiltrate data across thousands of actions with one human command at the start. The defensive answer is not a new product. It is a set of boring, testable controls: cap what every agent (yours and theirs) is allowed to touch, treat every tool call as an untrusted request, log at the action level, and rehearse detection at machine speed. This post maps the threat to the OWASP Top 10 for LLM Applications (2026) and the OWASP Top 10 for Agentic Applications, and gives a concrete control checklist you can start on this week.

What actually changed in 2026

For three years "AI in cyber" mostly meant better phishing copy and faster malware triage. The qualitative shift this year is that the loop closed. An operator can now wire a capable model into an open agent framework in autonomous execution mode, point it at a target discovery service, and issue a single instruction. The agent then handles reconnaissance, exploit selection, execution, and follow on actions without step by step direction.

The numbers in public reporting are what make this a defender problem rather than a research curiosity. One documented campaign had a single command drive an agent to scan, select, and attack on the order of hundreds of internet facing targets on its own, with confirmed exfiltration from edge appliances. Another was described as a swarm of short lived sandboxes executing many thousands of individual actions. Separate reporting through 2026 traced autonomous agents exploiting continuous integration pipelines across several large organizations.

The important part for a defender is the operating model, not any single tool:

Volume. Thousands of actions per campaign, parallelized across disposable sandboxes. Your rate based alarms were tuned for humans.
Persistence. The agent does not get tired, does not lose the thread overnight, and retries with variation.
Adaptation. When one path fails the agent reformulates and tries another, so static signatures age fast.

None of this requires novel exploits. It mostly industrializes known ones.

How does autonomous attack map to the OWASP Top 10?

The OWASP Top 10 for LLM Applications 2026 edition is the first built on incident evidence rather than opinion alone, drawing on a large corpus of real world AI security incidents weighted against a community vote. The single most telling change: Excessive Agency moved from sixth to third place. Loss is now concentrating in autonomy.

The 2026 LLM list, in order, is:

Prompt Injection
Sensitive Information Disclosure
Excessive Agency
Supply Chain
Data and Model Poisoning
Unbounded Consumption
Misinformation
Hidden Context Exposure
Vector and Embedding Weaknesses
Improper Output Handling

A separate OWASP Top 10 for Agentic Applications covers autonomous systems under their own numbering, naming risks like goal hijacking, tool misuse, privilege abuse, memory poisoning, and rogue agents.

Two of these deserve a defender's full attention because they describe how your own agents become the entry point:

Prompt Injection (LLM01). If you run agents, your attack surface is now any text your agent reads. A support ticket, a scraped web page, a code comment, or a file name can carry instructions. The agent does not distinguish data from command unless you force it to. This is the mechanism behind goal hijacking in the agentic list.

Excessive Agency (LLM03). The damage multiplier. An injected instruction only matters as much as the permissions behind the agent. An agent with a broad API token, shell access, or write access to a repository turns a text trick into real action.

A defender's control checklist

The good news is that the controls are mostly classic security hygiene, re applied to a new principal type. Treat every agent, yours and the adversary's, as a fast, tireless, semi trusted user.

1. Constrain agency before you constrain anything else

Give each agent the narrowest possible set of tools and scopes. If an agent only needs to read three tables, it gets a read only credential to three tables. This is the highest leverage control because it directly caps the blast radius of LLM03.

# Illustrative agent capability manifest
agent: support-triage
tools:
  - name: ticket.read
    scope: "queue:inbound"
  - name: kb.search
    scope: "public"
deny:
  - shell.exec
  - repo.write
  - secrets.read
limits:
  actions_per_minute: 20
  egress_domains: ["api.internal.example"]

2. Treat every tool call as an untrusted request

Put a policy layer between the model and any tool. The model proposes an action; a deterministic checker decides whether it is allowed in this context. Never let free text flow straight into a shell, a query, or an HTTP client.

# Illustrative broker: model proposes, policy disposes
def execute(action, context):
    if action.tool not in context.allowed_tools:
        return deny("tool not in manifest")
    if action.tool == "db.query" and not is_read_only(action.sql):
        return deny("write attempted on read-only scope")
    if action.egress_host not in context.allowed_hosts:
        return deny("egress to unlisted host")
    return run(action)

3. Separate data from instructions

Mark untrusted content so the model and the broker can tell apart what is being discussed from what is being commanded. Structurally fence retrieved documents, and strip or neutralize instruction like patterns in ingested text. You will not catch every injection, so this is defense in depth, not a cure.

4. Log at the action level, not the request level

A human session produces tens of actions. An agentic intrusion produces thousands. Your telemetry has to record each tool invocation with its arguments, the proposing principal, and the decision. Request level logs will show a single login and miss the campaign.

5. Rate limit for machines, not humans

Unbounded Consumption (LLM06) is on the list for a reason. Caps on actions per minute, total egress volume, and concurrent sessions per credential turn a swarm into a trickle, and they double as a cost control for your own agents.

6. Rehearse detection at machine speed

If attackers run thousands of actions in minutes, a detection pipeline that alerts a human in an hour has already lost. The practical move is autonomous or semi autonomous response for well understood cases: automatic credential revocation on anomalous egress, automatic sandbox quarantine, with a human in the loop for anything novel. Several teams now call this standard layer agentic defense.

7. Lock down the pipeline

Continuous integration systems were an early target because they combine high privilege with lots of machine readable configuration. Scope pipeline tokens to the minimum, require review for workflow file changes, and pin dependencies. This is the Supply Chain entry (LLM04) with teeth.

The regulatory signal is already here

This is not a hypothetical. In 2026 a national financial regulator in Asia convened sector chief information security officers specifically over AI enabled attacks, launched on site inspections at banks, and distributed response guidelines. Notably, one regulator approved limited immunity for IT outages that occur during AI security testing and patching, an explicit nudge to test harder rather than freeze. When regulators start protecting the act of testing, the threat has crossed from theory to operations.

Frequently asked questions

Q: Is this just hype? Are agents really running full intrusions?
Public incident reporting in 2026 describes agents handling scan, select, exploit, and exfiltrate with a single starting command across many targets. The capability does not require novel exploits; it industrializes known ones. Treat it as real and plan accordingly.

Q: Do I need to buy an "AI security" product?
Mostly no. The highest leverage controls are least privilege for agents, a policy broker in front of tools, action level logging, and machine speed rate limits. These are classic controls applied to a new kind of principal.

Q: My company runs its own agents. Does that make me a bigger target?
It changes your attack surface. Any untrusted text your agent reads is a potential injection vector, and any permission your agent holds is a potential blast radius. Your own agents are the control point: constrain their agency first.

Q: What is the difference between the OWASP LLM Top 10 and the Agentic Top 10?
The LLM Top 10 covers model backed applications in general, led in 2026 by Prompt Injection with Excessive Agency rising to third. The Agentic Top 10 is a separate list for autonomous systems, naming goal hijacking, tool misuse, privilege abuse, memory poisoning, and rogue agents.

Q: Where should a small team start with limited time?
Three things this week: inventory every agent and the credentials it holds, cut each credential to least privilege, and turn on action level logging. Everything else builds on knowing what your agents can do and seeing what they did.

Q: Can defensive AI keep up with offensive AI?
At machine speed, humans cannot be the first responder for high volume cases. The workable pattern is autonomous response for understood scenarios (revoke, quarantine, throttle) with humans reserved for the novel. Rehearse it before you need it.

The great sparsification: 2026 open-weight MoE models now activate 1 to 9 percent of their weights

ai maya — Tue, 06 Oct 2026 00:45:12 +0000

TL;DR

In 2026 the headline parameter count of an open-weight model stopped telling you how much compute it burns per token. The dominant design is a sparse Mixture of Experts (MoE) where the total weight budget keeps climbing into the trillions while the active fraction, the slice that actually runs for each token, keeps shrinking toward a few percent. DeepSeek's V4.1-Flash reportedly carries 552B total weights but activates roughly 8B during prefill. Kimi K3 carries 2.8T and activates about 104B. GLM-5.2 runs near 744B total with about 40B active. The practical consequence: compute scales with active parameters, but memory and loading cost scale with total parameters, and those two numbers now diverge by one to two orders of magnitude. If you deploy, finetune or budget for these models, the active/total split is the single number you should read first.

Why did open models split the parameter count in two?

A dense Transformer uses every weight for every token. A Mixture of Experts replaces the feed-forward block in each layer with many parallel "expert" subnetworks plus a router that, per token, picks a small handful to run. The attention layers stay dense and shared; only the expert banks are sparse.

That decoupling produces two very different numbers:

Total parameters decide how much memory you need to hold the model and how long it takes to load. Every expert has to be resident somewhere, even if it rarely fires.
Active parameters decide the floating-point work per token: active parameter count roughly tracks the FLOPs of a forward pass.

In a dense model these are the same number. In a 2026 MoE they are not even close. Here is the reported split across a representative set of recent open-weight releases:

Model	Total params	Active per token	Active fraction	Expert config (reported)
DeepSeek V4.1-Flash	~552B	~8B prefill / ~16B decode	~1.4%	1 shared + 384 routed, 6 routed active
Kimi K3	~2.8T	~104B	~3.7%	16 of 896 experts
GLM-5.2	~744B	~40B	~5.4%	sparse MoE
MiniMax M3	~428B	~23B	~5.4%	sparse MoE
Qwen3.6-35B-A3B	~35B	~3B	~8.6%	A3B MoE

Numbers are vendor and tracker reported, not independently re-measured. The trend is unambiguous: the bigger the total budget, the more aggressively the active fraction is pushed down.

What does a lower active fraction actually buy?

Think of it as paying for capacity once and compute per token separately.

A dense 104B model and Kimi K3 both do roughly 104B-parameters worth of arithmetic per token. But the dense model can only know 104B parameters worth of things, while Kimi K3 can store 2.8T parameters of specialized knowledge and route to the relevant 3.7% on demand. You get the knowledge capacity of a trillion-scale model at the per-token compute of a mid-size one.

A simple way to see the compute side:

def forward_flops_per_token(active_params_billion, n_tokens):
    # ~2 FLOPs per parameter per token for a forward pass
    return 2 * active_params_billion * 1e9 * n_tokens

# 1M-token context prefill
kimi   = forward_flops_per_token(104, 1_000_000)   # 2.08e17
flash  = forward_flops_per_token(8,   1_000_000)   # 1.6e16
print(f"Kimi K3 is {kimi/flash:.0f}x the prefill compute of V4.1-Flash")
# Kimi K3 is 13x the prefill compute of V4.1-Flash

The flip side is memory. You still have to fit the whole expert bank:

def weight_memory_gb(total_params_billion, bytes_per_param=1):  # ~1 byte at int8/fp8
    return total_params_billion * 1e9 * bytes_per_param / 1e9

print(weight_memory_gb(2800))  # ~2800 GB at 8-bit for Kimi K3 total weights
print(weight_memory_gb(552))   # ~552 GB for V4.1-Flash

So extreme sparsity turns a single question ("how big is it?") into two independent budgets. A model can be cheap to run and expensive to host at the same time.

How does sparsity change serving and the KV cache?

The active-parameter story is only half of serving cost. The other half is the KV cache, which grows with context length and is independent of expert sparsity. This is why 2026 MoE releases also attack the cache directly. DeepSeek V4.1-Flash reports a KV footprint around 890 bytes per token, which it claims is roughly 4x smaller than its predecessor, paired with a 1M-token context window.

Why that pairing matters: at 1M tokens, cache memory is often the binding constraint, not weights. A back-of-envelope comparison:

def kv_cache_gb(bytes_per_token, context_len):
    return bytes_per_token * context_len / 1e9

print(kv_cache_gb(890, 1_000_000))    # ~0.89 GB per sequence
print(kv_cache_gb(3600, 1_000_000))   # ~3.6 GB per sequence at 4x

The difference decides how many concurrent long-context sessions a given accelerator can hold. Shrinking the active fraction cuts per-token FLOPs; shrinking bytes-per-token KV cache cuts how many users you can batch. Both levers are being pulled hard at once, and you have to read both to predict throughput.

Operationally, a few things follow from very sparse MoE:

Load time dominates cold starts. You still stream the full total weight budget off disk, so a 2.8T model is slow to spin up even though it is fast per token once warm.
Expert placement is a scheduling problem. With hundreds of experts across many devices, all-to-all routing traffic between GPUs can become the bottleneck, not matrix multiply.
Batch composition affects utilization. If tokens in a batch route to disjoint experts, you activate more distinct experts than the per-token count suggests, eroding the efficiency win.

Does a smaller active fraction hurt quality?

Not obviously, which is the surprising part. On reported public benchmarks the trillion-scale sparse models are competitive at the frontier: Kimi K3 is cited near the top of permissive open-weight GPQA rankings, and GLM-5.2 is reported to lead an open-weights intelligence index and to post strong software-engineering benchmark numbers. Qwen's A3B line shows the trend at the small end, with roughly 3B active parameters claimed to reach agentic-coding scores that dense models of similar active size do not.

The honest caveat: benchmark leadership rotates monthly, these figures come from vendors and third-party trackers, and active-parameter counts can be reported differently for prefill versus decode (V4.1-Flash is a clear example, roughly 8B versus 16B). Treat any single ranking as a snapshot, and when you compare two MoE models, confirm whether the "active" number refers to the same phase.

How should I read a model card in 2026?

A short checklist that falls out of all this:

Find both numbers. Total parameters and active parameters per token. If a card only gives one, you cannot estimate either cost.
Separate your two budgets. Compute and latency track active params; memory, hosting and cold-start track total params.
Check the KV cache spec. Bytes per token times context length often dominates memory at long context, independent of expert sparsity.
Note prefill versus decode. Some models activate different expert counts in each phase; use the right one for your workload (long documents are prefill-heavy, chat is decode-heavy).
Mind the expert topology. Shared experts, routed experts and how many fire per token determine routing traffic and real utilization under batching.

FAQ

Is a model with fewer active parameters always cheaper to run?
Cheaper in per-token compute, yes. But not cheaper to host: you still need memory for all total parameters, and a very large total budget means slow loading and higher fixed infrastructure cost. The two budgets move independently.

Why keep weights you rarely use?
Because capacity and compute are being deliberately separated. A large expert bank stores specialized knowledge; the router pays only for the slice each token needs. You buy trillion-scale knowledge at mid-size per-token cost, at the price of trillion-scale memory.

What is the difference between prefill and decode active parameters?
Prefill processes the whole prompt in parallel; decode generates one token at a time. Some 2026 MoE models activate different expert counts in each phase, so a single "active parameters" figure can be ambiguous. Match the number to whether your workload is prompt-heavy or generation-heavy.

Does extreme sparsity reduce output quality?
Reported benchmarks suggest the top sparse models remain at the frontier, but rankings shift frequently and most numbers are vendor or tracker reported rather than independently reproduced. Verify on your own task before committing.

How do I estimate serving capacity for a long-context MoE?
Add two terms: resident weight memory (total params times bytes per param) plus KV cache (bytes per token times context length times concurrent sequences). At 1M-token context the KV term frequently dominates, so the per-token cache spec can matter more than the active fraction.

Is the sparsification trend likely to continue?
The direction is clear in 2026: total budgets climbing while active fractions fall toward a few percent, alongside aggressive KV-cache compression. Whether it holds depends on routing quality and interconnect bandwidth, both of which become the real bottlenecks as expert counts grow.

Hugging Face now has 48 official benchmarks. Here is what the map looks like

ai maya — Sun, 04 Oct 2026 15:15:38 +0000

Hugging Face marks a growing set of datasets as official benchmarks. Each one gets a leaderboard on its dataset page, filled automatically from the .eval_results files that model repositories publish. There are now 48 of them, from GPQA Diamond and MMLU-Pro to SWE-bench, Terminal-Bench, AIME 2026 and the Open ASR Leaderboard.

What has been missing is a way to see them together. Each leaderboard lives on its own page, so questions like "which fields do these benchmarks cover?" or "where do model makers actually submit?" are hard to answer.

A new Space, Hugging Face Official Benchmarks: Map and Participation, pulls every official leaderboard into one live page. It reads the public Hugging Face API in the browser on each visit, so the numbers below are a snapshot from October 4, 2026, and the page itself always shows the current state.

Here is what the map shows.

1. The benchmarks lean toward agents

Grouping the 48 benchmarks by field:

Field	Official benchmarks
Agents and terminal tasks	17
Science and knowledge	6
Documents and OCR	5
Coding	4
Vision and video	4
Law and finance	4
Math	3
Speech	2
Retrieval	2
Robotics	1

More than a third of the official set measures agents: terminal work, tool use, long-horizon tasks, browsing and multi-step workflows. That matches where the industry conversation has moved over the past year.

2. Submissions go somewhere else

The picture changes when you count leaderboard entries instead of benchmarks. Of roughly 1,020 entries:

Field	Entries
Science and knowledge	348
Agents and terminal tasks	197
Coding	158
Documents and OCR	89
Speech	72
Math	62
Vision and video	47
Law and finance	39
Retrieval	4
Robotics	4

Agents have the most benchmarks, but science and knowledge get the most submissions. Part of the reason is practical: GPQA and MMLU-Pro are cheap to run and widely reported, while agent benchmarks need harnesses, sandboxes and long runs. Many new agent benchmarks still have only a handful of entries.

3. Participation is very uneven

The median official benchmark has about 12 leaderboard entries.
14 of them have five entries or fewer.
The five busiest boards (MMLU-Pro about 141 entries, GPQA about 111, HLE about 89, SWE-bench Verified about 67, Open ASR about 62) hold 46% of all entries.

So a small group of benchmarks carries most of the comparison, and the long tail is thin. A #1 position on a board with 140 entries and one on a board with 3 entries are very different things, which is worth keeping in mind whenever a leaderboard result is quoted.

4. Who submits

About 410 models from 95 organizations appear across the official leaderboards. The most active publishers by number of entries:

Organization	Entries
Qwen	136
Z.ai (zai-org)	84
DeepSeek	65
Moonshot AI	65
NVIDIA	63
Ornith	48
OpenAI	47
MiniMax	36

Grouped by the country of the publishing organization, about 55% of entries come from China, 23% from the USA, 4% from Korea and 2% from France, with 13% from organizations without a clear public headquarters. These country labels are assigned by hand in the Space and are open for correction.

5. What else is on the page

#1 spots: for every benchmark, which model is ranked first and how many first places each organization holds, plus a heatmap of who leads which field.
Gaps: how far #1 is ahead of #2 on each board, in percent.
Boards entered vs. boards led: whether an organization wins by entering many boards or by winning most of the few it enters.
Every benchmark, sorted by number of entries, with #1 and #2.

How to read it

Two cautions apply to any view built on these leaderboards.

First, entries are published by model authors in their own repositories, and settings differ: sampling, majority voting, thinking budget and prompt format all vary between submissions. A leaderboard ranks the numbers authors report; it does not run the models itself.

Second, an organization is the namespace of the model repository. A fine-tune published under a different namespace counts for that namespace, not for the base model's creator.

With those in mind, the map is useful for a different kind of question than a single leaderboard answers: not "who is best at X", but "what is being measured, how crowded each measurement is, and who is showing up".

The Space: https://huggingface.co/spaces/quantid/huggingface-official-benchmark-leaderboards

A 180B model that tops seven Hugging Face leaderboards now runs on a laptop: no GPU needed

ai maya — Fri, 02 Oct 2026 05:40:54 +0000

Running a frontier-class model used to mean a rack of data-center GPUs. Not anymore.

VIDRAFT has released POCKET-Darwin-180B, a compressed build of Darwin-180B-RSI, the model that holds first place on seven Hugging Face official leaderboards (self-reported), that runs without a GPU. It is being released on both Hugging Face and ModelScope, Alibaba's AI model platform and China's largest; the download pages below open with the release.

Size: 111 GB (4-bit GGUF), down from 360 GB (BF16)
CPU only: a single server CPU (16 threads) generates 18.4–21.0 tokens/s, peak memory 78.8 GB
Laptop: RTX 5060 Laptop (8 GB VRAM) + 32 GB RAM: 4.17 tokens/s
Mini PC: a 128 GB-RAM mini PC holds the whole model in memory, no GPU required
Same accuracy: MMLU-Pro, 2,000 questions, paired comparison: 87.65% vs 87.65%

From an 8-GPU server to a laptop

	Darwin-180B-RSI (BF16)	POCKET-Darwin-180B
Model size	360 GB (131 files)	111 GB (4 files)
Hardware	4–8 NVIDIA B200, or 8× H100 (80 GB)	Laptop with 8 GB GPU + 32 GB RAM, 128 GB mini PC, CPU-only server, one DGX Spark
MMLU-Pro	87.65%	87.65%

An 8× H100 server is commonly estimated at around US$350K. A gaming laptop that runs POCKET costs roughly 1/250 of that.

Why a 180B model fits on a laptop

Darwin-180B is a Mixture-of-Experts model: of 512 experts, only 10 are used per token, so only about 3 billion of its 180 billion parameters are active at any moment. llama.cpp memory-maps the weights and reads the experts it needs directly from the SSD, so the full 111 GB never has to sit in RAM.

The compression itself is a graft: we took the widely used Unsloth UD-Q4_K_XL layout of the base model as a template and replaced only the 300 tensors that our self-improvement training changed (in Q8_0). Every other byte is identical to the base build, and all 300 replaced tensors passed read-back verification.

The model underneath: Model-level RSI

Darwin-180B-RSI is built on Qwen3.8-Flash-Next. It improves through Model-level Recursive Self-Improvement: the model solves verifiable problems, keeps only its own solutions that check out as correct, and trains on them. No human-written solutions or reasoning traces are used. Only the attention paths and shared experts are trained; the 512 routed experts and the router are left untouched.

The R3 round compressed here improved over the previous round on 1,000 held-out SuperGPQA questions never used for training or selection (+1.03 points, 95% CI [+0.05, +2.00]).

Leaderboard results of the original model (self-reported, majority voting):

Benchmark	Score
AIME 2026	100
HMMT Feb 2026	100
GPQA Diamond	94.44
MMLU-Pro	88.12
MMMU-Pro	79.48
LEXam (law)	68.94
LEXam-hard (law)	45.72

Verified as a research institution on ModelScope

VIDRAFT's organization on ModelScope (FINAL-Bench) carries an official Research Institution verification. Among the organizations we checked, verification is held by only a few, including Alibaba's Qwen, Shanghai AI Laboratory, Zhipu AI and OpenBMB, and VIDRAFT is the only Korean AI organization we found with it.

Who this is for

Organizations that cannot send data to an external cloud (defense, finance, public sector) can now run a top-tier model on an air-gapped server or a single mini PC.

Run it

Requires llama.cpp b11048 or newer.

# Laptop / desktop (8 GB+ VRAM, 32 GB+ RAM; experts streamed from SSD)
llama-server -m POCKET-Darwin-180B-UD-Q4_K_XL-00001-of-00004.gguf -ngl 999 --cpu-moe -fa on -c 8192 --jinja
# CPU only
llama-server -m POCKET-Darwin-180B-UD-Q4_K_XL-00001-of-00004.gguf -ngl 0 -t 16 -c 8192 --jinja --load-mode none

Recommended sampling: temperature 1.0, top-p 0.95, top-k 20. It is a reasoning model, so leave room for at least 2,048 output tokens.

Hugging Face: https://huggingface.co/FINAL-Bench/POCKET-Darwin-180B-GGUF
ModelScope: https://modelscope.cn/models/FINAL-Bench/POCKET-Darwin-180B-GGUF
Original model: https://huggingface.co/FINAL-Bench/Darwin-180B-RSI

A self-taught AI never trained on law just topped a Swiss law-exam benchmark

ai maya — Thu, 01 Oct 2026 02:48:48 +0000

Darwin-180B-RSI, an open-weight 180B model from the Korean startup VIDRAFT, is now #1 on LEXam and LEXam-hard, the two legal-reasoning leaderboards that Hugging Face lists as official benchmarks. The model was never trained on legal data.

With these two, the model holds first place on seven Hugging Face official leaderboards, the most of any organization on the Hub right now (Zhipu AI has 4; DeepSeek, Xiaomi and Moonshot AI have 3 each).

The benchmark: law-school exams, not trivia

LEXam was built by researchers at ETH Zurich, the University of Zurich and the Max Planck Institute, from 340 real law-school exams at Swiss universities, in German and English, with answers verified by legal experts. It doesn't test whether a model has memorized statutes. It tests whether the model can apply law to a set of facts and reason to a conclusion.

It is hard. The multiple-choice set has four options, so guessing gives 25 points, yet the strongest models on the leaderboard sit around 50. LEXam-hard is a subset of the open-ended questions that the strongest open models score lowest on.

The results

Leaderboard	Darwin-180B-RSI	Previous #1
LEXam (MCQ, 1,655 questions)	68.94	52.41 (DeepSeek-R1)
LEXam-hard (open-ended, 518 questions)	45.72	40.82 (Inkling)

Against the results the LEXam authors published themselves: GPT-5 62.65, Claude-4.5-Sonnet 58.01, Gemini-2.5-Pro 55.72. Darwin-180B-RSI scores 68.94 with a majority vote over 4 samples, and 60.54 with a single sample, which is still above Claude-4.5-Sonnet and Gemini-2.5-Pro.

How we measured: LEXam uses 4 samples per question and a majority vote, with a 32K-token thinking budget. LEXam-hard uses a single sample; the 60 answers that hit the 32K limit were regenerated with 120K. LEXam-hard answers were graded by DeepSeek-R1-0528, the judge named in the benchmark's official eval.yaml. Every protocol is on the model card.

How the model learned: model-level RSI

The interesting part is not the rank but how the model got there.

Darwin-180B-RSI uses model-level recursive self-improvement:

The model solves practice problems.
Each answer is checked automatically.
The model trains only on its own solutions that turned out correct.
The improved model becomes the solver for the next round.

The practice problems were math and science only, chosen because their answers can be checked automatically. No human-written solutions or reasoning traces were used. No legal text was in the training set.

So why does it do well on law? Our reading is that self-improvement on verifiable problems trains a habit of reasoning rather than knowledge: break the problem down, check each step, commit to a conclusion. That habit transfers to a domain the model never studied. We can't prove the transfer is caused by RSI alone, since we haven't measured the parent model on LEXam under the same protocol. One effect we did measure: the model reaches the same accuracy as its parent with about 11% shorter reasoning.

Model-level vs. harness-level RSI

Google recently released RRSI, which keeps the model frozen and lets an AI improve the prompts, tools and workflow around it. That is harness-level RSI. Darwin-180B-RSI is model-level: the weights change, so the gain ships inside the model file and works for anyone who downloads it, with no harness required. The two are complementary.

What's under the hood

Darwin evolves models instead of pretraining them from scratch. It diagnoses a parent model layer by layer and expert by expert, like an MRI, transplants the strongest parts from several models, and fuses them weighted by diagnostic trust. The Darwin family has 50+ official models and 400+ community derivatives (arXiv 2605.14386).
Darwin-180B-RSI changes only about 0.02% of the parent's parameters (attention paths and shared experts). All 512 routed experts, the router and the vision encoder are unchanged.
ZTC (Zero-Token Confidence) reads the model's internal state once, before generation, to estimate the probability that an answer will be correct.

All seven #1 results

Math: AIME 2026 (100), HMMT Feb 2026 (100)
Science: GPQA Diamond (94.44)
General knowledge: MMLU-Pro (88.12)
Visual reasoning: MMMU-Pro (79.48)
Law: LEXam (68.94), LEXam-hard (45.72)

Weights and evaluation settings: https://huggingface.co/FINAL-Bench/Darwin-180B-RSI

Four Kinds of Recursive Self-Improvement: What Exactly Is Improving Itself?

ai maya — Wed, 30 Sep 2026 06:52:15 +0000

Four Kinds of Recursive Self-Improvement: What Exactly Is Improving Itself?

Short answer: "Recursive self-improvement" (RSI) now covers at least four different things. The useful question is not whether a system improves itself, but what it improves: the harness around a model, the model's weights, the evaluator that scores it, or the research process that builds it. Each level ships a different artifact, fails in a different way, and needs a different kind of proof.

A 2026 survey of roughly 1,250 RSI papers (arXiv 2607.07663) organizes the field along two axes: what the system improves, and how closed the loop is. This post applies the first axis and turns it into a comparison you can use when reading the next "self-improving AI" claim.

The comparison table

	Harness-level RSI	Model-level RSI	Evaluator-level RSI	Research-level RSI
What changes	Prompts, tools, memory, control flow, agent code	The model's weights	The reward model / judge	Training methods, algorithms, experiments
Analogy	Rewriting an employee's manual	The employee gets smarter	The examiner gets sharper	The lab improves how it does research
Examples	Google's RRSI, Darwin Gödel Machine	TTRL, R-Zero, Self-Rewarding LMs, STaR, Darwin-RSI	Self-improving LLM judges and reward models	AlphaEvolve, autonomous post-training loops
What ships	A better configuration; the model is unchanged	A new model file	A better scorer	A new method or algorithm
Portability	The harness must travel with the model	The model alone carries the gain	The scorer must be used with it	The method must be re-applied
Main cost	Many calls to a strong external model (proposer, critic)	GPU training, once per round	Moderate	Very high
Typical failure	Overfits the evolve set (memorizes tasks)	Reinforces its own mistakes ("bad teacher")	Reward hacking: the scorer gets fooled	Hard to verify at all
Proof that convinces	Gains on benchmarks never used in selection	Gains on held-out benchmarks, paired tests	Agreement with independent human or ground-truth checks	Independent reproduction

1. Harness-level RSI: improve everything around the model

The model is frozen. An outer loop proposes edits to prompts, tools, memory and control flow, evaluates them, and keeps the winners.

Google's RRSI (Regularized RSI of Agent Harnesses, arXiv 2609.24972) is the clearest recent example. With a frozen policy it reports Terminal-Bench 2.1 going from 74.2 to 80.2, and gains of +2 to +5 points on benchmarks that never entered selection (SWE-bench Verified, GDPval, APEX-Agents, Frontier-Eng). With a weaker frozen model the in-distribution gain is larger (64.6 → 78.7).

The interesting part is not the loop but the regularizers that stop it from overfitting:

Noise-adjusted floor: a gain inside evaluation variance is not a gain.
Leakage critic: candidates are screened for benchmark-specific logic before evaluation.
Edit history: a falsified hypothesis is not proposed again.
Cost rule: extra inference tokens must be paid for by measured gain.
Pruning: components that stop helping are removed.

These rules generalize well beyond harnesses. Any self-improvement loop without them will eventually report gains that are noise or leakage.

2. Model-level RSI: the model itself improves

Here the weights change. The model solves problems, a check decides which of its own solutions were good, and it trains on those. Repeat.

The key design question is where the training signal comes from. In the label-free family (TTRL, R-Zero and our own Darwin-RSI), the model learns only from its own solutions; no human-written solutions or reasoning traces are used, and correctness is checked automatically — for example by agreement across the model's samples or by executing code.

Results in this family are real but modest per round:

System	Setup	Reported gain
R-Zero (ICLR 2026)	Qwen3-4B-Base, challenger/solver co-evolution	+6.49 math, +7.54 general reasoning
Darwin-27B-RSI	27B, two rounds of self-improvement	GPQA Diamond 72.85 → 78.09 (single sample), 79.80 → 83.59 (majority@16), same protocol for both

The characteristic failure is the bad teacher: if the check that selects "good" solutions is weak, the model trains on its own errors and gets confidently worse. The quality of the check is the ceiling of the whole loop.

Why this level matters in practice: the gain ships inside the model file. Anyone who downloads the model gets the improvement without the loop, the harness, or the checker.

3. Evaluator-level RSI: improve the judge

Every other loop depends on a scorer. If the scorer improves, every loop that uses it improves too. If the scorer is fooled, every loop optimizes the wrong thing.

This level is under-discussed and under-measured. The practical test is simple: does the improved judge agree more with independent ground truth on items it never trained on? If you cannot answer that, you have a reward-hacking risk, not an evaluator improvement.

4. Research-level RSI: improve how improvement is done

The system modifies the research process itself: search over algorithms, training recipes, or entire post-training pipelines. AlphaEvolve is the best-known example of evaluator-guided evolution applied to algorithms and AI training infrastructure. Autonomous post-training loops that run for weeks without a human are now reported on public leaderboards.

This is the most ambitious level and the hardest to verify. Independent reproduction is the only convincing evidence.

How the levels combine

The levels are complementary, not competing. A harness-level loop can run on top of a model-level RSI model. A better evaluator strengthens both. A reasonable roadmap for a lab is:

Model-level RSI to get a stronger base that ships as a file.
Harness-level RSI on top of it for task-specific gains.
Invest in the evaluator, because it is the ceiling of both.

A checklist for reading any "self-improving AI" claim

Which level? Harness, model, evaluator, or research process.
Where does the training signal come from? Human-written solutions, public answer keys, or the model's own solutions with an automatic check.
Is the gain outside the noise band? Look for paired tests or confidence intervals, not a single delta.
Was the evaluation set used for selection? If yes, discount the number; look for held-out results.
What ships? A config, a model file, a scorer, or a method.
What does it cost per round? Tokens for harness loops, GPU hours for model loops.

FAQ

What is recursive self-improvement (RSI) in AI?
A loop in which an AI system improves some part of itself and the improved version runs the next round. The part being improved can be the harness, the model weights, the evaluator, or the research process.

What is the difference between harness-level and model-level RSI?
Harness-level RSI changes the prompts, tools and workflow around a frozen model. Model-level RSI changes the model's weights, so the improvement ships inside the model file.

Is Google's RRSI model-level RSI?
No. RRSI keeps the model frozen and evolves the agent harness. It is harness-level RSI.

Can a model improve without human-written solutions?
Yes. Label-free methods train only on the model's own solutions and use an automatic correctness check, such as agreement across samples or code execution. The check is the limiting factor.

How do you know a self-improvement gain is real?
Measure on benchmarks that were never used for selection, use paired statistical tests, and treat any gain inside the evaluation noise band as zero.

References: RRSI code · RRSI paper · RSI survey, arXiv 2607.07663 · R-Zero · Darwin-27B-RSI model card

Reading an official Hugging Face leaderboard: protocols, majority vote and reproducibility

ai maya — Tue, 29 Sep 2026 12:20:42 +0000

TL;DR — Hugging Face now marks selected benchmark datasets as official and builds their leaderboards from .eval_results/*.yaml files in model repositories. Scores are self-reported, so the evaluation protocol is what makes a number comparable. This post walks through the protocol behind Darwin-180B-RSI, which ranks #1 on five of these boards.

How official leaderboards work

A benchmark dataset carries the benchmark:official tag and an eval.yaml defining its tasks.
A model publisher adds a file such as .eval_results/gpqa_diamond.yaml to their model repository with the dataset id, task id, value and date.
The dataset page aggregates those files into a ranked leaderboard.

You can query any board directly:

curl https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard

Because entries are self-reported, the leaderboard shows the value but not always how it was obtained. The notes field and the model card are where the protocol lives.

The protocol behind five #1 scores

Common settings for every run of Darwin-180B-RSI:

Setting	Value
Thinking budget	131,072 tokens per sample
Sampling	temperature 1.0 · top_p 0.95 · top_k 20
Precision	bf16
Engine	vLLM, tensor parallel, expert parallel

Per-benchmark settings and results:

Benchmark	Samples	Reported score	Value
AIME 2026	16	majority vote (mean 98.75)	100.0
HMMT Feb 2026	16	majority vote (mean 96.59)	100.0
GPQA Diamond	up to 16	majority vote	94.44
MMLU-Pro	1	single sample	88.12
MMMU-Pro (vision)	3	majority vote	79.48

Three things to check before comparing numbers

Samples and voting. A majority-of-16 score is a system score and is not directly comparable with a single-sample score. Look for both.
Thinking budget. Reasoning models lose points to truncation when the budget is small; a cut-off answer is not a wrong answer.
Contamination. Training data should be filtered against every reported test set. VIDRAFT reports an 8-gram overlap filter against all benchmarks it lists.

Reproducing

All five benchmarks are public datasets and the weights are open at huggingface.co/FINAL-Bench/Darwin-180B-RSI. Serving requires 8 × B200-class GPUs (4 at minimum) in bf16:

vllm serve FINAL-Bench/Darwin-180B-RSI \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --max-model-len 135168 --trust-remote-code

Darwin: evolving a 180B parent model by changing 0.02% of it

ai maya — Tue, 29 Sep 2026 12:20:05 +0000

TL;DR — Darwin-180B-RSI keeps its parent's 512 routed experts, router and vision encoder intact and modifies about 0.02% of the parameters. VIDRAFT's Darwin framework treats an open model as a "parent", diagnoses its weak paths, and strengthens only those — here combined with recursive self-improvement (RSI) on verified answers.

The parent

The father model is Qwen3.8-Flash-Next, a 180B mixture-of-experts vision-language model:


Layers / hidden size	48 / 2,560
Attention	36 linear-attention + 12 full-attention layers
Experts	512 routed (10 active per token) + shared expert
Context	262,144 tokens

What Darwin changed

According to the model card, Darwin updated three wiring paths and preserved the rest:

🔹 Full-attention layers (12) — the long-range path across the whole context
🔹 Linear-attention layers (36) — the fast inference path
🔹 Shared expert (48 layers) — the common path every token passes through
🔒 Preserved: all 512 routed experts, the router and the vision encoder

That is roughly 30 million trainable parameters out of 177 billion. Because the experts — where most factual knowledge is stored in an MoE model — are untouched, the parent's knowledge is retained.

Recursive self-improvement on verified answers

The RSI loop is simple to state:

The model solves practice problems that do not overlap with any reported benchmark (8-gram overlap filter).
Its answers are checked against verifiable references.
Only the correct reasoning is kept, and the model is retrained on it.
The improved model becomes the next solver.

Two design choices matter. Unverified solutions are never learned, so errors are not reinforced. And among correct solutions, shorter ones are preferred — which is why VIDRAFT reports the same accuracy with 11% shorter reasoning than the parent (MMLU-Pro: 4,320 → 3,833 tokens per answer).

How Darwin compares with evolutionary model merging

The best-known prior work in this area is Sakana AI's evolutionary model merge (Nature Machine Intelligence, 2025), which searches merge recipes over hundreds of generations and demonstrated 7–10B models. Darwin takes a diagnosis-first approach — measuring layer- and expert-level strengths before transplanting or tuning — which VIDRAFT says reduces search cost and has scaled from 4B to 397B models.

On Hugging Face, the Darwin family lists 50+ official models and 400+ community derivatives. The framework is described in Darwin Family (arXiv:2605.14386).

Zero-Token Confidence

VIDRAFT pairs RSI with ZTC, a lightweight readout of the model's final-layer hidden state that estimates, before any token is generated, the probability that the upcoming answer is correct. In an RSI loop, low-confidence areas point to what the model should learn next; in deployment, the same signal can gate tool calls or escalate to a human. The company's earlier Darwin-397B-ZTC ships such a readout.

Perfect scores on AIME 2026 and HMMT 2026: what an open 180B model got right, and why thinking budget mattered

ai maya — Tue, 29 Sep 2026 12:19:28 +0000

TL;DR — Darwin-180B-RSI, an open-weight model from Korean startup VIDRAFT, reports 100% on AIME 2026 (30/30) and 100% on HMMT February 2026 (33/33) under 16-sample majority vote with a 131K-token thinking budget. They are the first perfect scores on these two Hugging Face official leaderboards.

The numbers

Benchmark	Majority of 16	Mean over 16 samples	Single sample	Previous #1
AIME 2026	100.0	98.75	96.67	97.1
HMMT Feb 2026	100.0	96.59	93.94	92.7

Even the mean accuracy over 16 samples (98.75 on AIME, 96.59 on HMMT) is above the previous top entries on both boards.

About the two competitions

AIME (American Invitational Mathematics Examination) is the qualifier toward the USA Mathematical Olympiad. Every answer is an integer from 0 to 999, so guessing rarely works.
HMMT (Harvard-MIT Mathematics Tournament) is one of the most difficult high-school competitions, attended by national-olympiad-level students. Answers include fractions and radicals, so grading requires symbolic equivalence (for example, 2√3 = √12).

The finding: misses were truncations, not wrong math

VIDRAFT's engineers first measured the parent model with a 65K-token budget. The model missed two AIME 2026 problems, and on inspection every miss was a truncated solution — the reasoning ran out of tokens before reaching an answer. Solutions that finished were correct.

Re-running those problems with a 131K-token budget produced 8/8 correct answers. With the budget raised for all problems, the 16-sample run scored 30/30.

The practical lesson for anyone evaluating reasoning models:

Truncation is not an error. Count cut-off answers separately; otherwise a budget limit looks like a capability limit.
Budget is part of the protocol. A score without its thinking budget is not comparable.
Grade symbolically. String matching undercounts fractions and radicals; a math-equivalence checker is required.

How to read "majority of 16"

Majority vote is a system score: the model answers each question 16 times and the most frequent answer is graded. VIDRAFT reports the majority score and the mean single-sample accuracy side by side on the model card and in the leaderboard submission notes, so readers can compare with entries that use a single sample.

Try it

Model: huggingface.co/FINAL-Bench/Darwin-180B-RSI — the model card contains the full evaluation protocol.

JEV vs ZTC: Statistically Tied on Accuracy, 1.4 Points Apart Where It Counts

ai maya — Sun, 20 Sep 2026 09:50:48 +0000

Short answer: on accuracy, JEV (0.7350 AUC) and ZTC (0.7364) are statistically tied — the 0.0014
gap has a 95% interval of −0.019 to +0.032. They separate on two other axes: deployment shape (API
only vs open weights) and downstream agent accuracy, where ZTC gained 1.34 points and JEV lost 0.07
at the same retry budget.

Both numbers come from the same test set: 2,018 items, 508 of them incorrect, identical labels,
published grading code —
Typed Decision Leaderboard.

First, what these two things are

JEV is TypeSafe AI's answer verifier, the product that established this category under the name
System One models. ZTC (Zero-Token Confidence) is VIDRAFT's, shipping as two open-weight
models.

Both are typed decision models: they return a structured verdict — boolean, choice, or score —
from a single forward pass, emitting zero generated tokens. That is what makes both of them one
to three orders of magnitude cheaper than asking a frontier model "is this answer right?"

So the choice between them is not a choice of architecture. It is a choice of accuracy, shape, and
behaviour under load.

Axis 1 — accuracy: a tie, and it should be reported as one

System	AUC	95% interval vs. the other
ZTC (397B)	0.7364	—
JEV	0.7350	−0.019 to +0.032
ZTC-Judge-27B	0.7282	−0.034 to +0.020

The interval contains zero in both comparisons. Under the rule that a gap indistinguishable from
noise produces no rank, neither system can be called more accurate than the other on this set.

For context, the same table's reference row — a logistic model over answer length and formatting —
scores 0.7036, and eight of the thirteen measured systems fall below it. Both JEV and ZTC are
comfortably above that bar. Most of the category is not.

Axis 2 — deployment shape, where they genuinely differ

This is the axis that usually decides the purchase, and it is not close.

	JEV	ZTC
Weights published	no	yes (Apache-2.0)
Runs inside a private network	no	yes
Adaptable to your own data	no	yes
Hosted API	yes	yes
Cost	$0.024 / 1,000 calls	your own compute
Generated tokens	0	0

If your data cannot leave your network, the accuracy tie is irrelevant — only one of these is
installable. If you would rather not operate a GPU, the same logic runs the other way, and $0.024
per thousand calls is cheap enough that "just use the API" is a defensible answer.

For comparison, asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls — about
23× more — and scores 0.7148, below both.

Axis 3 — per domain, the ranking inverts

The headline is an average of five domains. They disagree.

Domain	n	JEV	ZTC 397B	ZTC 27B
Professional exams (law/math/bio)	400	0.7981	0.8660	0.8462
Biology & medicine	917	0.7185	0.7433	0.7154
Commonsense & multi-step	278	0.6006	0.6072	0.6172
Disaster & safety guidance	225	0.7620	0.7319	0.6961
Scientific reasoning	198	0.8421	0.6287	0.7410

JEV wins scientific reasoning outright at 0.8421 — 0.21 ahead of the overall leader. If that is
your workload, the overall ranking is the wrong table to read, and the answer to "which one" is JEV.

Two other things in that table are worth carrying away regardless of vendor:

Bigger is not better. Inside one family, the 397B scores 0.11 below the 27B on scientific
reasoning. A separate measurement puts a 180B-class model at 0.7146, under a 4B at 0.7284.
Verification quality tracks representation geometry, not parameter count.

Commonsense is unsolved by everyone. Every system sits near 0.60 on multi-step commonsense. No
product on this board fixes that domain, and knowing it is more useful than any ranking.

Axis 4 — as an agent gate, they are not tied at all

AUC answers "which separates right from wrong answers best." It does not answer "which helps my
agent." Those turned out to be different questions.

Setup. For each item: score the answer with a verifier, send the lowest-scoring 20% to a
stronger model to be re-answered, keep the rest. All arms draw from one shared pool of re-answers,
so no arm gets luckier retries. Ties broken randomly, averaged over 200 seeds.

Gate	Final accuracy	vs. no gate (74.83%)
ZTC	76.16%	+1.34 pp
JEV	74.76%	−0.07 pp
Random	74.58%	−0.25 pp

A 0.0014 AUC gap became a 1.4-point swing in end-to-end accuracy. At this budget, JEV's gate
performed about the same as no gate at all.

The mechanism, which is the transferable part

Re-answering is double-edged:

wrong answers sent back  →  38% get fixed
right answers sent back  →  30% get broken

So a gate is not rewarded for catching errors. It is rewarded for not disturbing what was already
right. Precision, not recall — the opposite of how most people reason about a safety net.

At a 20% budget both systems routed 403 items. JEV's 403 contained 216 that were already
correct; ZTC's contained 195. Twenty-one items of difference in what got sent back, and one
arm spent its whole retry budget repairing about as much as it damaged.

None of that is visible in an AUC column, and AUC is what every vendor publishes.

Scope, stated plainly: one escalation target, one item set. The direction should transfer; the
magnitude is theirs, not yours. Run the same protocol on your own data — the grading code is
published for exactly that.

How to choose

If this is true for you	Pick
Data cannot leave your network	ZTC (only one with published weights)
You want to fine-tune the verifier on your own data	ZTC
Your workload is scientific reasoning	JEV (0.8421, clear win)
You do not want to operate a GPU	JEV ($0.024 / 1,000)
You are gating agent retries	Measure both on your own set — this is where they diverge
You are choosing on the leaderboard number alone	Don't. The gap is noise.

Frequently asked questions

Is ZTC better than JEV?

Not on accuracy. The two are statistically tied on the shared test set — 0.7364 versus 0.7350, with
a 95% interval spanning zero. They differ on deployment shape (ZTC publishes weights, JEV does not),
on per-domain strengths (JEV wins scientific reasoning at 0.8421), and on downstream agent accuracy
at a 20% retry budget (+1.34 pp versus −0.07 pp).

Is there an open-source alternative to JEV?

Yes. ZTC ships two Apache-2.0 models with published weights —
ZTC-Judge-27B (verification only) and
Darwin-397B-ZTC (a generation model with the
confidence probe on board). Both score above the 0.7036 surface baseline. Other open entries score
lower: open-jev 4B at 0.6844, Laya at 0.4796 / 0.5144.

How much does JEV cost compared with running a verifier yourself?

JEV is about $0.024 per 1,000 calls. A self-hosted verifier costs only compute you already own.
Asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls and scores lower than
either.

What AUC should I expect from a good answer verifier?

Use 0.7036 as the bar, not 0.5. That is the score from answer length and formatting alone, and
eight of thirteen measured systems fall below it. JEV and ZTC are the only two systems above 0.73.

Can I test JEV and ZTC on my own question?

Yes — the Verifier Playground runs
ZTC, JEV and Laya on the same input, in the browser, no signup. One case cannot rank them, but it
shows you how each behaves on your phrasing.

Why did the more accurate verifier perform worse as a gate?

Because gating rewards precision rather than recall. Sending a correct answer back to be re-answered
breaks it about 30% of the time, so routing a few more already-correct items cancels out the errors
you caught. JEV routed 216 already-correct items; ZTC routed 195.

They Put 7 Attention Mechanisms on a Latin Square. Then Removed Them One by One.

ai maya — Fri, 18 Sep 2026 03:48:33 +0000

Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight
identical blocks, differing only in learned weights.

Nobody tested that. It is a convention, not a conclusion.

A paper out of VIDRAFT AI Research (arXiv:2609.20269, CC BY 4.0)
tests it, and the interesting part is not the headline. The headline is placement is free,
composition is not. The interesting part is what happens when you read the ablation table.

The confound that makes this hard to measure

If you build a stack with several different sequence mixers and it beats a uniform stack, you
cannot say why. Was it which mechanisms you used, or where you put them? Those two are
tangled in every heterogeneous architecture paper, because any concrete stack is one particular
arrangement of one particular set.

The paper's move is to remove the second variable by construction rather than by search.

A Latin square is an N×N grid where each symbol appears exactly once in every row and every
column. Sudoku is a Latin square with extra constraints. Put seven mechanisms on a 7×7 square and
read it out across 49 layers, and every mechanism is guaranteed to be spread evenly through depth.

layer  0..6   A B C D E F G
layer  7..13  B C D E F G A
layer 14..20  C D E F G A B
       ...

No mechanism can cluster. Not "did not cluster in this run" — cannot, structurally. That is the
whole point: you get balance without running an architecture search to find it.

The flagship is Aether-7B-5Attn — 6.59B parameters MoE, ~2.98B active, 49 layers, seven mixers
on a 7×7 square. Trained on 16× B200 (2-node FSDP), 162,000 steps, 144.2B tokens, ~11,700
B200-hours in the final stage.

The actual experiment

You cannot ablate a 6.59B flagship eight ways with eight seeds each. So the paper builds a
parameter-matched proxy: four mechanisms on a 4×4 square over sixteen layers, 700.9M parameters,
eight seeds per arm.

Four arms, identical parameter counts, differing only in arrangement:

Arm	Mean CE	SD	Δ vs latin	2·pooled SD	Verdict
`latin`	5.28639	0.00867	—	—	reference
`periodic`	5.29484	0.01611	+0.16%	0.02588	null (within noise)
`block`	5.31754	0.01520	+0.59%	0.02475	real (2.5× pooled SD)
`homo_F`	—	—	+1.68%	—	real

Read it as a gradient of how tightly each mechanism is confined in depth:

Distributed, any order (latin vs periodic) → indistinguishable. Shuffling a balanced schedule buys you nothing.
Confined to contiguous bands (block) → you pay 0.59%.
One mechanism everywhere (homo_F) → you pay 1.68%.

So the first practical result: stop burning GPU hours searching for the right layer order. If
your schedule is balanced and distributed, the permutation is noise. That is a real cost saving,
and it is the kind of null result that rarely gets published.

Now the part worth your attention

The paper then removes one mechanism at a time from the four-mechanism stack, re-matching parameters
and cycling the remaining three through depth:

Removed	Mean CE	Δ vs latin	Verdict
sliding window	5.28495	−0.03%	null
differential	5.28450	−0.04%	null
full attention	5.29615	+0.18%	null
Mamba-2 / SSM	5.39940	+2.14%	real
(baseline)	5.28639	—	8 seeds

Three of the four mechanisms can be deleted with no measurable effect. Two of them are slightly
negative — the model is nominally better without them, well inside noise.

One is load-bearing.

Look at which one. Sliding window, differential, and full attention are all members of the
attention family — variations on the same operator. Mamba-2 is a state-space model, a
structurally different way to mix along the sequence.

The honest reading of this table is narrower and more useful than "heterogeneity helps":

What pays is having at least one mechanism from a different family. Seven flavors of attention
is still, functionally, a homogeneous stack.

That is a concrete design rule you can apply tomorrow, and it explains why the homogeneous arm
(1.68%) and the no-SSM arm (2.14%) land in the same neighborhood — from the loss's point of view,
removing the SSM is collapsing toward homogeneity.

Does it survive scale?

The obvious objection: 700.9M with 1,500 steps is a pilot, not a shipped model.

The paper re-runs the three decisive arms — latin, homo_F, no_M — at 1.514B parameters, a
2.16× increase, everything else held constant, three seeds each. Both penalties get bigger:

	700.9M	1.514B
homogeneous stack	+1.68%	+2.63%
remove SSM family	+2.14%	+3.20%

Penalties that grow with scale are the good kind of evidence. A finding that shrinks as you scale is
usually a small-model artifact; one that widens is more likely structural.

Note what was not re-run: the placement axis (periodic, block) stays at 700.9M, and neither
axis was tested at N=7 or at flagship size. The paper says so in its limitations rather than leaving
you to notice.

Two methodology details worth stealing

1. A pre-registered decision rule.

The threshold is |Δ| > 2 · pooled_SD, fixed on 2026-07-22 — before the final seeds ran. Section
6.4 explains why that matters: an early two-seed run showed latin and periodic cleanly
separated, which would have been a placement effect, which is the more marketable story. Four seeds
dissolved it. Eight confirmed the dissolution, at Δ 0.008 — a third of the threshold.

The rule that killed the authors' preferred result is the same rule that certifies block and
homo_F as real. As the paper puts it: a rule that only ever confirms what you hoped for is not a
rule.

If you run architecture experiments, the two habits here are cheap and they work: no claim below
four seeds, and fix your threshold before the last run.

2. A positive control in the safety audit.

Mixing operators along the time axis is exactly where causality violations hide — a stack that leaks
future information will show beautiful loss curves for the wrong reason. All 49 layers pass a
negative control, and, on the same loaded checkpoint, a positive control: 16 of 16 deliberately
injected faults localized exactly, across NSA, hybrid, and linear-attention layers.

This is the right shape for any correctness gate. A clean pass proves nothing unless you have also
shown the detector fires when something is actually broken. Most audits skip the second half.

What is released

Weights, training-data recipes, training code, logs, and architecture source. There is also
Aether-6B-11Attn-base — eleven mechanisms (attention, Mamba-2, Hyena, GDN, MLA among them) on an
11×11 square over 121 layers — shipped as-is, a mid-training artifact whose only job is to show
the construction is not specific to N=7. Whether the composition findings hold at N=11 is explicitly
left open.

The takeaway for practitioners

Placement search on a balanced schedule is probably wasted compute. 0.16%, inside noise.
Family diversity is the lever, not mechanism count. Three attention variants were free to delete; the one SSM was not.
A Latin square is a cheap way to get guaranteed balance without searching for it. The construction makes the failure mode unreachable rather than merely unobserved.
Pre-register your threshold and run four seeds minimum. This paper's most valuable page is the one where the method deleted the result the authors wanted.

Paper: arXiv:2609.20269 — Placement Is Free, Composition Is
Not: The Latin Square as a Provably-Balanced Construction for Heterogeneous Sequence-Mixer Stacks

DEV Community: ai maya

Decision AI Is Becoming Its Own Category, and the Metrics Are Not Ready

TL;DR

From writing to judging

Why this is a category, not a feature

The metrics are not ready

What to watch

FAQ

Closing

When the attacker is an agent: a defender's field guide to autonomous AI intrusions in 2026

TL;DR

What actually changed in 2026

How does autonomous attack map to the OWASP Top 10?

A defender's control checklist

1. Constrain agency before you constrain anything else

2. Treat every tool call as an untrusted request

3. Separate data from instructions

4. Log at the action level, not the request level

5. Rate limit for machines, not humans

6. Rehearse detection at machine speed

7. Lock down the pipeline

The regulatory signal is already here

Frequently asked questions

Further reading

The great sparsification: 2026 open-weight MoE models now activate 1 to 9 percent of their weights

TL;DR

Why did open models split the parameter count in two?

What does a lower active fraction actually buy?

How does sparsity change serving and the KV cache?

Does a smaller active fraction hurt quality?

How should I read a model card in 2026?

FAQ

Further reading

Hugging Face now has 48 official benchmarks. Here is what the map looks like

1. The benchmarks lean toward agents

2. Submissions go somewhere else

3. Participation is very uneven

4. Who submits

5. What else is on the page

How to read it

A 180B model that tops seven Hugging Face leaderboards now runs on a laptop: no GPU needed

From an 8-GPU server to a laptop

Why a 180B model fits on a laptop

The model underneath: Model-level RSI

Verified as a research institution on ModelScope

Who this is for

Run it

A self-taught AI never trained on law just topped a Swiss law-exam benchmark

The benchmark: law-school exams, not trivia

The results

How the model learned: model-level RSI

Model-level vs. harness-level RSI

What's under the hood

All seven #1 results

Four Kinds of Recursive Self-Improvement: What Exactly Is Improving Itself?

Four Kinds of Recursive Self-Improvement: What Exactly Is Improving Itself?

The comparison table

1. Harness-level RSI: improve everything around the model

2. Model-level RSI: the model itself improves

3. Evaluator-level RSI: improve the judge

4. Research-level RSI: improve how improvement is done

How the levels combine

A checklist for reading any "self-improving AI" claim

FAQ

Reading an official Hugging Face leaderboard: protocols, majority vote and reproducibility

How official leaderboards work

The protocol behind five #1 scores

Three things to check before comparing numbers

Reproducing

Darwin: evolving a 180B parent model by changing 0.02% of it

The parent

What Darwin changed

Recursive self-improvement on verified answers

How Darwin compares with evolutionary model merging

Zero-Token Confidence

Links

Perfect scores on AIME 2026 and HMMT 2026: what an open 180B model got right, and why thinking budget mattered

The numbers

About the two competitions

The finding: misses were truncations, not wrong math