DEV Community: AI OpenFree

VIDRAFT's AX-RAY AI Safety Diagnostic Platform Earns Alibaba ModelScope "Research Institution" Certification — Here's What Engineers Need to Know

AI OpenFree — Fri, 02 Oct 2026 07:01:22 +0000

VIDRAFT's AX-RAY AI Safety Diagnostic Platform Earns Alibaba ModelScope "Research Institution" Certification — Here's What Engineers Need to Know

TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has become the first Korean AI organization to receive official "Research Institution" certification on Alibaba's ModelScope platform, joining a short list that includes Qwen, Shanghai AI Lab, and Z.ai (Zhipu AI). The company is now pushing AX-RAY — its structured AI safety diagnostic framework covering 12 categories and 117 risk items — into the Chinese AI ecosystem. If you're building or auditing LLMs for regulatory compliance, AX-RAY's public evaluation reports on Hugging Face are worth your attention.

What it is

ModelScope is a large-scale open-source AI model community co-founded by Alibaba and the China Computer Federation (CCF). It hosts over 140,000 models and serves more than 20 million users, making it China's largest AI distribution platform. While major organizations including Google, Meta, and DeepSeek maintain presences on the platform, none of them currently hold the formal "Research Institution" certification badge — a distinction VIDRAFT has now secured roughly seven months after opening its ModelScope organization page (FINAL-Bench) in February 2026.

AX-RAY is VIDRAFT's AI safety diagnostic platform. It evaluates AI models across:

12 categories of safety concern
117 risk items aligned with nine global regulatory frameworks, including:
- EU AI Act
- GDPR
- U.S. NIST AI Risk Management Framework (AI RMF)
- ISO/IEC 42001
- Domestic Korean statutes

Models receive a letter grade from A through F based on their evaluated risk profile. Diagnostic results and detailed reports are published publicly on Hugging Face.

VIDRAFT's ModelScope organization currently hosts 91 public models and 9 datasets, including:

Darwin — a Korean-language-specialized LLM
POCKET — a lightweight on-device model
AX-RAY datasets — the AI safety evaluation benchmark data

How it works

AX-RAY operates as a structured audit framework rather than a single benchmark task. At a conceptual level:

Risk taxonomy construction: The 117 risk items are derived from a cross-mapping of nine international regulatory and standards frameworks, enabling the same diagnostic run to generate compliance-relevant documentation for multiple jurisdictions simultaneously.
Multi-category evaluation: Each model under test is probed across all 12 safety categories. This appears to cover areas including AI agent behavior, harmful content generation, and small-model-specific failure modes — based on the public diagnostic results VIDRAFT has shared.
Graded output: The A–F grading system produces a human-readable safety score that can be attached to regulatory filing documentation, making it practically useful for teams preparing AI Act conformity assessments or NIST AI RMF profiles.

VIDRAFT is currently participating in a security-specialized AI foundation model development project commissioned by South Korea's Ministry of Science and ICT (MSIT) and the National IT Industry Promotion Agency (NIPA), as part of the Naver Cloud consortium.

Benchmarks & results

VIDRAFT has published results from an AX-RAY diagnostic run covering 40 publicly available AI models (domestic and international). Key findings:

45% of models evaluated (18 out of 40) received the lowest possible grade (F).
In the "AI Agent Safety" category, 23 out of 25 models (92%) showed detected risk signals.
All 12 small models with fewer than 4 billion parameters that were evaluated received a risk determination — a 100% failure rate in that cohort.

These results suggest that agent-mode safety and small-model safety are currently the weakest points in the broader open-model ecosystem, at least under AX-RAY's evaluation criteria.

On Hugging Face, VIDRAFT's 82 public models and datasets accumulated 1.01 million downloads in the most recent 30-day window at time of reporting.

How to try it

VIDRAFT's models, datasets, and AX-RAY diagnostic reports are publicly accessible on both Hugging Face and ModelScope:

Hugging Face — browse VIDRAFT's published models and datasets (including AX-RAY evaluation reports):

https://huggingface.co/FINAL-Bench

ModelScope — access the certified FINAL-Bench organization page on ModelScope:

https://modelscope.cn/organization/FINAL-Bench

You can use the Hugging Face CLI to explore available assets:

pip install huggingface_hub
huggingface-cli scan-cache

No private API access, waitlist, or paid tier is mentioned for the public models and benchmark reports. The AX-RAY diagnostic platform (as a service for evaluating your own models) has not yet been described as publicly self-serve — check the Hugging Face organization page for current availability.

FAQ

Q: How does AX-RAY differ from existing safety benchmarks like MT-Bench or HELM?
A: AX-RAY is explicitly designed for regulatory compliance output, not just relative model ranking. Its 117 risk items are cross-referenced against nine specific legal and standards frameworks, so a diagnostic run can directly support documentation for the EU AI Act, NIST AI RMF, and ISO/IEC 42001 in a single pass. Most academic benchmarks don't map to legal instruments in this way.

Q: Why does ModelScope certification matter for developers outside China?
A: ModelScope's 20M+ user base and its co-governance by the China Computer Federation make it a primary distribution channel for AI models in Chinese research and industry contexts. For teams shipping models that need to reach Chinese enterprise or academic users, having a verified "Research Institution" presence there carries similar trust signaling to a verified organization on Hugging Face. It also opens distribution to users who may not access Hugging Face directly.

Q: Are the AX-RAY evaluation datasets themselves open?
A: Yes — per the source, AX-RAY datasets are listed among the 9 datasets registered in VIDRAFT's ModelScope organization and are part of the 82 assets publicly available on Hugging Face.

Originally reported by AI타임스 (2026-10-02) — source article.

Darwin-180B-RSI: How VIDRAFT's Recursive Self-Improvement Model Topped 7 Hugging Face Leaderboards — Without Legal Training Data

AI OpenFree — Thu, 01 Oct 2026 11:01:38 +0000

Darwin-180B-RSI: How VIDRAFT's Recursive Self-Improvement Model Topped 7 Hugging Face Leaderboards — Without Legal Training Data

TL;DR: VIDRAFT's Darwin-180B-RSI is a 180-billion-parameter reasoning model trained via a Recursive Self-Improvement (RSI) framework that updates model weights directly — no human-annotated chain-of-thought required. It just claimed #1 on both the LEXam and LEXam-hard Hugging Face-certified legal reasoning benchmarks, despite never being trained on legal domain data. Its checkpoint is publicly available on Hugging Face for external reproduction.

What it is

Darwin-180B-RSI is VIDRAFT's self-improving large language model, built around a model-level Recursive Self-Improvement (RSI) architecture. Key facts from the announcement:

180 billion parameters in size
Achieves top scores across 7 Hugging Face-certified leaderboard categories — the most of any single AI organization across all 47 officially certified Hugging Face benchmark leaderboards at time of publication
Covers math, science, general knowledge, visual reasoning, and now legal reasoning (LEXam, LEXam-hard) — the last two domains added without any legal-specific training data
Outperforms models up to 685 billion parameters (e.g., DeepSeek-R1) on legal benchmarks despite being less than a third of the parameter count
Model checkpoint and evaluation environment are open-sourced on Hugging Face for independent verification

How it works

The core design principle is Recursive Self-Improvement at the weight level — a conceptually distinct approach from system-level self-improvement (like prompt tuning or tool-orchestration loops that leave base weights unchanged).

Here's the high-level loop:

Autonomous problem solving: The model attempts problems from domains where correctness can be verified mechanically — primarily mathematics and science.
Automated verification: A rule-based or formal checker determines whether each candidate solution is correct. No human-written intermediate reasoning traces are injected.
Selective retraining: Only the reasoning paths that led to verified correct answers are used to update the model's weights. Incorrect paths are discarded.
Iterative bootstrapping: The improved model becomes the compute substrate for the next improvement round, creating a recursive loop.

The practical result is that reasoning capabilities generalized from math and science transfer to unseen domains like legal analysis — the model isn't memorizing legal statutes, it's applying structured multi-step logical inference. As a secondary efficiency gain, the RSI-trained model reportedly shortened reasoning token length by 11% relative to the base model, pruning redundant inference steps without sacrificing accuracy.

This is architecturally different from Google's system-level RRSI approach (which adjusts prompts, external tools, and workflows while keeping base weights frozen). VIDRAFT's approach bakes improvements permanently into a single weight file, meaning users get the full capability from the model artifact alone, with no additional scaffolding required at inference time.

Benchmarks & results

All scores below are from the Hugging Face-certified LEXam benchmark, developed jointly by ETH Zurich, University of Zurich, and the Max Planck Institute. LEXam is built from 340 real law school final exams and tests applied legal reasoning under civil law — not statute memorization.

LEXam (multiple-choice, 1,655 questions):

Model	Score
Darwin-180B-RSI (4-pass majority vote)	68.94
Darwin-180B-RSI (1-pass, single inference)	60.54
GPT-5 (benchmark authors' measurement)	62.65
Claude-4.5-Sonnet (benchmark authors' measurement)	58.01
Gemini-2.5-Pro (benchmark authors' measurement)	55.72
DeepSeek-R1 (685B)	52.41
Qwen3-235B-Thinking	48.19
gpt-oss-120b	47.71

LEXam-hard (open-ended, 518 questions — scored by official grading model):

Model	Score
Darwin-180B-RSI	45.72
Inkling (Thinking Machines)	40.82
DeepSeek-V4-Pro	38.93

Context: random-chance baseline on the 4-option multiple-choice LEXam is 25 points. Most top LLMs average ~50 points on it; LEXam-hard's previous best was in the low 40s.

Across all 47 Hugging Face certified leaderboards, VIDRAFT now holds #1 in 7 categories — ahead of Zhipu AI (4), DeepSeek / Xiaomi / Moonshot AI (3 each).

How to try it

The Darwin-180B-RSI model checkpoint and its evaluation environment are publicly available on Hugging Face as open-source. VIDRAFT has also disclosed the full benchmark evaluation parameters — number of inference passes, ensemble majority-vote settings, and reasoning token budget — to enable independent reproduction.

To pull the model via the Hugging Face CLI:

huggingface-cli download VIDRAFT/Darwin-180B-RSI

⚠️ Verify the exact repository path on huggingface.co/VIDRAFT before downloading. The source confirms open availability but does not specify a direct URL string.

FAQ

Q: Does Darwin-180B-RSI require legal fine-tuning data to perform well on legal benchmarks?
A: No. The model was explicitly evaluated without any prior legal domain training. Its legal reasoning performance comes from generalizing multi-step logical inference skills acquired during math and science self-improvement cycles.

Q: What's the practical difference between VIDRAFT's model-level RSI and system-level self-improvement (like Google's RRSI)?
A: System-level approaches adjust prompts, tools, and orchestration workflows at runtime while leaving the underlying model weights unchanged. VIDRAFT's RSI updates the model's weights directly, so improvements are permanently embedded in the weight file. You don't need any additional runtime infrastructure — just load the model and run inference. VIDRAFT notes both approaches are complementary rather than competing.

Q: How do I know the benchmark results are legitimate and not cherry-picked evaluation conditions?
A: VIDRAFT publicly disclosed all evaluation parameters: inference pass count, majority-vote configuration, and reasoning token budget. The checkpoint is on Hugging Face so anyone can re-run the LEXam evaluation independently. The benchmark itself was built by academic researchers at ETH Zurich, University of Zurich, and the Max Planck Institute, with answers validated by legal professionals.

Originally reported by 로이슈 (2026-10-01) — source article.

Darwin-180B-RSI Tops Global Legal Reasoning Benchmarks — Without Ever Training on Legal Data

AI OpenFree — Thu, 01 Oct 2026 07:01:23 +0000

Darwin-180B-RSI Tops Global Legal Reasoning Benchmarks — Without Ever Training on Legal Data

TL;DR: VIDRAFT's Darwin-180B-RSI reasoning model has claimed #1 on both the LEXam and LEXam-hard Hugging Face-certified legal reasoning benchmarks, despite never being trained on legal data. It achieves this through Recursive Self-Improvement (RSI) — a self-supervised loop driven entirely by math and science problems. Developers working in legal tech, public sector AI, or high-stakes reasoning applications should take note.

What it is

Darwin-180B-RSI is a 180-billion-parameter reasoning model developed by VIDRAFT (비드래프트), a Korean AI startup. Its headline result: #1 on both LEXam and LEXam-hard, two Hugging Face-certified legal reasoning benchmarks developed by ETH Zurich and collaborators, without the model ever having been fine-tuned on legal corpora.

The Darwin family of models is built on a foundational approach that merges and combines the strengths of existing pre-trained models rather than training from scratch — a design philosophy aimed at dramatically reducing the compute cost typically associated with large-scale pre-training.

The -RSI suffix designates models that have additionally undergone VIDRAFT's Recursive Self-Improvement training procedure (described below).

How it works

Darwin-180B-RSI's performance on legal reasoning is the product of two conceptually distinct techniques applied on top of the base Darwin architecture:

1. Recursive Self-Improvement (RSI)
Instead of relying on human-authored chain-of-thought annotations, the model enters a self-supervised loop:

The model attempts to solve problems (sourced from math and science domains only).
An automated verifier checks whether the model's solution is correct.
Solutions that pass verification are fed back as training signal.
This cycle repeats, letting the model iteratively refine its own reasoning capability without human labeling.

The key insight is that general reasoning ability — not domain-specific knowledge — is what transfers to legal exam performance. Legal reasoning requires applying rules to facts and drawing conclusions, a structure that maps closely to rigorous mathematical reasoning.

2. Zero-Token Confidence (ZTC)
ZTC is VIDRAFT's technique for estimating the model's internal confidence in a given answer without requiring extra inference tokens. This is used during the self-improvement loop to help the model gauge answer reliability, and it contributes to a reported 11% reduction in average reasoning chain length relative to the parent model — meaning fewer tokens consumed at inference time with no accuracy regression.

The base Darwin architecture itself achieves cost efficiency by merging and composing existing model checkpoints rather than running a full pre-training run from scratch. Only 0.02% of total parameters are modified during the self-improvement phase.

Benchmarks & results

All scores below are sourced directly from the AI타임스 report and reflect results on Hugging Face-certified leaderboards.

LEXam (multiple-choice legal reasoning):

Model	Score
Darwin-180B-RSI	68.94
GPT-5	62.65
Claude 4.5 Sonnet	58.01
DeepSeek-R1	52.41
Qwen-235B-Thinking	48.19
gpt-oss-120b	47.71

LEXam-hard (open-ended / essay-style legal reasoning):

Model	Score
Darwin-180B-RSI	45.72
Inkling (Thinking Machines) — previous #1	40.82

Darwin-180B-RSI's 180B parameter count is notably smaller than DeepSeek-R1's 685B, yet it substantially outperforms it on both benchmarks.

Broader leaderboard standing: With these results, VIDRAFT now holds #1 positions across 7 of 47 categories on Hugging Face-certified leaderboards, spanning math, science, general knowledge, visual reasoning, and legal reasoning — the highest count among Korean AI companies, and ahead of Zhipu AI (4), DeepSeek, Xiaomi, and Moonshot AI (3 each).

LEXam benchmark context: LEXam is based on real university law school exams and is designed to test the ability to apply statutes to factual scenarios and derive legal conclusions — not rote memorization of law. This makes it a meaningful proxy for practical legal reasoning ability.

How to try it

The article states that the model and evaluation conditions are publicly available on Hugging Face for independent verification. You can explore VIDRAFT's public presence at:

🤗 Hugging Face: search for VIDRAFT or Darwin-180B-RSI on huggingface.co

At the time of writing, no public GitHub repository URL, API endpoint, or install command has been announced in this report. Check VIDRAFT's official channels for the latest on API access or model weights availability.

FAQ

Q: How can a model with zero legal training data outperform models that presumably have seen legal text?
A: The results suggest that LEXam is testing structured reasoning — applying rules to facts to reach conclusions — rather than legal memorization. RSI trained on math and science problems appears to build exactly that kind of generalizable deductive capability, which transfers surprisingly well to legal exam formats.

Q: Is Darwin-180B-RSI smaller or larger than the models it beat?
A: Smaller in some key comparisons. At 180B parameters it outperforms DeepSeek-R1 (685B parameters) on both LEXam benchmarks. Parameter count is clearly not the dominant factor here.

Q: What domains is VIDRAFT targeting with this technology?
A: According to CEO Minsik Kim, the target verticals are legal services, public administration, and financial services — domains where accurate, auditable judgment under uncertainty is critical.

Q: Does ZTC mean faster inference?
A: ZTC contributed to an 11% reduction in reasoning chain length compared to the parent model, which translates directly to lower token consumption at inference time while maintaining parent-model-level accuracy.

Originally reported by AI타임스 (2026-10-01) — source article.

VIDRAFT Darwin-180B-RSI Tops 5 Hugging Face Leaderboards: AIME & HMMT Perfect Scores, 94.44% on GPQA Diamond

AI OpenFree — Wed, 30 Sep 2026 03:01:24 +0000

VIDRAFT Darwin-180B-RSI Tops 5 Hugging Face Leaderboards: AIME & HMMT Perfect Scores, 94.44% on GPQA Diamond

TL;DR: VIDRAFT's Darwin-180B-RSI is a 180B-parameter reasoning model that achieves #1 rankings across five Hugging Face-certified leaderboards — including the first-ever perfect scores on AIME 2026 and HMMT 2026 benchmarks. It combines three proprietary techniques (Darwin selective merging, Recursive Self-Improvement, and Zero-Token Confidence) to push frontier-level math, science, and multimodal reasoning. If you're evaluating reasoning models for hard STEM workloads, this is a result worth tracking.

What it is

Darwin-180B-RSI is VIDRAFT's latest reasoning-focused large language model. The "180B" refers to its parameter count, and "RSI" stands for Recursive Self-Improvement — a core training methodology baked into the model's name.

Key characteristics from the source:

Scale: 180 billion parameters (VIDRAFT has separately extended the same Darwin merging technology to a ~397-billion-parameter model class)
Focus: High-difficulty reasoning across mathematics, science, and multimodal domains
Certification: Rankings verified on Hugging Face's officially certified leaderboards — not self-reported numbers
Efficiency signal: Achieves the same accuracy as prior iterations while reducing inference reasoning length by 11%

How it works

Darwin-180B-RSI is built on three proprietary technologies that VIDRAFT describes at a conceptual level:

1. Darwin — Selective Model Merging

Darwin analyzes performance at the layer and expert (MoE-style) unit level across multiple source models, then selectively identifies and combines the strongest components from each. Rather than averaging model weights indiscriminately, it surgically extracts high-performing substructures to produce a merged model that inherits complementary strengths. VIDRAFT has scaled this technique up to the ~397B parameter range.

2. RSI — Recursive Self-Improvement

RSI is a training loop in which the model:

Attempts to solve problems autonomously
Verifies whether its own answers are correct
Trains on the verified, high-quality reasoning traces

The improved model then becomes the solver for the next iteration of this loop — a self-bootstrapping cycle that incrementally raises reasoning quality without requiring human-labeled solutions at each step.

3. ZTC — Zero-Token Confidence

Before generating a response, ZTC analyzes the model's internal state to compute a confidence estimate for the candidate answer. If confidence falls below a threshold, the model is designed to withhold its response rather than produce a low-confidence output. This is a calibration mechanism aimed at reducing hallucination by giving the model a principled "abstain" option.

Benchmarks & results

All figures below are from Hugging Face-certified leaderboards as reported by VIDRAFT on 2026-09-28:

Benchmark	Darwin-180B-RSI	Previous SOTA (on that leaderboard)	Notes
AIME 2026	100%	97.1%	Math Olympiad-level problems; first perfect score on this leaderboard
HMMT 2026	100%	92.7%	Math Olympiad-level problems; first perfect score on this leaderboard
GPQA Diamond	94.44%	93.5% (Kimi-K3)	PhD-level expert-authored science questions
MMLU-Pro	88.12%	—	14 professional domains incl. law, medicine, economics
MMMU-Pro	79.48%	—	Multimodal reasoning: charts, sheet music, medical imaging

Darwin-180B-RSI ranked #1 on all five leaderboards simultaneously. The AIME and HMMT perfect scores are noted as the first time any model has achieved 100% on those Hugging Face-certified boards.

How to try it

The source article does not include explicit Hugging Face model IDs, GitHub repository links, or API endpoint details for Darwin-180B-RSI at the time of publication. Access details have not been publicly confirmed in this report.

What is publicly known from related VIDRAFT coverage:

VIDRAFT models have been distributed via Hugging Face (the company has surpassed 1.6 million model downloads on the platform as of September 2026)
VIDRAFT has previously open-released evaluation datasets and diagnostic tooling (AX-RAY) on Hugging Face

To follow release announcements, monitor VIDRAFT's Hugging Face organization page and their official channels directly. As soon as a model card or repository goes public, standard access patterns would apply.

FAQ

Q: Are the benchmark results independently verified or self-reported?
A: VIDRAFT explicitly states these rankings come from Hugging Face's certified leaderboards — meaning the evaluation pipeline is managed by Hugging Face, not run in-house by VIDRAFT. That's a meaningful distinction from self-reported numbers.

Q: What makes the 11% reasoning-length reduction significant?
A: Shorter reasoning chains at equivalent accuracy directly reduce inference compute costs and latency. For production deployments where you're paying per token or running latency-sensitive workloads, a model that reasons more concisely without sacrificing correctness is practically valuable — not just a benchmark curiosity.

Q: How does Darwin differ from standard model merging techniques like SLERP or TIES-merging?
A: Standard merging methods operate on full weight tensors with relatively coarse granularity. Darwin's distinguishing claim is that it evaluates and selects at the layer and expert unit level, enabling more surgical recombination of specialized capabilities. The specific algorithmic details beyond this conceptual description have not been publicly disclosed.

Q: Is ZTC related to existing confidence calibration or selective prediction research?
A: Conceptually it occupies similar territory — the model abstains when uncertain rather than guessing. VIDRAFT's specific implementation approach has not been detailed in public materials beyond the mechanism described above.

Originally reported by 지디넷코리아 (2026-09-28) — source article.

Darwin-180B-RSI: VIDRAFT's Recursive Self-Improvement MoE Model Claims Five Leaderboard Tops

AI OpenFree — Tue, 29 Sep 2026 23:01:18 +0000

Darwin-180B-RSI: VIDRAFT's Recursive Self-Improvement MoE Model Claims Five Leaderboard Tops

TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has released Darwin-180B-RSI — a 180-billion-parameter mixture-of-experts reasoning model built on Qwen3.8-Flash-Next that combines selective model merging with recursive self-improvement. The model reportedly tops five Hugging Face leaderboards across math, science, general, and multimodal reasoning benchmarks. If the reported gains hold on task-specific workloads, it's worth a closer look for reasoning-heavy applications.

What it is

Darwin-180B-RSI is VIDRAFT's open reasoning model, released in late September 2026. Here are the key technical facts as reported:

Base model: Built on top of Qwen3.8-Flash-Next via selective model merging — making it a high-performance model adaptation rather than a ground-up pretrain
Architecture: Mixture-of-Experts (MoE) with 180 billion total parameters and 512 experts, of which 10 experts are activated per inference request
Context window: Approximately 260,000 tokens
Availability: Described as an open model, with reported leaderboard results hosted on Hugging Face

The sparse expert activation design (10 of 512 experts per token routing) means that despite the 180B parameter count, the effective compute per forward pass is substantially lower than a dense 180B model — a meaningful practical consideration for deployment cost and latency.

How it works

VIDRAFT describes Darwin-180B-RSI as combining two high-level techniques: selective model merging and recursive self-improvement (RSI).

At a conceptual level, the pipeline works roughly like this:

Model merging as initialization: Rather than training from scratch, VIDRAFT merges capabilities from existing checkpoints selectively — preserving strengths from the base model while injecting targeted improvements.
Recursive self-improvement loop: The model generates candidate solutions to reasoning problems. Those solutions are automatically evaluated against verifiable ground-truth answers. Only solutions with confirmed correct answers are retained and used to retrain the model in the next iteration.
Iteration: This generate → filter → retrain cycle repeats, theoretically allowing the model to bootstrap higher reasoning quality from its own verified outputs over multiple passes.

This approach is conceptually related to ideas like rejection sampling fine-tuning and self-play reinforcement learning, but VIDRAFT's specific implementation details — including iteration count, reward modeling, and filtering criteria — are not disclosed in the source reporting.

The MoE routing means that at inference time, each token activates only a small subset of the expert network. This is a well-established technique (see Mixtral, DeepSeek-MoE) for scaling parameter count while keeping per-token FLOPs tractable.

Benchmarks & results

VIDRAFT reports the following scores on publicly recognized benchmarks, with Darwin-180B-RSI claiming first place on five Hugging Face leaderboards:

Benchmark	Reported Score	Domain
AIME 2026	100% (perfect)	Mathematical reasoning
HMMT 2026	100% (perfect)	Mathematical competition
GPQA Diamond	94.44%	Graduate-level science Q&A
MMLU-Pro	88.12%	Multi-domain professional knowledge
MMMU-Pro	79.48%	Multimodal understanding & reasoning

⚠️ Important caveat: These numbers are self-reported by VIDRAFT. The source article explicitly notes that no independent third-party validation of these results has been performed. Leaderboard rankings reflect performance on fixed benchmark datasets and may not generalize to production workloads, novel problem distributions, or domain-specific tasks. Engineers evaluating this model should run it against their own held-out test sets before drawing conclusions about real-world reasoning quality.

How to try it

The source reports Darwin-180B-RSI is an open model available on Hugging Face. If VIDRAFT follows its prior release pattern (the company had a prior Hugging Face feature covered in September), the model weights should be downloadable via the standard Hugging Face toolchain.

To check availability and download weights once the repository is public:

# Search for the model on Hugging Face
huggingface-cli search vidraft/darwin-180b-rsi

# Download when available
huggingface-cli download vidraft/darwin-180b-rsi

Note: The exact repository path is not confirmed in the source. Search for VIDRAFT or Darwin-180B-RSI directly on huggingface.co to find the correct model card and any associated inference instructions.

No GitHub repository, API endpoint, or OpenAI-compatible serving URL is mentioned in the source reporting. Check VIDRAFT's Hugging Face profile and official channels for inference API access details.

FAQ

Q: Is Darwin-180B-RSI practical to self-host given its 180B parameter count?
A: The MoE architecture activates only 10 of 512 experts per request, so active-parameter FLOPs are much lower than a dense 180B model. That said, you still need sufficient memory to load all expert weights. Quantized or offloaded serving strategies (e.g., llama.cpp, vLLM with expert offloading) may be necessary depending on your hardware. Check the model card for official serving recommendations once it is published.

Q: How does the recursive self-improvement approach differ from standard RLHF or SFT?
A: Standard supervised fine-tuning (SFT) trains on human-labeled data; RLHF uses a human preference reward model. VIDRAFT's RSI loop instead uses the model's own outputs filtered by verifiable correctness — closer in spirit to rejection sampling fine-tuning or outcome-reward RL (like GRPO or STaR), but without requiring a separate reward model or human annotators for each iteration. The practical implication is that it scales more easily to domains with automated answer verification, like math and formal science.

Q: Should I trust the perfect AIME and HMMT scores at face value?
A: Treat them as a signal worth investigating, not a guarantee. Perfect scores on competition math benchmarks from self-reported results warrant independent replication. Run the model on problems from your domain and measure it yourself before committing to it for production use cases.

Originally reported by AI Market Watch (미국) (2026-09-28) — source article.

VIDRAFT Beats Global Big Tech on Five AI Benchmarks Simultaneously

AI OpenFree — Tue, 29 Sep 2026 03:01:21 +0000

VIDRAFT Beats Global Big Tech on Five AI Benchmarks Simultaneously

TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has claimed top positions across five separate AI performance benchmarks, outperforming major global tech companies. This is a notable signal for the broader ML engineering community that competitive frontier AI development is no longer exclusive to the largest Western or East Asian incumbents — and worth tracking if you're evaluating state-of-the-art model performance.

What it is

VIDRAFT (비드래프트) is a Korean AI startup positioning itself in the Pre-AGI space — meaning its research and engineering efforts are oriented toward systems that approach, but have not yet reached, general artificial intelligence. The company's latest models have achieved what the press is calling a "five-crown" (5관왕) performance result: top rankings across five distinct AI capability evaluations, surpassing well-resourced global big tech competitors in each of those categories simultaneously.

This kind of multi-benchmark dominance is significant because it is relatively rare for any single model or model family to lead across multiple diverse evaluation axes at the same time. Typically, optimization for one benchmark involves trade-offs that hurt performance elsewhere. Achieving five concurrent top positions suggests a degree of generalization and architectural robustness that goes beyond narrow task-specific tuning.

How it works

While specific architectural internals and training configurations are not publicly disclosed, the conceptual approach VIDRAFT appears to be pursuing is consistent with the broader frontier AI research direction: building models that generalize across diverse task types rather than being narrowly optimized for a single capability domain.

At a high level, leading multi-benchmark AI systems typically achieve this through some combination of:

Diverse, high-quality training data curation — ensuring coverage across reasoning, language understanding, code, math, and multimodal tasks
Scalable model architectures — designs that maintain performance improvements as compute and parameter counts grow
Robust evaluation-informed development — iterative refinement guided by signal from a wide range of capability assessments, not just a single leaderboard target
Alignment and instruction-following quality — ensuring that raw capability translates into usable, reliable outputs for real-world developer tasks

The "five benchmark" result implies that VIDRAFT's approach is not over-indexed on any single evaluation, which is a meaningful technical signal about the underlying model's generality.

Benchmarks & results

The source article reports that VIDRAFT achieved first-place rankings across five AI performance benchmarks, surpassing global big tech companies across all five simultaneously. The reporting describes this as a "5관왕" (five-crown) outcome — a sweep of top positions in a multi-benchmark context.

Specific benchmark names, numeric scores, and comparative tables are not provided in the available source material. What the coverage does establish qualitatively:

The competing organizations include major global technology companies with substantially larger engineering headcounts and infrastructure resources
The result spans multiple evaluation dimensions (not a single repeated metric)
This was reported by Chosun Biz, one of Korea's major business news outlets, lending it a degree of editorial scrutiny

As specific benchmark names and scores become publicly available — for example, through Hugging Face leaderboards, VIDRAFT's own technical blog, or arXiv preprints — those would be the primary sources engineers should consult for reproducible, comparable numbers.

How to try it

Based on the information currently available from this source, public developer access channels (Hugging Face model hub, GitHub repositories, or OpenAI-compatible API endpoints) have not been announced in conjunction with this benchmark result.

If VIDRAFT follows the pattern of other frontier AI labs, future access may come through one or more of:

A Hugging Face organization page for model weights or demos
An API with OpenAI-compatible endpoints for integration into existing tooling
A GitHub repository for evaluation code or technical reports

Engineers interested in following VIDRAFT's releases should monitor their official channels directly. This article will not speculate on endpoints, model names, or access details that have not been publicly confirmed.

FAQ

Q: How should I interpret "five benchmark crowns" without seeing the specific benchmark names and scores?
A: Treat it as a directional signal worth investigating rather than a verified technical claim you can act on immediately. The meaningful next step is to wait for VIDRAFT to publish a technical report or leaderboard entries with reproducible evaluation details — benchmark name, version, evaluation protocol, and scores — before making architectural or tooling decisions based on this result.

Q: Is VIDRAFT's work peer-reviewed or publicly documented in any preprint?
A: The current source article does not reference an accompanying arXiv paper, technical report, or public evaluation submission. Until such documentation is available, the result is reported through press coverage only. Engineers should watch for official technical publications before drawing deep conclusions about methodology.

Q: Does this affect my choice of model for production use today?
A: Not directly, until public access and reproducible evaluation details are available. However, it is a useful reminder to keep your model evaluation pipeline flexible — leaderboard standings shift frequently, and building your stack to swap foundation models with minimal friction is always good engineering practice.

Originally reported by 조선비즈 (2026-09-29) — source article.

Korean startup's open 180B model tops five Hugging Face official leaderboards, including perfect AIME 2026 and HMMT 2026 scores

AI OpenFree — Mon, 28 Sep 2026 08:03:40 +0000

TL;DR — Seoul-based AI startup VIDRAFT has released Darwin-180B-RSI, an open-weight 180B mixture-of-experts model that now sits at #1 on five Hugging Face official benchmark leaderboards: AIME 2026 (100%), HMMT February 2026 (100%), GPQA Diamond (94.44%), MMLU-Pro (88.12%) and MMMU-Pro (79.48%). The full weights are downloadable at huggingface.co/FINAL-Bench/Darwin-180B-RSI.

What happened

On September 28, 2026, VIDRAFT published Darwin-180B-RSI and submitted its self-measured scores to five benchmark leaderboards that Hugging Face lists as official benchmarks (datasets carrying the benchmark:official tag). Leaderboard entries on these datasets are populated from .eval_results files in each model repository.

Benchmark	Darwin-180B-RSI	Previous #1
AIME 2026	100.0	97.1 (Inkling, A.X-K2)
HMMT Feb 2026	100.0	92.7 (Kimi-K2.6)
GPQA Diamond	94.44	93.5 (Kimi-K3)
MMLU-Pro	88.12	88.0 (MiniMax-M2.1, Intern-S2-Preview)
MMMU-Pro (vision)	79.48	79.4 (Kimi-K2.6)

According to the public leaderboards, no model had previously reported 100% on the AIME 2026 or HMMT February 2026 boards.

Why these five benchmarks matter

GPQA Diamond — 198 graduate-level physics, chemistry and biology questions written by domain PhDs; experts in the field reach roughly 65%.
MMLU-Pro — 12,032 ten-option questions across 14 disciplines, a harder successor to MMLU.
MMMU-Pro (vision) — 1,730 college-level questions where the question itself is embedded in an image (charts, scores, medical images) across 30 subjects.
AIME 2026 / HMMT Feb 2026 — the 2026 American Invitational Mathematics Examination and the Harvard-MIT Mathematics Tournament, both olympiad-track competitions with exact answers.

How the scores were measured

The model card publishes the protocol. Every run used a 131,072-token thinking budget, sampling at temperature 1.0 / top_p 0.95 / top_k 20, in bf16.

Benchmark	Samples per question	Reported
AIME 2026	16	majority vote (mean over 16 = 98.75)
HMMT Feb 2026	16	majority vote (mean over 16 = 96.59)
GPQA Diamond	up to 16	majority vote
MMLU-Pro	1	single sample
MMMU-Pro	3	majority vote

Readers comparing numbers should note that leaderboard values are self-reported by each publisher and that settings (sample count, voting, thinking budget) differ between models.

What is inside

The model's parent is Qwen3.8-Flash-Next (180B MoE, Qwen Community License). VIDRAFT says it modified only about 0.02% of the parameters — attention paths and shared experts — while leaving all 512 routed experts, the router and the vision encoder unchanged. The company attributes the result to three in-house techniques:

Darwin — a model "diagnose-then-evolve" framework (arXiv:2605.14386)
RSI (recursive self-improvement) — the model solves practice problems, its answers are checked against verifiable references, and it is retrained on its correct reasoning. VIDRAFT reports the same accuracy with 11% shorter reasoning than the parent.
ZTC (Zero-Token Confidence) — a readout of the model's internal state that estimates, before generation, the probability that an answer will be correct.

Deep dives

FAQ

Is the model open? Yes. Weights, config and tokenizer are public on Hugging Face under the Qwen Community License 1.0.

Who is VIDRAFT? A Korean AI deep-tech startup (CEO Minsik Kim) that describes itself as an "AI foundry". Its Darwin model family counts 50+ official models and 400+ community derivatives on Hugging Face.

Can I reproduce the scores? The model card lists the full evaluation protocol; all five benchmarks are public datasets.

Reading an official Hugging Face leaderboard: protocols, majority vote and reproducibility

AI OpenFree — Mon, 28 Sep 2026 08:02:56 +0000

TL;DR — Hugging Face now marks selected benchmark datasets as official and builds their leaderboards from .eval_results/*.yaml files in model repositories. Scores are self-reported, so the evaluation protocol is what makes a number comparable. This post walks through the protocol behind Darwin-180B-RSI, which ranks #1 on five of these boards.

How official leaderboards work

A benchmark dataset carries the benchmark:official tag and an eval.yaml defining its tasks.
A model publisher adds a file such as .eval_results/gpqa_diamond.yaml to their model repository with the dataset id, task id, value and date.
The dataset page aggregates those files into a ranked leaderboard.

You can query any board directly:

curl https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard

Because entries are self-reported, the leaderboard shows the value but not always how it was obtained. The notes field and the model card are where the protocol lives.

The protocol behind five #1 scores

Common settings for every run of Darwin-180B-RSI:

Setting	Value
Thinking budget	131,072 tokens per sample
Sampling	temperature 1.0 · top_p 0.95 · top_k 20
Precision	bf16
Engine	vLLM, tensor parallel, expert parallel

Per-benchmark settings and results:

Benchmark	Samples	Reported score	Value
AIME 2026	16	majority vote (mean 98.75)	100.0
HMMT Feb 2026	16	majority vote (mean 96.59)	100.0
GPQA Diamond	up to 16	majority vote	94.44
MMLU-Pro	1	single sample	88.12
MMMU-Pro (vision)	3	majority vote	79.48

Three things to check before comparing numbers

Samples and voting. A majority-of-16 score is a system score and is not directly comparable with a single-sample score. Look for both.
Thinking budget. Reasoning models lose points to truncation when the budget is small; a cut-off answer is not a wrong answer.
Contamination. Training data should be filtered against every reported test set. VIDRAFT reports an 8-gram overlap filter against all benchmarks it lists.

Reproducing

All five benchmarks are public datasets and the weights are open at huggingface.co/FINAL-Bench/Darwin-180B-RSI. Serving requires 8 × B200-class GPUs (4 at minimum) in bf16:

vllm serve FINAL-Bench/Darwin-180B-RSI \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --max-model-len 135168 --trust-remote-code

Darwin: evolving a 180B parent model by changing 0.02% of it

AI OpenFree — Mon, 28 Sep 2026 08:02:19 +0000

TL;DR — Darwin-180B-RSI keeps its parent's 512 routed experts, router and vision encoder intact and modifies about 0.02% of the parameters. VIDRAFT's Darwin framework treats an open model as a "parent", diagnoses its weak paths, and strengthens only those — here combined with recursive self-improvement (RSI) on verified answers.

The parent

The father model is Qwen3.8-Flash-Next, a 180B mixture-of-experts vision-language model:


Layers / hidden size	48 / 2,560
Attention	36 linear-attention + 12 full-attention layers
Experts	512 routed (10 active per token) + shared expert
Context	262,144 tokens

What Darwin changed

According to the model card, Darwin updated three wiring paths and preserved the rest:

🔹 Full-attention layers (12) — the long-range path across the whole context
🔹 Linear-attention layers (36) — the fast inference path
🔹 Shared expert (48 layers) — the common path every token passes through
🔒 Preserved: all 512 routed experts, the router and the vision encoder

That is roughly 30 million trainable parameters out of 177 billion. Because the experts — where most factual knowledge is stored in an MoE model — are untouched, the parent's knowledge is retained.

Recursive self-improvement on verified answers

The RSI loop is simple to state:

The model solves practice problems that do not overlap with any reported benchmark (8-gram overlap filter).
Its answers are checked against verifiable references.
Only the correct reasoning is kept, and the model is retrained on it.
The improved model becomes the next solver.

Two design choices matter. Unverified solutions are never learned, so errors are not reinforced. And among correct solutions, shorter ones are preferred — which is why VIDRAFT reports the same accuracy with 11% shorter reasoning than the parent (MMLU-Pro: 4,320 → 3,833 tokens per answer).

How Darwin compares with evolutionary model merging

The best-known prior work in this area is Sakana AI's evolutionary model merge (Nature Machine Intelligence, 2025), which searches merge recipes over hundreds of generations and demonstrated 7–10B models. Darwin takes a diagnosis-first approach — measuring layer- and expert-level strengths before transplanting or tuning — which VIDRAFT says reduces search cost and has scaled from 4B to 397B models.

On Hugging Face, the Darwin family lists 50+ official models and 400+ community derivatives. The framework is described in Darwin Family (arXiv:2605.14386).

Zero-Token Confidence

VIDRAFT pairs RSI with ZTC, a lightweight readout of the model's final-layer hidden state that estimates, before any token is generated, the probability that the upcoming answer is correct. In an RSI loop, low-confidence areas point to what the model should learn next; in deployment, the same signal can gate tool calls or escalate to a human. The company's earlier Darwin-397B-ZTC ships such a readout.

Perfect scores on AIME 2026 and HMMT 2026: what an open 180B model got right, and why thinking budget mattered

AI OpenFree — Mon, 28 Sep 2026 08:01:42 +0000

TL;DR — Darwin-180B-RSI, an open-weight model from Korean startup VIDRAFT, reports 100% on AIME 2026 (30/30) and 100% on HMMT February 2026 (33/33) under 16-sample majority vote with a 131K-token thinking budget. They are the first perfect scores on these two Hugging Face official leaderboards.

The numbers

Benchmark	Majority of 16	Mean over 16 samples	Single sample	Previous #1
AIME 2026	100.0	98.75	96.67	97.1
HMMT Feb 2026	100.0	96.59	93.94	92.7

Even the mean accuracy over 16 samples (98.75 on AIME, 96.59 on HMMT) is above the previous top entries on both boards.

About the two competitions

AIME (American Invitational Mathematics Examination) is the qualifier toward the USA Mathematical Olympiad. Every answer is an integer from 0 to 999, so guessing rarely works.
HMMT (Harvard-MIT Mathematics Tournament) is one of the most difficult high-school competitions, attended by national-olympiad-level students. Answers include fractions and radicals, so grading requires symbolic equivalence (for example, 2√3 = √12).

The finding: misses were truncations, not wrong math

VIDRAFT's engineers first measured the parent model with a 65K-token budget. The model missed two AIME 2026 problems, and on inspection every miss was a truncated solution — the reasoning ran out of tokens before reaching an answer. Solutions that finished were correct.

Re-running those problems with a 131K-token budget produced 8/8 correct answers. With the budget raised for all problems, the 16-sample run scored 30/30.

The practical lesson for anyone evaluating reasoning models:

Truncation is not an error. Count cut-off answers separately; otherwise a budget limit looks like a capability limit.
Budget is part of the protocol. A score without its thinking budget is not comparable.
Grade symbolically. String matching undercounts fractions and radicals; a math-equivalence checker is required.

How to read "majority of 16"

Majority vote is a system score: the model answers each question 16 times and the most frequent answer is graded. VIDRAFT reports the majority score and the mean single-sample accuracy side by side on the model card and in the leaderboard submission notes, so readers can compare with entries that use a single sample.

Try it

Model: huggingface.co/FINAL-Bench/Darwin-180B-RSI — the model card contains the full evaluation protocol.

South Korea's Government–Naver Consortium Is Building Dual 700B-Class Cybersecurity-Specialized AI Models with 4,512 GPUs

AI OpenFree — Wed, 23 Sep 2026 07:01:09 +0000

South Korea's Government–Naver Consortium Is Building Dual 700B-Class Cybersecurity-Specialized AI Models with 4,512 GPUs

TL;DR: A South Korean government–Naver Cloud consortium has officially launched development of two frontier-scale, cybersecurity-specialized large language models — one focused on offensive (red-team) capabilities and one on defensive (blue-team) capabilities — backed by 4,512 GPUs and ~830 TB of security-domain training data. The models are planned for open-source release and potential international export, making them worth watching for security engineers and ML practitioners building in the threat-intelligence and vulnerability-research space.

What it is

The South Korean Ministry of Science and ICT (MSIT), in consortium with Naver Cloud, LG AI Research, and 41 partner organizations, has kicked off a cybersecurity-specialized AI model development program. The official launch event was held on September 22, 2026, in Seoul.

Key facts from the source:

Two distinct 700B-class models are being developed in parallel:
- A red-team (offensive) model — focused on source code analysis, automated vulnerability discovery, root-cause inference, and attack-path reasoning.
- A blue-team (defensive) model — focused on correlating distributed system events and threat intelligence, classifying attack behavior, assessing threat severity, and recommending response actions.
The models are security-domain fine-tunes of existing foundation models: Naver Cloud is building on HyperCLOVA X for the defensive variant; LG AI Research is contributing an offensive variant based on its own foundation model.
Training data: approximately 830 TB of security-domain data, sourced from national critical infrastructure and industrial security environments, with a dedicated preprocessing pipeline to convert raw source data into training-ready format.
The program includes field validation across sectors including finance and defense.
Planned open-source release with international distribution ambitions.

How it works

The architectural concept here is a dual-model adversarial co-training loop — sometimes called a "spear-and-shield" or red/blue feedback cycle:

Red-team model analyzes code and system artifacts to surface vulnerabilities, classify their type and origin, and reason about exploitable attack chains.
Blue-team model ingests multi-source event logs and threat feeds to reconstruct attack behavior, score threat severity, and propose mitigations.
Critically, the two models inform each other's training: vulnerabilities found by the red-team model become labeled training signal for the blue-team model; successful defenses by the blue-team model raise the bar the red-team model must clear in future iterations.

This adversarial feedback loop is conceptually analogous to self-play in game-theoretic AI systems, applied here to the offensive/defensive security domain at foundation-model scale. The data preprocessing pipeline handles conversion of raw operational security data — including real incident records — into formats suitable for supervised and instruction-tuning workflows.

Benchmarks & results

The project is at launch/kickoff stage (September 2026), so no public benchmark results are available yet. The source article does not report evaluation scores, leaderboard placements, or ablation outcomes.

What is publicly stated:

The Deputy Prime Minister and MSIT Minister described the ambition as reaching "frontier-level" performance in the security-specialized domain.
A mid-program evaluation is scheduled for February 2027, at which point continued GPU resource allocation from the government side will be decided.
Field validation ("실증") will be conducted against real national infrastructure and industrial environments — implying evaluation on operational security tasks rather than purely academic benchmarks.

Engineers should watch for public benchmark disclosures around or after the February 2027 checkpoint.

How to try it

Public developer access is not available at this time. The program launched on September 22, 2026, and the models are still in active development. The source article states that open-source release is planned, but no Hugging Face repository, GitHub organization, API endpoint, or release timeline has been announced publicly.

When the models are released, expect distribution channels typical of Korean government-backed open-source AI initiatives (Hugging Face Hub, potential OpenAI-compatible API wrappers). Follow Naver Cloud and LG AI Research's official channels for release announcements.

FAQ

Q: Why develop two separate models instead of one unified security model?
A: The red/blue split allows each model to specialize deeply — offensive reasoning (exploit path analysis, vulnerability classification) and defensive reasoning (event correlation, threat scoring, response recommendation) have distinct input distributions and output requirements. More importantly, keeping them separate enables the adversarial co-training loop: each model's outputs become harder training targets for the other, driving iterative capability improvement on both sides.

Q: Is the ~830 TB training dataset going to be publicly released alongside the models?
A: The source article does not state this. Given that much of the data originates from national critical infrastructure and live security environments, full public release of the raw dataset seems unlikely, though curated or preprocessed subsets could accompany an open-source model release. No official announcement on dataset availability has been made.

Q: What is the GPU breakdown across consortium partners?
A: Naver Cloud self-funded 4,000 GPUs (NVIDIA B200) starting August 2026, LG AI Research is contributing 256 GPUs (NVIDIA H200), and the Korean government is supplying 256 GPUs (NVIDIA B200) over a 10-month support window — totaling 4,512 GPUs across the consortium.

Originally reported by EBN (2026-09-22) — source article.

JEV vs ZTC: Statistically Tied on Accuracy, 1.4 Points Apart Where It Counts

AI OpenFree — Sun, 20 Sep 2026 10:09:29 +0000

Short answer: on accuracy, JEV (0.7350 AUC) and ZTC (0.7364) are statistically tied — the 0.0014
gap has a 95% interval of −0.019 to +0.032. They separate on two other axes: deployment shape (API
only vs open weights) and downstream agent accuracy, where ZTC gained 1.34 points and JEV lost 0.07
at the same retry budget.

Both numbers come from the same test set: 2,018 items, 508 of them incorrect, identical labels,
published grading code —
Typed Decision Leaderboard.

First, what these two things are

JEV is TypeSafe AI's answer verifier, the product that established this category under the name
System One models. ZTC (Zero-Token Confidence) is VIDRAFT's, shipping as two open-weight
models.

Both are typed decision models: they return a structured verdict — boolean, choice, or score —
from a single forward pass, emitting zero generated tokens. That is what makes both of them one
to three orders of magnitude cheaper than asking a frontier model "is this answer right?"

So the choice between them is not a choice of architecture. It is a choice of accuracy, shape, and
behaviour under load.

Axis 1 — accuracy: a tie, and it should be reported as one

System	AUC	95% interval vs. the other
ZTC (397B)	0.7364	—
JEV	0.7350	−0.019 to +0.032
ZTC-Judge-27B	0.7282	−0.034 to +0.020

The interval contains zero in both comparisons. Under the rule that a gap indistinguishable from
noise produces no rank, neither system can be called more accurate than the other on this set.

For context, the same table's reference row — a logistic model over answer length and formatting —
scores 0.7036, and eight of the thirteen measured systems fall below it. Both JEV and ZTC are
comfortably above that bar. Most of the category is not.

Axis 2 — deployment shape, where they genuinely differ

This is the axis that usually decides the purchase, and it is not close.

	JEV	ZTC
Weights published	no	yes (Apache-2.0)
Runs inside a private network	no	yes
Adaptable to your own data	no	yes
Hosted API	yes	yes
Cost	$0.024 / 1,000 calls	your own compute
Generated tokens	0	0

If your data cannot leave your network, the accuracy tie is irrelevant — only one of these is
installable. If you would rather not operate a GPU, the same logic runs the other way, and $0.024
per thousand calls is cheap enough that "just use the API" is a defensible answer.

For comparison, asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls — about
23× more — and scores 0.7148, below both.

Axis 3 — per domain, the ranking inverts

The headline is an average of five domains. They disagree.

Domain	n	JEV	ZTC 397B	ZTC 27B
Professional exams (law/math/bio)	400	0.7981	0.8660	0.8462
Biology & medicine	917	0.7185	0.7433	0.7154
Commonsense & multi-step	278	0.6006	0.6072	0.6172
Disaster & safety guidance	225	0.7620	0.7319	0.6961
Scientific reasoning	198	0.8421	0.6287	0.7410

JEV wins scientific reasoning outright at 0.8421 — 0.21 ahead of the overall leader. If that is
your workload, the overall ranking is the wrong table to read, and the answer to "which one" is JEV.

Two other things in that table are worth carrying away regardless of vendor:

Bigger is not better. Inside one family, the 397B scores 0.11 below the 27B on scientific
reasoning. A separate measurement puts a 180B-class model at 0.7146, under a 4B at 0.7284.
Verification quality tracks representation geometry, not parameter count.

Commonsense is unsolved by everyone. Every system sits near 0.60 on multi-step commonsense. No
product on this board fixes that domain, and knowing it is more useful than any ranking.

Axis 4 — as an agent gate, they are not tied at all

AUC answers "which separates right from wrong answers best." It does not answer "which helps my
agent." Those turned out to be different questions.

Setup. For each item: score the answer with a verifier, send the lowest-scoring 20% to a
stronger model to be re-answered, keep the rest. All arms draw from one shared pool of re-answers,
so no arm gets luckier retries. Ties broken randomly, averaged over 200 seeds.

Gate	Final accuracy	vs. no gate (74.83%)
ZTC	76.16%	+1.34 pp
JEV	74.76%	−0.07 pp
Random	74.58%	−0.25 pp

A 0.0014 AUC gap became a 1.4-point swing in end-to-end accuracy. At this budget, JEV's gate
performed about the same as no gate at all.

The mechanism, which is the transferable part

Re-answering is double-edged:

wrong answers sent back  →  38% get fixed
right answers sent back  →  30% get broken

So a gate is not rewarded for catching errors. It is rewarded for not disturbing what was already
right. Precision, not recall — the opposite of how most people reason about a safety net.

At a 20% budget both systems routed 403 items. JEV's 403 contained 216 that were already
correct; ZTC's contained 195. Twenty-one items of difference in what got sent back, and one
arm spent its whole retry budget repairing about as much as it damaged.

None of that is visible in an AUC column, and AUC is what every vendor publishes.

Scope, stated plainly: one escalation target, one item set. The direction should transfer; the
magnitude is theirs, not yours. Run the same protocol on your own data — the grading code is
published for exactly that.

How to choose

If this is true for you	Pick
Data cannot leave your network	ZTC (only one with published weights)
You want to fine-tune the verifier on your own data	ZTC
Your workload is scientific reasoning	JEV (0.8421, clear win)
You do not want to operate a GPU	JEV ($0.024 / 1,000)
You are gating agent retries	Measure both on your own set — this is where they diverge
You are choosing on the leaderboard number alone	Don't. The gap is noise.

Frequently asked questions

Is ZTC better than JEV?

Not on accuracy. The two are statistically tied on the shared test set — 0.7364 versus 0.7350, with
a 95% interval spanning zero. They differ on deployment shape (ZTC publishes weights, JEV does not),
on per-domain strengths (JEV wins scientific reasoning at 0.8421), and on downstream agent accuracy
at a 20% retry budget (+1.34 pp versus −0.07 pp).

Is there an open-source alternative to JEV?

Yes. ZTC ships two Apache-2.0 models with published weights —
ZTC-Judge-27B (verification only) and
Darwin-397B-ZTC (a generation model with the
confidence probe on board). Both score above the 0.7036 surface baseline. Other open entries score
lower: open-jev 4B at 0.6844, Laya at 0.4796 / 0.5144.

How much does JEV cost compared with running a verifier yourself?

JEV is about $0.024 per 1,000 calls. A self-hosted verifier costs only compute you already own.
Asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls and scores lower than
either.

What AUC should I expect from a good answer verifier?

Use 0.7036 as the bar, not 0.5. That is the score from answer length and formatting alone, and
eight of thirteen measured systems fall below it. JEV and ZTC are the only two systems above 0.73.

Can I test JEV and ZTC on my own question?

Yes — the Verifier Playground runs
ZTC, JEV and Laya on the same input, in the browser, no signup. One case cannot rank them, but it
shows you how each behaves on your phrasing.

Why did the more accurate verifier perform worse as a gate?

Because gating rewards precision rather than recall. Sending a correct answer back to be re-answered
breaks it about 30% of the time, so routing a few more already-correct items cancels out the errors
you caught. JEV routed 216 already-correct items; ZTC routed 195.

DEV Community: AI OpenFree

VIDRAFT's AX-RAY AI Safety Diagnostic Platform Earns Alibaba ModelScope "Research Institution" Certification — Here's What Engineers Need to Know

VIDRAFT's AX-RAY AI Safety Diagnostic Platform Earns Alibaba ModelScope "Research Institution" Certification — Here's What Engineers Need to Know

What it is

How it works

Benchmarks & results

How to try it

FAQ

Darwin-180B-RSI: How VIDRAFT's Recursive Self-Improvement Model Topped 7 Hugging Face Leaderboards — Without Legal Training Data

Darwin-180B-RSI: How VIDRAFT's Recursive Self-Improvement Model Topped 7 Hugging Face Leaderboards — Without Legal Training Data

What it is

How it works

Benchmarks & results

How to try it

FAQ

Darwin-180B-RSI Tops Global Legal Reasoning Benchmarks — Without Ever Training on Legal Data

Darwin-180B-RSI Tops Global Legal Reasoning Benchmarks — Without Ever Training on Legal Data

What it is

How it works

Benchmarks & results

How to try it

FAQ

VIDRAFT Darwin-180B-RSI Tops 5 Hugging Face Leaderboards: AIME & HMMT Perfect Scores, 94.44% on GPQA Diamond

VIDRAFT Darwin-180B-RSI Tops 5 Hugging Face Leaderboards: AIME & HMMT Perfect Scores, 94.44% on GPQA Diamond

What it is

How it works

1. Darwin — Selective Model Merging

2. RSI — Recursive Self-Improvement

3. ZTC — Zero-Token Confidence

Benchmarks & results

How to try it

FAQ

Darwin-180B-RSI: VIDRAFT's Recursive Self-Improvement MoE Model Claims Five Leaderboard Tops

Darwin-180B-RSI: VIDRAFT's Recursive Self-Improvement MoE Model Claims Five Leaderboard Tops

What it is

How it works

Benchmarks & results

How to try it

FAQ

VIDRAFT Beats Global Big Tech on Five AI Benchmarks Simultaneously

VIDRAFT Beats Global Big Tech on Five AI Benchmarks Simultaneously

What it is

How it works

Benchmarks & results

How to try it

FAQ

Korean startup's open 180B model tops five Hugging Face official leaderboards, including perfect AIME 2026 and HMMT 2026 scores

What happened

Why these five benchmarks matter

How the scores were measured

What is inside

Deep dives

FAQ

Reading an official Hugging Face leaderboard: protocols, majority vote and reproducibility

How official leaderboards work

The protocol behind five #1 scores

Three things to check before comparing numbers

Reproducing

Darwin: evolving a 180B parent model by changing 0.02% of it

The parent

What Darwin changed

Recursive self-improvement on verified answers

How Darwin compares with evolutionary model merging

Zero-Token Confidence

Links

Perfect scores on AIME 2026 and HMMT 2026: what an open 180B model got right, and why thinking budget mattered

The numbers

About the two competitions

The finding: misses were truncations, not wrong math

How to read "majority of 16"

Try it

South Korea's Government–Naver Consortium Is Building Dual 700B-Class Cybersecurity-Specialized AI Models with 4,512 GPUs

South Korea's Government–Naver Consortium Is Building Dual 700B-Class Cybersecurity-Specialized AI Models with 4,512 GPUs

What it is

How it works

Benchmarks & results

How to try it

FAQ

JEV vs ZTC: Statistically Tied on Accuracy, 1.4 Points Apart Where It Counts

First, what these two things are