AX-RAY: VIDRAFT's Open Safety Benchmark That Detects "Causal Leakage" in General-Purpose LLMs
TL;DR: VIDRAFT has released AX-RAY, an AI safety evaluation leaderboard and dataset on Hugging Face, designed to detect a phenomenon called causal leakage in general-purpose LLMs. The benchmark covers 117 safety diagnostic items and maps findings to real national laws and cultural norms. If you work on LLM evaluation, red-teaming, or AI safety, this is a public resource worth exploring.
What it is
AX-RAY is an open AI safety diagnostic framework released by Korean deep-tech AI startup VIDRAFT (비드래프트). It consists of two publicly available artifacts on Hugging Face:
- An evaluation dataset structured around 117 safety diagnostic items
- A leaderboard tracking how general-purpose LLMs perform across those items
The central technical claim is notable: AX-RAY is designed to detect causal leakage — a failure mode in AI models where outputs are influenced by hidden or unintended causal pathways rather than the model's explicit, transparent reasoning chain. Until now, causal leakage has largely been treated as a theoretical concern in AI safety literature. VIDRAFT's evaluation framework claims to operationalize its detection in real, deployed general-purpose LLMs.
In their initial evaluation run, VIDRAFT reports detecting causal leakage signals in two models, including one model from NVIDIA.
A distinctive design choice is the framework's legal and normative mapping layer: each of the 117 diagnostic items is linked to existing national laws and regulatory frameworks, not just abstract safety principles. For Arabic-speaking regions, this extends to mapping against religious and social norm systems (Sharia-informed legal structures), making AX-RAY one of the few AI safety benchmarks explicitly designed for cross-cultural and cross-jurisdictional evaluation.
How it works
At a conceptual level, AX-RAY probes models with scenarios designed to surface behavior that cannot be explained by the model's stated or observable reasoning path. The key insight driving the benchmark is that a model can appear to reason correctly while still being influenced by latent causal signals embedded in its training distribution — signals that could, in principle, cause unpredictable or unsafe behavior in agentic, high-stakes deployments.
The 117 diagnostic items are organized to stress-test:
- Hidden dependency behavior — whether a model's outputs shift based on implicit contextual cues it should not be sensitive to
- Safety rail circumvention tendencies — patterns that might, in edge cases, lead a model to take unexpected actions in autonomous settings
- Normative alignment — whether behavior stays within the bounds of specific legal and regulatory systems across different jurisdictions and cultural contexts
The legal mapping layer is particularly relevant for teams building production systems in regulated industries. Rather than asking only "did the model refuse the bad prompt?" AX-RAY asks "does this model's behavior comply with the specific legal framework that governs deployment in country X?" This shifts evaluation from a binary safety gate to a jurisdiction-aware compliance diagnostic.
Benchmarks & results
VIDRAFT's initial evaluation using AX-RAY surfaced causal leakage signals in two general-purpose LLMs, one of which is attributed to NVIDIA. The source article does not publish granular per-model scores or aggregate leaderboard rankings beyond this finding.
For context on VIDRAFT's broader model development track record, the article notes several publicly reported figures for their existing models:
- Hugging Face cumulative downloads: 1 million+
- K-AI Leaderboard ranking: #1
- GPQA score: 90.9%
- Google's Fast Gemma Challenge: #1 globally
These figures relate to VIDRAFT's foundation models (AETHER and Darwin), not to AX-RAY itself. AX-RAY is an evaluation tool, not a model — its "results" are the safety diagnostics it produces on other models.
How to try it
VIDRAFT has made both the AX-RAY leaderboard and the evaluation dataset publicly available on Hugging Face. You can explore them directly at the Hugging Face organization page for VIDRAFT.
To browse or download the dataset using the Hugging Face CLI:
pip install huggingface_hub
huggingface-cli download vidraft/AX-RAY
Note: The exact dataset repository slug above is illustrative of the standard CLI pattern — verify the precise repository name on the VIDRAFT Hugging Face page before running. No API key is required for public datasets.
The leaderboard is viewable directly in the Hugging Face Spaces or Datasets UI. No GitHub repository or OpenAI-compatible API endpoint for AX-RAY has been announced at time of writing.
FAQ
Q: What exactly is "causal leakage" and why should ML engineers care?
A: Causal leakage refers to a model's behavior being shaped by unintended hidden causal pathways — inputs or correlations the model shouldn't be acting on, but is. In production agentic systems (think autonomous financial or medical decision-making), this can manifest as unpredictable or unsafe actions that aren't caught by standard prompt-level red-teaming because the failure mode lives in the model's internal causal structure, not in surface-level outputs.
Q: Is AX-RAY only relevant for safety researchers, or is it useful for application developers too?
A: Both. Safety researchers will care about the causal leakage detection methodology. Application developers building in regulated industries — finance, healthcare, public services — will find the jurisdiction-specific legal mapping layer directly relevant: it lets you evaluate whether a model's behavior is compliant with the specific laws governing your deployment context, rather than relying on generic safety benchmarks.
Originally reported by AI타임스 (2026-08-17) — source article.
Top comments (0)