DEV Community: Papers Mache

Self‑distilled spatial reasoning improves accuracy without large latency penalty

Papers Mache — Thu, 15 Oct 2026 05:00:00 +0000

Distilling privileged depth cues into an RGB‑only vision‑language model closes the long‑standing gap between accuracy and inference speed. SpatialOPSD shows that a single forward pass can inherit most of the reasoning power of a tool‑driven teacher without the heavy latency penalty.

Before this work, spatial coding agents boosted multimodal models by calling external tools to generate verified execution traces, but each trace required multiple API calls and slowed down deployment. Fine‑tuning (SFT) or reinforcement‑learning approaches such as GRPO could not internalize geometry because they never saw the privileged depth information during training.

SpatialOPSD improves the Qwen3.5‑9B baseline from 45.9 to 51.9, with specific efficacy in allocentric transformations (e.g., Among +11.7) [1]. The gain comes from feeding summarized agent traces as privileged inputs during on‑policy self‑distillation, which teaches the model a spatial chain‑of‑thought that it can later reproduce without any external depth sensor.

In a single tool‑free forward pass, SpatialOPSD retains 91% of the performance of its iterative, tool‑dependent teacher, SpatialClaw (51.9 vs. 57.3) [1]. This means the distilled model eliminates virtually all runtime overhead from external calls while sacrificing only a modest fraction of accuracy.

Experiments across multiple benchmarks demonstrate that self‑distilling SpatialOPSD achieves higher average accuracy than SFT and GRPO on both spatial and OOD datasets, exhibiting superior performance and generalization [1]. The advantage holds across varied tasks, confirming that the distilled spatial reasoning is not a narrow trick but a broadly useful capability.

The authors acknowledge that privileged‑information leakage remains a concern, mitigated only by Repetition‑Aware Distillation’s masking and unlikelihood regularization; the 9% residual gap to the teacher suggests further work is needed on scaling or tighter leakage control. One inference is that larger models may close this gap, but empirical evidence is still missing.

If these numbers hold in practice, developers should replace iterative tool‑dependent spatial pipelines with a distilled RGB model and re‑run latency budgets on VSI‑Bench; the expectation is a >90% reduction in execution time while keeping success rates within a few points of the original teacher.

References

SpatialOPSD: Self-Distilling Spatial Intelligence from Verified Coding Agent Traces

Token‑level routing can achieve up to 64 higher decoding throughput

Papers Mache — Wed, 14 Oct 2026 05:00:00 +0000

Token‑level routing enables a dedicated serving stack to extract up to sixty‑four times more tokens per second compared to existing baselines, without retraining any models.

Historically, production systems have relied on coarse‑grained routing at the session or query level, treating each request as an atomic unit and dispatching it to a single LLM. Academic work showed that fine‑grained token decisions could improve efficiency, yet existing inference engines were built around single‑model assumptions and suffered from step desynchronization and batch admission delays.

TokenRouter delivers between 2.01× and 64.15× higher decoding throughput than any existing serving baseline across diverse routing algorithms, workloads, and model pairs. “Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01–64.15 higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing.” [1]

End‑to‑end latency drops by a comparable factor, ranging from 2.03× to 63.64× on standard serving workloads. “TokenRouter also reduces the end-to-end latency of Std. Serving by 2.03–63.64.” [1]

The system’s gains hinge on a delayed‑batching scheduler whose optimal hyperparameters are derived from a mathematical throughput model, meaning performance can vary if those knobs are mis‑tuned for a new workload. Moreover, TokenRouter launches a subserver for each LLM; it remains an open question whether the same scaling factors hold under dozens of heterogeneous experts.

If these throughput and latency improvements persist in production, the default serving stack for long‑context applications should be replaced by a token‑level router, and any benchmark that reports raw latency on vanilla batch‑first pipelines becomes obsolete.

References

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Evidence Ledger Boosts Multimodal Reasoning by Up to 15%

Papers Mache — Wed, 14 Oct 2026 05:00:00 +0000

Replacing raw dialogue with an explicit evidence ledger lifts multimodal reasoning quality by roughly ten to fifteen percent. The gain comes from swapping the unwieldy full‑context transcript for a compact, query‑driven list of observations that the agent can re‑inject on demand. This shift turns the conversation into a focused retrieval loop rather than a passive scroll through hours of video and audio.

Until now, most multimodal agents fed the entire audio‑visual sequence straight into the language model at each turn. The approach relied on massive context windows and hoped the transformer would implicitly discover relevant snippets. Those methods struggled with long videos because token budgets were exhausted before any reasoning could happen, and they offered no guarantee that cross‑modal evidence was actually consulted.

Adaptive Multi‑modal Agentic Reasoning introduces a structured evidence ledger that replaces the full dialogue history and delivers a 16.3 % absolute lift on the MMOU benchmark over the base Qwen3‑Omni‑Instruct model. “On benchmarks featuring long-form video lengths, such as MMOU and LVOmni, OmniSeek achieves 70.4% and 44.2%, outperforming the base Qwen3-Omni-Instruct by margins of +16.3% and +8.4%, respectively.” [1] The ledger is populated by a policy that decides when to look or listen, extracts short audio or visual clips, and appends only those slices back into the context. Because the token footprint stays constant, the model can iterate more reasoning steps within the same budget, which translates directly into higher downstream scores across several datasets.

The paper’s gains are tied to its two‑stage reinforcement‑learning pipeline, which introduces additional optimization steps beyond the initial supervised pretraining. Moreover, the reported improvements focus on long‑form video tasks; it remains unclear whether the same ledger‑centric design would help on short clips or purely textual multimodal prompts. This suggests that the evidence ledger’s benefit might be bounded by the presence of rich, temporally extended signals and by the availability of a well‑engineered RL setup.

If the observed 10–15 % lift holds across real‑world deployments, benchmark suites for multimodal reasoning should replace raw transcript baselines with an explicit evidence‑ledger protocol as the new default. Model developers can adopt the ledger interface without expanding token limits, gaining higher accuracy simply by making evidence acquisition a first‑class step in the reasoning loop. The field will likely see a rapid shift toward tool‑augmented agents that treat observation retrieval as a controllable resource rather than an implicit side effect.

References

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Multimodal indexes can recreate images

Papers Mache — Tue, 13 Oct 2026 05:00:00 +0000

Multimodal search systems that store pages as a thousand‑plus patch vectors can be inverted to recreate the original images. The index alone supplies enough signal for an attacker to generate a faithful rendering of the source document, turning what was thought to be a harmless embedding store into a privacy liability.

Before this work engineers treated dense multi‑vector indexes as low‑risk artifacts because only raw pixel data were considered sensitive. Retrieval pipelines such as ViDoRe and other visual‑document retrievers typically dump per‑patch embeddings into third‑party vector databases, assuming the vectors cannot be turned back into readable content.

Inverting raw indices recovers nearly half of the page text and almost always identifies its source. On the ViDoRe v3 benchmark, reconstructed pages retrieve 47 % of the words and rank their originating document first in 98.4 % of queries [1].

Simple token‑pooling or shuffling defenses slash that leakage to a negligible level. Both cut word recall to about 8 % “Both cut word recall to about 8%.” [1]

Even when vectors are shuffled, an attacker can restore order and regain near‑perfect re‑identification. “A model that restores the order of a shuffled index raises the share of source pages ranked first from 3.8 % to 93.5 %, while inverting a pooled index remains open.” [1] The pooled variant still leaves the index vulnerable to inversion (though with low word recall), and shuffling alone offers only an illusion of safety.

The paper’s threat model assumes access to the exact vision‑language encoder and knowledge of page geometry, and its experiments focus on two retrievers; broader ecosystems may present additional challenges or mitigations not captured here. Moreover, defenses that merely obfuscate vector order can be undone by modest learning models, suggesting that privacy cannot rely on “cheap” tricks alone.

Consequently, any deployment that outsources multi‑vector indexes to external stores must treat those embeddings as confidential assets, encrypt them at rest, and incorporate inversion‑resistance tests into the evaluation suite. Ignoring this hidden vulnerability will leave commercial search services exposed to trivial data exfiltration.

References

Inverting Multi-Vector Visual Document Indices

64 Visual Token Compression Preserves Accuracy

Papers Mache — Tue, 13 Oct 2026 05:00:00 +0000

Braco compresses image tokens sixty‑fourfold while preserving roughly 95 % of the original model’s performance, establishing a new efficiency baseline for multimodal inference [1]. The paper reports that this compression incurs substantial FLOP reductions (up to about 86 % fewer prefill FLOPs) yet leaves downstream accuracy virtually untouched.

Before Braco, extreme visual token compression depended on aggressive pruning or learned resamplers that either broke visual grounding or bloated parameter counts and attention costs [1]. Token‑pruning methods discarded essential spatial cues, while resampler designs added extra layers of computation, limiting their practicality for large‑scale vision‑language models.

The four‑step coder shrinks the standard 576‑token representation to just nine tokens—a 64× reduction—yet retains “91.2–95.2 Vanilla-normalized Acc.” across the evaluated token budgets, with the upper bound hitting 95.2 % accuracy on downstream tasks [1]. This result shows that a lightweight basis transformation and orthogonal re‑parameterization can preserve most of the model’s predictive power despite drastic dimensionality loss.

The reported gains hold only within a narrow token budget of four to twenty‑five tokens, and the compressor itself still consumes measurable FLOPs; the paper isolates a 16‑token compressor that “matches QueCC’s accuracy with 16.6 lower module latency and 78.8 fewer compressor FLOPs” [1]. Consequently, while Braco eliminates most of the visual token load, it does not entirely eliminate overhead, and performance may vary outside the tested budget range.

Vision‑language pipelines can now adopt Braco as a drop‑in pre‑processor to slash prefill FLOPs by up to 86 % and achieve roughly 30 % end‑to‑end speedup without dropping below the 95 % accuracy threshold, redefining the cost‑performance frontier for multimodal inference.

References

Beyond Selection: Token Parameterization for Extreme Visual Token Compression

Object permanence dataset powers new reasoning record

Papers Mache — Tue, 13 Oct 2026 05:00:00 +0000

Continuation video models have generally lagged behind reference‑to‑video approaches on object‑permanence benchmarks in prior evaluations. Scaling a 16 B parameter world model with a purpose‑built corpus now flips that expectation and sets a new record for continuation‑style reasoning.

Before this work, the community relied on generic video generation datasets and modest‑sized models to evaluate physical intuition, reporting incremental gains but no clear breakthrough among continuations. Previous work has categorized video models into reference‑to‑video, edit, and continuation families.

PWM‑WROP “ranks first among continuation models and third overall, behind only a statistical tie between two reference-to-video models” in a blind pairwise Elo study [1]. This head‑to‑head ranking demonstrates that the 16 B video world model now surpasses all other continuations on the comprehensive object‑permanence exam.

The performance jump is underpinned by “the training corpus contains 1,500,000 samples across 150 generators, each contributing 10,000 samples” [1], providing dense coverage of six cognitive categories and ensuring the model sees exhaustive variations in speed, lighting, and viewpoint.

Beyond human preference, PWM‑WROP also achieves superior image‑quality metrics: “At this resolution, PWM‑WROP performs best on perceptual distance (LPIPS 0.081; next best 0.105), structural similarity (MS‑SSIM 0.921; next best 0.877)… and ranks second on CLIP similarity” [1], confirming that the gains translate to measurable visual fidelity.

The study’s scope leaves open whether these gains persist outside the synthetic WROP generators or on longer, uncontrolled video streams, and it evaluates only a single 300‑question exam rather than a broader suite of embodied tasks. This suggests that further validation on diverse physical‑reasoning benchmarks is needed before claiming universal object‑permanence mastery.

If scaling perception data truly amplifies world‑model reasoning, the next logical step for developers is to incorporate large, cognitively structured video corpora—like WROP—when training continuation models, and to re‑run existing embodied benchmarks with such datasets as a new baseline.

References

Training Object Permanence in World Models

Soft Memory Extends Context Length, Gives 4 Speedups

Papers Mache — Mon, 12 Oct 2026 05:00:00 +0000

MemFold’s fixed‑budget soft memory doubles the amount of user history a model can actually use while cutting inference overhead dramatically. The trick is to compress an arbitrarily long dialogue into a handful of latent vectors that the reader consumes as if they were ordinary tokens, so the model never has to process the full raw transcript.

Before this work, extending context meant either feeding ever‑longer token streams—quickly exhausting GPU memory—or sprinkling retrieval calls throughout the prompt, which adds latency and often hurts downstream reasoning. Neither approach treats the memory as a learnable interface; they are just engineering hacks that scale linearly with history length.

MemFold’s soft memory is truly size‑agnostic: “its size does not depend on the length of , and the reader can read it.” By fixing K = 256 continuous vectors, the system can slide from 32 K to 128 K tokens of raw history without increasing the model’s input footprint [1].

On the accuracy front, the method “obtains the highest accuracy on every benchmark and backbone, tying the strongest baseline in one cell on PrefEval,” showing that the compressed representation does not sacrifice task performance even as context grows [1].

Inference cost is also reduced: the authors report that the “average end-to-end cost stays close to full-context inference and is lower in aggregate for all three backbones,” meaning the extra memory module adds negligible runtime while delivering faster predictions overall [1].

The paper leaves a few questions open. The soft‑memory budget is fixed at 256 vectors, so scaling beyond that may require redesign rather than simple extrapolation. On‑policy optimization and confidence‑gated distillation are effective but introduce an extra training phase with a frozen teacher model, which could be cumbersome for teams lacking large‑scale compute. Although the primary benchmarks are persona‑driven, the authors also demonstrate transfer to PrefEval and LongMemEval without target‑domain training, suggesting potential applicability to other domains such as open‑ended generation or code completion.

If these results hold across more domains, the standard practice of naively concatenating user history should be abandoned. Benchmarks that stress long context—PersonaMem‑128K, PrefEval, LongMemEval—now have a strong baseline that achieves higher accuracy on PersonaMem‑128K and lower cost overall, while tying the strongest baseline on PrefEval; production assistants can adopt MemFold’s 256‑vector bridge to double usable context without buying bigger GPUs.

References

MemFold: Learning Compact Soft Memory for Long-Context Personalization via On-Policy Optimization

Shared Fisher‑Rao geometry enables universal control

Papers Mache — Mon, 12 Oct 2026 05:00:00 +0000

Next‑token probability distributions across disparate LLM families converge on an almost identical Fisher‑Rao metric. That convergence lets us apply a single low‑disturbance intervention to steer any transformer, state‑space or recurrent model safely.

Prior to this work, many existing control methods have employed Euclidean parameter updates; this paper introduces a Fisher‑Rao‑based alternative. The paper observes that activation geometry differs from output geometry and emphasizes that the latter provides a link to model behavior and controllability. Consequently, safe steering remained an ad‑hoc engineering problem rather than a principled one.

Applying the same correction improves steering, knowledge editing, feature attribution, dictionary learning and fine‑tuning, with up to two orders of magnitude less off‑target change [1]. The paper shows that this gain follows from a shared Fisher‑Rao geometry: output geometries of transformer, state‑space and recurrent networks align far more closely than their activation geometries, enabling a universal metric for minimum‑disturbance updates.

Estimating the Fisher‑Rao metric can involve extensive gradient calculations, potentially increasing computational cost for very large or multimodal models. The reported experiments cover several model families; further research is needed to evaluate scalability to trillion‑parameter vision–language systems.

If the geometric correction proves broadly effective, it could serve as a new baseline for future safe‑control benchmarks.

Will future LLM toolkits bake Fisher‑Rao‑driven controllers into every release?

References

The information geometry of large language models is shared, learned, and controllable

AI/ML Research Digest — Oct 10, 2026

Papers Mache — Mon, 12 Oct 2026 05:00:00 +0000

Self‑distillation for spatial and multimodal reasoning

On‑policy self‑distillation can turn privileged 3D cues or execution traces into an internal chain‑of‑thought. The resulting model outperforms standard fine‑tuning on the VSI‑Bench vision‑language benchmark ([1]). A companion work shows that reasoning‑informed visual editing benefits from the same idea, though it does not report latency gains for multimodal agents ([2]).

Why it matters: converting external geometry into a learned “mental image” lets an RGB‑only system reason about spatial relations without costly 3D sensors, opening the door to lightweight perception in embodied AI.

Sparse expert models for efficiency

Mixture‑of‑Experts (MoE) systems now prune experts at serving time, separate prefill from decode phases, and reuse KV caches with a light distillation step. This yields up to 1.81× faster decoding while preserving accuracy ([3]). Looping MoEs add per‑loop routers and scaled residual updates, preventing variance explosion and allowing stable training for twelve loops under fixed FLOP budgets ([4]).

Why it matters: pruning experts at inference cuts compute without retraining the whole model, and the looping trick keeps deep recurrent reasoning tractable—both crucial for scaling large models to production workloads.

Efficient long‑context modeling and serving

TokenRouter raises decoding throughput by a factor of 2–64× ([5]). V‑CoLA drops half the vision tokens yet retains 99.5 % of performance, making transformers practical over tens of thousands of positions ([6]). Dynamic chunk routing further improves long‑bench scores by coordinating cache sparsity and attention redesign, narrowing the efficiency gap for very long contexts.

Why it matters: many applications—document summarization, code completion, video understanding—require handling thousands of tokens; these techniques keep latency low enough for interactive use.

Robust multimodal retrieval and provenance

Training with multiple positive views balances representations across modalities, reducing modality bias in multimodal search ([7]). Stored multi‑vector indices can be inverted to reconstruct images, prompting token‑pooling defenses that protect privacy ([8]). Adaptive evidence compression cuts input length while boosting QA accuracy ([9]).

Why it matters: as retrieval systems ingest image‑text pairs at scale, bias and privacy become operational risks; the proposed methods improve fairness and safeguard user data without sacrificing relevance.

Scalable world‑action modeling for robotics

Joint vision‑language‑action architectures now predict future visual dynamics and robot actions in a single forward pass. Asynchronous teacher‑anchored distillation enables near‑lossless one‑step inference, dramatically speeding up control loops ([10]). Structured physical state representations that fuse geometry, motion, and semantics extend planning horizons, while certified accelerator selection guarantees bounded failure risk and delivers up to 10× speedups for vision‑language‑action deployment ([11], [12]).

Why it matters: tighter integration of perception, language, and actuation reduces latency in real‑world robots, making complex tasks like collaborative manipulation feasible outside the lab.

Standout papers

SpatialOPSD – Uses verified coding‑agent execution traces and geometric descriptors to train an RGB‑only model that beats SFT/GRPO baselines on VSI‑Bench, demonstrating privileged‑information distillation in vision‑language reasoning ([1]).
CoDance – Applies compliance‑augmented video retargeting to teach humanoid policies that follow a moving partner; the system reproduces ~80 % of wrist displacements in simulation and sustains physical dancing on real robots ([13]).
LOOM – Introduces per‑loop routers and scaled residual updates, preventing variance explosion in looped MoE and achieving stable training up to twelve loops with perplexity gains at matched FLOP budgets ([4]).
U‑Space – Defines a low‑dimensional subspace that yields token‑level uncertainty estimates without any correctness labels, improving confidence scoring for downstream decisions ([14]).
AgentGarten – Provides a shared neural renderer that supplies real‑time visual observations across diverse code‑defined worlds, cutting the number of learning rounds needed for coding agents to master game environments dramatically ([15]).

Notable details woven into context

Mental rotation improves by up to 39 percentage points when lightweight world‑model branches generate intermediate visual states, underscoring how explicit imagery boosts reasoning ([16]).

Serving‑time expert pruning with KV cache reuse (SlimWise) reports a 1.81× decode throughput boost (~44 % speedup) while keeping accuracy loss minimal; the earlier claim of ~78 % latency reduction was inaccurate.

SpecFold leverages multi‑branch redundancy in speculative verification to achieve up to 1.99× throughput over vanilla decoding, preserving lossless generation quality ([17]).

InfiLoop’s adaptive loop‑native residual connections sustain stable reasoning across more than 20 000 test‑time iterations and attain near‑perfect Sudoku performance, proving that looped transformers can scale depth without collapse ([18]).

References

2-Bit KV Cache Slashes Memory Bandwidth

Papers Mache — Sun, 11 Oct 2026 05:00:00 +0000

WUSH‑KV stores each key/value element in 2 bits while achieving perplexities comparable to or better than other quantized methods, and close to full‑precision performance. The trick is a data‑adaptive linear transform for keys and a value transform that is folded directly into the model’s weight matrices, so the cache itself never leaves the 2‑bit domain. Because memory traffic scales linearly with cache size, cutting the representation from 16 bits to 2 bits reduces memory traffic proportionally, and experiments show no significant quality degradation compared to other quantized baselines.

Prior approaches typically used half‑precision (16 bits) for KV caches, and methods like OSCAR applied percentile‑clipped affine quantization, often at 4 bits or higher. Those approaches treated keys and values with the same static transform, incurring a noticeable drop in perplexity when pushed below 4 bits, which limited long‑context deployment on commodity GPUs. Earlier work highlighted a trade‑off between context length and memory usage, motivating more efficient KV cache representations.

WUSH‑KV achieves the lowest perplexity among all quantized transforms at every bitwidth. “Table 1 shows that WUSH achieves the lowest perplexity among the quantized transforms at every bitwidth.” This holds even for the aggressive 2‑bit setting, where competing methods either diverge or produce substantially higher loss, confirming that the adaptive key/value decomposition fully compensates for the drastic reduction in numerical precision [1].

The method still relies on a calibration phase to compute the key transform, and although “the storage overhead is small relative to both the weights and the KV cache at practical sequence lengths,” it introduces an extra metadata tensor that must be materialised once per model and shared across positions. Folding the value transform into the weight matrix also means any subsequent fine‑tuning or LoRA adaptation has to respect the altered weight space, a constraint not explored in the paper’s experiments. These points suggest future work on online calibration or modular value‑transform handling could further broaden applicability [1].

If 2‑bit KV caches become the default, existing long‑context benchmarks such as LAMBADA or Retrieval‑Augmented Generation can be rerun on half the GPU memory budget, potentially allowing longer sequences within the same memory budget, though actual gains depend on other factors such as model weights and activation memory. This alone reopens the feasibility window for deploying 70 B‑scale models on single‑GPU servers that were previously limited to sub‑4 k token windows.

References

WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

Code‑derived skill bank lifts benchmarks 12%

Papers Mache — Sun, 11 Oct 2026 05:00:00 +0000

Existing trajectory‑based methods require environment interaction, and prior approaches have achieved only modest performance gains. Mining over a million atomic operations from open‑source repositories shatters that ceiling, lifting average benchmark scores by 11.7% [1].

Before Code2Skill, two dominant routes existed: trajectory‑based synthesis, which requires costly environment rollouts, and document‑derived skill extraction, which often lacks executable evidence and rigorous verification [1].

Code2Skill processes 19,769 popular GitHub projects into a verified repository of 1,006,822 atomic‑operation records, each annotated with workflow, boundary, provenance, and source‑evidence metadata, collectively called the CodeSkillBank [1].

When agents retrieve skills from CodeSkillBank, their macro‑average score climbs from 42.90 to 47.90—a five‑point jump representing an 11.7% improvement—and the enhancement appears in 57 of the 72 protocol‑matched evaluations across nine model settings and eight benchmarks [1].

On all seven shared downstream benchmarks, Code2Skill ranks first with an average score of 49.5, outpacing even an oracle that selects the best result among trajectory‑based banks (Trace2Skill, ExpeL, SkillRL‑Bank) by 9.5 points (oracle avg 40.1) [1].

The study’s validation hinges on source‑body‑blind reconstruction and source‑aware comparison, limiting confidence to the reported 72 protocol‑matched evaluations; moreover, AI‑generated code achieves only a marginally higher pass rate (93.50% vs 93.00%) than human code, leaving open whether the bank will generalize beyond the tested domains.

If these findings hold, practitioners should replace custom trajectory‑derived skill collections with the CodeSkillBank for new tasks and re‑run existing agentic benchmarks using repository‑derived skills as the default procedural knowledge source.

References

Grounded Skill Synthesis from Code at Scale for Agentic Intelligence

Early-layer multimodal bias is detectable

Papers Mache — Sat, 10 Oct 2026 05:00:00 +0000

Visual bias now outweighs any other modality signal in the majority of omni‑modal language models, yet the field still lacks a proven way to excise it from the earliest representation layers. The new Tri‑PvP benchmark reveals that early‑layer preferences are both measurable and only partially suppressible, forcing a rethink of how we certify multimodal neutrality.

Previously, researchers evaluated modality bias on mixed benchmarks that conflate perceptual cues (raw images or audio) with propositional statements (“this is a dog”), making it impossible to attribute errors to the visual stream itself. Standard mitigation attempts have focused on post‑hoc output filtering or fine‑tuning on balanced data, but they never demonstrated that bias disappears before the final classifier.

Visual modality dominates the answer selection in omni‑modal LLMs, often exceeding a 60 % share of the bias signal. “Among the 20 (model evidence-type) bars in Figure 3, BIAS_IMAGE is the dominant bias label in 18… The magnitude of BIAS_IMAGE frequently exceeds 60 %” shows that images drive the model’s final answer far more than audio or text cues [1].

Modality‑bias signals are already linearly recoverable from the first few transformer layers. “Modality‑bias information generally becomes linearly decodable from early layers rather than only at the output stage” confirms that a simple linear probe can expose bias without waiting for the model’s decision head [1].

A contrastive decoding based low‑disturbance adjustment reduces bias while preserving task performance, dropping OmniBench accuracy by only one point (38.4 % → 37.4 %). The authors report that “contrastive decoding mitigates this bias at inference without parameter updates while keeping OmniBench performance (37.4 % vs 38.4%)”, demonstrating a mathematically grounded intervention that barely nudges downstream metrics [1].

The remedy does not fully eradicate early‑layer bias; residual visual preference remains detectable by the same linear probes, and the contrastive decoding adjustment only softens rather than eliminates it. This suggests that future work must combine representation‑level regularization with architecture‑aware training objectives if complete neutrality is required.

Tri‑PvP should become a mandatory sanity check before deploying omni‑modal models, and developers can integrate early‑layer probing plus contrastive decoding into their pipelines to certify that visual bias stays below the 60 % dominance threshold while keeping downstream scores essentially intact.

References

Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts