DEV Community: Papers Mache

Optimal Transport Converts Dense Layers to Sparse Experts

Papers Mache — Mon, 15 Jun 2026 05:00:00 +0000

Differentiable optimal transport rewrites a dense feed‑forward layer into a balanced mixture‑of‑experts without any hand‑crafted routing or expert sizing. The paper treats neuron assignment as a transport problem and solves it with Sinkhorn‑Knopp iterations, so the conversion is fully differentiable and end‑to‑end trainable.

Before DOT‑MoE, turning a pretrained dense model into a sparse expert system required heuristic clustering of neurons or random splits, and training MoEs from scratch was notoriously unstable. Those approaches offered no principled way to guarantee expert capacity or to jointly learn token routing.

DOT‑MoE “retaining 90% of the original dense model’s performance while reducing active parameters by 50%” across multiple architectures and benchmarks, which is the most compelling evidence that the transport‑based refactor preserves quality at half the compute cost [1].

In addition, “DOT‑MoE achieves the lowest perplexity (7.99) among all existing methods, outperforming the state‑of‑the‑art DISP‑LLM (9.84) by a substantial margin,” demonstrating that the OT formulation does more than match pruning baselines—it actually improves predictive fidelity [1].

The method still hinges on differentiable Sinkhorn iterations, which add overhead to the conversion pipeline and may limit scalability to extremely large models. Moreover, the evaluation focuses on feed‑forward networks; extending the transport‑based decomposition to attention heads or other architectural components remains unexplored.

If the reported gains hold broadly, the default workflow for model compression can shift from ad‑hoc expert design to a systematic DOT‑MoE conversion, letting engineers halve active parameters while staying within a 10% performance envelope. This eliminates a major source of manual tuning and opens a scalable path to sparse inference for large‑scale pretrained models.

References

DOT-MoE: Differentiable Optimal Transport for MoEfication

AI/ML Research Digest — Jun 13, 2026

Papers Mache — Mon, 15 Jun 2026 05:00:00 +0000

Infrastructure and inference optimization for scale

Sparse‑attention mechanisms cut the quadratic cost of self‑attention, making longer contexts feasible [1].

Intra‑model routing lets a decoder run speculative steps ahead of the true sequence, reducing latency without hurting quality [2].

PACI keeps weight updates locally, so pipeline stages never wait on each other; this removes bubbles and yields up to 1.69× faster time‑to‑accuracy at unchanged memory use [3].

Together these tricks shrink compute budgets and open the door to real‑time LLM services at larger scales.

Agentic reasoning and environment interaction

SpatialClaw replaces fixed API calls with a persistent Python kernel that VLMs can query repeatedly, enabling iterative construction of geometric primitives. The change lifts performance on 3D/4D reasoning benchmarks dramatically [4].

Dynamic benchmarks now simulate evolving software stacks and social settings, forcing agents to plan over time rather than react to a single prompt [5].

These moves push agents from static question‑answering toward genuine tool use and continual decision making.

Stabilizing RL and distillation for reasoning

Replacing hard gradient clipping with smooth divergence regularization stabilizes policy updates, leading to higher success rates in reasoning‑heavy RL tasks [6].

Recursive composition of verifiable environments lets distilled models inherit generalization abilities from deeper hierarchies, scaling reasoning performance without extra data [7].

SpatialClaw’s stateful VLM interaction

The system embeds a live Python interpreter inside a vision‑language model, so the model can call, modify, and re‑call code as part of a single inference pass. This stateful loop reduces error propagation and yields large gains on spatial reasoning suites that require multi‑step geometry manipulation [4].

MoVerse real‑time video generation

MoVerse expands a 360° panorama into a continuous scene using a persistent 3D Gaussian scaffold. The scaffold reuses geometry across frames, allowing the model to synthesize video at 8 FPS on a consumer GPU—a speed previously limited to offline pipelines [8].

PACI pipeline optimization

By accumulating gradients locally and enforcing a bound on weight inconsistency, PACI eliminates idle time between pipeline stages. Experiments show up to 1.69× faster convergence to target accuracy while keeping the memory footprint identical to a standard pipeline [3].

Critical failure in audio editing accuracy

The MMAE benchmark measures exact‑match edits on complex audio tasks. Current systems achieve less than 5 % exact match, exposing a severe gap between research claims and practical audio manipulation capability [9].

Denoising step reduction in world models

Lip Forcing collapses the diffusion process to two denoising steps, raising inference speed to 31 FPS and preserving visual fidelity [10].

Next Forcing improves training dynamics with multi‑chunk predictions; it speeds inference but does not depend on the two‑step schedule, offering a separate path to efficiency [11].

Harness design impact on SWE‑Bench

Evaluations with Claw‑SWE‑Bench show that the structure of the agent harness—how tools are exposed and state is managed—often explains more of the pass‑rate rise than changes to the underlying language model itself [12].

These findings collectively show where the field is extracting more performance: tighter compute kernels, tighter integration with mutable tools, and tighter control over training dynamics. Each advance reduces a concrete bottleneck—memory, latency, or instability—making large‑scale, interactive AI systems more practical.

References

90% Less Memory Enables Infinite Video Generation

Papers Mache — Sun, 14 Jun 2026 05:00:00 +0000

A shared low‑rank cache slashes the memory footprint of autoregressive video diffusion by more than nine‑tenths while still permitting arbitrarily long rollouts.

Before these contributions, streaming video diffusion relied on a per‑head key‑value cache that grows linearly with the temporal window, forcing practitioners to cap video length or provision prohibitively large GPUs.

VideoMLA reduces per‑token KV cache memory by 92.7 % while preserving compatibility with standard chunk‑causal generation. The paper shows that this compression does not hurt visual fidelity; on VBench the method matches short‑horizon baselines and even secures the best long‑horizon score, while Table 3 reports the highest throughput and lowest latency among chunk‑wise autoregressive models, translating to a 1.23× speedup on a single B200 [1].

Echo‑Infinity achieves state‑of‑the‑art performance and, to our knowledge, demonstrates promising 24‑hour (>1.3 M frames) real‑time rollouts for the first time, suggesting a practical path toward infinite video generation. In practice the system runs at 18.5 FPS on a single NVIDIA H100 and incurs only a 10.6 % throughput overhead compared with a memory‑free baseline, proving that constant‑cost, evolving memory can sustain day‑scale generation without exploding resource use [2].

These results leave open several questions. VideoMLA’s latent dimension must be chosen manually, and although the bottleneck rank appears sufficient for the evaluated datasets, it is unclear how the approach scales to higher‑resolution or multi‑modal streams. Echo‑Infinity’s learnable memory, while effective up to a million frames, has not been stress‑tested on content that requires very long‑range narrative coherence, and the unified RoPE recipe may still encounter extrapolation limits on unseen motion dynamics.

If the combined system lives up to the reported numbers, developers can abandon the practice of over‑provisioning GPU memory for long video generation. Benchmarks that previously capped at a few seconds should be rerun with the minute‑scale configs shipped in the VideoMLA and Echo‑Infinity repositories, and production pipelines can target hour‑ or day‑scale output on a single H100 without redesigning the hardware stack.

References

Linear Ensembles Can Erase LLM Watermarks

Papers Mache — Sat, 13 Jun 2026 05:00:00 +0000

Watermarking schemes that embed distributional perturbations into LLM outputs are effectively broken by linear ensembles of a few independently trained models. The intuition behind most provenance tools is that a tiny bias introduced at generation time survives any downstream processing, making it detectable by a statistical test. In practice, that assumption collapses as soon as an application draws from more than one provider. The result is a hidden amplifier for hallucination‑free text that simultaneously wipes out the very signal we trusted for attribution.

Before this work, the community treated watermark perturbations as immutable once rendered into the probability distribution. Methods such as the z‑score detector or the binary‑mask classifier were evaluated on single‑model generations and consistently reported true‑positive rates above 90 % at a 5 % false‑positive budget. The detection threshold of 4 on the z‑score became the de facto benchmark for a “detectable” watermark, and no prior study had examined how ensemble decoding would interact with that statistic.

A linear ensemble of just three to five models eradicates the watermark signal in practice. “Empirically, simply averaging 3-5 models cancels out these perturbations.” [1] The cancellation works because each provider injects an independent perturbation; averaging restores the underlying, unwatermarked distribution up to a second‑order error term. No extra training or fine‑tuning is required—plain probability averaging suffices.

Averaging three models drives detection z‑scores below the standard threshold of 4, cuts true‑positive rate at 5 % false‑positive rate to under 50 %, and simultaneously boosts text quality by 27.5 % while running six times faster than the strongest baseline. “Experiments across six watermarking schemes and three LLMs show that averaging across 3 models suppresses detection z-scores from 5-300 to below 2 (below the detection threshold of 4) and reduces TPR@5%FPR to below 50%, while improving quality by 27.5% and running 6 faster than the best baseline on the long sequence generation.” [1] The authors also introduce WASH, a lightweight pipeline that aligns vocabularies and tokenisers across heterogeneous models, making the ensemble practical even when the constituent LLMs differ in architecture.

The study leaves open several important questions. It evaluates only six watermarking schemes and three LLM families, so it is unclear whether more sophisticated, non‑linear perturbations would survive averaging. WASH mitigates vocabulary mismatches, yet scaling to dozens of providers may introduce latency or memory bottlenecks not captured in the reported six‑fold speedup. Moreover, the analysis assumes that perturbations are statistically independent; coordinated watermarking could deliberately inject correlated noise to resist cancellation, but the feasibility of such industry‑wide coordination remains speculative.

Robust provenance tracking can no longer rely on simple distributional watermarks unless model providers adopt a universally shared signing key or shift to cryptographic signatures that survive ensemble decoding. The immediate consequence is that any service that aggregates outputs from multiple LLM APIs must treat watermark‑based detection as unreliable and consider alternative attribution mechanisms.

References

Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs

Benchmarks Evaluate Memory Quality and Adaptive Planning in LLM Agents

Papers Mache — Fri, 12 Jun 2026 05:00:00 +0000

Newly released test suites expose two blind spots that have long lurked behind headline scores: how faithfully an LLM‑augmented agent preserves useful information across millions of tokens, and whether it can reshuffle its plan when hidden rules surface mid‑game. The field has been chasing end‑task success while silently assuming that memory and planning stay reliable under the hood.

Prior memory‑policy work trained agents with only outcome‑level rewards, which “introduce a severe credit assignment problem: it fails to localize intermediate memory degradation and provides no explicit supervision to suppress noise accumulation during recursive summarization.” [1] This leaves agents blind to the gradual erosion of task‑relevant facts. Likewise, planning benchmarks have treated constraints as a static checklist, ignoring the reality that world and user rules often emerge only after an initial proposal.

MMPO tackles the memory blind spot by attaching a self‑supervised Belief Entropy signal to each summary, and the authors report that “experiments show that MMPO consistently outperforms existing methods on diverse long‑horizon tasks, maintaining 97.1% performance even when scaled to 1.75M‑token contexts.” [1]

AdaPlanBench flips the planning script: agents must propose a plan, receive feedback about hidden violations, and then revise. Under this pressure “experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy.” [2]

The benchmark also quantifies the scaling of difficulty: “performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness.” [2] This mirrors the authors’ observation that “User‑Constraint Only is consistently harder than World‑Constraint Only, while Both Constraints is the most demanding setting.” [2]

A third suite probes stance simulation, revealing how easily a model’s inferred opinion can be nudged. The multimodal revision that injects a meme “produces an average directional shift of +49.3%, compared with +44.8% for the add strategy and –4% for the paraphrase control.” [3] Even purely textual edits can swing simulated stance by a large margin, with the add strategy producing an average directional shift of +44.8%.

These results leave several questions open. MMPO’s belief‑entropy proxy is still a heuristic; it is unclear whether it scales beyond the reported 1.75 M tokens or how it interacts with external knowledge sources. AdaPlanBench, while extensive, confines itself to 307 household scenarios, so its conclusions may not transfer to industrial or high‑risk domains. The stance‑shift audit tests simulated users, not real people, so the measured volatility may over‑ or under‑estimate true societal impact. A natural next step is to ask whether a single agent architecture can jointly minimize belief entropy while tracking an evolving constraint graph, and how such a system would behave under adversarial context injections.

If these benchmarks are taken as the new reliability baseline, the immediate effect will be a reshuffling of agent leaderboards: any model that cannot keep above ~97% performance in the MMPO benchmark (which evaluates up to 1.75 M‑token contexts) or achieve the best reported adaptive‑planning accuracy of 67.75% on AdaPlanBench should be considered for further safety evaluation. Re‑evaluating existing agents on MMPO, AdaPlanBench, and the stance‑revision suite will surface latent failure modes that current safety glosses overlook.

References

AI/ML Research Digest — Jun 06, 2026

Papers Mache — Thu, 11 Jun 2026 05:00:00 +0000

Scaling Long‑Horizon Video Generation

Recent work replaces sliding‑window attention with memory‑centric designs. A learnable evolving memory compresses the entire history at constant cost, enabling real‑time infinite rollouts [1]. Low‑rank KV caches achieve roughly a 90 % memory reduction while preserving visual fidelity [2]. A complementary low‑rank latent with 3‑D RoPE further shrinks per‑head caches without quality loss [2]. Together these tricks make video models scalable to lengths that were previously infeasible and cut GPU memory footprints dramatically [3].

Stabilizing On‑Policy Distillation for LLMs

Distilling policies from RL‑trained LLMs suffers from high KL variance and distribution drift. Aligning hidden representations reduces this variance, yielding smoother student updates [4]. Trust‑region constraints enforce a bounded policy shift, preventing collapse during distillation [5]. Two orthogonal tricks—logit‑free chunk verification and self‑distilled policy gradients—provide additional stability for RL fine‑tuning [6], [7].

Agentic Reliability and Safety Frameworks

Benchmarks now probe LLM agents on extended reasoning tasks that require memory‑policy optimization and plan adaptation under shifting constraints [8], [9]. Counterfactual context revision audits reveal how agents change stance when presented with altered evidence, exposing hidden failure modes [10]. Self‑evolving prompt agents automatically refine system prompts, improving alignment without human intervention [11].

Standout Papers

VideoMLA: Efficient KV Cache Reduction – Introduces a shared low‑rank latent and 3‑D RoPE to replace per‑head KV caches, cutting memory use by over 90 % while keeping generation quality intact [2].

Echo‑Infinity: Infinite Video Rollouts – Proposes a learnable evolving memory that stores compressed history at fixed cost, allowing autoregressive video generation to run indefinitely in real time [1].

Hamilton‑Jacobi Evolution for NN Dynamics – Shows that gradient steps follow a viscous Hamilton‑Jacobi PDE, unifying ResNets, Transformers, and RNNs under a single mathematical lens [12].

Other Notable Findings

Distributional watermarking is fragile – Linear ensembles can erase watermark perturbations, indicating that current watermarking schemes may be easy to bypass [13].

Speculative decoding throughput – A pipeline‑parallel speculative decoding framework processes multiple tokens per step, delivering a theoretical speedup far beyond prior baselines [14].

Sparse MoE via optimal transport – Differentiable optimal transport converts dense feed‑forward layers into sparse experts, offering a principled recipe for Mixture‑of‑Experts specialization [15].

These results tighten the gap between research prototypes and production‑ready systems. Memory‑efficient video models can now power longer streams, stable distillation makes LLM policy fine‑tuning safer, and new safety benchmarks expose hidden brittleness in agentic behavior. The accompanying techniques—low‑rank caches, evolving memories, trust‑region distillation, and optimal‑transport MoEs—provide concrete tools for engineers building the next generation of generative and autonomous AI.

References

Raw waveform diffusion matches autoencoder quality

Papers Mache — Fri, 05 Jun 2026 05:00:00 +0000

Raw waveform diffusion can now deliver the same—or even higher—audio fidelity that autoencoder‑based pipelines have long claimed as their exclusive domain. By discarding any latent compression step, WavFlow produces samples that listeners cannot distinguish from those generated by established latent diffusion models.

For years the community has built audio generators on top of semantic‑acoustic autoencoders, a strategy epitomized by Stable Audio 3, which first compresses waveforms into a compact latent space before applying diffusion. This two‑stage design has been justified as necessary to tame the high dimensionality of raw audio and to keep training tractable.

WavFlow’s VGGSound results prove that a pure‑waveform approach is competitive: “Experimental results show that WavFlow achieves competitive results on the video‑to‑audio benchmark VGGSound (FD 59.98, IS 17.40, DeSync 0.44) … matching or exceeding the performance of established latent‑based methods” [1]. The FD score of 59.98 sits squarely within the range of top latent models, while the IS and DeSync numbers confirm comparable perceptual quality and temporal alignment.

On the text‑to‑audio front, WavFlow even sets new records: “Our model attains the best FD (10.63) and IS (12.62) reported to date, rivaling dedicated T2A systems” [1]. Those figures surpass the best published latent‑based scores on AudioCaps, demonstrating that raw‑space diffusion does not sacrifice semantic relevance for fidelity.

When the architecture is scaled to the 16 kHz “L” variant, the gap widens: “Scaling to WavFlow‑L‑16kHz yields consistent improvements, surpassing MMAudio‑L‑44.1kHz in distributional fidelity (FD: 59.98 vs. 60.60) while matching its performance in perceptual and alignment metrics (IS 17.40, DeSync 0.44)” [1]. This head‑to‑head comparison shows that raw‑waveform diffusion can outpace a leading latent system on the most demanding distributional metric while staying on par elsewhere.

The study’s scope still leaves open several practical concerns. Training required five million video‑text‑audio triplets and a custom amplitude‑lifting scheme to keep optimization stable, implying a higher data and compute budget than many latent pipelines. Moreover, the best results are reported at 16 kHz, whereas many production scenarios demand 44.1 kHz or higher fidelity, raising the question of whether the same gains will hold at those rates.

If these results generalize, the default assumption that audio synthesis must pass through an encoder‑decoder bottleneck should be revisited. Future benchmark suites ought to include a raw‑waveform diffusion baseline, and engineers can consider dropping the autoencoder stage altogether when building multimodal generation systems.

References

WavFlow: Audio Generation in Waveform Space

Agents still fail 38% of real CLI tasks

Papers Mache — Thu, 04 Jun 2026 05:00:00 +0000

State‑of‑the‑art agents succeed on just 62.5 % of authentic command‑line workflows. The TerminalWorld benchmark, built from tens of thousands of real developer recordings, evaluates agents in a zero‑shot setting on tasks that span simple one‑liners to multi‑step deployment pipelines. That success ceiling shatters the prevailing belief that large language models can already replace shell scripts for everyday use.

Existing evaluations have leaned on hand‑crafted command suites that capture only a narrow slice of developer activity. Benchmarks such as Terminal‑Bench present curated queries and score agents on idealized subtasks, but they miss the messy, iterative patterns seen in production terminals. Consequently, reported numbers have long over‑estimated practical reliability.

The best evaluated agent reaches a max pass rate of only 62.5 % — “Comprehensive benchmarking on TerminalWorld‑Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%.” [1] This figure comes from a fully automated pipeline that reverse‑engineers tasks from 80 k+ asciinema recordings, ensuring the evaluation mirrors what developers actually type.

Even the strongest model fails on more than a third of the tasks, with overall pass rates ranging between 49.0 % and 62.5 % and an average of 54.8 % — “Overall, all evaluated models achieve modest pass rates (49.0%–62.5%, avg. 54.8%), with even the best model (i.e., Claude Opus 4.7) failing on over one‑third of the tasks, confirming that the real‑world terminal tasks in TerminalWorld pose a substantial challenge to frontier LLMs.” [1] The gap persists across both small‑scale utilities and long‑running build scripts.

Agents typically reach the correct outcome via a different set of commands than the human practitioner used, with a median overlap of only 21.4 % — “The median overlap is only 21.4%, meaning agents typically reach the correct outcome via a different set of commands than the human practitioner used.” [1] This low overlap signals brittle tool use and limited error‑recovery strategies, as models opt for shortcuts that happen to succeed rather than faithfully reproducing expert workflows.

The benchmark measures only zero‑shot performance using the recorded terminal sessions, ignoring iterative prompting, tool‑specific fine‑tuning, or external memory that a real assistant could exploit. Moreover, the tasks, while diverse, are still bounded by the recordings the engine could parse, leaving open how agents handle completely novel commands or privileged operations. These constraints suggest the reported 62.5 % ceiling is a lower bound on what could be achieved with richer interaction loops.

Assuming an AI assistant can fully automate routine CLI chores is premature; teams should continue to treat agents as aides, not replacements, and re‑evaluate new models against TerminalWorld before deployment. Will the next generation finally break the 70 % barrier, or is CLI automation a harder problem than we thought?

References

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Generative models now output simulation‑ready 3D assets

Papers Mache — Wed, 03 Jun 2026 05:00:00 +0000

Vision‑language transformers paired with geometric primitives now output metric‑scale, simulation‑ready 3D assets. PhysX‑Omni demonstrates that a single autoregressive transformer can ingest image, text, and spatial priors and directly emit meshes, materials, and physics descriptors that import straight into simulation engines that support URDF/XML formats.

Before this work, most generative 3D pipelines either ignored physical properties or were constrained to a single object class—rigid, deformable, or articulated—forcing developers to stitch together separate tools for geometry and physics. The authors explicitly point out this fragmentation as the motivation for a unified approach [1].

"On PhysXVerse, our method achieves a PSNR of 21.52, CD of 2.95, and F-score of 91.28, substantially surpassing the previous best results." These figures indicate that the generated geometry is not only visually faithful but also quantitatively closer to ground‑truth scans than any prior model evaluated on the same dataset.

"On PhysXVerse, the absolute scale error is reduced from 309.31 in PhysXGen and 298.19 in PhysX-Anything to only 2.79 in PhysX-Omni." This two‑order‑of‑magnitude improvement means the objects come out at real‑world dimensions, eliminating the tedious manual rescaling step that has long plagued simulation pipelines.

The released codebase ships an inference script that writes URDF and XML files, so a single forward pass yields a complete physics‑enabled asset ready for drop‑in use with any simulator that understands these standards. while the pipeline aims to produce ready‑to‑use assets, some post‑hoc rigging or parameter tuning may still be necessary depending on the target simulator, and the pipeline runs end‑to‑end from prompt to asset.

The paper does not report generation latency or memory consumption, leaving open whether PhysX‑Omni can serve interactive authoring tools where instant feedback is crucial. Moreover, PhysXVerse currently spans indoor and outdoor categories, covering a range of object types, though its robustness on highly articulated creatures or large‑scale terrains has yet to be thoroughly evaluated.

If the reported fidelity and scale accuracy translate to production workloads, studios can replace hours of manual modeling and physics authoring with a single text prompt, fundamentally reshaping asset pipelines that have long treated geometry and simulation as separate stages.

References

PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects

Agent compute drops substantially with online skill distillation and graph‑guided knowledge

Papers Mache — Wed, 03 Jun 2026 05:00:00 +0000

Online skill distillation and graph‑guided knowledge substantially reduce the compute bill of LLM agents while keeping success rates competitive. These reductions could enable shifting agents from cloud‑only services to modest on‑device hardware.

Earlier web and mobile agents leaned on heavyweight tricks: multi‑rollout searches, separate verifier passes, and stacks of specialist vision‑language models that balloon token counts and memory footprints. Such pipelines achieved high task‑success but only by paying huge inference costs.

PANDO reaches a 58.3 % success rate on the full 910 VisualWebArena suite while using 115 K tokens per task, and it does so “while using 58% fewer tokens than SGV and 61% fewer than WALT, with no pre‑evaluation discovery budget” [1]. This headline figure shows that a single‑rollout, online distillation loop can surpass strong baselines without the token overhead that traditionally powers web agents.

Beyond success, PANDO also delivers the best intrinsic efficiency: it records the lowest Action Repetition Rate (9.1 %), the lowest Step Overhead Ratio (1.8), and the highest prompt‑cache utilization (72.4 %) among automated methods [1]. These metrics translate directly into reduced memory churn and faster per‑step latency.

UI‑KOBE lifts a lightweight 4 B‑parameter backbone to 70.7 % success on mobile GUI tasks, “substantially outperforming the same backbone model without graph guidance, which achieves 58.6 %” [2]. The gain comes from re‑using an app‑specific knowledge graph that steers the agent through local decisions instead of forcing a monolithic end‑to‑end planner.

The results leave open two practical questions. PANDO still exhibits a 9.1 % action‑repetition rate and a step‑overhead ratio of 1.8, meaning some inefficiency remains even after distillation. UI‑KOBE “does introduce an exploration cost, averaging $6.2 and 6.4 hours per app” [2], and its graph is only as good as the explored UI surface, limiting portability to unseen applications. Moreover, the formulation “reduces GUI task execution to a sequence of guided local decisions, significantly lowering the reasoning burden on small models” [2], which may not hold for highly dynamic or multimodal interfaces.

If these savings hold across domains, the community should start evaluating agents under strict token‑budget constraints rather than raw success alone. Revisiting VisualWebArena and MobileGUI benchmarks with a ≤ 120 K token ceiling per task would surface designs that truly scale to on‑device deployment.

References

Verifiable rewards improve LLM math accuracy

Papers Mache — Tue, 02 Jun 2026 05:00:00 +0000

RL from verifiable rewards now beats GRPO baselines by a comfortable margin, and the advantage comes from assigning credit at far finer granularity than whole‑response scores. By turning verification into token‑ and subproblem‑level signals, the newest methods extract learning from progress that would otherwise be discarded.

Before these works, reinforcement learning for reasoning relied on a single scalar reward per generated answer. GRPO and similar RL‑HF pipelines treated the whole response as the unit of credit, which made credit assignment noisy and left hard problems stuck in “gradient dead zones.” No mechanism existed to reward partial solves or to isolate the effect of a single token on the final verdict.

DelTA’s discriminative token credit assignment reshapes the RL update into a linear discriminator over token‑gradient vectors, amplifying side‑specific directions while suppressing shared noise. “DelTA consistently outperforms all same‑scale RL baselines on both Qwen3-8B-Base and Qwen3-14B-Base, achieving the best result on every benchmark and the highest average score at both scales” [1]. The paper reports average gains of +3.26 points for the 8 B model and +2.62 points for the 14 B model across seven math suites, turning a marginal RL improvement into a systematic boost.

SCRL converts a reasoning chain into verifiable subproblems and normalizes rewards at each position, so that the longest consecutively solved subproblem sequence determines the advantage. “The gain is especially clear on Qwen3-4B, where SCRL reaches an average score of 35.0%, improving over the second‑best baseline QuestA (32.0%) by 3.0 points and over vanilla GRPO (30.9%) by 4.1 points” [2]. Across the same seven benchmarks the method adds +4.1 average points for the 4 B model and +1.9 points for the 14 B model, and on hard AIME/IMO sets it lifts pass@1 by +3.7 points and pass@64 by +4.6 points.

RELEX shows that RLVR trajectories live in an almost one‑dimensional subspace, making most of the performance gain capturable by a rank‑1 projection that grows near‑linearly with training steps. “Specifically, we find that the majority of downstream performance gains are captured by a rank‑1 approximation of the parameter deltas, where the magnitude of this projection evolves near‑linearly with training steps” [3]. Extrapolating from only 15–20 % of the usual RLVR steps, RELEX matches GRPO on Qwen2.5‑Math‑1.5B (71.6 % vs 71.5 %) and slightly exceeds it on Qwen3‑4B‑Base (85.6 % vs 85.5 %), but falls short on Qwen3‑8B‑Base (87.4 % vs 88.5 %) on the in‑domain MATH benchmark, while also beating RLVR on five out‑of‑domain tests.

The three papers leave open questions about scalability and universality. DelTA’s centroid reweighting still risks being dominated by high‑frequency formatting tokens, so its discriminative edge may shrink on longer, more heterogeneous sequences. SCRL depends on high‑quality reference chains; constructing those for novel domains could re‑introduce costly annotation. RELEX assumes a linear, rank‑1 evolution that has only been demonstrated on math‑oriented backbones; whether the same simplicity holds for dialog or retrieval‑augmented models remains to be seen.

If fine‑grained verification truly captures the lion’s share of RLVR learning, developers should replace monolithic reward wrappers with token‑ or subproblem‑level credit pipelines as the new default. Moreover, RELEX’s cheap extrapolation suggests that, by training only a short RLVR run and then extrapolating, comparable checkpoints can be obtained at a fraction of the compute budget, potentially enabling a rapid rollout of more reliable reasoning across deployed LLM services. The next wave of RL‑enhanced models will likely be judged not by how many steps they train, but by how sharply they can slice the verification signal.

References

ScientistOne achieves perfect citation verification

Papers Mache — Mon, 01 Jun 2026 19:13:24 +0000

Chain‑of‑evidence pipelines erase the citation hallucination problem that has long plagued autonomous research agents. By insisting that every factual claim be anchored to a concrete source, the system forces the generator to expose its evidence at generation time, making fabricated references impossible to hide. In practice this means a literature‑review bot can now be trusted to point you to the exact paper it is quoting, instead of inventing a bibliography entry that looks plausible but does not exist.

Before ScientistOne, every baseline system exhibited at least one verifiability failure, with hallucinated reference rates soaring to 21 % and score verification succeeding in as few as 42 % of generated papers. The gap was not a fringe bug; it was a systemic property of the current research‑assistant paradigm, where surface‑level fluency masked deep inconsistencies. Those numbers made any downstream reliance on automatically written surveys a gamble.

ScientistOne eliminates the hallucination risk entirely, reporting “zero hallucinated references (0/337 bibliography entries)” across its entire evaluation suite [1]. The framework constructs an evidence chain for each citation, ensuring that each claim can be traced to its source as required by the Chain‑of‑Evidence framework. The audit checks the evidence chain, and any discrepancy would cause the reference verification to fail.

Score verification becomes a certainty: “perfect score verification (12/12)” means every claimed result reproduces exactly under independent re‑evaluation [1]. The pipeline reruns the reported experiment, compares the numeric outcome to the manuscript, and only admits the result if the difference lies within a negligible tolerance. This eliminates the classic “the numbers look right but can’t be reproduced” loophole that has rendered many AI‑generated papers useless.

Method‑code alignment also reaches the top of the leaderboard, with ScientistOne attaining “the highest method–code alignment (14/15)” while matching or exceeding human expert performance on all five frontier tasks [1]. Each algorithmic description is paired with the exact source code snippet that implements it, and a static analysis check confirms that the signature and hyper‑parameters line up. The result is a paper where the methods section is no longer a prose summary but a verifiable map to runnable artifacts.

The triumphs are bounded by the scope of the study: 75 papers covering five research tasks and a handful of extensions to medical imaging, fine‑grained recognition, 3D perception, and language modeling. Although the evidence chain held up across these domains, it remains an open question whether the same zero‑failure rate will persist on large‑scale, multi‑disciplinary corpora or under adversarial prompt engineering. Moreover, the audit relies on deterministic reproducibility of downstream experiments, which can be brittle when external services change.

If the zero‑hallucination claim holds under broader scrutiny, citation verification should become a mandatory step in any automated scientific‑writing pipeline. Existing benchmarks that score only linguistic quality must be augmented with a verifiability metric, and developers of literature‑review assistants ought to embed a chain‑of‑evidence module by default. In short, the paper‑writing landscape will shift from “does it read well?” to “can every claim be traced and reproduced.”

References

ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence