DEV Community: Papers Mache

Autoregressive retriever training raises BEIR scores

Papers Mache — Mon, 06 Jul 2026 05:00:00 +0000

Contrastive training still dominates dense retrieval despite its data hunger. DREAM takes a different approach by deriving supervision directly from the next‑token prediction objective of a frozen LLM rather than from pre‑constructed positive and negative pairs[1].

Most dense retrievers such as DPR, ANCE, or ColBERT rely on contrastive loss built from mined query‑document pairs, often requiring billions of examples and expensive mining pipelines.

Training on three backbone sizes—0.5 B, 1 B, and 3 B parameters—DREAM consistently outperforms existing baselines on BEIR and RTEB[1].

The method hinges on injecting retriever scores into selected attention heads of a frozen LLM, a non‑standard alteration that integrates the retriever with the LLM’s attention mechanism[1]. It also leaves unanswered how well the approach scales to multimodal corpora or ultra‑large models beyond 3 B.

If the claim holds, the default pipeline for retrieval‑augmented generation can drop the contrastive data construction stage and adopt a frozen‑LLM supervised retriever as the new baseline.

References

DREAM: Dense Retrieval Embeddings via Autoregressive Modeling

Binary chunk trees cut RAG latency

Papers Mache — Sun, 05 Jul 2026 05:00:00 +0000

Binary chunking trees boost information efficiency by roughly 6 percent while delivering relevance on par with conventional RAG pipelines. The improvement comes without any extra LLM inference at retrieval time, making it a pure systems win [1].

Before SproutRAG, most long‑document retrievers leaned on external LLMs for chunking, fixed‑size context expansion, or hierarchical summarization, each adding latency or discarding signal. “Unlike prior approaches that rely on external LLMs, fixed context expansion, or lossy summarization, SproutRAG learns which attention heads and layers best capture semantic document structure, enabling multi‑granularity retrieval without additional LLM calls or compressed summaries.” [1]

The core metric—information efficiency (IE)— rises 6.1 percent over the strongest baseline across four heterogeneous benchmarks. “We present SproutRAG … improving information efficiency (IE) by 6.1 % on average over the strongest baseline.” [1]

Relevance does not suffer; retrieval quality matches that of flat vector‑store RAG despite the hierarchical search. The paper reports reduced latency and maintains generation quality comparable to baselines, though specific speedup figures are not detailed in the abstract [1].

The study stops at four benchmark suites and does not report indexing cost or behavior on corpora with billions of chunks, leaving open whether the tree construction scales linearly or incurs hidden memory pressure. This suggests a need for large‑scale ablations and profiling of the tree‑building pipeline before production roll‑out.

If the latency gains hold at scale, swapping a flat vector store for SproutRAG’s binary chunk tree becomes a zero‑change upgrade: drop the new index format into the existing retrieval stack and expect a modest speedup without retuning downstream prompts.

References

SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG

JSON-Schema masks can block needed tool calls

Papers Mache — Sat, 04 Jul 2026 05:00:00 +0000

Grammar‑based token masks can silently block the very function calls an LLM agent must emit. A lightweight two‑pass inference hack sidesteps the problem without retraining the model.

Before this work, engineers routinely combined JSON‑Schema output constraints with tool‑calling APIs, assuming the two constraints coexist harmlessly. Existing agents simply turned on the schema validator and let the model decide when to invoke a tool.

The suppression stems from the way schemas are enforced: “JSON Schema constraints are compiled into grammar‑based token masks that render tool‑call tokens unreachable during decoding” [1]. The mask prunes every token that would start a function call, so decoding never reaches a valid tool‑call token even though the rest of the response complies with the schema.

Running the model in a transparent two‑pass mode eliminates the dead‑end. In the second pass the mask is dropped, allowing the model to emit the missing call, and the paper reports that “Tool Invocation Rate increased from 0% to 100%” [1]. The fix preserves full schema compliance while recovering every required tool activation.

The study evaluates open‑weight model families in a production pipeline, but does not assess closed‑source models or more complex multi‑tool workflows, leaving it unclear whether they suffer the same mask‑induced deadlock. Moreover, the extra decoding pass adds additional latency, which may be a concern for real‑time agents. This suggests a need for smarter mask designs that exclude only truly illegal tokens rather than bluntly cutting off all call prefixes.

If the two‑pass pattern holds across deployments, any benchmark that measures tool use under schema constraints should be re‑run with the mask disabled in a second pass. More importantly, production agents can adopt the transparent two‑pass strategy as a default safety net, ensuring that tool calls are never silently lost while still guaranteeing structured output.

References

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints

Tiered models separate public and private capabilities

Papers Mache — Fri, 03 Jul 2026 05:00:00 +0000

Open‑weight checkpoints now hand over every model capability to anyone who can download the file. A tiered architecture splits the network into public and private branches, and a secret key flips a small permutation of parameters to route computation through the private sub‑graph. Because the key governs the internal graph rather than the input, the extra abilities remain invisible to users who lack it. This moves capability control from a post‑hoc filter to the model’s core design.

Historically, open‑weight releases have no mechanism to hide dangerous or proprietary functions; practitioners either prune the capabilities at pre‑training time or hide the model behind a closed API. Both strategies either cripple the model for all users or require a separate service that defeats the purpose of open weights. The Tiered Language Model (TLM) framework proposes a single weight set that can be run in multiple configurations without altering the underlying parameters. The authors train a public configuration that behaves like a conventional LLM and a keyed configuration that unlocks extra skills [1].

The keyed configuration attains perfect recall of private facts while the public side stays at zero throughout training. “The keyed configuration reaches perfect recall of all facts while the public configuration remains at zero throughout training (see Figure˜3 (right)).” [1] This separation is achieved by “a compact secret key specifies a permutation over a small parameter subset, inducing an alternative computation graph over the same weights that exposes additional capabilities.” [1] Training jointly from scratch and then fine‑tuning the private branch preserves the public behavior, even when the private model learns a new language and instruction‑following ability.

The design’s security guarantees rely on the key affecting only about 5 % of the parameters, leaving the bulk of the model unchanged. “Because authorization operates on the model’s weight structure rather than in the input space, the mechanism resists fine‑tuning‑based extraction and partial key compromise.” [1] Yet the experiments are limited to 180 M‑ and 650 M‑parameter models, so scaling to billions of parameters remains unproven. It also assumes the secret key remains undisclosed; a full key leak would instantly expose the private branch, an edge case not explored in the paper.

If the tiered approach scales, companies can ship fully open‑weight checkpoints that embed IP‑protected capabilities, eliminating the need for separate, closed‑service endpoints. Deployments could expose a harmless public model to the community while retaining a licensed private tier for commercial use, all under a single weight file. This shifts the burden of model‑stealing mitigation from monitoring APIs to safeguarding a tiny cryptographic token. The ecosystem may soon see open‑weight releases accompanied by a standard key‑distribution layer, redefining how proprietary LLM functionalities are protected.

References

Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs

Head-level attention fusion trims compute

Papers Mache — Thu, 02 Jul 2026 05:00:00 +0000

Merging full‑attention and linear‑attention at the head granularity slashes transformer FLOPs without appreciably hurting downstream quality. The trick is to keep the expensive quadratic path only where it truly matters and let the cheap linear path handle the rest.

Before HydraHead, most hybrid designs operated at the layer level, swapping an entire layer’s attention mechanism for a linear variant or arranging a fixed ratio of full to linear layers. Those schemes struggled to reconcile the distributional mismatch between the two attention families, and researchers largely assumed the layer axis was the only practical mixing point.

HydraHead conserves full‑attention computation for just a quarter of the heads, delegating the remaining three‑quarters to its GDN (as described in the paper) linear module. “By default, we retain FA computation for 25% of the heads, while the remaining 75% use the GDN structure.” [1]

Despite this aggressive pruning, the model still reaches the same benchmark scores as a traditional 3:1 layer‑wise hybrid, even when the linear‑to‑full head ratio climbs to seven‑to‑one. “Interpretability‑based head selection enables aggressive FA compression with minimal performance degradation… matches the overall performance of a 3:1 layer‑wise hybrid even at substantially higher LA‑to‑FA mixing ratios (e.g., 7:1).” [1]

The authors evaluate HydraHead primarily on long‑context reading and reasoning suites, training on 15 B tokens before distillation. This leaves open whether the head‑selection pipeline scales down to smaller pre‑training budgets, or how it behaves on tasks that rely heavily on fine‑grained token‑level interactions such as token‑level classification or generation with strict fidelity constraints.

If the reported ratios hold across typical workloads, swapping a vanilla transformer block for a HydraHead block should reduce attention‑related FLOPs by up to roughly 40% (around one‑third) while preserving accuracy, potentially enabling larger context windows or fitting larger models on edge‑class GPUs.

References

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

RL-driven data mixing boosts evaluation scores

Papers Mache — Wed, 01 Jul 2026 05:00:00 +0000

An RL‑driven data scheduler can lift MMLU performance by 27.5 % relative while achieving a 2.23× higher HumanEval pass@1, and it does so with virtually no extra compute [1]. The scheduler learns a policy that decides, at each step, how many examples from each source task to present to the model. Because the policy operates online, the training loop sees only a 0.4 % wall‑clock increase per step.

Before AC‑ODM, most LLM pre‑training pipelines relied on static or uniform mixing of source corpora, assuming that larger models or longer training were the only ways to close downstream gaps. Researchers experimented with hand‑crafted curricula, but those schedules lacked feedback from the model’s evolving gradients. Consequently, improvements from smarter data allocation remained anecdotal.

AC‑ODM delivers those gains by learning a policy that allocates examples across tasks on the fly. “On Pythia-1B, it reaches optimal validation perplexity using up to 66% fewer training steps than competitive baselines, delivering a 27.5% relative improvement in MMLU accuracy and a 2.23× higher pass@1 on HumanEval, all while incurring a virtually negligible ( 0.4%) per-step wall-clock increase and only 2% additional memory overhead.” [1] This translates to a 7.2 % absolute lift in 0‑shot MMLU accuracy and a more than two‑fold jump in HumanEval pass@1, with the same hardware budget.

The study leaves open how the approach scales beyond a 1 B‑parameter backbone. All reported numbers come from a Pythia‑1B experiment, and the paper does not present results on larger, production‑scale models [1]. The proxy mode, which transfers a policy learned on a small model to a larger target, introduces an extra training phase; the paper does not report a quantified cost for this phase. An open question is whether the same relative gains survive when the model’s capacity dwarfs the data‑mixing policy’s representational power.

If the reported efficiency carries over, replacing uniform sampling with an AC‑ODM scheduler should become the new default in pre‑training scripts. Practitioners can drop a few lines of RL‑policy code, keep the memory footprint within 2 % of the baseline, and re‑run standard benchmarks to harvest immediate gains. The community ought to treat data mixing as a tunable hyper‑parameter on par with model depth, rather than an afterthought.

References

AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining

Coordinate-space diffusion improves video consistency

Papers Mache — Tue, 30 Jun 2026 05:00:00 +0000

Leveraging multi‑view point tracking as geometric supervision for video diffusion models reduces the cross‑view jitter that has plagued monocular pipelines. By routing attention features through an auxiliary tracking head, the generated novel‑view videos maintain better alignment with the physical scene across camera motions.

Before this work, two families dominated novel‑view video synthesis. Explicit 3‑D reconstructions fed geometry into renderers, but off‑the‑shelf modules faltered on dynamic objects, producing warped artifacts. Purely camera‑conditioning diffusion models delivered eye‑catching visuals yet drifted as the viewpoint changed, betraying the underlying motion. Both routes left a gap between visual fidelity and geometric consistency.

The core contribution of MVTrack4Gen is an auxiliary multi‑view tracking head that restores those lost correspondences. The authors observe that “specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency” [1]. By routing the attention features into a point‑tracking objective, the model learns to keep motion aligned across perspectives, and “across diverse benchmarks, our method achieves state‑of‑the‑art geometric consistency and competitive camera accuracy” [1].

The paper’s scope stops short of a turnkey solution. The codebase and pretrained checkpoints are promised but not yet released, so reproducibility hinges on a future pull‑request rather than an immediate drop‑in. Moreover, the tracking supervision assumes access to multi‑view point tracks, a requirement that may be costly for bespoke datasets. This suggests that scaling the approach to truly in‑the‑wild video collections will demand either synthetic supervision or more efficient tracking pipelines.

If the reported gains hold, any video diffusion stack that currently conditions only on camera pose should be retrofitted with a lightweight correspondence head. Running a standard multi‑view consistency benchmark on the augmented model will reveal whether the modest architectural addition truly closes the realism gap that has constrained AI‑generated video for production use.

References

MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

Step‑level RL sharpens LLM reasoning credit

Papers Mache — Mon, 29 Jun 2026 05:00:00 +0000

Finer‑grained credit assignment is a promising approach that may help large language model agents turn a series of observations into coherent, multi‑step reasoning. By moving the learning signal from a single episode reward to step‑by‑step feedback, the agent can preserve delayed signals that would otherwise wash out across long interactions.

Until now, most post‑training RL for LLMs has treated each token as an independent decision point, mirroring the paradigm of RLHF and RLVR. That token‑centric view forces the optimizer to infer the value of a whole action from a string of token‑level gradients, a mismatch that becomes severe when the agent’s behavior is defined by higher‑level actions such as “search the web”, “fetch a diagram”, or “execute a tool”. The result is a brittle credit signal that struggles on tasks requiring deep, multi‑turn planning.

StepPO flips the hierarchy: it reshapes the Markov Decision Process so that each interaction step, not each token, is the basic trajectory unit, and it applies step‑level Generalized Advantage Estimation to align policy updates with the natural granularity of agent decisions. “Experiments across multi‑hop QA, academic paper search, and text‑world action tasks show that StepPO consistently outperforms various RL algorithms” [1]. The consistent edge appears across three very different domains, suggesting the improvement is not a quirk of a single benchmark.

On the flagship HotpotQA benchmark, StepPO beats every token‑centric baseline in‑domain and maintains the lead on the out‑of‑domain 2Wiki and MuSiQue suites. “StepPO outperforms all baselines on in‑domain HotpotQA and remains strongest on out‑of‑domain 2Wiki and MuSiQue, indicating that step‑centric optimization transfers beyond training” [1]. This transferability could be relevant for multimodal pipelines that aim to generalize from textual reasoning to visual grounding or tool use, though the paper does not evaluate such settings.

The authors attribute the gains to a better preservation of delayed reward signals. “These results show that step‑level credit assignment better preserves delayed reward signals in long‑text, multi‑step agent interactions” [1]. The analysis, however, is limited to language‑only environments; it remains unclear how the same mechanism copes with noisy visual embeddings or asynchronous tool calls, where step boundaries may be ambiguous.

If step‑level credit assignment truly safeguards long‑range reward information, the next logical step is to retrofit existing multimodal agents with a step‑aligned RL head and re‑evaluate on benchmarks like VQA‑2 with tool‑augmented reasoning. The expectation is that the same performance boost observed on pure text tasks will translate into sharper, more reliable multimodal problem solving.

References

StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

AI/ML Research Digest — Jun 27, 2026

Papers Mache — Mon, 29 Jun 2026 05:00:00 +0000

RL‑Driven Agentic Optimization

Training agents with only sparse rewards often yields unstable behavior. Recent work replaces explicit reward models with dense, token‑level supervision. Hindsight skill distillation supplies per‑token guidance, stabilizing learning curves [1]. A complementary “progress advantage” signal predicts future improvement and serves as a learned reward, eliminating the need for hand‑crafted reward functions [2]. Both approaches make large‑scale RL more sample‑efficient, which matters for deploying agents in complex, open‑ended environments.

Geometric Integration in Video Generation

Diffusion transformers that ignore 3D structure generate physically implausible motions. PhysiFormer injects explicit world‑coordinate reasoning, allowing the model to predict mesh dynamics directly in 3‑D space and produce more realistic animations [3]. A separate line of work adds multi‑view point tracking to the diffusion pipeline, enforcing cross‑view consistency and reducing jitter across camera angles [4]. These geometric cues are crucial for applications like virtual production and robotics where realism is non‑negotiable.

Efficient Retrieval‑Augmented Generation (RAG)

RAG pipelines often suffer from latency because each retrieval step invokes a heavy encoder. One paper compresses topic metadata into lightweight embeddings that guide the retriever without full passes through the encoder, cutting inference time dramatically [5]. Another introduces a binary chunking tree that supports retrieval at multiple granularities in a single pass, removing the need for extra LLM calls when refining context windows [6]. Faster RAG widens the gap between research prototypes and interactive products.

Tiered Language Models for Capability Separation

A new architecture partitions a model into public and private sub‑networks linked by a secret key. The secret‑key‑controlled computation graph activates private capabilities only when authorized, preventing extraction attacks that exploit prompt engineering alone [7]. This structural defense goes beyond brittle prompting constraints and offers a practical path toward safer model deployment.

PhysiFormer for 3D Mesh Dynamics

PhysiFormer predicts mesh deformations directly in world coordinates using a diffusion transformer that learns physics‑consistent transitions without handcrafted priors [3]. By removing hand‑engineered inductive biases, the model adapts to diverse materials and forces, opening doors for automated animation and simulation pipelines.

DREAM: Autoregressive Retriever Training

Instead of contrastive pairs, DREAM trains dense retrievers with the autoregressive loss of a frozen LLM. The retriever learns to produce passages that the language model would naturally generate, removing the need for costly labelled relevance data. Benchmarks on BEIR show consistent improvements over traditional contrastive methods [8].

Other Observations

JSON‑Schema Tool Suppression – Grammar‑based token masks designed to enforce JSON‑Schema constraints sometimes block legitimate tool calls, hurting performance. A two‑pass execution scheme that postpones masking resolves the issue without retraining the model [9].
RL‑Based Data Mixing Gains – An RL scheduler that selects training sources during pre‑training yields a 7.2 % boost on MMLU and a 2.23× increase in HumanEval pass@1, demonstrating that dynamic data curricula can markedly improve downstream reasoning abilities [10].
Transformer Attention Latency Reduction – Merging full and linear attention at the head level, combined with a mixture‑of‑experts query‑head selector, cuts compute cost while keeping accuracy on par with dense attention models [11], [12].

These developments collectively push toward more stable agents, physically grounded generation, faster retrieval, and safer deployment—key stepping stones for bringing advanced AI into real‑world workflows.

References

Linear temporal attention gives agents memory across gaps

Papers Mache — Sun, 28 Jun 2026 05:00:00 +0000

Persistent world models are what keep long‑horizon tasks coherent when the agent’s sensors go dark. Linear temporal attention and associative graph memories give embodied systems a way to write and read state across those dark intervals, eliminating the drift that has long plagued simulation‑to‑real pipelines.

Before these advances, world models behaved like camera‑following renderers: they could generate plausible frames while observed, but they fell apart the moment the viewpoint changed. An analysis of 9 600 videos across 23 models showed a systematic “preservation‑access‑re‑observed‑consistency” gap, with the re‑observed state almost never correct regardless of image quality or model size [1].

Kairos demonstrates that hybrid linear temporal attention can close that gap without exploding compute. Its gated linear attention runs in O(n) time, allowing a 5‑second video to be processed in ≈ 11.7 seconds on an NVIDIA A800—real‑time edge inference that would be impossible with quadratic attention [2].

MRAgent shows that a graph‑structured episodic memory can make the same persistence cheap. By integrating reasoning into memory access, it “reduces prompt tokens to 118 k, a significant decrease from baselines like A‑Mem (632 k)” —an 81 % reduction that also slashes runtime cost while preserving expressive power [3].

The new components are not a silver bullet. Kairos still relies on sliding‑window and dilated windows, so events that span beyond the window length may be truncated, and MRAgent’s reconstruction loop grows with the number of graph nodes, raising concerns about scalability to truly open‑ended lifespans. The papers discuss these design trade‑offs but do not claim a specific solution for fully unbounded persistence.

If persistent state is the missing piece, then the community’s evaluation suite must reflect it. WRBench’s three‑stage diagnostic—intervention, continuity, and re‑observed‑state correctness—should become a default test for any world‑model rollout, forcing developers to measure not just visual fidelity but the ability to keep the world moving while nobody is looking.

References

Sparse autoencoders trade interpretability for fragility

Papers Mache — Sat, 27 Jun 2026 05:00:00 +0000

Interpretability via sparse autoencoders can mask extreme fragility in the very neurons we treat as safety handles. A single clamped unit may look like a clean lever, yet the model can reroute around it without our notice. The illusion of control grows as practitioners equate “human‑readable” with “trustworthy.”

Sparse autoencoders have become the de‑facto tool for dissecting residual‑stream activations, and many recent defenses assume that clamping a identified unsafe feature will reliably suppress the corresponding misbehavior. This assumption underpins latent‑space steering, refusal‑steering, and unlearning pipelines that intervene directly on SAE latents.

Stable SAE features concentrate the predictive power, while unstable features barely move the needle. The authors report a “functional asymmetry of stable and unstable features. Stable features carry most of the reconstruction‑ and prediction‑relevant signal, whereas unstable features have weak marginal impact and are dominated by low‑frequency surface‑form triggers” [1]. Moreover, unstable features fire on average only 0.18 % of tokens versus 0.44 % for stable ones, highlighting their sporadic influence.

Despite their individual non‑reproducibility, unstable features live in a reproducible low‑rank subspace. “Decoder‑space analysis shows that unstable features are individually non‑reproducible but collectively span reproducible lower‑rank subspaces” [1], suggesting that seed‑dependent basis choices shuffle the same underlying geometry rather than generate pure noise.

Even a hard‑wired clamp on a harmful feature can be undone on almost every test case. In safety‑critical refusal‑steering, the authors achieve a 95.8 % recovery rate on valid samples while keeping defended‑feature relative drift to 0.131, “substantially below suffix‑based baselines” [2].

The SAE reconstruction residual, not the clamped concepts, carries the recovered unsafe behavior. “Figure 6 identifies the SAE reconstruction residual as the dominant carrier. Residual replay nearly matches full recovery, while clamped‑feature replay fails and non‑clamped SAE‑feature replay remains weak” [2].

These results expose a gap between feature‑level control and behavioral completeness: “SAE features can support causal intervention, but controlling them does not guarantee control over the underlying behavior” [2]. The interventions block a visible route but leave the underlying functionality intact.

The recovery experiments target a narrow set of prompts and keep the clamp active throughout optimization, so it remains unclear how the phenomenon scales to open‑ended generation or to interventions that are applied intermittently. Likewise, the seed‑stability analysis focuses on reconstruction statistics rather than downstream safety metrics, leaving open whether the identified unstable subspaces actually matter for real‑world harms.

Safety pipelines that treat SAE latents as definitive intervention points should be revised to monitor the residual stream as well. Deployments that rely solely on clamped neurons risk a false sense of security, because the model can re‑express the same undesirable behavior through the unexplained residual.

If every interpretable neuron can be sidestepped, what concrete metric will certify that a model is truly safe?

References

Adaptive token compression halves diffusion model cost

Papers Mache — Fri, 26 Jun 2026 05:00:00 +0000

Latency still blocks diffusion models from being practical in interactive editing tools. HiLo‑Token proves that input‑adaptive high‑low frequency token compression can halve that latency while keeping generation quality unchanged, delivering up to a 3.13× speedup on typical edits [1].

In current pipelines the Diffusion Transformer (DiT) consumes the bulk of compute, accounting for about 73 % of total model latency even after being distilled from 50 timesteps down to 8 timesteps. Earlier optimizations therefore focused on timestep reduction, leaving the token‑wise cost untouched.

The authors allocate full‑resolution tokens to a dilated edit mask, prune low‑frequency tokens outside it, and supplement the remainder with a 16× downsampled representation, achieving up to 3.13× DiT speedup on small‑mask edits. Across mask sizes the average token ratios drop to 6.38 % (small), 15.92 % (medium) and 35.36 % (large), corresponding to DiT speedups of 3.13×, 2.59×, and 1.67× respectively [1].

The reported gains diminish as the mask grows, falling to 1.67× speedup for large masks where 35 % of tokens are retained, exposing a trade‑off between compression ratio and runtime. Moreover, the evaluation is limited to mask‑guided editing; how the same token budget strategy fares on unrestricted image‑to‑image or text‑to‑image diffusion remains an open question.

Deployments that power Photoshop’s Remove feature can cut their Amazon AWS p5.48xlarge node count by a third thanks to the reduced token budget. Practitioners should therefore treat adaptive token allocation as a default optimisation layer when building diffusion‑based vision editors for constrained hardware.

References

HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing