DEV Community: Papers Mache

Mixed‑precision routing speeds up attention prefill

Papers Mache — Sat, 05 Sep 2026 05:00:00 +0000

TileMix doubles prefill throughput while leaving the model untouched. The speedup comes from routing attention‑score tiles to INT8 Tensor Cores instead of running everything in FP16. By keeping the dense connectivity graph intact, the method avoids any retraining or architectural tweaks. “TileMix improves throughput over FlashAttention by routing score-tile groups to INT8 Tensor Cores.” [1]

Before TileMix, practitioners had to choose between a uniform low‑precision path that hurt long‑context quality or token‑selection schemes that broke the full attention graph. FlashAttention remains the de‑facto FP16 fused kernel for dense self‑attention, and existing mixed‑precision tricks apply a single precision globally rather than per tile. Consequently, speed gains were modest and often came at the cost of accuracy on long prompts.

According to the paper, SpTrans75 reaches 31.80 K tokens /s on a 4 k‑token prompt, more than double FlashAttention’s 14.33 K tokens /s while staying ahead of other INT8 baselines. The paper does not provide a direct TileMix vs. FlashAttention throughput comparison for this setting. [1] The result holds across LLaMA, Qwen and Vicuna models, showing a consistent accuracy‑efficiency frontier.

TileMix’s design hinges on hardware‑aligned score tiles and compact bitmask routing, which may limit portability to GPUs lacking INT8 Tensor Core support or to future attention kernels that change tile granularity. Although the authors emphasize that “by routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped‑query attention, variable‑length batches, and INT8 key/value caches,” [1] it remains an open question how well the approach scales to non‑A100 architectures or to inference frameworks that cannot fuse the mixed‑precision path.

Deployments that spend most of their latency in prefill should replace FlashAttention with TileMix on supported hardware, cutting end‑to‑end latency roughly by half without sacrificing model quality.

References

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

Query‑aware routing lifts retrieval recall

Papers Mache — Fri, 04 Sep 2026 05:00:00 +0000

Retrieval pipelines have long forced engineers to pick either a fast but brittle retriever or a powerful yet latency‑heavy one, and the field has accepted that trade‑off as immutable. RetrievalRouter flips that assumption by learning, from the query text alone, which backend will serve each request, delivering measurable recall lifts for retrieval‑augmented generation (RAG) systems.

Before this work, practitioners built static pipelines around dense encoders such as DPR or late‑interaction models like ColBERT, and multimodal variants were treated as separate tracks. Benchmarks showed that no single architecture dominated across domains, so teams settled on the best compromise for their whole workload.

RetrievalRouter is 2.5 % more accurate and faster than the best static baseline. The authors train a lightweight classifier that predicts the optimal pipeline per query, exposing a single tunable knob that spans the full accuracy‑latency frontier, and then report that “Against the best static baseline, RetrievalRouter is 2.5% more accurate and faster.” [1] This improvement holds across financial and scientific corpora without any architectural changes to the underlying retrievers.

At its sweet spot RetrievalRouter reaches 0.755 nDCG@5 at 0.666 s, dominating all four late‑interaction pipelines. The paper highlights that “At , RetrievalRouter reaches 0.755 nDCG@5 at 0.666s and dominates all four late-interaction pipelines.” [1] The result places the router on a Pareto‑optimal curve where both relevance and response time improve together, a regime previously unattainable with any static configuration.

Routing adds a fixed 15 ms overhead per query, limiting gains in ultra‑low‑latency scenarios. The authors acknowledge that “Routing incurs 15 ms of overhead.” [1] While the net latency remains lower than most dense or multimodal backends, the extra step becomes noticeable when the target service time is already sub‑200 ms, suggesting that further engineering—such as batching router predictions or embedding them in inference servers—may be required.

If these numbers hold across production workloads, every RAG benchmark should treat RetrievalRouter as the new baseline rather than a single static retriever. Re‑evaluating existing pipelines with query‑aware routing will reveal hidden recall gains and may eliminate the need for costly, heavyweight backends in many applications.

References

RetrievalRouter: Joint Modality and Architecture Selection for Document Retrieval

Test‑time policy optimization replaces supervised labels

Papers Mache — Thu, 03 Sep 2026 05:00:00 +0000

An asymmetric test‑time objective lets on‑policy distillation hit supervised OPSD performance without any hand‑labeled data. By rewarding rollouts that agree with a teacher and penalizing those that diverge, the method turns unlabeled interaction streams into a reliable supervisory signal. The result is label‑free test‑time training that reaches the same accuracy as pipelines built on costly annotation.

Before this work, reinforcement‑learning fine‑tuning and On‑Policy Self‑Distillation (OPSD) both required ground‑truth labels to steer the model toward correct reasoning paths. A natural workaround—using majority‑vote pseudo‑labels—proved brittle: a single incorrect vote could corrupt the teacher and mislead every token in subsequent generations.

Even when the pseudo‑label is wrong, 79 % of the rollouts that disagree with it are wrong too, confirming an asymmetric error pattern that can be exploited safely. This observation underpins the design of TTPO’s dual‑branch loss, where agreement triggers distillation and disagreement triggers a penalizing RL signal[1].

Without any labels, TTPO matches label‑supervised OPSD on five competition‑level benchmarks and lifts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, delivering a +25.2 % to +36.4 % gain when the model is evaluated without additional thinking time[1].

TTPO exceeds the label‑dependent OPSD across all three model scales—40.1 vs. 39.7 on the 1.7B variant, 58.6 vs. 58.4 on 4B, and 62.6 vs. 61.7 on 8B in average score—despite operating without ground‑truth supervision[1].

The approach still leans on majority‑vote pseudo‑labels as a routing signal; if the vote is consistently noisy in a new domain, the asymmetry may weaken and the penalty branch could misfire. Moreover, the experiments focus on mathematical reasoning tasks, leaving open whether the same gains translate to language generation or classification settings.

If these results hold broadly, the standard practice of building a labeled OPSD dataset before test‑time training should be abandoned for math‑heavy pipelines; simply run TTPO at inference time and cut the entire annotation stage out.

References

TTPO: Test-Time Policy Optimization

On‑the‑fly harnesses boost LLM agents

Papers Mache — Wed, 02 Sep 2026 05:00:00 +0000

Custom harness generation at inference time can lift LLM‑agent success rates by as much as twenty percentage points without touching the underlying model. JIT‑Agent shows that this leap comes from a trainable meta‑model that spits out task‑specific wrappers on the fly, turning a manual engineering bottleneck into an automated plug‑in.

Before JIT‑Agent, agent scaffolds were hand‑crafted prompt wrappers that had to be rewritten for each new tool or workflow. Researchers treated these harnesses as static artefacts, and any performance gain required either painstaking redesign or a larger foundation model.

Across all 18 directly matched backbone–benchmark pairs, replacing the default scaffold with a JIT‑generated harness improves performance. The paper reports gains ranging from five to twenty points, with the most dramatic jump of +20.2 on GLM‑5.2, proving that a modest meta‑model can outpace heavyweight retraining cycles [1].

DeepSeek‑V4‑Flash equipped with JIT‑Agent surpasses GPT‑5.6 on DeepSearchQA by +9.1 and on OdysseyBench by +4.3, demonstrating that the benefit persists even when the backbone is already competitive [1]. These results hold across heterogeneous model families such as Mimo‑V2.5 and Qwen3.6, underscoring the method’s portability.

JIT‑Agent makes these choices explicit through four interoperable modules: how history is compressed, how local intent is formed, how tools and skills are orchestrated, and how control advances. The design assumes an archive of prior harnesses to distill from, so completely novel toolkits may still require a bootstrap phase, and the additional inference step introduces latency that has not been quantified in the benchmarks.

If this line of work scales, product teams can upgrade existing LLM agents simply by inserting JIT‑Agent as a harness generator, re‑running their internal test suites to capture the 5–20 pp lift without buying bigger GPUs or retraining. The immediate next experiment is to replace hand‑written scaffolds in any deployed agent stack with a JIT‑generated counterpart and measure the real‑world throughput impact.

References

JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

Blockwise diffusion achieves >5 faster inference

Papers Mache — Tue, 01 Sep 2026 05:00:00 +0000

Text‑to‑3D generation still burns tens of seconds per output, making interactive use impractical. By partitioning the token sequence into confidence‑guided blocks, a new inference path sidesteps the sequential bottleneck and preserves the fine geometric detail that users expect.

Prior pipelines either decode shape tokens autoregressively or run global diffusion over the entire representation, both of which are costly. Autoregressive decoding cannot revise early mistakes, while diffusion repeatedly processes the full 3‑D state, inflating runtime as fidelity improves [1].

Block3D reduces mean end‑to‑end generation time to 4.99 seconds, a 5.15× speedup over the fine‑tuned autoregressive baseline, while keeping geometric fidelity on par with existing methods. “Its mean end-to-end generation time is 4.99 seconds, corresponding to a speedup over the controlled Cube baseline,” and “it improves paired geometry and reduces Cube’s mean generation time from 25.71 to 4.99 seconds while retaining competitive text‑shape alignment” [1].

The reported gains are limited to a held‑out split of TRELLIS‑500K and rely on confidence‑guided intra‑block correction, whose overhead may grow with larger token vocabularies or higher‑resolution meshes. This suggests an open question: will the same >5× acceleration hold when scaling to industrial datasets or multi‑GPU deployments [1].

Adopting blockwise diffusion as the default generation loop can reduce latency, and using the released pretrained checkpoint enables inference without additional model training. Practitioners can drop the standard diffusion scheduler, plug in the Block3D checkpoint, and re‑run CD‑L1 and F‑score evaluations to verify that speed gains come with no measurable loss in quality.

References

Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion

AI/ML Research Digest — Aug 29, 2026

Papers Mache — Mon, 31 Aug 2026 05:00:00 +0000

Efficiency through latent compression and adaptive decoding

Latent compression, block‑wise inference, and mixed‑precision routing slash compute by roughly 5–10× while keeping output quality intact [1] [2] [3].

The gain matters because it makes large generative models viable on cheaper hardware and reduces energy consumption—two practical bottlenecks for deployment.

LLM agent performance hinges on curated harnesses and evaluation frameworks

Task‑specific harnesses that generate, patch, or evaluate code at inference time lift success rates of language‑model agents by 5–20 percentage points, even though the underlying model stays unchanged [4] [5].

This shows that much of the current performance gap is engineering rather than model capacity, suggesting a low‑cost path to more reliable agents.

Multimodal world modeling and interactive RL

Linking large multimodal models with simulators and reinforcement learning creates agents capable of planning, acting, and adapting inside dynamic visual environments [6].

Such closed‑loop systems move us from static perception toward embodied AI that can learn by interaction.

Standout papers

Stream4D – 4D reward for coherent video generation

Replacing static 3‑D critics with a feed‑forward 4‑D reconstruction loss and a motion prior preserves long‑range dynamics, improves visual fidelity, and aligns better with human preferences [7].

Gated Recurrent Transformer (GRT) depth sharing

A three‑layer GRT reuses a shared core updated by a gate. Under identical FLOP budgets it matches the performance of a twelve‑layer GPT‑2 Small, proving that clever weight reuse can replace raw depth [1].

JIT‑Agent – on‑the‑fly harness synthesis

JIT‑Agent learns to produce bespoke harness code during inference, boosting LLM agent task success by 5–20 pp without any retraining of the base model [4].

Quantization‑Aware Healing (QAH)

QAH teaches a 4‑bit student directly from a full‑precision teacher via distillation. The resulting model reaches or exceeds the original accuracy while converging dramatically faster than traditional quantization pipelines [2].

Test‑Time Policy Optimization (TTPO)

TTPO uses an asymmetric objective that rewards agreement and penalizes disagreement during on‑policy distillation. It attains supervised OPSD performance without any external labels, simplifying data collection for policy learning [3].

Notable details

Blockwise diffusion with confidence‑guided intra‑block correction cuts text‑to‑3D inference time by more than fivefold while preserving geometric fidelity [8].

Faster diffusion expands the range of interactive 3‑D applications.
Entropy‑Valley length selection picks denoising‑friendly target lengths for masked diffusion translation, yielding sizable adequacy gains in machine translation benchmarks [9].
TileMix mixed‑precision routing directs tiles of the attention matrix through low‑precision kernels, boosting dense‑attention prefill throughput without retraining and retaining long‑context quality [10].
OraRL oracle rollouts introduce expert rollouts and a decoupled advantage estimator, slashing the sample budget needed for video‑grounded multimodal language models while preserving performance [11].
RetrievalRouter query‑aware routing learns to select the most suitable dense or multimodal retriever per query, delivering noticeable recall improvements in retrieval‑augmented generation pipelines [12].

References

AdaptiveMem reduces reasoning fixation errors

Papers Mache — Mon, 31 Aug 2026 05:00:00 +0000

Dynamic external memory can slash stale‑belief mistakes at inference time without any retraining. AdaptiveMem achieves this by prompting the model at inference time to reconsider and avoid reliance on potentially outdated or misleading retrieved memories.

Before AdaptiveMem, adding long‑term memory to LLMs often backfired: all evaluated memory strategies underperform the no‑memory setting, with even the strongest methods suffering drops of more than 10 % on the new MemTrapBench benchmark. This revealed a systematic “cognitive trap” where faithfully stored memories distort current reasoning.

AdaptiveMem is a simple yet effective prompt skill for mitigating memory‑induced cognitive traps. The authors describe it as “a simple yet effective prompt skill for mitigating memory‑induced cognitive traps.” [1]

It can be directly integrated into diverse memory frameworks and guides the model to reconsider how retrieved memories should be used. The paper notes that “it can be directly integrated into diverse memory frameworks and guides the model to rethink how retrieved memories should be used.” [1]

AdaptiveMem improves performance on MemTrapBench while preserving performance on typical memory benchmarks such as LongMemEval. As reported, “AdaptiveMem improves performance on MemTrapBench while preserving performance on typical memory benchmarks such as LongMemEval.” [1]

The authors evaluate scenarios covered by MemTrapBench; they do not discuss performance on other downstream tasks. moreover, the approach hinges on prompt engineering, which can be sensitive to phrasing variations. This suggests an open question: will similar inference‑time rewrites work for multimodal or chain‑of‑thought settings where memory interaction is less explicit?

If AdaptiveMem reduces reasoning fixation without extra training, it could be considered as an alternative to static retrieved snippets in production pipelines, though the paper does not make specific deployment recommendations. The expected gain—eliminating a major source of error while keeping overall performance steady—makes it a compelling plug‑in for deployed systems.

References

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

Multimodal robots learn more efficiently

Papers Mache — Sun, 30 Aug 2026 05:00:00 +0000

A vision‑language backbone can significantly reduce robot training data requirements, addressing the reliance on extensive teleoperation recordings. EXIMO flips that script: a pretrained multimodal encoder drives exploration, letting the robot learn long‑horizon manipulation with dramatically fewer environment interactions.

Before EXIMO, state‑of‑the‑art manipulation relied on behaviour cloning from massive teleoperation datasets and on‑policy reinforcement learning that spirals in sample cost. The dominant VLA models sit at billions of parameters and demand hundreds of hours of human control to cover new tasks — a bottleneck that has limited rapid iteration in embodied AI.

EXIMO compresses the finetuning workflow into three stages—explore, imitate, and optimise—thereby eliminating redundant data collection loops. The three‑stage design lets a vision‑language model plan subgoals for the VLA, which then gathers an orchestrated dataset before a brief off‑policy RL polish [1].

When the VLM planner is added, success rates jump sharply while wall‑clock time to solve each task stays unchanged. “VLM orchestration significantly increases the success rate of GROD while maintaining similar time to success,” the authors report, showing that richer visual‑language reasoning does not slow execution [1].

Fine‑tuning on the VLM‑generated data gives an immediate boost: “GROD + SFT starts at a higher success rate than the base model and also obtains higher performance at convergence compared to the base GROD model.” This early advantage persists through training, confirming that shared multimodal representations shrink the data bottleneck [1].

The paper’s own ablations reveal two open gaps. First, the experiments in the paper focus on a set of tabletop manipulation tasks; it remains an open question how well the approach scales to mobile manipulation or deformable objects. Second, the approach hinges on the VLM’s planning fidelity—errors in subgoal decomposition can propagate through the orchestrated dataset, a risk that the authors acknowledge but do not quantify.

If EXIMO’s efficiency holds across domains, the community should rerun standard embodied benchmarks (e.g., RLBench) with VLM‑guided exploration as the default finetuning baseline. Doing so will expose whether shared visual‑language backbones can consistently replace costly teleoperation pipelines and accelerate real‑world robot deployment.

References

EXIMO: VLM Guided Exploration of VLA Policies

Object‑agnostic planners grasp unseen objects zero‑shot

Papers Mache — Sat, 29 Aug 2026 05:00:00 +0000

Grasp planners that hinge on object models are being eclipsed by approaches that synthesize contacts directly on the hand. A new generation of generative planners learns only the geometry of a gripper’s contact surface and can produce viable grasps for any unseen item in a single inference step.

Before these works, data‑driven grasp synthesis required large, object‑specific datasets collected either from painstaking annotation or from expensive simulation pipelines. The resulting models generalized poorly to novel shapes and incurred substantial engineering overhead to maintain per‑object training corpora.

GOAG reaches an average success rate of 86.93% on the MultiDex benchmark despite never seeing object‑specific data during training. While being trained in an object‑agnostic manner, GOAG achieved a higher average success rate for generating accurate grasps across all three grippers compared to the baselines [1].

The entire dataset creation for GOAG consumed only about 1 GPU hour on a single Nvidia RTX 4090, whereas comparable object‑specific pipelines reported generation times of roughly 1,400 GPU hours on an Nvidia A100. “The entire dataset creation required approximately 1 GPU hour on a single Nvidia RTX 4090.” “In contrast, a previous work [12] reported a much longer generation time of 1,400 GPU hours using Nvidia A100.” [1]

CoToGrasp attains state‑of‑the‑art performance on the large‑scale DexGraspNet dataset, surpassing existing taxonomy‑guided planners while remaining completely object‑agnostic. “Extensive evaluations on the large‑scale DexGraspNet dataset demonstrate that CoToGrasp achieves state-of-the-art performance, outperforming existing taxonomy-guided planners.” “By learning the intrinsic contact manifold of the gripper within this workspace, our model achieves zero-shot generalization to unseen objects at inference.” [2]

Both papers evaluate primarily on static benchmark collections and rely on simulated depth or point‑cloud inputs; real‑world clutter, occlusions, and dynamic interactions remain underexplored. This suggests that while contact‑surface learning removes the need for object‑specific datasets, additional research is required to confirm robustness under uncontrolled perception conditions.

Benchmarks such as MultiDex and DexGraspNet should incorporate an object‑agnostic baseline, because these results demonstrate that eliminating per‑object data does not sacrifice grasp success and dramatically cuts dataset generation time.

References

IAR embeds documents directly into model weights

Papers Mache — Fri, 28 Aug 2026 05:00:00 +0000

Retrieval‑free internalization now beats standard fine‑tuning on domain‑specific question answering, and it does so without sacrificing the model’s broad linguistic competence. The IAR framework makes this possible by turning a static document collection into parametric knowledge that lives directly in the weights.

Before IAR, most systems relied on an external retriever at inference time, coupling a dense index with a language model or augmenting the model via LoRA adapters to expose domain data. Those approaches still incurred latency from the retrieval step and often required separate pipelines for general‑purpose tasks.

IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset‑model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench[1]. The three‑stage post‑training—Inject, Align, Recover—delivers a consistent uplift while keeping the model usable for unrelated benchmarks.

Qwen3‑4B CC is the clearest example, reaching 50.5 % versus 42.4 % domain accuracy while improving all three general metrics[1]. This single model jump illustrates how weight‑level knowledge injection can eclipse a vanilla supervised fine‑tune even on a mid‑size 4B parameter backbone.

Relative to Vanilla SFT, Best IA improves domain accuracy by 2.8, 7.7, 5.3, and 4.7 points on CC for Llama, Phi, Qwen3‑4B, and SmolLM respectively; the corresponding CCI gains are 5.6, 6.1, 0.4, and 2.3 points[1]. The injection‑and‑align stages alone already capture a sizable fraction of the final performance boost before any recovery step.

The paper’s own limitation is that achieving these gains requires a three‑stage training pipeline, which adds engineering overhead and may not scale trivially to corpora larger than those tested. This suggests an open question: can a single‑pass or lighter‑weight internalization procedure retain both domain fidelity and general capability at scale?

If the reported improvements hold across broader settings, developers should replace external retriever components with IAR‑enhanced models for latency‑critical services, cutting inference time and infrastructure complexity while preserving—or even improving—answer quality.

References

Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization

Self‑evolving agents acquire skills without scaling

Papers Mache — Thu, 27 Aug 2026 05:00:00 +0000

Agents that can rewrite their own simulated worlds and distill those rewrites into reusable modules now eclipse raw model scaling as the dominant path to higher accuracy. Two independent systems released this month demonstrate a co‑evolutionary loop where workflows become skills and feedback loops sustain evolution without any extra training.

Previously, LLM agents constructed ad‑hoc workflows at inference time but discarded the resulting procedures once a task finished, while skill libraries were handcrafted or assembled offline [1]. This static pipeline prevented agents from capitalizing on successes they had already achieved, leaving long‑term capability growth to be driven solely by larger backbones.

Earlier attempts to evolve skills relied on a single‑turn question‑answer signal, which quickly ran out of trustworthy gradients and left multi‑turn defects invisible [2]. Without a persistent source of feedback the evolution process stalled after the first round of fixes.

FlowEvo reaches 85.6% accuracy on ALFWorld, a 26.4‑point lead over the strongest baseline, while consuming roughly one third of the tokens ["On ALFWorld, it reaches 85.6%, 26.4 points above the strongest baseline, while using roughly one third as many tokens." 2607.21596]. The system compiles each successful workflow into a callable skill, stores it in a persistent bank, and reuses it either directly or as context for new plans, turning transient traces into permanent capabilities.

Across ten base models ranging from 7 B to 671 B parameters, FlowEvo outperforms the ExpeL baseline in 49 of 50 model‑dataset comparisons, showing that the gain comes from skill accumulation rather than raw scaling [1]. The authors track downstream utility for each skill and suppress those that cause negative transfer, ensuring that the bank grows only with beneficial routines.

When compiled skills are made available for reuse, performance jumps from 38.8% to 80.6%, confirming that the feedback loop of skill compilation is the dominant factor ["Enabling skill reuse on top of compilation produces the dominant improvement (38.8% to 80.6%), confirming that the gain is driven by skill feedback rather than curation." 2607.21596]. This leap dwarfs gains from merely curating a larger static library, because each skill carries its execution trace and contextual cues.

SkillEvo lifts production‑skill success by 23 points over self‑reflection evolution and by 15.4 points over single‑turn QA‑driven evolution across six cloud‑service categories, nine skills, and ninety‑eight reference files ["Across six categories of cloud services, 9 production Skills, and 98 skill-reference files, SkillEvo surpasses self‑reflection‑based evolution by 23.0 points and single‑turn‑QA-driven evolution by 15.4 points." 2608.13120]. The framework replaces a scalar rejection gate with an active governance layer that repairs factual degradation and structural bloat, keeping the skill set healthy over many iterations.

Its multi‑turn simulation rewrites defects layer by layer—each follow‑up question both consumes feedback and creates fresh gradients—keeping the evolutionary signal alive without expanding model size ["Follow‑up questions expose defects layer by layer, so that every round of revision both consumes feedback and produces new feedback." 2608.13120]. By treating interaction as a continuous gradient generator, SkillEvo avoids the decay observed in single‑turn pipelines and sustains improvement across long dialogues.

The reported gains were achieved using a powerful GPT‑4o‑mini backbone; the paper does not evaluate whether smaller or open models would achieve similar token efficiency [1]. Moreover, maintaining a persistent skill bank introduces overhead for indexing and versioning, and mismanagement could lead to subtle interference between skills.

SkillEvo relies on simulated user feedback, where follow‑up questions are used to expose defects layer by layer; however, the paper does not claim that these simulations faithfully capture real‑world defects, which could limit performance in noisy production settings. If the feedback generator is biased or incomplete, the evolution gradient can drift toward spurious optimizations rather than true capability growth.

Teams can now prioritize building persistent skill banks and multi‑turn evaluation pipelines over chasing larger parameter counts, re‑running existing benchmarks with half‑sized models to achieve comparable scores. The co‑evolutionary recipe suggests that future agent performance will be defined more by the richness of their reusable skill ecosystems than by raw scaling.

If this loop holds, will the next generation of agents be judged by the size of their skill libraries rather than the size of their parameters?

References

FlashPrefillV2 gives 47 long‑context speedup

Papers Mache — Wed, 26 Aug 2026 05:00:00 +0000

Sparse block‑prefill kernels now make 128 K token prompts tractable for dense LLMs, shattering the long‑standing quadratic bottleneck that forces most services to truncate inputs. FlashPrefill V2 flips the script by delivering an order‑of‑magnitude speedup while keeping output quality essentially intact.

Before this work, inference on very long contexts relied on dense attention implementations such as FlashAttention‑2, whose quadratic scaling still dominated latency even after low‑level optimizations. The community accepted that serving 100 K‑plus tokens would require either massive hardware or severe approximation, leaving high‑quality long‑context use cases out of reach.

FlashPrefill V2 achieves a 47.26× speedup over FlashAttention‑2 at 128 K context length in FP8 precision, and still delivers 27.19× faster inference in BF16 while achieving substantially lower latency than dense baselines, delivering up to 30.49× speedup over an FA3/4‑aligned dense baseline in FP8 precision [1]. The authors attribute this gain to a mean‑corrected block‑sparse operator coupled with warp‑specialized kernels, ping‑pong pipelining, and native support for paged KV caches that align with the latest FlashAttention‑3/4 stack.

The method “confining the accuracy loss to within about one point on RULER and LongBench averages, and to within points at 128K on RULER where fewer than of the blocks are computed” demonstrates that the aggressive sparsity does not materially degrade generation quality even at extreme context lengths [1]. This bound holds across both FP8 and BF16 runs, suggesting the mean‑correction term successfully mitigates approximation error.

The paper’s evaluation stops at 128 K tokens and focuses on homogeneous workloads; it does not report performance on mixed‑length batches, multi‑GPU scaling, or models larger than those tested on Hopper GPUs. Moreover, integration is demonstrated only with SGLang, leaving open whether the same gains translate to other serving stacks or to inference under heavy request concurrency.

If these results hold in production, any service that currently falls back to truncation for inputs beyond 64 K tokens should swap its dense attention backend for FlashPrefill V2, cutting end‑to‑end latency by roughly an order of magnitude and unlocking real‑time summarisation or code completion on massive documents.

References

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving