Why Vision-Language Models Should Reroute, Not Remove Visual Tokens

Vision-language models are getting better at reading charts, spotting objects, and answering questions about images. But that progress comes with a familiar cost: more visual detail usually means more visual tokens, and more tokens means more compute, more memory, and slower inference.

A recent paper, Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models, proposes a small but important change in how we cut that cost. Instead of permanently deleting low-priority visual tokens, the method lets them stay in play and re-enter the candidate pool later. In other words: reroute them instead of removing them.

That sounds like a minor implementation detail, but it changes the trade-off quite a bit.

Why visual tokens are expensive

A modern vision-language model does not usually hand an image to the language model as a single blob. It turns the image into many patch-level embeddings, then feeds those embeddings into the decoder alongside text tokens.

That gives the model the chance to reason over fine-grained visual evidence, but it also creates a scaling problem:

• Higher-resolution images create more tokens.
• Video multiplies the problem across frames.
• Attention cost grows with sequence length.
• KV-cache memory grows too, which matters during long conversations or multi-step reasoning.

If you are building a multimodal app, this is not an abstract concern. It is the difference between a model that can inspect a dense document or video clip and one that times out or runs out of memory.

The current generation of efficiency work is therefore less about making models "smaller" in the abstract and more about deciding which parts of the input really deserve compute.

The problem with irreversible pruning

Most visual token reduction methods follow a simple logic:

1. Score the visual tokens.
2. Keep the most important ones.
3. Delete the rest.

That works when importance is stable, but importance in a vision-language model is not always stable. The paper argues that a token that looks unimportant in an early stage may become useful later, especially for grounding-sensitive tasks.

Think about an image of a cluttered desk. A token covering the corner of a notebook may not matter when the model first sees the scene. But later, if the question becomes "What brand is written on the notebook cover?", that token suddenly matters.

Once a token is physically removed, the model cannot recover it. That is the core weakness of rank-and-remove pruning.

What recoverable routing changes

The key idea in Reroute, Don't Remove is to treat reduction as a routing problem rather than a deletion problem.

Instead of throwing away deferred tokens, the model lets them bypass a stage and stay eligible for later selection. Tokens that are selected pass through the current decoder block, while deferred tokens are not destroyed. They simply wait for the next routing decision.

The authors describe this as a training-free plug-in that can sit on top of existing token reduction methods such as FastV and PDrop. That is useful because it means the method does not require redesigning the whole vision-language stack.

The practical effect is straightforward:

• You still get a smaller active set of tokens at each stage.
• You preserve the chance to recover visually important tokens later.
• You reduce the risk of losing grounding information too early.

The paper reports that this helps under aggressive token reduction, especially on tasks where spatial evidence matters.

Why that is a better fit for multimodal reasoning

This approach lines up with a broader lesson from multimodal systems: not all redundancy is wasted.

Sometimes the model needs a rough pass first, then a second look. A token that seems unimportant during global scene understanding may become important during object grounding, OCR-like reading, or fine-grained comparison.

Recoverable routing gives the model a second chance to notice those details without paying the full cost of keeping every token live all the time.

That is a more realistic compromise than hard deletion. It accepts that multimodal inputs are messy and that importance can change across depth.

How this fits into the 2026 efficiency trend

The paper is part of a wider shift in multimodal efficiency research. Instead of treating token reduction as a simple compression task, recent work is moving toward methods that are more adaptive and more task-aware.

For example, the broader token-reduction landscape now includes training-free acceleration methods such as FlashVID, which uses attention and diversity-based token selection plus tree-based spatiotemporal token merging for video models. There is also growing interest in surveys and collections that map the field’s many pruning, merging, and compression variants, such as Awesome-Collection-Token-Reduction.

Meanwhile, product teams are also making efficiency choices visible in the architecture itself. Meta’s SAM 3.1 release introduced object multiplexing for real-time video detection and tracking, reducing redundant passes by tracking multiple objects in a single forward pass. It is not the same technique, but it points in the same direction: multimodal systems are being built around explicit compute budgets, not just raw capability.

What developers should take away

If you work on multimodal systems, the paper suggests a few practical rules of thumb:

1. Measure grounding, not just latency

A faster model that misses the relevant object is not better for many real workflows. Benchmarks need to include grounding-sensitive tasks, not only throughput and FLOPs.

2. Be careful with early pruning

The earliest pruning decision is not always the safest one. If your task needs iterative reasoning, preserve room for later recovery.

3. Think in stages

A layered routing scheme can be easier to reason about than a one-shot keep-or-delete choice. Different layers can make different decisions about the same token.

4. Prefer methods that preserve optionality

In multimodal systems, optionality has value. Keeping a token eligible for later selection can be cheaper than discovering too late that it was needed.

Closing thought

The most interesting part of Reroute, Don't Remove is not just that it saves compute. It is that it reframes visual token reduction as a reversible process.

That is a useful design principle for vision-language models in general. If the model is still deciding what matters, do not be too eager to delete evidence. Route it, defer it, and let later layers make the final call.

For multimodal systems, that small shift can make the difference between efficient and brittle.