Key Takeaways
- New research identifies a “Compression Gap” in Vision-Language-Action (VLA) models, showing that discrete action tokenization can bottleneck performance scaling even as vision encoders improve.
- The bottleneck stems from fixed-capacity codebooks in discrete action representations — richer visual inputs simply cannot propagate through a constrained action vocabulary.
- Solutions include continuous action representations like diffusion policies, improved learned tokenizers, and hybrid architectures that pair discrete reasoning with continuous action decoding. Scaling up a robot’s vision system should make it better at physical tasks — but new research shows that assumption breaks down when actions are encoded as discrete tokens. A paper published this week on arXiv, “The Compression Gap: Why Discrete Tokenization Limits Vision-Language-Action Model Scaling,” identifies why upgrading vision encoders in many VLA models delivers diminishing returns: the action tokenizer becomes the tightest bottleneck in the pipeline, swallowing the gains before they reach the motors. Here’s what that means in practice, and what the field is doing about it.
Phase 1: Understanding the Compression Gap in VLA Models
Acknowledge the Role of Discrete Tokenization
Discrete tokenization — converting continuous action signals into a fixed vocabulary of tokens — has been a foundational design choice in VLA models, including RT-1, RT-2 and OpenVLA. The appeal is obvious: by treating robot actions as sequences of discrete symbols, VLA models can plug directly into the transformer architectures and training pipelines developed for large language models (LLMs). A gripper position or joint angle gets binned into discrete IDs, much like words in a sentence, letting the model process visual inputs, language instructions and robot actions within a single unified framework. That simplicity has real advantages — but it comes with a hidden cost.
Identify the “Compression Gap” as an Information Bottleneck
The core finding is an information-theoretic problem. When actions are discretized through a fixed-capacity codebook, that codebook becomes the tightest constraint in the entire visuomotor pipeline. No matter how rich the upstream visual representation becomes, the fixed action vocabulary limits how much of that information actually reaches execution. It’s a classic compression ceiling: the encoder improves, but the channel doesn’t.
Experiments on the LIBERO benchmark illustrate this clearly. Continuous action policies such as Diffusion Policy show substantial performance gains when the vision encoder is upgraded. Models that rely on discrete action tokenization — such as OAT — show attenuated gains across the same scaling range. The codebook’s capacity, not discreteness per se, is the limiting factor. Fine-grained details critical for precise physical manipulation get lost in compression before the robot ever moves.
Phase 2: Implementing Continuous Action Representations
Embrace Diffusion-Based Policies for Fine-Grained Control
The most direct way around the discrete bottleneck is to drop it entirely and move to continuous action representations. Diffusion policies have emerged as the leading approach here — they generate continuous action trajectories directly, providing the high-frequency precision that dexterous manipulation demands. Unlike autoregressive discrete token generation, diffusion models can produce action sequences in parallel, which matters for latency on long-horizon tasks.
Two notable implementations show what this looks like in practice. “Discrete Diffusion VLA” models discretized action chunks using discrete diffusion, retaining the progressive refinement of the diffusion paradigm while staying compatible with the discrete token interface of standard VLMs — including adaptive decoding order and improved error correction. A separate framework, E0, formulates action generation as an iterative denoising process over quantized action tokens, offering flexible control over discretization granularity and planning horizon. Both approaches demonstrate strong generalisation across simulation and real-world environments.
Explore Flow Matching for Smooth Trajectory Generation
Flow matching is another continuous-action technique gaining traction in VLA frameworks. Rather than binning actions, it directly models the transformation from a simple distribution to the complex distribution of target actions — producing smooth, continuous trajectories rather than the potentially jerky outputs of discrete binning. Combined with vision-language backbones, flow matching gives the generated actions the continuity that fluid physical interaction requires. It’s particularly well-suited to general-purpose robot control where the action space is high-dimensional and smooth motion matters.
Phase 3: Advancing Discrete Tokenization Techniques
Continuous representations aren’t always the right trade-off — there are good reasons to keep leveraging existing VLM architectures. The alternative is to make discrete tokenization itself smarter, moving beyond fixed-capacity codebooks and naive per-dimension binning.
Develop Scalable Learned Tokenizers (VQ-VAE based)
Vector Quantized Variational AutoEncoder (VQ-VAE) based tokenizers take a learned approach to building the codebook, adaptively capturing the spatio-temporal dynamics of robot actions rather than relying on hand-designed bins. Research on VQ-VLA shows that tokenizer precision correlates directly with improvements in long-horizon action modeling — and that synthetic action data can be used to scale tokenizer training without meaningful performance loss in real-world deployment, because the domain gap between simulated and real action trajectories is small. As synthetic data volume increases, these tokenizers show linear scaling properties in task success rate, inference speed and cumulative error reduction. Critically, it’s computationally far cheaper to scale the tokenizer than to scale the entire VLA model.
Utilize Frequency-Domain Compression (FAST)
Frequency-space Action Sequence Tokenization (FAST) takes a different angle, using discrete cosine transforms (DCT) to compress action signals in the frequency domain rather than binning them dimension by dimension. This makes it viable for highly dexterous, high-frequency tasks where per-timestep binning schemes break down. FAST+, a universal robot action tokenizer trained on millions of real robot action trajectories, operates as a black-box tokenizer across diverse action spaces and control frequencies. According to the research, when combined with VLA models, FAST reduces training time significantly while matching the performance of diffusion VLAs — making it a strong efficiency play for teams that want to stay within an autoregressive framework.
Consider Ordered Tokenization (OAT)
Ordered Tokenization (OAT) attempts to satisfy compression, decodability and causal structure simultaneously through structured discrete representations. The “Compression Gap” paper highlights OAT’s current limitations with fixed-capacity codebooks, but it remains an active research direction. Future work is exploring adaptive codebooks and hybrid schemes that extend OAT’s capacity without abandoning the causal structure that makes it attractive for autoregressive generation.
Phase 4: Implementing Hybrid Architectures
The most architecturally ambitious response to the compression gap is to stop choosing between discrete reasoning and continuous control — and build systems that use both.
Integrate Collaborative Diffusion and Autoregression (HybridVLA)
HybridVLA is a unified framework that addresses the weaknesses of both approaches head-on. Purely autoregressive discrete methods disrupt action continuity; purely diffusion-based methods don’t fully exploit the pretrained reasoning capabilities of VLMs. HybridVLA incorporates diffusion denoising directly into the next-token prediction process within a single large language model, using a training recipe designed to prevent the two generation paradigms from interfering with each other.
The results show that discrete and continuous prediction methods can reinforce rather than compete with each other, with each showing relative strengths on different task types. A collaborative action ensemble mechanism adaptively fuses both predictions at inference time, producing control that is reportedly more robust than either approach alone on both simulation and real-world benchmarks.
Utilize Dual-System Designs for High-Frequency Control
A second hybrid approach separates the problem architecturally: a large vision-language backbone handles high-level reasoning and task understanding, while a separate fast visuomotor policy converts those internal representations into continuous control signals at the frequency real hardware demands. Figure AI’s Helix VLA model for humanoid robots follows this pattern — System 2 (slow, language-grounded reasoning) handles instruction parsing and scene analysis, while System 1 (fast, reactive control) generates the smooth motor commands. This split sidesteps the compression gap entirely in the control loop, because discrete tokens never need to carry fine motor information — that’s handled downstream by a dedicated continuous policy.
Phase 5: Data and Evaluation Strategies for Scaling VLA Models
Leverage Diverse and High-Quality Data
Data scale and diversity remain foundational to VLA performance regardless of the action representation chosen. Initiatives like Open X-Embodiment, Droid and BridgeData compile large demonstration sets across varied tasks, environments and robot platforms. For learned tokenizers specifically, the ability to scale on synthetic data is a practical advantage — research suggests action trajectories show minimal domain gap between simulation and reality, meaning synthetic data can train tokenizers without meaningfully hurting real-world performance.
Adopt Robust Evaluation Metrics and Protocols
Evaluation methodology matters as much as architecture. Beyond simple success rates, metrics should capture action precision, trajectory smoothness and generalisation across novel environments. For real-world testing, bias in outcome assessment is a genuine risk — a “Grouped Blind Ensemble protocol,” which blinds operators to model identity and separates policy execution from outcome judgment, is one approach designed to reduce experimenter bias. Rigorous evaluation separates genuine progress from results that don’t survive contact with uncontrolled environments.
The compression gap is a real architectural constraint, and it explains why throwing better vision encoders at discrete-tokenization VLA models has produced disappointing returns. The solutions — continuous diffusion policies, smarter learned tokenizers and hybrid architectures that route reasoning and control through different subsystems — each make different trade-offs in complexity, compute and compatibility with existing VLM infrastructure. There’s no single right answer yet, but the research direction is clear: the action representation layer needs as much engineering attention as the vision and language components that feed it. For more coverage of AI chips and infrastructure, visit our AI Hardware section.
Originally published at https://autonainews.com/how-to-overcome-discrete-tokenization-limits-in-vision-language-action-models/
Top comments (0)