The prevailing trend in multimodal foundation models is to collapse vision, language, and action into a single transformer backbone. VideoChat3 breaks that mold by deploying a dedicated video‑centric backbone, while RoboTTT achieves long‑context robot capabilities through test‑time training integrated into existing vision‑language‑action policies rather than a separate backbone.
Until now most open‑source multi‑modal systems reuse the same ViT‑style encoder for images, video frames, and even robotic observations, which forces a compromise between spatial fidelity and temporal depth. This shared design often inflates token counts or limits context length, making it hard to scale efficiently across domains.
VideoChat3 delivers state‑of‑the‑art video understanding with a modest 4 B‑parameter transformer while halving visual token count. “Experiments across general, long‑form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open‑source models with equal or larger parameter counts with only 4B parameters and higher efficiency.”[1]
RoboTTT lifts robot manipulation performance by 87 % and solves a five‑minute, ten‑stage assembly that no prior policy completes. “On challenging real‑robot manipulation tasks, RoboTTT improves overall performance by 87% over the single‑step context baseline and fully completes a five‑minute, ten‑stage assembly task, which no baseline ever does.”[2]
Both advances leave open the question of how far separate backbones can scale without prohibitive training costs. RoboTTT’s gains hinge on “integrates Test‑Time Training into robot foundation models, … parameters updated by gradient descent during both training and inference.”[2] VideoChat3 is evaluated only on general, long‑form, and streaming video benchmarks, so its behavior on interactive or ultra‑low‑latency streams remains untested.[1]
If dedicated backbones continue to outpace monolithic designs, engineering pipelines should abandon the one‑size‑fits‑all model in favor of modality‑specific foundations, cutting integration effort and token budgets while preserving performance.
Top comments (0)