Current visual‑grounding pipelines waste cycles by decoding box coordinates token‑by‑token, turning a simple localization problem into an autoregressive sequence generation task. The community has accepted this overhead as the price of expressive multimodal reasoning. Replacing the classification head with a direct regression module slashes latency and boosts precision at once.
Before Jag, most systems combined a pretrained language encoder with a visual merger and then emitted box corners through a sequential decoder such as an LLM or a transformer‑based pointer network. Works like ReferItGAN and GLIP demonstrated strong accuracy but required multiple forward passes to parse the generated coordinate string, inflating end‑to‑end runtime. Benchmarks such as RefCOCO measured success with Acc@0.5 while ignoring the hidden cost of text‑based box synthesis.
Jag achieves the lowest measured mean end‑to‑end latency in single‑request inference, 65.97 ms (Figure 2) [1]. The regression head reads the final token state of the fused image–language stream and emits all four corners simultaneously, eliminating the need for a generation loop. Because the visual encoder and language backbone are reused unchanged, the whole pipeline runs in one forward pass on a single GPU.
Jag improves grounding accuracy over Base on every test split, with gains ranging from 9.63 to 17.51 accuracy points (Table 2) [1]. The joint supervision forces the multimodal representations to align with geometric targets, so even small referring expressions benefit from tighter box predictions. Across RefCOCO, RefCOCO+, and RefCOCOg the model consistently outperforms its autoregressive predecessor despite a leaner architecture.
Jointly updating the language backbone, visual merger, and regression head improves Acc@0.5 by more than 60 points on each dataset relative to head‑only training [1]. This requirement means that Jag’s speedup presumes access to the full pretrained checkpoint and additional fine‑tuning compute; freezing any component collapses the accuracy advantage. Moreover, the reported gains stem from a compact encoder, leaving open whether the same head redesign scales to massive multimodal LLMs without prohibitive memory overhead.
Benchmarks that currently rely on autoregressive grounding models should replace their output heads with Jag‑style direct regressors, expecting substantially lower latency and a double‑digit accuracy lift on standard GUI datasets.
Top comments (0)