RelateAnything shatters the low‑recall ceiling that has limited zero‑shot visual relationship detection, delivering 2.3–3.5× higher mean recall while staying under a 20 ms per‑frame budget. Prior models were shackled to a fixed set of 50‑56 predicates and relied on object‑label conditioning, preventing them from handling arbitrary relation vocabularies at inference time.
The field has long been constrained by three intertwined obstacles: no free‑text, verified relation corpus; architectures that condition on learned object labels; and evaluation metrics that reward agreement with the training taxonomy rather than true open‑vocab performance. Consequently, scene‑graph systems remained tied to a single detector and could not ingest novel predicate strings.
RelateAnything achieves a mean recall 2.3–3.5× higher than the strongest open‑vocabulary baseline of comparable scale across three cross‑dataset benchmarks and one zero‑shot setting [1]. The boost persists even when paired with a real detector, and it outperforms a 3B‑parameter VLM scene‑graph model while using under 2 % of its parameters.
The relation model runs at 20 ms per frame on a single GPU, alongside a real‑time object detector [1]. This speed translates to roughly 49 FPS end‑to‑end, enabling truly live video processing without sacrificing the recall gains.
The authors acknowledge that training still depends on the RA‑4M corpus covering 10 k free‑text predicates and that positive‑unlabeled supervision may limit performance on completely novel vocabularies; moreover, in‑domain measurements can overstate transfer benefits by up to fivefold. This suggests further work is needed to verify robustness when the predicate bank expands beyond the curated set and to devise metrics that capture true open‑vocab generalisation without bias toward the training corpus.
If these numbers hold under broader scrutiny, developers should replace fixed‑predicate scene‑graph pipelines with RelateAnything for any real‑time video stream, re‑evaluating their systems on OV‑SGG‑Bench or similar cross‑dataset suites to expose hidden recall deficits.
Top comments (0)