Tencent has released RxBrain, an open roughly 6.2-billion-parameter robot model that plans not just in words but by generating images of the world it wants to create. Instead of producing a text plan and handing it to a controller, RxBrain interleaves textual reasoning with generated pictures of each subgoal's desired physical state, then feeds both to a downstream action model. It is a concrete bet on one side of a live debate in robotics: whether a robot needs an explicit imagined picture of its goal, or just enough real-world data to learn the motions directly.
Key facts
- What it is: a ~6.2B-parameter unified embodied-cognition model, released on Hugging Face under Apache-2.0.
- The mechanism: it alternates text reasoning with generated goal images, associating each subgoal with a target physical state (paper).
- Training data: Tencent reports about 50,177 hours of pre-training data and roughly 210 million examples.
- The gap: goal-image quality (0.52) lags observation understanding (0.83) on its own benchmark.
The background: most robot foundation models are vision-language-action systems, explained in the vision-language-action models lesson, that map what the robot sees and is told into motor commands. RxBrain adds a layer above that. It generates an interleaved planning sequence where text and generated images take turns: reason about the task, then produce an image of what the scene should look like after the next step, then reason again. Tencent presents the text and the goal image as complementary high-level conditions for a separate action model that actually moves the arm.
Under the hood, RxBrain routes different modalities, text, observed vision, and generated vision, through their own pathways while letting them share attention, an architecture related to a mixture of experts. The generated frames are produced with flow matching in a compressed latent space, the technique in the flow matching explainer, and then fed back into the model's context so later reasoning can use them. The picture to hold: a robot that sketches what it is aiming for, checks its sketch, and uses the sketch to guide its hands.
Why it matters: this reframes robot planning around imagination rather than pure trajectory imitation. Tencent reports that extending RxBrain with an action module reaches 87% average success across three multi-stage real-robot tasks on two arm types, versus 82% for a strong baseline. In the company's own words, the goal image and the text act as "complementary high-level conditions" for the controller. It arrives alongside a contrasting bet: Xiaomi-Robotics-1, which is not yet released, argues general behavior comes from industrial-scale real data, pre-training on more than 100,000 hours of manipulation trajectories. One approach tries to learn the hands; the other tries to supply the mind's eye.
The honest caveats are specific. RxBrain's own weak point is exactly the imagination it is built around: on its unreleased benchmark, its goal-image component scores 0.52, well below observation understanding at 0.83 and subgoal planning at 0.78, and its free-running joint-planning score falls from 0.69 at two steps to 0.55 at eight. So long-horizon visual imagination is the reported soft spot, not a solved problem. And "open" needs scoping: the weights and inference code are Apache-2.0, but the official repository still lists the RxBrain benchmark and the action-model fine-tuning code as to-do, and the public scripts cover reasoning, image generation, and planning inference rather than a full action-training entry point. The 87% success figure is author-reported and not independently testable from the released materials. The model is genuinely usable today; the reproducible, full-stack version is still coming.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)