Registration for the WebRetriever Global Challenge is now open. Hosted by Mininglamp Technology, the competition is co-organized with Peking University, the Institute of Automation at the Chinese Academy of Sciences, the AI and Robotics Innovation Center at the CAS Hong Kong Institute for Advanced Study, and Synced (机器之心).
The total prize pool is $15,000 USD. Both individuals and teams are welcome, with no restrictions on nationality or institutional affiliation.
Why this benchmark exists
When an AI agent steps into a real browser, can it actually complete a task on its own, the way a human would, across the messy and ever-changing open web?
This remains the central bottleneck holding Web Agents back from real-world deployment. Existing benchmarks mostly rely on a small number of simulated or self-hosted sites that fall far short of the complexity of the live internet. On the evaluation side, current methods focus heavily on whether individual actions were executed correctly, but lack a systematic way to measure whether the agent actually delivered the end result the task demanded.
WebRetriever is our attempt at building a more honest measuring stick. The paper has been accepted at ECCV 2026.
Scale: 800 real live websites, 1,550 cross-domain tasks spanning eight verticals including tech, finance, healthcare, education, and government, all running against the actual public internet.
Evaluation accuracy: Our NavEval framework achieves 91.2% agreement with human expert judgments, compared to roughly 81% from prior best methods, making automated large-scale evaluation reliable for the first time at this scale.
What the numbers say: Even the best single model achieves under 50% on basic navigation success, and end-to-end task completion hovers around 20%. Getting there is not the same as getting it done.
How to register
Competition platform (Octo): https://im.deepminer.com.cn/
Invite code: 0f351ca01bb4c4dd
Step 1 — Sign up for an Octo account (skip if you already have one). You can register via browser (email recommended for timely updates), or if you use Claude Code, ChatGPT Codex, Cursor, or similar AI coding tools, register directly from your terminal via https://mininglamp-ai.github.io/WebRetriever_Challenge/join/ — no browser required.
Step 2 — Join the competition space using the invite code above.
Step 3 — Submit your team information (team name, members, etc.) following the in-space instructions.
Resources
Paper: https://arxiv.org/abs/2607.06118
Dataset: https://huggingface.co/datasets/Mininglamp-2718/WebRetriever
Code & leaderboard: https://mininglamp-ai.github.io/WebRetriever
Challenge page: https://mininglamp-ai.github.io/WebRetriever_Challenge/
Top comments (0)