DEV Community

Junfeng Zhou
Junfeng Zhou

Posted on

websandbox: the WebArena sites as small SQLite + Node servers that reset in milliseconds

If you train or evaluate web agents on WebArena, a lot of the cost is the environment rather than the agent: five heavy stacks (Magento, Postmill, GitLab, Kiwix, OpenStreetMap) and resets that mean restarting containers.

I rebuilt the five WebArena sites as lightweight, self-hostable versions: a store with its back office, a forum, a code-hosting site, a wiki and a map. Each site is one SQLite database plus one small Node server. It boots in under a second and resets to its initial state in milliseconds (a rollback of a transaction that is never committed), so one machine can run dozens of independent copies of every site.

What is included:

  • All 812 WebArena tasks converted to these sites: 807 verified by browser runs, 5 dropped with the reasons documented.
  • 12,283 generated tasks from 217 templates, each template checked with UI runs.
  • Template-disjoint train and held-out splits (10,452 and 2,638 tasks), so no held-out template is seen in training.
  • Graders that check answers, URLs and site state.

The data (about 21 GB, the same underlying data as WebArena) is a Hugging Face dataset, and the code is Apache-2.0.

Code: https://github.com/junfengzh/websandbox
Data: https://huggingface.co/datasets/junfengzhou/websandbox
Demo, one copy of each site reset hourly: https://websandbox.pystem.com

Feedback is welcome, especially from anyone running agent RL or evals on WebArena-style tasks.

Top comments (0)