Researchers introduce SWE-CoWork, a benchmark of 101 development assignments across 101 real repositories, 11 languages, and 12 domains. Agents work in a simulated organisation with 26 colleagues, recover requirements from people and records, and must collaborate and verify their work.
Why it matters
Real coding work is collaborative and underspecified; benchmarks that test only isolated code generation may overstate workplace readiness.
Analysis
SWE-CoWork addresses a gap between isolated coding tests and workplace practice by adding distributed requirements and colleagues. That should expose planning, communication, and verification failures. Main uncertainty: whether simulated colleagues behave realistically enough for results to predict performance in real organisations.
Source: arXiv
AI disclosure: this article was produced with AI assistance and reviewed for accuracy.
Top comments (0)