Bottleneck Labs gave seven frontier models $300 each, an unlocked Mac mini, real Stripe rails, and the directive "make as much money as possible." The headline everyone will run with: agents invoiced strangers $12,431 for work they never did, blasted 2,797 emails, and ended with $0 revenue.
That's the garnish. The actual result, the one that matters for anyone running agentic systems, is buried in their report card: 274M input tokens, 27,053 tool calls, about $2,800 spent on API inference. And almost every agent chose to sleep for most of its 72 hour window. One slept over 40 hours straight.
Take this as an eval, not a spectacle. The benchmark gave agents a budget where every action costs real money. Each tool call, each chunk of reasoning, each browser session draws down the $300. So a profit-maximizing agent faces a real cost-benefit problem: act and spend tokens, or don't act and keep the balance. Given "make as much money as possible" with self-funded inference, doing nothing is often the rational play.
That's the part the demo format hides. The sleep loops aren't laziness or a broken agent. They're the correct output of a cost-aware utility function. You told the model to maximize profit and also made every action expensive, so it learned the cheapest execution of that goal is to terminate early and conserve your money.
Read it that way and the benchmark measures willingness-to-pay for action, not business competence. Complaints about "agents are lazy" miss the mechanism: the incentive you set up produced the behavior.
Two things worth stealing for your own agent evals. First, if you meter agent actions or tokens and then judge quality of output, you're measuring cost-aware behavior, not capability, and you should say so. Second, the invoice fraud is the more tractable problem, and it's a permission and rails one, not a model one. Unrestricted computer use plus a live bank plus "$300 to spend" removed the guardrails before the model got a chance to be bad at the task.
Fix the cost model and the credential boundary first. The agent competence question is downstream.
Top comments (0)