How I Measure Whether My Agent Is Actually Saving Me Time

Every "my agent did X in Y minutes" post measures the happy path. The honest question is different: did the whole week get shorter?

Here is the measurement that survived my own bias.

Measure the task, not the demo

For each task I track four numbers:

• human baseline - how long it took me by hand, averaged over 3 runs
• agent wall time - clock time from start to a result I accepted
• review time - how long I spent checking the diff/output afterwards
• rework time - anything I had to redo because the result was wrong

Agent wall time is the number everyone quotes and the least interesting one. review + rework is where the truth lives. A task that takes the agent 2 minutes and costs me 25 minutes of review is a loss, and it looks like a win in every screenshot.

The threshold that changed my priorities

My rule: a task is worth automating when

agent wall time + review + rework < human baseline

but only after at least 5 runs. Run 1 is always slow - the agent does not know the environment, and I do not know what the failure modes look like. Runs 1-3 usually look like regressions, and that is expected. Deciding after one run is how people conclude agents do not work.

Track the interruptions

The metric I did not expect to matter: how many times the agent stopped and waited for me. Each interruption costs a context switch, not just the seconds. Ten 30-second interruptions are worse than one 5-minute wait, because a 5-minute wait can be filled with other work and a context switch cannot.

So: batch the approvals. If a task needs six decisions, do not ask six times - collect them and ask once with all the payloads.

Kill the tasks that never converge

Some tasks are stable at 40% success no matter what you do: anything with a login wall, anything behind a captcha, anything where the UI is a single opaque canvas. After ten runs, if the success rate is not climbing, stop. Negative results are data. Keeping a flaky automation in the loop is worse than doing it by hand, because you keep paying the review cost for a result you cannot trust.

Numbers from a real month

Roughly 400 tasks. About 30% ended up cheaper than doing them by hand, and those accounted for most of the saved time. About 20% were abandoned after the fifth honest attempt. The rest sat in the middle and stay there until the tooling around them improves.

That distribution is the honest pitch for agents: not a replacement, a coverage map of where delegation currently pays off.

If you track anything similar, I would like to know what metric you use that I am missing - especially one that made you stop automating something you had already shipped.