A browser job can finish in thirty seconds and still leave someone with ten minutes of reconciliation. If your evaluation counts only browser runtime, the slower implementation can look cheaper while doing more useful work.
When comparing a framework such as Playwright with an agent workflow product such as BrowserAct, I would start with a small outcome ledger. The purpose is to expose who pays for retries, review, and maintenance. This is a proposed measurement method, not a benchmark of either product.
Disclosure: this article is published by BrowserAct. The numbers below are fictional arithmetic examples, not product prices or customer results.
Define one unit of useful work
Take a monthly invoice collection job. A useful unit could be one supplier account and one billing month, with an agreed list of expected documents.
A completed browser session is not automatically a completed unit. Your acceptance rule might require readable files, matching account and date range, and a manifest that identifies missing documents. If the expected set is unknown, record completeness as unknown instead of silently counting it as complete.
Use the same acceptance rule for both implementations. Otherwise you are comparing a file collector with a page navigator and giving them the same success label.
Keep attempts inside the original job
Suppose a request needs three attempts before the files are verified. That is one successful business job with three attempts. It is not three jobs, and it is not three successful completions.
Assign a job ID before starting. Record attempt IDs underneath it. Keep failed and uncertain attempts in the cost numerator, even when a later attempt succeeds.
Here is a fictional application-side record. It is a design sketch, not a BrowserAct or Playwright API schema:
{
"job_id": "demo-supplier-2026-10",
"acceptance_version": "invoice-set-v1",
"attempt_count": 3,
"outcome": "verified",
"runtime_cost_usd": 0.18,
"model_cost_usd": 0.12,
"operator_minutes": 8,
"review_minutes": 2,
"evidence_ref": "internal://demo/manifest-001"
}
Use references to protected evidence rather than putting invoice contents or credentials into the evaluation table. Track operator time and review time separately so you can see whether a change removes work or merely moves it to someone else.
Show recurring and setup costs separately
For a fixed evaluation window, recurring cost per verified job can be calculated as:
(runtime + model + service allocation + human time + maintenance)
----------------------------------------------------------------
unique verified jobs
Service allocation means the portion of a subscription or shared infrastructure assigned to this workload. Write down the allocation rule. Do not charge the entire shared service to one candidate and exclude equivalent infrastructure from the other.
Report initial implementation time separately. If you amortize it, state the assumed volume and lifetime, then show a second result without amortization. A short pilot cannot establish how many future jobs will actually run.
Elapsed waiting and active human work also differ. A twenty-minute export may require no operator attention. A two-minute export that interrupts someone five times has a different operating cost. Record both elapsed duration and active minutes.
Work through a fictional month
Imagine 100 requested jobs, of which 90 meet the acceptance rule. Runtime and model costs total $45 across all attempts. Allocated services cost $25. Operators and reviewers spend four hours, valued at a hypothetical $30 per hour. Maintenance takes another two hours at that same assumed rate.
Recurring cost is $250: $45 + $25 + $120 + $60. Dividing by 90 verified jobs gives about $2.78 per verified job.
If you counted only runtime and model usage, the result would be $0.50. Neither number is a product quote. The gap shows why the ledger needs human work and maintenance.
The ten unresolved jobs remain visible. Report the completion rate alongside cost, since an implementation could appear cheap by abandoning difficult work. If there are zero verified jobs, report cost per verified job as undefined, not zero.
Make the comparison fair enough to act on
Use matched tasks, accounts, periods, and acceptance rules. Separate normal jobs from expired-login cases and changed-page cases. Keep retry budgets and operator access comparable. Record why a job was excluded before calculating results.
Do not assume a framework has no authentication or debugging support. Playwright documents both authenticated-state reuse and a Trace Viewer. Your cost question is how your implementation uses those capabilities and who maintains the surrounding workflow.
Apply the same scrutiny to a managed product: measure actual operator minutes and maintenance rather than assuming a subscription eliminates them. Include time spent updating the reusable instructions when a website changes.
Use the ledger to decide what to fix next
High review time points toward a clearer acceptance rule or better evidence. High recovery time points toward login handling and escalation. High maintenance time points toward workflow ownership and change management. Expensive model usage deserves attention, but it is only one line.
Our BrowserAct vs Playwright guide covers the broader responsibility split. The practical next step is a matched pilot with a ledger you can audit, followed by a second measurement after one real maintenance event.
Which cost do you currently record least reliably: review, recovery, or maintenance?
Top comments (0)