A tool shortlist often compares unlike things. One candidate returns extracted data. Another gives you a browser connection. A third includes an agent that decides what to click. Their demos can all look successful while stopping at different points in your actual job.
Before scoring them, define the same finish line. Then ask each candidate stack to reach it, including whatever application code and operator work it requires.
This is a proposed evaluation method for browser automation projects. The export scenario below is fictional; it is not a benchmark of any product.
Define the job in terms a reviewer can verify
Consider a monthly report export from a permitted test account. “Open the dashboard” is a useful intermediate step. “Return a report for October from the requested account, with evidence of its scope” is a deliverable.
Write the acceptance contract before building any adapters:
{
"case_id": "monthly-report-01",
"requested_account": "test-account-a",
"requested_period": "2026-10",
"required_output": "report_file",
"acceptance": [
"file_is_readable",
"account_matches_request",
"period_matches_request"
],
"allowed_intervention": "operator_handles_authentication",
"submission_policy": "inspect_uncertain_result_before_retry"
}
This JSON is an example of an application-owned contract, not a vendor API schema. Replace its fields with the checks your real task needs. A collection task might require source URLs and field coverage; a report task might require document dates and a file format.
Do not let the easiest candidate redefine completion. A screenshot, a live browser connection, and a verified file can each be useful outputs. They satisfy different contracts.
Describe the whole composition for each candidate
Keep a small manifest beside every implementation. List the browser runtime, action-selection component, application adapter, verifier, and operator responsibilities. Record versions or revisions so the result can be interpreted later.
For example, a browser-hosting option may be paired with your existing automation code. An extraction service may need a small application wrapper. An agent interface may require instructions for account checks and recovery. These are possible compositions to document, not claims about the current feature boundaries of particular vendors.
If your project already has a reliable component, say that you reused it. If you wrote new glue code during the trial, count that work too. Otherwise you can accidentally compare a complete implementation against an isolated component and attribute the difference to the wrong layer.
Separate missing capability from missing integration
A failure deserves a useful label before it receives a score:
| Outcome | What the evaluation can conclude |
|---|---|
| Requested output passed every acceptance check | This case completed under the recorded conditions |
| A required action has no supported path in the evaluated setup | This setup does not cover the case |
| The adapter has not been implemented | Coverage is still unknown; implementation work remains |
| Authentication needs an allowed operator step | Continue under the same intervention policy for all candidates |
| Submission may have succeeded but cannot be confirmed | Mark uncertain and inspect before another submission |
Do not convert “we did not build that integration” into “the product cannot do it.” Equally, a feature in documentation does not establish that your configured stack completed the case.
Record the evidence behind each label: the observed state, the returned artifact, or the documented gap. Keep a confirmed failure separate from an untested claim.
Use the same verifier at the finish line
Where practical, have one application-owned verifier inspect outputs from every candidate. It should not accept an output simply because an adapter reports success.
For the fictional export, the verifier could open the file, check the expected account and period, and record a content hash. If the format does not expose those values reliably, define an alternative evidence requirement before the trial. Do not invent a confident match from a filename alone.
The verifier should return both a decision and the checks performed. Keep the submitted request beside that result. A correct file for a different account or month is still the wrong deliverable.
Report two stages of cost
Separate the work needed to make the first accepted case possible from the work needed to repeat it. Setup, adapter development, instruction writing, and verification design belong in the first stage. Runtime, model usage, retries, and operator intervention belong in the repeated run record.
Avoid collapsing an incomplete run into a low-cost success. Track accepted, rejected, uncertain, and untested cases individually. State whether stopped attempts and operator recovery are included in any aggregate figure.
If scope differs, report that difference before presenting a ranking. A stack that covers one simple extraction case cannot establish an advantage on a suite that also includes authenticated exports and uncertain submissions.
Make the decision conditional on coverage
Choose the composition that covers the required cases within your team's operating constraints. Keep optional capabilities separate from requirements, and retain explicit unknowns when a trial has not exercised them.
This can lead to a mixed stack: an existing component handles a stable part of the job while a different interface handles exploration or operator checkpoints. The acceptance contract keeps that composition reviewable when its parts change.
I work on BrowserAct. This article expands the category-selection argument in our browser automation tools comparison. It proposes a way to compare complete implementations; it does not claim a measured winner, product pricing, or a tested customer-portal workflow.
Top comments (0)