Two browser jobs can both report success and deliver different results. One may leave original invoice files in the finance archive; another may return a table of links that expire before anyone retrieves them. Comparing their completion times as if they performed the same job produces a misleading result.
Our BrowserAct and Skyvern invoice-workflow comparison uses documentation to shortlist execution routes. It does not establish a same-portal speed, cost or success-rate winner. This post proposes the measurement harness that would be needed before making those claims.
Freeze the task, including its destination
Start with one authorized account, an explicit invoice-date interval and a known expected document set. Record the supplier region and account variant. Decide whether the task needs invoice PDFs, shipment details, credit memos or a subset of those documents.
The task ends at the agreed destination, not at the final browser click. If finance needs durable files in an approved archive, both candidates must be evaluated against that same handoff. A temporary result link is an intermediate artifact unless that is explicitly the requested deliverable.
Store the approved scope separately from each run's observations. If a candidate silently changes an empty date filter or switches to another account, preserve the mismatch rather than accepting the files as equivalent output.
Record stages rather than a single success flag
A comparison record should make intermediate success visible without confusing it with delivery. An illustrative application-side record could contain:
{
"experiment_ref": "invoice-pilot-A",
"scope_ref": "approved-scope-v1",
"candidate_ref": "configured-route-A",
"stages": {
"request_accepted": true,
"portal_access_confirmed": true,
"retrieval_finished": true,
"archive_reconciled": false
},
"unresolved": [
{
"expected_document_ref": "example-invoice-004",
"reason": "retrieved_file_identity_mismatch"
}
]
}
This is a synthetic record, not an output schema from either product or evidence of a real run. Its purpose is to show how a finished retrieval can coexist with an incomplete archive. Counts and timestamps should come from actual observations, with the evidence needed to audit them.
A file manifest can carry document reference, account reference, document type, saved path, byte count, checksum and verification result. Check business identity independently of byte validity. Two valid PDFs are not interchangeable merely because they both open.
Keep human work inside the observation window
Record when a human had to renew login, confirm an account, handle a second factor or resolve an exception. Capture elapsed job time and operator time separately: waiting for a person is different from the person spending ten minutes diagnosing a mismatch.
Keep the operating model explicit. A local runtime needs a responsible machine and process; a managed route still needs ownership for account access and output handling. These are evaluation questions, not assumptions that every product in a category has identical requirements.
Also distinguish setup from recurring work. A reusable workflow may have a substantial initial build cost. Excluding that cost from a first-month comparison is different from amortizing it across a declared number of future runs. Show the assumption instead of hiding it in a single headline price.
Make the second run part of the experiment
After reconciling the first run, repeat the same scope. Observe duplicate handling, version changes, missing-item recovery and interrupted retrieval. Do not intentionally invalidate a live account or manufacture destructive failure conditions for the sake of a test.
Use ordinary, authorized exceptions when they occur. Record the cause as unknown when evidence is insufficient. A challenge encountered once is not proof of a platform-wide failure rate, and a successful demonstration is not proof of broad supplier coverage.
Retain candidate configuration and observation dates so a later rerun is interpretable. Browser mode, workflow revision and output settings can change the comparison even when the product names remain constant.
Compare cost only against comparable delivery
Track charged usage, setup effort, recurring operator effort and archiving work separately. If one candidate delivers only part of the expected set, report the missing work alongside its cost; do not present a low spend as the price of a complete task.
BrowserAct and Skyvern can each be shortlisted for a particular workflow, but documentation and a demo answer different questions from a controlled pilot. A useful comparison report lets the reader inspect the approved scope, delivered files, human interventions and unresolved items before drawing a conclusion.
Top comments (0)