Choose an AI tool by checking whether a specific workflow produces an acceptable result with less total effort and manageable risk. A fluent demonstration, a long integration list, and an employee's enthusiasm answer different questions. None establishes that the tool can handle your actual inputs, permissions, exceptions, and review workload.
The following eleven tests form a proposed evaluation method, not a product ranking. They apply to a shortlist for customer communication, document processing, voice intake, marketing, internal knowledge, or sales operations. The examples are hypothetical. No vendor trial, client result, or guaranteed productivity improvement is claimed here.
The starting point is the business AI tool selection guide. Its different business needs are useful categories for a shortlist; the decision below moves from categories to observable acceptance conditions. You can use the same worksheet for an AI assistant, ordinary automation, and your existing manual process. Sometimes the simplest alternative wins because it solves the real problem without introducing another system to maintain.
1. Can you define the finished job without naming a tool?
Write one sentence describing what arrives, what must leave, and who accepts it. For example: a support request arrives with an order reference; a reviewer receives a proposed answer with the correct order facts, a relevant policy reference, and any unresolved question. Sending the answer is a separate action. This definition makes the boundary visible before the demonstration begins.
Avoid objectives such as “improve customer experience with AI.” They leave the evaluator free to count almost anything as success. A narrower objective might require reducing the time spent gathering order information while preserving the current approval step. That can be observed. Whether customers are more satisfied remains a separate measurement, not something inferred from faster text generation.
List the exclusions beside the objective. The support assistant may retrieve shipping facts but not issue refunds. A sales assistant may prepare a qualification note but not invent a prospect's budget. A meeting tool may suggest action items but not assign commitments to someone who never agreed to them. These exclusions are part of the job definition, not footnotes for the security team.
Then test whether a simpler change could solve the same problem. A missing knowledge-base article, an unclear form, or a broken handoff may explain more of the delay than writing speed. Fixing that bottleneck can remain worthwhile even if the AI pilot is cancelled. Keeping a non-AI alternative in the comparison prevents the buying process from assuming its own conclusion.
The output of this test is a task contract: input, acceptable output, exclusions, recipient, and acceptance owner. If two stakeholders describe different finished jobs, resolve that disagreement before scoring vendors. Otherwise, the comparison will reward whichever demonstration happens to resemble each person's private expectation.
2. Does the baseline represent work you actually receive?
Build a test set from cases you are authorised to use, with confidential details removed where needed. Separate normal cases from exceptions, incomplete requests, and cases that should be refused or escalated. Preserve the original distribution for one operational sample, and keep a second challenge set for rare but consequential failures. Mixing the two without labels makes the overall score difficult to interpret.
A challenge set deliberately overrepresents difficult cases. That is useful for finding weaknesses, but it is not an estimate of everyday failure frequency. Conversely, a random week of routine work may contain no serious permission conflict. Passing that week does not demonstrate that the system handles a conflict safely. Report the two results separately instead of averaging away their different purposes.
Record what the current process produces on these cases, including unanswered requests and work returned for clarification. Measure the same beginning and end points for both alternatives. If the manual baseline starts when a message arrives but the AI measurement starts after someone has cleaned the message, you have hidden preparation work inside the comparison.
The OECD's survey discussion distinguishes generative-AI use from use in a firm's core activities, and explains how definitions and sampling affect adoption estimates. That distinction is useful here: adoption is evidence that a tool is used, not proof that a particular workflow improved. Read the OECD account of SME use and survey limitations before borrowing an adoption statistic for a business case.
Keep a manifest of case identifiers and the sampling method. Do not quietly replace a difficult case after seeing a poor result. If the case was invalid, retain the exclusion reason and recalculate both alternatives on the same corrected set. A small transparent comparison is more useful than a large score whose denominator changes between demonstrations.
3. What happens when the input is incomplete or contradictory?
Provide the same business request with a missing identifier, an obsolete attachment, and conflicting dates. Inspect whether the proposed output preserves the uncertainty or fills the gap with a plausible guess. An answer can be grammatical and still be unusable because the missing field determines what the business is allowed to do next.
For document intake, distinguish extraction from interpretation. Extracting a date printed on a form is not the same as deciding whether that date is the correct deadline. Your test should identify the authoritative document and the rule for resolving disagreement. When neither is available, the acceptable result may be a request for clarification rather than a completed record.
Try a negative case with no answer in the permitted material. A knowledge assistant should not need to produce an answer at any cost. Evaluate how clearly it explains the gap and whether it directs the request to an appropriate person. Count a correct refusal as a successful outcome when refusal is part of the task contract.
Also check input boundaries that are easy to miss during a polished demonstration: empty files, duplicated attachments, broken text extraction, unusually long messages, and a language different from the one the workflow normally receives. These are test categories, not a claim that every business needs every category. Select those that could actually reach the proposed system.
Write the expected response before running the case. “Ask for the order identifier and do not prepare a refund instruction” is assessable. “Be cautious” is not. Store both the observed output and the evaluator's reason, so that a disagreement can be reviewed without rerunning a nondeterministic system and obtaining a different answer.
4. Can a reviewer verify the evidence efficiently?
An answer with several links is not automatically well supported. Ask a reviewer to trace a material claim to the exact passage or record that supports it. Check whether the cited source is current for the task, whether the quoted scope matches the recommendation, and whether the assistant has combined two records that refer to different customers or periods.
For an internal-policy answer, preserve the document version. For an order response, preserve the record timestamp. For a financial summary, preserve the units and reporting period. The evidence should let a reviewer answer “what supported this output at the time?” without relying on whatever the live database happens to contain today.
Measure verification effort as well as answer accuracy. A tool that saves drafting time but forces an expert to reconstruct every source may move work rather than remove it. Capture how often reviewers reopen documents, consult another system, or ask a colleague to resolve an ambiguity. These are useful operational observations even before a monetary estimate is possible.
The NIST AI Risk Management Framework provides a voluntary framework for incorporating trustworthiness into the design, use, and evaluation of AI systems. It is not a product certification. In this evaluation method, its role is to keep risk and trustworthiness inside the buying decision instead of treating them as a later paperwork exercise.
Evidence retention also needs limits. Do not copy entire customer files into an evaluation report when a protected reference and a redacted excerpt will do. Restrict access to the retained material and define when it should be deleted. A reproducible test should make the decision auditable without creating a second, poorly governed repository of sensitive information.
5. Are you scoring the right outcome for each workflow?
Use a scorecard that follows the business consequence of an error. A voice intake system and a marketing drafting tool should not share a single generic “quality” score. For the former, an incorrect appointment date can matter more than an awkward sentence. For the latter, an unsupported product claim can matter more than whether the first draft matched the preferred tone.
| Workflow under evaluation | Acceptable result | Important failure to test | What not to mistake for success |
|---|---|---|---|
| Customer-message drafting | Correct facts and a reviewable proposed reply | Unsupported promise or wrong customer context | A fast, friendly response |
| Voice appointment intake | Confirmed details and explicit unresolved fields | Incorrect date, name, or escalation | Natural-sounding speech alone |
| Document extraction | Traceable fields with missing values preserved | Confident completion of absent information | Every output field being populated |
| Internal knowledge search | Answer supported by authorised current material | Cross-team disclosure or obsolete policy | A long list of citations |
| Sales preparation | Relevant account facts and labelled hypotheses | Invented buying intent or contact details | More prepared messages |
Decide which failures are disqualifying and which are repairable by the ordinary reviewer. Keep these categories explicit. Do not allow a high average score for easy cases to compensate for a serious permission breach. Equally, do not reject a useful drafting tool because it occasionally needs an expected stylistic edit that your process already includes.
Where reviewers disagree, inspect the rubric before blaming the system. An ambiguous policy can produce inconsistent human labels too. Resolve the interpretation with the process owner, version the rubric, and rerun affected comparisons. Retain the earlier verdict rather than rewriting history to make the new score appear continuous.
This table is a starting structure. It is not a set of universal thresholds. Set acceptance limits according to the workflow's consequences, the organisation's requirements, and the ability to detect and recover from mistakes. Record who owns those limits before the final evaluation.
6. Is the net workload lower after review and correction?
Measure the whole human contribution: preparation, interaction, verification, correction, exception handling, and recovery. Separate machine waiting time from human effort, while also recording elapsed turnaround when that matters to customers. A task that runs unattended for several minutes may reduce human effort without improving turnaround; a rapid answer may do the reverse if it requires extensive checking.
A useful arithmetic check is: net human minutes saved equals comparable baseline minutes minus preparation, interaction, review, correction, and recovery minutes in the assisted process. This is an accounting identity, not a productivity forecast. Its usefulness depends on whether the inputs were measured consistently and whether the resulting outputs meet the same acceptance conditions.
For illustration only, suppose a task takes twenty human minutes in the baseline. The assisted attempt consumes three minutes of preparation, two of interaction, six of review, four of correction, and one of recovery. The observed saving for that hypothetical case is four minutes. Calling it an eighteen-minute saving because generation took two minutes would omit most of the work.
Independent findings caution against assuming a universal benefit. METR's early-2025 randomised study involved sixteen experienced open-source developers working on tasks in familiar repositories and reported slower completion with the studied AI assistance. Its population and tools limit generalisation; it is a reason to measure your own task, not a verdict on every tool. See the primary study and its experimental scope.
Inspect the distribution, not just a total. A favourable average can hide a small class of cases with large correction costs. Report which case families improved, which deteriorated, and which remain too sparse to assess. The deployment boundary may belong around a subset of the workflow rather than around the entire product.
7. Can it stay inside its permission boundary?
Document the actions available to the tool separately from the words it can generate. Reading a customer record, proposing a change, committing that change, and contacting the customer are different permissions. A system that only needs to prepare a draft should not inherit the full privileges of an administrator account used to connect the integration.
Use a sandbox or a non-mutating test mode for deliberate boundary checks. Include a request to read an unrelated account, an instruction embedded in an untrusted document, and an attempted action outside the agreed workflow. Inspect the system's actual behaviour rather than accepting a reassuring sentence about its policy. A refusal message is not sufficient if the prohibited side effect still occurred.
For a proposed external message, verify the recipient, the approved content, and the authority to send. For a record update, verify the target identifier and the fields permitted to change. Business identifiers should not be guessed from a similar name. Ambiguity should return the case to a person with access to the necessary context.
Keep the evaluation proportionate. You are not conducting an unrestricted security test against a vendor's systems. You are checking your authorised configuration and the actions the proposed workflow would take in your environment. Agree on safe test data and scope with the people responsible for the system before running adverse cases.
The acceptance evidence should include the denied action, the observed result, and confirmation that no unintended record or message was created. This test is especially important when a successful demonstration depends on broad credentials that would never be acceptable for routine operation. Convenience during setup must not silently become the production security model.
8. Failure modes: does recovery work when a dependency fails?
Interrupt the workflow at a safe test boundary: make a dependency unavailable, return a temporary error from a test service, or provide a deliberately expired sandbox credential. Observe whether the system stops clearly, loses the case, repeats a completed action, or leaves an ambiguous partial result. Recovery is part of the workflow, not merely an infrastructure detail.
An ambiguous write deserves different handling from a confirmed failure before a write. If an external system may have accepted an operation before the connection failed, blindly repeating it can create duplicate records or messages. The evaluator should ask how the integration determines whether the action already happened and how a person resolves uncertainty when that cannot be determined automatically.
Check the human handoff. Can another operator find the case, see the last confirmed step, understand what remains uncertain, and continue without starting over? A dashboard marked “failed” is not enough if it hides which business action succeeded. Preserve relevant identifiers and timestamps without exposing credentials or unnecessary personal data in the log.
Define an exit from the pilot itself. Keep the manual process available, document how to stop scheduled jobs, and know how to revoke the integration's access. Test the stop procedure rather than assuming that cancelling a subscription immediately disables every connected workflow. The exact mechanism depends on your deployment and must be verified there.
This is also a staffing question. Identify who handles a failure outside the pilot team's working hours, or restrict the trial to supported hours. An unattended workflow that creates an urgent review queue overnight may be a poor fit even when its daytime demonstration looks efficient.
9. Can the business retain control of data and configuration?
Ask where input data, generated outputs, logs, and backups are retained in the proposed configuration. Distinguish the product's advertised options from the plan and region you would actually use. Record the applicable documentation and contractual review instead of treating a general security page as proof that every setting meets your requirements.
Inspect exportability with a harmless test record. Can you recover the information needed to continue the business process elsewhere? Does the export preserve identifiers, relationships, timestamps, and review history, or only a readable transcript? A visually attractive export can still be operationally incomplete if another system cannot reconstruct what happened.
Inventory the parts you will own: mappings, instructions, templates, access rules, reference documents, evaluation cases, and integration code. Assign someone to maintain each material component. If the only person who understands the workflow is the salesperson who demonstrated it, the implementation is not yet ready for handover.
Treat legal and procurement requirements as their own review, not as a checkbox inferred from technical success. The relevant obligations depend on the data, geography, use case, contract, and organisational role. This method does not certify compliance. An unresolved requirement should have an owner and a resolution condition before real sensitive data enters the trial.
BCG's analysis of the AI value gap emphasises organisational and workflow changes alongside technology. That is a consulting interpretation, not evidence that buying a particular tool will produce a return. The practical implication for this scorecard is to include maintenance and ownership work in the decision rather than evaluating only the interface.
10. Is the comparison fair and repeatable?
Freeze a configuration for the final evaluation: product version where available, enabled features, reference material, instructions, integrations, and human review procedure. Record what cannot be pinned. A hosted product may change between runs; that limitation belongs in the result instead of disappearing behind a single score.
Separate the cases used to improve the configuration from the cases used to judge it. Once a team has repeatedly tuned against an example, success on that example is less informative about unfamiliar work. Keep a held-out set that still represents the intended workload. Do not present repeated attempts on the same case as independent new cases.
Allow comparable assistance for each alternative. If one tool receives cleaned data and expert coaching while another receives raw requests, report that difference. You may legitimately compare complete service packages, but then the coaching and preparation are part of what is being bought and maintained. They should not be omitted from the workload or dependency analysis.
Review outputs without a vendor label when practical, especially for subjective judgments such as clarity or completeness. The evaluator still needs enough context to apply the rubric. The aim is not to remove business knowledge; it is to avoid rewarding an output because it came from the favoured demonstration.
Retain the unsuccessful attempts. They show what users experienced and what recovery cost. A final showcase assembled from the best result of several retries measures the quality of a curated presentation, not the reliability of a normal workflow. State the retry policy and apply it consistently across the comparison.
11. Can you make a bounded release decision?
Choose among rejection, another specific experiment, and a limited rollout. “Continue exploring” is not a decision unless it names the uncertainty to resolve, the next test, and the condition for stopping. A tool can be suitable for one task family and unsuitable for another; your rollout should preserve that distinction.
For a limited rollout, specify the accepted inputs, the users, the action permissions, the review responsibility, and the stop conditions. Retain the baseline and evaluation configuration. Name the changes that require reassessment, such as a new data source, a broader permission, a new customer segment, or a materially changed model behaviour.
Do not convert saved minutes directly into realised revenue. Time saved may be absorbed by other work, training, or idle capacity. If the business case depends on handling more demand or reducing a specific expense, measure that outcome separately. Likewise, a higher employee acceptance rate does not prove that customer outcomes improved.
Use the following decision flow as a proposed operating model. It is not a claim that any named product implements these controls automatically.
flowchart TD
A[Defined task and baseline] --> B[Representative evaluation]
B --> C{Boundary or serious error?}
C -->|Yes| D[Reject or redesign]
C -->|No| E{Acceptable output and workload?}
E -->|No| F[Targeted retest]
E -->|Yes| G[Limited rollout with review]
G --> H[Observe exceptions and changes]
H --> BKeep the outcome understandable to the people who run the process. They should know which cases are eligible, what they must verify, and when they should return to the manual route. A technically impressive system with an unclear operating boundary is not a completed adoption decision.
An executable workload check
This small Python example accounts for human effort in a synthetic fixture. It does not evaluate factual correctness or authorise deployment. An evaluator supplies the accepted/rejected labels. Failed attempts remain in the totals; the program deliberately refuses to describe a batch with rejected outputs as a clean time-saving comparison.
Save it as workload_check.py and run python workload_check.py. It uses only the standard library and makes no network calls. Replace the fixture only with authorised, consistently measured cases. Keep actual customer identifiers outside a shared example file.
from math import isfinite
def evaluate(cases):
if not cases:
raise ValueError("No evaluated cases")
fields = ("baseline", "prepare", "interact", "review", "correct", "recover")
seen = set()
for case in cases:
if case["id"] in seen:
raise ValueError("Duplicate case identifier")
seen.add(case["id"])
if type(case["accepted"]) is not bool:
raise ValueError("accepted must be boolean")
for field in fields:
value = case[field]
if type(value) not in (int, float) or not isfinite(value) or value < 0:
raise ValueError("Minutes must be finite and non-negative")
baseline = sum(c["baseline"] for c in cases)
assisted = sum(sum(c[f] for f in fields[1:]) for c in cases)
rejected = sum(not c["accepted"] for c in cases)
return {
"cases": len(cases),
"rejected": rejected,
"baseline_minutes": baseline,
"assisted_minutes": assisted,
"comparable_saving_minutes": None if rejected else baseline - assisted,
}
fixture = [
dict(id="synthetic-1", accepted=True, baseline=20,
prepare=3, interact=2, review=6, correct=4, recover=1),
dict(id="synthetic-2", accepted=False, baseline=15,
prepare=2, interact=2, review=5, correct=8, recover=3),
]
assert evaluate(fixture[:1])["comparable_saving_minutes"] == 4
assert evaluate(fixture)["comparable_saving_minutes"] is None
print(evaluate(fixture))
The rejected case does not vanish. Its work remains visible, and the result withholds a clean savings figure. In a real evaluation, inspect why it failed and whether recovery completed the original job. Then decide whether the comparison should include a verified manual completion, a restricted scope, or a rejected pilot. Do not simply change the acceptance label to obtain a favourable total.
Validation checklist before adoption
- The task contract names the input, accepted output, exclusions, and decision owner.
- The baseline and assisted measurements use the same boundaries and comparable cases.
- Routine and challenge-set results remain separate, with exclusions explained.
- Material outputs can be traced to authorised evidence without excessive reconstruction.
- Review, correction, recovery, and maintenance work are visible.
- Permission and failure-mode tests confirm actual behaviour in an authorised environment.
- Data handling, export, and contractual questions have named owners and resolved requirements.
- The final configuration and evaluation rubric are recorded.
- Rejected cases and unsuccessful retries remain in the evidence.
- The rollout, stop conditions, and reassessment triggers are explicit.
Start with the task that has a clear owner and an observable finished result. Apply the scorecard before deciding which product deserves a broader trial. If the evidence supports only a narrow use, keep the use narrow. If it supports no improvement, preserving the current process is a valid outcome of the evaluation.
Top comments (0)