Over the last two weeks I published a series on practical AI adoption: pilots, thresholds, documents, review queues, acceptance sets, data, costs. This is the whole thing on one page, in the order you would use it. It is written for a small business or a small team with no data science department.
Before you build anything
- Pick one process, named in one sentence, with one owner who can say "accepted". Not "customer experience", but "incoming support emails" or "supplier invoices".
- Check whether you need AI at all. If the rules are stable, the volume is small, or one error is very expensive, a script, a template or a lookup table is the better answer.
- Measure the baseline. Time a real sample of at least 100 items: minutes per item, error rate, monthly volume.
- Write the target as a number before any result exists: share handled without a person, accuracy on those, cost per item.
- Do the payback arithmetic on one page: saving minus review, running and error cost, divided into the build cost. Run it again with pessimistic inputs.
Design the pilot
- Choose the cheapest rung that could hit the target: rules, then classic ML, then a constrained LLM, and only then agents.
- Decide what goes to a person: a confidence threshold plus hard rules, a named reviewer, a time limit.
- Size the review queue: items per day times share to review times minutes per review. If it exceeds the people you have, change the scope.
- Settle the data question with whoever owns it: what is in the input, whether it may leave your systems, what the provider keeps, what lands in logs.
- Build a golden set of about 100 real, labelled items covering typical, hard and must-refuse cases. Freeze it.
- Write the one-page brief: scope, baseline, target, data, golden set, human path, timeline, cost, risks, and the decision rule.
Run and judge it
- Run for 2 to 6 weeks, not months, with a checkpoint in the middle.
- Judge on the golden set and on live traffic, against the number you wrote down. Read every failure.
- Decide by the rule you fixed in advance: scale, change, or stop. Stopping is a valid, cheap outcome.
After go-live
- Monitor input shape, confidence distribution, share going to humans, and the override rate.
- Audit a random sample of the automated path weekly, because nobody reviews it otherwise.
- Rerun the golden set on every change of model, prompt, threshold or data.
- Watch cost and latency as quality signals, and give every alert an owner.
- Review monthly for half an hour: trends, the ten worst items, fresh examples for the golden set, the threshold.
If you hire someone
- Ask how you will know it works, what happens when it is wrong, where your data goes, what it costs to run after launch, what is out of scope, and what happens if you stop after the pilot.
The shortest version
Pick one process. Measure it. Write the target as a number. Use the simplest tool that can reach it. Keep a human behind the doubtful cases. Test on frozen examples. Watch it after launch. Decide by a rule you wrote before you started.
All demos behind this series use public data and are open: ticket triage, invoice extraction, document Q&A. Their numbers are test numbers, not client results. Portfolio and contacts: @Kotenev01_bot.
I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: @kotenev01_en
Top comments (1)
This is the kind of checklist I wish more small operators had before buying anything. The two points I would underline are picking one process with one owner, and sizing the review queue to the people you actually have.
In property work, that second one is where pilots quietly die — the automation looks fast, but the human review pile grows faster than anyone can clear it, and then either mistakes slip through or the tool just gets switched off.
I also like that stopping is a valid, cheap outcome. Naming the target as a number before you start makes that decision honest instead of emotional. Thanks for putting the whole series on one page.