Originally published on the NuWay Biz Solutions blog.
✦ Cover image: Made with ChatGPT — we're transparent about AI. See the exact prompt on the original post.
"It's an AI agent. It handles your inbound automatically."
"Handles it how?"
"Automatically."
That exchange is a fair summary of the market right now, and it is not really the salesperson's fault. The word arrived faster than the meaning did.
Gartner, the technology research firm a lot of corporate software budgets get built on, has a name for the resulting mess. They call it agent washing: taking an assistant, a chatbot or an old automation tool, and reissuing it with the new label. Of the thousands of vendors selling agents, they counted about 130 selling a real one in mid-2025.
Gartner counted about 130 real agent vendors in mid-2025, out of thousands
You do not need to work out which 130. You need five answers, and every one of them is a question about your business rather than a question about the software.
1. What does it do when nobody is watching it?
This is the question the label is hiding. Ask the vendor to name one thing their software does with no person approving it first.
If every answer comes back as it suggests, it drafts, it recommends, it surfaces, then you are being shown an assistant. Which is genuinely useful and worth paying for. It is just a different product at a different price, and you should know which one is on the invoice.
An assistant
- Waits for you to ask
- Hands the work back for you to approve
- Its mistakes stay inside your building
- Usually priced per person per month
An agent
- Decides on its own that something needs doing
- Does it, then tells you
- Its mistakes reach customers before you see them
- Usually priced per action, per outcome, or per run
Gartner's analyst on this, Anushree Verma, put the useful half of it more bluntly than a vendor ever will: “Many use cases positioned as agentic today don't require agentic implementations.”
Plenty of jobs are better done by something that asks first.
2. What can it do that I can't take back?
Everything an agent is allowed to touch falls into two piles, and the piles are not the same size.
One pile is reversible. It tagged the wrong lead, it filed a job under the wrong customer, it wrote a note nobody needed. You fix it in a minute and no one outside the business ever knew.
The other pile leaves the building. It emailed a customer. It issued a refund. It quoted a price. It cancelled tomorrow's appointment.
You cannot un-send an email to a customer, and the agent will not know it should have asked.
So get the list. The one that names each action it can take with no human in between, and where that action lands. Vendors lead with a longer and friendlier list, so ask for this one specifically.
Then say out loud which of those you are comfortable with on a Saturday when nobody is looking at a screen. That is the real permission you are granting, and it is worth writing into the contract rather than discovering later.
3. Which of my numbers does it read?
An agent acts on what it reads. That single fact is why this question sits in the middle of the list rather than at the end.
We have made this point about what AI-ready data actually means for a while, and about why the AI you already pay for stalled out. An agent raises the stakes on it. A dashboard built on numbers that disagree wastes your afternoon. An agent built on the same numbers sends the customer a bill for the wrong amount.
So ask which systems it reads from, and then ask yourself the harder half: would you personally bet on those numbers being right this morning?
If the honest answer is no, do that work first. It pays for itself whether or not you ever sign. Match the build to the mess, same as always.
4. What does this cost in my busiest month?
The software you already buy costs the same in February as it does in July. Agents mostly are not sold that way.
Intercom publishes its price plainly, which is more than most vendors do, so it makes a clean example: its Fin agent is $0.99 per outcome, charged once per conversation. An outcome counts when the customer confirms the issue is resolved, when the agent completes a workflow, or when the customer doesn't ask for more help after it responds.
Read that last one twice. Silence counts as success, which is a reasonable way to measure it and also exactly the number you want to watch.
Usage pricing is a perfectly reasonable way to buy this. It just asks one piece of arithmetic of you first: take your worst week last year, the one where the phone did not stop, and work out what that week would have cost at the quoted rate.
Then ask what happens when you hit the ceiling on your plan, because the answer is rarely nothing.
5. How would I know it did the job right?
We wrote recently that a stopped automation looks exactly like a quiet week. An agent inherits that blind spot and adds a worse one of its own.
An automation does one fixed thing every time its trigger fires, so at least you know what it was trying to do. An agent picked the thing. It can run perfectly, do exactly what it decided to do, and still have done the wrong thing, confidently, in your name.
There is a good measurement of how often an agent finishes the job at all. Researchers at Carnegie Mellon built a fake company, a simulated software business with its own internal websites and data, and turned the best agents available when they ran it loose on the work its staff would handle. The most competitive agent finished 30% of the tasks on its own. Their conclusion is worth keeping in your head during the sales call: simpler tasks often go fine, and longer multi-step ones are still beyond what these systems can do.
That number will improve. It has not improved enough for you to skip the check.
So ask what it produces that a person can look at afterwards. A record of what it decided and why, in language you can read on a phone, with a way to find the ones it got wrong before your customer does. A dashboard of how busy it has been will not tell you any of that.
Ask this one on the call
"Show me the last hundred things it did in an account like mine, and point at the ones it got wrong."
A vendor with a real product will have a screen for this, because their other customers ask. A vendor who offers you a demo environment instead has told you something useful, for free.
When the answer is yes
✦ Made with ChatGPT — we're transparent about AI. See the exact prompt on the original post.
None of this is an argument against buying one. There are jobs where an agent earns its keep quickly, and they have a recognisable pattern: the same decision, many times a week, against a rule you could write down on an index card.
Booking requests. First-pass triage on inbound. Chasing the same three pieces of missing information off every new job. Work where being right ninety-something percent of the time and flagging the rest is a genuine improvement on what happens today, which is that it waits until Thursday.
Verma's own advice on where to start says it better than we can: “They can start by using AI agents when decisions are needed, automation for routine workflows and assistants for simple retrieval.” Three different tools. Most of what gets pitched as the first one is doing the job of the other two.
The businesses that get value out of this are the ones that could answer the five questions before they signed.
- [ ] I can name one thing it does with no person approving it
- [ ] I have the list of actions it can take that reach a customer
- [ ] I know which of my systems it reads, and I would bet on those numbers this morning
- [ ] I have worked out the bill for my busiest week
- [ ] I know what it produces that I can check afterwards
Four or more and you are ready to have a serious conversation with a vendor. Fewer, and the gaps are your agenda for the next call: whichever one you couldn't tick is the one that will cost you.
Take the list to the next call. You will know inside ten minutes whether the person on the other end has thought about their product as hard as you are about to think about your business.
And if you get the answers and still cannot tell whether the thing is worth buying, send us what you were pitched and we will tell you what it actually is. Sometimes the answer is that you already own something that does the job.
Practical AI. Clear process. Real business value.
— Brian, NuWay Biz Solutions
P.S. One more to try on the next demo, and it is the cheapest of the lot. Ask what the software does when it is not sure. A real answer sounds like a rule: it stops, it flags, it asks a human. If what comes back is a shrug, or that it uses its best judgement, you have just learned the most important thing on this list.
Top comments (0)