Before any AI tool or so-called agent goes anywhere near a client project, I want to watch it fail in front of me. Sales pages tell you what it does on a good day. My job is to know what it does on a bad one, who is accountable when it is wrong, and what it costs at ten times the demo volume. Those answers decide whether it gets in, and most tools do not.
What is the first test?
Ugly inputs. I keep a folder of them, and every tool gets the folder before it gets a meeting. A scanned invoice photographed at an angle. An email written half in Hindi and half in English. A spreadsheet with merged cells and a total row sitting in the middle of the data.
There is also a support ticket that runs as one long sentence with no punctuation. Nothing confidential, all real in shape.
What I am watching for is how it fails, since every tool fails somewhere. Does it say "I'm not sure" or does it produce a confident wrong answer with a straight face? A tool that flags uncertainty can be put in front of a human. A tool that bluffs cannot, whatever its accuracy on the vendor's own examples.
Is it an agent, or a form with a chat window?
A lot of what is sold as an "AI agent" is a chatbot stapled to a form. That is fine as a product, and it changes what we can honestly promise a client, so I ask four blunt questions before the demo starts. The answers sort the two categories inside ten minutes.
| Question I ask | A real agent | A form with a chat window |
|---|---|---|
| What does it produce? | An action inside a system | Text for a person to act on |
| Who approves? | A checkpoint I can make mandatory | Nobody, because nothing happens |
| Is there a log? | Every step, in plain language | The conversation, and that is all |
| Wrong at 2am, then what? | It stops and escalates | It waits to be read |
The distinction matters more than the demo does. I lean on the way agent-washing, real agents versus fake ones is laid out because it gives me a vocabulary for the vendor call: what does the thing decide on its own, and what happens when that decision is wrong at two in the morning.
What about cost and control?
Four questions, asked before anyone on my side falls in love with the tool. They are dull questions and they are the ones that decide whether a tool can carry client work, because a tool that fails any of them becomes somebody's problem to unwind eighteen months later.
- What does it cost at ten times our expected volume? AI pricing is gentle at pilot scale and brutal at production scale.
- Where does the data go, and can the client's legal team live with that answer?
- Can we get our data and our configuration out if we leave?
- What happens when the vendor swaps the underlying model, which they will, without telling us?
The last one has bitten more projects than any bug. A prompt tuned for months quietly starts behaving differently, and nobody on the vendor's side considers it a change worth announcing.
What is the rule once a tool passes?
It still does not get to act alone. Every AI tool enters a client workflow with a human checkpoint in front of it, and it earns autonomy over weeks by being measured: how often did the human change the output, and how often did that change matter?
Only when those numbers are boring for a sustained stretch do we discuss removing the checkpoint, and even then the client decides.
At Shanti Infosoft the vetting sits with delivery rather than with whoever found the tool first, because the person who found it is usually excited and excitement is a poor reviewer. The same folder of ugly inputs goes into every AI integration we scope, before anyone writes a line of glue code.
One more thing I have learned to ask on the vendor call: "Show me a customer who turned it off, and tell me why." The good vendors have an answer. The rest change the subject. If you want a second pair of eyes on a tool your team is about to sign for, book a slot.
Which AI tool on your current project has never been shown a genuinely ugly input?
Sonal Jain leads delivery at Shanti Infosoft, a CMMI Level 5 company that has shipped for 700+ companies, and she keeps the ugly-input folder herself.
Top comments (0)