The first AI agent a business puts in front of customers is often treated like a launch: announce it, turn it on and hope the volume proves it was a good idea. You get a mix of successes, quiet failures and no clear rule for whether to keep going.
A better first step is a 30-day pilot. Pick one narrow job, set a success number before you start and decide on day 30 whether to expand, fix or stop.
Why a pilot beats a launch
A launch asks the agent to be ready for every kind of customer question on day one. A first agent is rarely ready for that. If you turn the whole range on at once, you will spend the month reacting instead of learning.
A pilot puts a fence around the work: one job, a defined set of cases and people ready to take over. Thirty days is long enough to see patterns and short enough that you can still change course before the agent becomes "the way we do things now" by default.
Pick one narrow job
Narrow means you can describe the job in one sentence, and you can tell when a request is outside it. Good first jobs have a clear boundary: answer a known set of product questions, route a request to the right person, draft a reply for a human to send or book a simple appointment. The agent has a job, and it also has a "not my job."
Avoid a first job that includes refunds, legal questions, complaints or anything you cannot easily reverse, unless a person must approve those steps.
Write the boundary down and share it with the people who will take over when a request falls outside it. If they do not know where the fence is, the agent will start taking work it was not meant to take.
Set a success number before day one
If you wait until day 30 to decide what "good" looks like, you will argue from impressions. Set the number first. Pick a small set of measures you can actually count, such as:
- How often the agent finishes the job without a person stepping in
- How often a person has to correct the agent's answer or action
- How often customers come back with the same question, or ask to speak with someone
You need a written target for each measure, set before the pilot starts, and a way to record the results as you go. Choose numbers that fit your business, and write them down in advance, not at the decision meeting.
Also write down what "stop" looks like. If corrections pile up, if customers cannot get to a person or if the agent steps outside its job, you should be able to pause it without a debate.
Decide on day 30: expand, fix or stop
Put a decision meeting on the calendar when you start the pilot. Include the owner of the agent, the people who took over from it and someone who hears from customers. On day 30, you have three honest options:
- Expand. The agent met the targets and the handoffs worked. Widen the job, raise the volume or add a closely related task, one change at a time.
- Fix. The idea is sound, but the answers, the handoffs or the boundary need work. Keep the fence where it is, make the changes and run another short pilot on the same job.
- Stop. The agent missed the targets or is not a fit for this job. Turning it off is a successful pilot if it saved you from a messy launch.
Write the decision down, including why. The next pilot will be easier if you can see what you learned.
Questions to ask whoever runs your agent
- What is the one job this agent is doing in the pilot, and what is outside the boundary?
- What success numbers did we write down before day one, and where do we record the results?
- Who takes over when a request is out of scope, and have we practiced that handoff?
- What would cause us to pause the agent before day 30?
- When is the decision meeting, and are the options still expand, fix or stop?
A pilot is how you give the agent a fair test, and how you keep a first attempt from becoming a permanent one by accident.
I work with businesses on putting AI agents to work safely, through training, pilots and safety checks. If you have already tried a first agent, I'd like to hear whether you ran it as a pilot or a launch, and what you would do differently. Reply and tell me how you handled it.
Top comments (0)