Most teams start an AI agent project with a model API key and a scaffold. The harder work is quieter: naming the job, the data the agent may touch, which actions need a human before they run, and which pages or tools must stay reliable for the loop to finish. When those questions stay blank, the usual result is chatbot cosplay with write access, not an agent you can defend in a client review.
What follows is a pre-build checklist for agencies and product teams who are about to wire tools into an LLM. It is not a protocol tutorial and not an underwriting standard. It is the set of answers that usually fit on one page before anyone opens a pull request.
Is this a job for an agent, or a chatbot with tools?
An agent is useful when a goal needs repeated tool use until a verifiable outcome exists: open a ticket, update a CRM field, pull a report, or click through a documented UI path. A chatbot is enough when the useful output is text the human will check and paste somewhere else. Mixing the two in one SOW is how teams grant write access to something that should have stayed a draft assistant.
Three plain questions usually separate the cases. Can a reviewer check the final answer against a source of truth without redoing the whole task? Is the path mostly routine process, with judgment reserved for named checkpoints? Is there a specific bottleneck (queue depth, copy-paste between systems, nightly spreadsheet) rather than a vague wish for “an AI that helps”? Nesta’s public framing of suitability questions lands in a similar place; the agency version adds one more: will the client accept a machine acting inside their systems under your account?
When those answers stay fuzzy, a read-only prototype is the cheaper path. Chat with retrieval is easier to unwind than an agent that already created records nobody can explain. Tools can be added later; reversing silent writes is the expensive path.
What data and tools does the agent get (least privilege)?
Teams that stay out of trouble usually inventory every system the agent may call, then drop anything that is not required for the job. Read scopes until a write is proven necessary, one client workspace instead of a global admin token, and short-lived credentials instead of shared “ops” keys that outlive the prototype all reduce blast radius. Where the platform allows it, a dedicated agent identity beats borrowing a human admin session that also unlocks billing and invites.
An allowlist written as if it will appear in a security review (tables, APIs, folders, browser origins) tends to survive review better than a vague “access to the CRM.” Anything not on that list failing closed is the safer default. When the agent will sit inside a multi-tenant product or shared agency login, role design belongs in the build brief, not as a cleanup ticket. Our product spotlight on team roles and access control is the human side of the same problem: who can change thresholds, invite users, or see every organisation.
Least privilege also means tool privilege. An agent that can “browse the web” is not the same as one that can only call three named MCP servers. Narrow tools reduce blast radius when prompts go sideways. Identity vendors make the same point at length for enterprise agents; the agency version is smaller but identical in shape: separate identity, narrow scopes, no shared God mode.
Where must a human approve before the agent continues?
Irreversible or high-cost actions (send email to customers, change prices, delete data, publish content, move money, grant access) are the ones that need an explicit human gate with a logged decision, not a confidence score buried in a system prompt. If the gates cannot be listed, the tools are usually premature. That gap shows up fast once the agent can change live data.
Design that matches how people actually behave works better than theatre. When every low-risk step requires approval, humans rubber-stamp and stop reading. When nothing requires approval until after damage, the gate is decorative. A short list of hard stops, a named on-call owner for each, and observability for the rest is the pattern that tends to hold.
Accountability also needs a human name. “The model” is not an owner. Account lead, on-call engineer, or client admin, plus a clear halt path, belongs on the same one-pager as the allowlist so ownership is not rediscovered during an incident.
How will you evaluate success and failure modes?
Prompt tuning is more useful after a small golden set exists: real examples of the task, including messy ones. “Done” works better when it is measurable (correct CRM fields, matching ticket state, matching report totals). “Stop and escalate” works better when incomplete inputs, tool errors, and loops each have an explicit exit.
Adversarial and boring cases (missing fields, duplicate records, stale pages, rate limits, consent banners that block a click path) expose gaps that demos hide. Timing a full run and logging tool failures separately from model mistakes gives a baseline you can regress against. Without that baseline, every demo looks like success and every production week looks like mystery.
Failure modes belong in the brief, not the post-mortem. The three ways the agent is most likely to embarrass the team, each with a detection signal, are the useful sketch. When the signal is missing, the failure usually arrives as a client email.
What cost and latency budgets are acceptable?
Agents burn tokens and wall clock. A classification that takes forty seconds and three tool calls may lose to a human who finishes in ten. A budget per successful completion (currency and seconds), with a hard abort when either is exceeded, keeps demos honest.
Interactive agents (someone waiting) and batch agents (overnight queues) need different budgets even when the architecture looks the same. Cost also includes the people who review escalations. When half the runs need a human, the system is mostly a queue with extra steps.
When do you need WebMCP, MCP, or both?
Chrome’s guidance is clear: WebMCP does not replace MCP. They solve different jobs and often sit in the same product. MCP is for backend tools and data that agents can call from anywhere, on a persistent server. WebMCP is for a live website in a browser tab: ephemeral tools registered on the page, aimed at browser agents acting while a user (or session) is on that origin. Official comparison: When to use WebMCP and MCP.
Which surface owns the work is the useful question before a scaffold is chosen.
| Need | Lean toward | Why |
|---|---|---|
| Background data and actions available anytime | MCP / APIs | Persistent, platform-agnostic tools with typed contracts |
| In-tab actions on your own live UI | WebMCP (where supported) | Browser-integrated tools; not a guess-the-button scrape |
| Measuring whether pages are agent-ready | Lab agentic audits (+ page tooling) | Complements Performance scores; see Lighthouse agentic browsing work |
| High write risk on third-party SaaS you do not control | Prefer their API + human gate | You cannot ship WebMCP on someone else’s product |
When the agent still has to infer clicks from layout because no WebMCP tools exist, that path stays fragile. Chrome’s Lighthouse work on agentic browsing scoring treats machine interaction and layout stability as their own concern, not a rebrand of Performance. For agents that will click production money URLs, Lighthouse agentic browsing scoring is useful lineage before a green Performance score is treated as proof the agent path is safe. Layout shift that is merely annoying for a shopper can be fatal for a coordinate-based click.
Protocol plumbing (how MCP servers are wired, how sessions authenticate) is a separate build track. Which surface first, then transport, is the order that usually avoids rebuilding auth twice. Teams that reverse it often discover the second rebuild during a client pilot.
Who owns access, and what destinations do you monitor?
An agent inherits whatever credentials it is given. Mapping those credentials to human roles (who rotates keys, who can expand scopes, who sees audit logs) keeps the blast radius visible. In agency work, that often means one organisation at a time, viewer versus manager versus admin, and no shared God mode for contractors.
The destinations the agent depends on (dashboards, checkout flows, documentation URLs, internal app screens) are part of reliability too. When those pages regress on Largest Contentful Paint, Interaction to Next Paint, or Cumulative Layout Shift, the agent fails for the same reasons users bounce. Scheduled lab and field monitoring on that URL list is part of agent reliability, not a nice-to-have after launch. Monitoring layers onto the stack you already run; the LLM vendor’s status page does not cover your client’s product pages.
FAQ
What should I ask before building an AI agent if the stakeholder only wants a demo?
A demo that stays read-only, uses synthetic or sampled data, and has a written stop condition is usually enough to prove the job. Demo write access against live customer data is how prototypes become incidents. A recorded read-only walkthrough still shows the workflow without minting production credentials on day one.
Is an AI agent checklist the same as an AI governance checklist?
No. Governance and underwriting frameworks ask whether you can defend the system under risk and policy. A pre-build checklist asks whether the job, scopes, gates, and evals are clear enough to start. Both matter before production; they answer different meetings. Confusing them is how teams either stall forever or ship without owners.
When should we use WebMCP versus MCP for agents?
MCP (or plain APIs) fits persistent backend tools available outside a single tab. WebMCP fits when the agent must act on your live site in the browser and you can expose explicit page tools. Chrome treats them as partners, not rivals; many products need both, with different privilege levels. The narrower surface that still completes the job is usually the better place to start.
Do we need performance monitoring for an internal agent?
If the agent loads or clicks real URLs, those URLs still benefit from monitoring. Internal does not mean invisible: slow or shifting layouts still break tool steps and burn retries. The same monitoring list you would use for a human critical path is a sensible default.
A one-pager before the framework choice
The answers in the sections above usually fit on a single page before a framework is chosen. A blank row is a useful first ticket. When the agent will touch client workspaces or public money URLs, access roles and destination monitoring sorted early mean the build invents fewer privileges under deadline pressure.
A concrete next step on the monitoring side is a free Watcher trial with the agent’s critical URLs on a schedule and budgets you already trust for human visitors: Start free trial.
References
- Before you build an AI agent, ask yourself these five questions (WeAreBrain; SERP peer for the title query)
- Five questions to ask before building an AI agent (Nesta)
- A practical guide to building agents (OpenAI; ranks on related “before building” queries)
- When to use WebMCP and MCP (Chrome Developers)
- Lighthouse agentic browsing scoring (Chrome Developers)
- How to implement least privilege for AI agents (Okta)
- Product spotlight: team roles and access control (Apogee Watcher)
- Lighthouse agentic browsing scoring explained for monitoring teams (Apogee Watcher)
Top comments (0)