Most vendor security reviews were written for tools that store data. AI agents also act: they read content nobody vetted, hold credentials, and call tools. A questionnaire that stops at "where is data hosted?" and "do you have SOC 2?" can pass a product whose agent holds one broad admin token and decides on its own when to ask a human.
Here are the questions I'd add before switching on any AI agent or AI feature in a SaaS tool, grouped by what can actually go wrong. Each comes with the answer you want and the answer that should stop the rollout.
1. What can the agent do without a human?
Ask: List every action the agent can take on its own: send, delete, pay, deploy, change permissions.
Good: a complete tool list with a consequence level, and high-consequence tools off or approval-gated by default.
Red flag: "The agent can do anything the user can do."
Ask: Where is approval enforced: in code outside the model, or by the model deciding to ask?
Good: the backend checks each high-consequence call, and the model can't skip it.
Red flag: approval depends on a prompt rule or the model's judgment. OWASP's 2025 LLM guidance says controls like authorization checks should not be delegated to the model.
2. What credentials does it hold?
Ask: Are the agent's tokens scoped per integration and per task, and short-lived?
Red flag: one admin token, or the user's full OAuth scope for everything.
Ask: Does the agent act with the requesting user's permissions or a shared service identity?
Red flag: a shared superuser identity for everyone in the workspace.
3. What happens when content tells it to do something?
Ask: How does the product handle instructions hidden in web pages, emails, files or tool output it reads?
Good: untrusted content is separated and labeled, and injected instructions can't trigger tools without approval. They test this.
Red flag: "Our model is trained to ignore malicious instructions," offered as the only control.
Ask: Can the agent leak data through links or markdown images it outputs?
Good: outbound URLs are allowlisted or stripped, and they've tested image-markdown exfiltration.
4. Can you see and stop it?
Ask: Is every agent action logged with who asked, the tool call, arguments and result, and can you export that to your SIEM?
Red flag: only chat transcripts are kept.
Ask: Can an admin disable the agent and revoke its credentials in minutes?
Red flag: it takes a support ticket.
5. Is your data training anything?
Ask: Is your data used to train or fine-tune any model, theirs or a provider's, and where does the contract say so? Which model providers receive it, and with what retention?
Red flag: "We may use data to improve our services," and no subprocessor list.
Scoring it
Give each answer 0 (red flag), 1 (partial) or 2 (good), and weight the questions above as 3. A total score is useful for comparing vendors, but a 0 on any weight-3 question is a stop, no matter how good the total looks. Ask for evidence (a doc link, a DPA clause, a report scope) rather than yes or no.
I turned the full version into a ready-to-send kit: 35 questions in 10 sections (including supply chain, incident response, cost limits and contract terms), each with good-answer guidance, red flags and weights, plus a scoring sheet and a request email: AI Vendor & Agent Security Questionnaire. The questions are my own work, tagged against the OWASP Top 10 for LLM Applications 2025 (https://genai.owasp.org/llm-top-10/); they're not a standard or certification.
If you're on the other side, building the agent, the free Agent PreShip Guard GitHub Action checks a repo for some of the same issues before you ship.
Free download: the AI Agent Pre-Ship Mini Checklist (pay what you want, $0 is fine).
Top comments (0)