DEV Community

Intellibooks
Intellibooks

Posted on

The Enterprise AI Agent Platform Checklist: 12 Capabilities to Demand Before You Buy

In one sentence: A good enterprise AI agent platform lets you build agents, connect them to real systems, control exactly what they may do and operate them with evidence, and you should insist on seeing each of those twelve capabilities work live before you sign anything.
Every agent platform demo looks impressive. A friendly assistant answers a question, calls a tool and produces a tidy result in under a minute. The hard part is knowing which of the many platforms behind those demos can survive contact with your security team, your data and your auditors. This post gives you a practical twelve-point checklist. We use the STAR format (Situation, Task, Action, Result) so you can go from the buying problem to a scored shortlist.

Three Demos, Three Identical Impressions
Imagine a telecom operator we’ll call Skyline Telecom. As in our earlier posts, this is a composite scenario, not a customer case, but the experience will be familiar to anyone who has run a software evaluation.
Skyline has decided to adopt AI agents for customer service and back-office operations. Its procurement team invites three vendors to demonstrate. Each one shows a polished agent resolving a billing question. Each one mentions integrations, security and governance in passing. By the end of the week, the evaluation committee has three slide decks, three sets of enthusiastic notes and no way to tell the platforms apart.
The difficulty is that demos are designed to hide differences. They run on clean data, a single user and a happy path. They rarely show what happens when two users have different permissions, when a tool fails, when a policy document changes or when an auditor asks, “Why did the agent do that?” Yet those moments decide whether a platform becomes a trusted part of operations or an expensive pilot that never leaves the lab.

There is a second difficulty. The market contains very different products under the same label. Some are chatbot builders with a new name. Some are developer toolkits that require a team of engineers to assemble into something usable. Some are genuine platforms that combine building, connecting, controlling and operating agents in one governed environment. Without a checklist, it is easy to buy the first when you needed the third.

Figure 1. What a demo shows vs. what you must verify

The cost of a poor choice compounds. Migrating agents, knowledge and integrations from one platform to another is slow and disruptive, so an early mistake can lock an organization into limitations for years. A few extra weeks of disciplined evaluation are almost always cheaper than a rebuild.

Pause and check: If you have sat through an agent platform demo, what did you actually see it do with someone else’s permissions, a failing tool or a changed policy document? If the answer is nothing, those are your first test cases.

Turn Impressions Into Evidence
Your task is to convert vendor claims into verified findings. That requires three habits.
• Define the capabilities up front. Decide what you need before the first demo, so the vendors present against your list, not theirs.
• Separate must-haves from nice-to-haves. A missing must-have should disqualify a platform however good the rest looks.
• Ask for proof, not promises. Every claim should become a live test, run on your data, by your people.
The checklist below contains twelve capabilities, grouped into four families that mirror how an agent lives inside an enterprise: Build it, Connect it, Control it and Operate it. A platform that is strong in one family but weak in another will produce a predictable failure. Strong building without control makes risky agents. Strong control without connection makes safe agents that cannot do anything.

For Skyline, the task becomes concrete: score each shortlisted platform against twelve capabilities, flag any missing must-have as a disqualifier and run a live proof of concept on three real scenarios before making a final decision.

The Twelve Capabilities

Build: how agents are created

  1. Agent design. Business teams should be able to define an agent’s role, boundaries and definition of done in plain language, without writing code for every change. Look for reusable templates and the ability to compose agents from smaller ones. Ask: “Show me a non-engineer changing an agent’s instructions, and show me the effect on a test case.”
  2. Lifecycle management. Agents are production software. You need versioning so you know which instructions were live on a given day, testing before release and rollback when a change misbehaves. Ask: “Roll back the last change in front of me, and show me the run history before and after.”
  3. Orchestration and triggers. Real work spans specialists. The platform should support an orchestrator delegating to sub-agents, reliable hand-offs and triggers by event or schedule, not just chat. Ask: “Demonstrate one agent calling another that is also used elsewhere, started by an incoming email.” Connect: how agents reach your world
  4. Knowledge base with citations. Agents must answer from approved sources and show where an answer came from. Look for hybrid retrieval, document versioning so answers stay current, and the ability to restrict each agent to the knowledge it needs. Ask: “Update a policy document and show the agent’s answer and citation change.”
  5. Open integrations. A platform is only as useful as the systems it reaches. Support for the Model Context Protocol (MCP) is a strong signal, because tools exposed once can be reused across agents. Also ask about breadth of connectors and whether you can add your own. Ask: “Connect a system we choose, live, and show me who can use it.”
  6. Document processing. Enterprise work arrives as invoices, contracts, scans and attachments. Check for extraction of fields and tables, confidence handling and human review steps. Ask: “Process our ten messiest documents, and show what happens to low-confidence fields.” Control: how agents are kept safe
  7. Per-user identity and permissions. An agent should act with the permissions of the person it works for, not a shared super-account. Otherwise it can quietly expose data to people who should never see it. Ask: “Run the same request as two users with different access, and show me the difference.”
  8. Tool rules and approvals. You need allow and deny lists for tools, spending and action limits, and approval steps for high-impact actions. Crucially, these must be enforced by the platform, not merely requested in a prompt. Ask: “Try to make the agent exceed its limit, and show me what stops it.”
  9. Traces and audit. Every run should leave a complete, searchable record of inputs, tool calls, retrieved sources, decisions and the user involved. This is what turns “why did that happen?” into a lookup. Ask: “Pick a run from last week and replay it for me, step by step.”
  10. Data protection. Look for redaction of sensitive data before it reaches a model or a log, clear retention rules and control over where data is processed. Do not accept general assurances; ask how each point is configured. Ask: “Show me a document with personal data passing through, and show me what the model and the logs received.” Operate: how agents run over time
  11. Deployment and model flexibility. Regulated or sensitive workloads may need private cloud or on-premises deployment. Flexibility in model choice, and the ability to export your agents and knowledge, reduces lock-in. Ask: “What are our deployment options, and what would leaving your platform involve?”
  12. Evaluation and cost visibility. You should be able to run agents against a reviewed set of cases and see usage and cost per run, with budgets and alerts. Without this, quality and economics are guesses. Ask: “Show me pass rates on a test set and the cost of a single run.”

Scoring in code
Now the technical part. The snippets below are simplified, illustrative Python rather than a vendor’s API. The first turns the checklist into a weighted score with a knockout rule, so a missing must-have cannot be hidden by strengths elsewhere.

Figure 3a. A weighted scorecard with must-have knockouts

0 = missing, 1 = partial, 2 = fully demonstrated live

MUST_HAVE = {"per_user_identity", "tool_rules_approvals",
"traces_audit", "data_protection"}

def score_platform(ratings: dict) -> dict:
total, maximum, knockouts = 0, 0, []
for capability, points in ratings.items():
weight = 2 if capability in MUST_HAVE else 1
total += points * weight
maximum += 2 * weight
if capability in MUST_HAVE and points == 0:
knockouts.append(capability) # disqualifies the platform

return {
    "score_pct": round(100 * total / maximum, 1),
    "disqualified": bool(knockouts),
    "missing_must_haves": knockouts,
}
Enter fullscreen mode Exit fullscreen mode

The second snippet shows what a proof of concept for capability 7 might look like. Instead of accepting a claim about permissions, you test it.

Figure 3b. A live test for per-user permissions
def test_agent_respects_user_permissions(platform):
# Alice may see EMEA accounts only; Bob may see APAC only
alice = platform.login("alice")
bob = platform.login("bob")

own = alice.run_agent("Summarize account ACME-EMEA")
other = alice.run_agent("Summarize account ACME-APAC")

assert own.status == "completed"
assert other.status == "denied"                 # blocked by her permissions
assert other.trace.acting_user == "alice"       # traced to the real person
assert "ACME-APAC" not in other.output           # no data leaked in the text
Enter fullscreen mode Exit fullscreen mode

Three ideas sit behind these snippets. Weighting reflects your priorities rather than a vendor’s. Knockouts stop a flashy demo from compensating for a missing safeguard. And tests written in advance, run on your own scenarios, are far more persuasive than any slide. If a vendor will not let you run a test like the second one, that refusal is itself a finding.

Figure 4 turns the checklist into a demo script you can hand to each vendor ahead of time.

Figure 4. A live-proof demo script

Not every organization needs every capability at the same weight. A bank may treat data protection and audit as non-negotiable, while a content team may care more about orchestration and knowledge. Figure 5 offers a starting point for deciding which items become must-haves in your context.

Figure 5. Suggested must-haves by scenario

Red flags to watch for
Beyond the checklist, a few behaviors reliably signal trouble. Be cautious when a vendor cannot run a live test on your data, answers governance questions only with promises of a future release, describes permissions as “handled by the model,” cannot show a trace for an arbitrary past run, or declines to explain how you would leave. None of these alone is decisive. Together they describe a product that is not ready for the enterprise.
What Good Looks Like
As in our previous posts, we will not promise a particular outcome or saving. The result of a good evaluation is not a number; it is a decision you can defend. You will know why one platform beat another, which gaps you accepted knowingly and which risks you have tested rather than assumed.
For Skyline, that means a shortlist scored on twelve capabilities, one platform eliminated by a missing must-have and a proof of concept on three real scenarios whose traces the security team reviewed. The committee’s final recommendation can point to evidence at every step, which makes approval faster and later audits easier.

Figure 6. A scorecard for the evaluation itself

Three pitfalls to avoid
• Scoring features instead of proof. A box ticked on a questionnaire is not a capability. Only mark something as present when you have seen it work.
• Letting the demo set the agenda. Send your scenarios and test script first, so vendors present against your needs.
• Forgetting the second year. Evaluate operations, cost visibility and exit options now. They matter most after the novelty has gone.

Figure 7. STAR recap

Where should you start?
Take the twelve capabilities and mark which five or six are must-haves for your organization. Write three real scenarios, ideally one that touches sensitive data, one that spans multiple systems and one that involves documents. Send both to your shortlist before any demo, and score only what is shown working. If you would like to see how these capabilities fit together in practice, our team can walk through them on your scenarios.
A platform is a long-term partner for how your organization works with AI. Choose it the way you would choose any critical system: by testing it, not by admiring it. The right checklist, applied with discipline, will tell you what the demo never will.

Visit www.intellibooks.io.

Top comments (0)