AI Agent Observability: Tracing Tool Calls, Costs and Outcomes

AI agents introduce a new monitoring problem.

Traditional software is mostly deterministic:

Request
→ Code
→ Database
→ Response

An AI agent can look more like:

Request
→ Context retrieval
→ Model
→ Tool selection
→ API
→ Model
→ Another tool
→ Validation
→ Action
→ Response

The execution path can change from one request to another.

That's why AI agent observability needs to go beyond application logs.

What to Trace

A useful agent trace should capture:

session_id
agent_id
model
context_version
tool_calls
latency
tokens
errors
retries
policy_checks
approvals
outcome

Don't store sensitive model reasoning simply because you can.

Structured execution events are usually much more useful operationally.

Tool Calls

For an agent with:

get_customer()
get_order()
check_inventory()
create_ticket()
update_crm()

log:

Tool
Input
Output/status
Duration
Authorization
Retry count

This lets developers find the exact point where a workflow failed.

Cost Observability

One user request may trigger multiple model calls.

Request
├── Model call
├── Retrieval
├── Model call
├── Tool
├── Model call
└── Validation

Track:

Tokens
Model
Cost
Tool calls
Retries
Cost per successful task

Cisco and Splunk's latest observability work specifically adds visibility into token spend and coding-agent usage.

Evaluation

HTTP 200 doesn't mean the agent succeeded.

Evaluate:

Task completion
Accuracy
Groundedness
Tool selection
Policy compliance
Escalation quality
Security

Also monitor:

Unexpected tool calls
Unauthorized access
Sensitive data retrieval
Policy violations
Abnormal activity

This is especially important when agents can modify production systems.

Agent Trace Example
User
↓
Agent
↓
retrieve_customer()
↓
get_order()
↓
check_policy()
↓
human approval
↓
refund_order()
↓
validate_result()
↓
response

If the customer later reports a problem, the developer can inspect the execution trace rather than guessing what happened.

Control Plane

At larger scale, observability becomes part of an agent control plane.

          AGENTS
             |
      CONTROL PLANE
   /       |       \

Identity Policy Observability
\ | /
APIs
|
Data Systems

Salesforce's current AI Control Plane follows this direction by combining agent discovery, identity, policy, lifecycle management, evaluation, observability and cost controls.

Final Thought

Traditional monitoring asks:

Is the application healthy?

AI agent observability asks:

What did the agent do, why did the workflow fail, what did it cost, and did the final outcome actually meet the task?

That is the monitoring model production agentic software needs.