TL;DR
Nango is the best platform for building, running, and observing AI agent integrations. Teams can investigate a failed operation and change the integration that produced it on the same platform.
Datadog, LangSmith, Langfuse, Arize Phoenix, and Braintrust address application tracing or evaluation needs. Use them alongside your integration runtime when you need additional visibility into model behavior, agent control flow, or response quality.
Why AI agent integrations need execution visibility
An agent that takes thirty seconds to create a support ticket might be waiting on the model, retrying an API request, or recovering from an expired access token. A useful observability setup lets you identify which step caused the delay and which customer’s connection was affected.
Finding that cause requires evidence from each stage of the request. An agent observability tool collects and analyzes traces, logs, and metrics across model calls, tool executions, and application services. For agent integrations, this includes the external requests, retries, and customer connection context behind each tool call.
Best AI agent observability tools at a glance
| Tool | Best for | Main strength | Main tradeoff |
|---|---|---|---|
| Nango | Building, running, and observing customer API integrations | Auth, tools, triggers, syncs, and execution context | Add evaluation tooling for model quality |
| Datadog | Existing Datadog monitoring setups | Agent and service trace correlation | API execution stays in your application or integration runtime |
| LangSmith | LangGraph and LangChain debugging | Agent traces and evaluations | Provider-level detail depends on instrumentation |
| Langfuse | Teams operating their own tracing stack | Sessions and prompt workflows | Self-hosting includes several storage services |
| Arize Phoenix | Local trace analysis and evaluation | OpenTelemetry and OpenInference | Production operations and API execution remain your responsibility |
| Braintrust | Regression testing from production examples | Trace-to-dataset workflows | Enterprise self-hosting retains a hosted control plane |
What to look for in observability tools for agent integrations
The tools above cover different parts of an agent’s execution. To choose between them, check what each can tell you about a failed or incorrect API operation.
Start with a tool call such as create_ticket. An agent trace can show its arguments, output, duration, and error. To investigate a provider failure, you also need the HTTP requests inside that call: the response status, failed attempts, and the connection used. A single successful tool span can conceal retries that slow the operation.
Those request details must also identify the affected customer. If one account starts returning 403 Forbidden, you need to isolate that account’s failures. Connection identifiers and integration names help you follow the same customer’s background activity, including a sync that stops updating records when no agent request is running.
Evaluation answers another question: did the agent take the right action? Creating a ticket in the wrong project can produce a successful API response. Deterministic checks, human review, and model-based evaluations help assess correctness using the evidence in your traces.
To make those investigations possible in production, check how much instrumentation you must supply, where telemetry is stored, and how long you can retrieve it. A customer report may arrive weeks after the original run, so retention and deployment requirements should be included in the comparison.
1. Nango
Best for: Teams that need visibility and control for multi-tenant, customer-facing agent integrations.
Nango provides 7,000+ prebuilt tools for 1,000+ APIs, with managed auth and all integration patterns covered: tool calls, triggers, and syncs. For observability, that means you can investigate the connection, the operation an agent called, and the background work that supplies its data within the same integration platform.
You can quickly get started with prebuilt tools, then customize them as needed using coding agents such as Claude Code, Cursor, and Codex. Nango’s runtime handles authentication, retries, rate limits, and execution.
Once an integration is running, Nango records its activity as operations with associated HTTP requests, errors, and custom messages. You can filter these records by integration, connection, function, and status to isolate a customer’s failed tool call or sync. The observability documentation describes these filters. In the GitHub example below, list_issues and update_issue actions appear alongside webhook operations, with execution times and connection identifiers.
If an update_issue action fails, those filters let you locate the run for the affected customer and inspect its HTTP requests and error messages. Because you own the action’s code, you can use that evidence to adjust its behavior or add logging for the next run.
When the fix requires access beyond a prebuilt tool, you can drop down to native API requests through the request proxy. The proxy uses the same customer connections and logs as actions and syncs, preserving that context as you customize the integration.
Coding agents can access execution evidence too. Nango’s Logs MCP tools let compatible clients list and filter operations and retrieve their messages through the Management MCP server. This gives the agent modifying an integration access to its logs during debugging.
For investigations that extend into your application monitoring stack, Nango exports OpenTelemetry traces for action executions, sync executions, third-party webhook executions, and proxied requests. Send them to a compatible collector to use your backend’s dashboards and alerts. Export alone does not guarantee that integration operations appear as child spans of an agent trace; verify context propagation and correlation in your setup.
Enterprise teams may also need control over where the integration runtime and telemetry operate. BYOC provides a Nango-managed deployment in your choice of cloud and region.
Pros
- Build, run, and investigate auth, tool calls, triggers, and syncs on one platform.
- Start with 7,000+ pre-built tools, then customize functions and native API requests as requirements grow.
- Give coding agents access to execution evidence through the Nango Management MCP.
- Choose enterprise BYOC or self-hosting to gain control over the runtime and telemetry infrastructure.
Cons
- Model response quality and prompt experiments need a separate evaluation tool.
2. Datadog
Best for: Teams that already monitor their application and infrastructure in Datadog.
Datadog Agent Observability collects model, tool, and workflow spans alongside application performance monitoring (APM). This lets teams already using Datadog investigate agent activity within their service monitoring workflow. It also supports evaluations, datasets, and experiments.
To connect agent activity with service traces, send spans through Datadog’s SDK instrumentation or HTTP ingestion API. Its span API includes an apm_trace_id field for that relationship. Your instrumentation still needs to capture the relevant operations.
Pros
- Correlates agent spans with application traces.
- Includes evaluations and experiments.
Cons
- Free and Pro include 15-day trace retention; longer retention adds cost.
- Budget for the Datadog products you use across the complete application, including APM and logs.
- Agent Observability consumes telemetry; your application or integration runtime remains responsible for executing customer API calls.
3. LangSmith
Best for: Teams building with LangGraph or LangChain that want tracing and evaluations in the same development workflow.
LangSmith captures nested agent traces, groups interactions into threads, and connects trace inspection with evaluation datasets. Its close integration with LangGraph and LangChain suits teams already using those frameworks. It also supports other frameworks and custom applications through SDKs and OpenTelemetry ingestion.
For API integrations, trace inspection helps identify runs where an agent chooses the wrong tool or supplies incorrect arguments. You can use problematic runs as evaluation examples when revising a tool description or routing strategy. Capturing the provider requests inside the tool remains part of your instrumentation work.
Pros
- Integrates with LangGraph and LangChain.
- Supports evaluations and framework-agnostic tracing.
Cons
- Self-hosted and hybrid deployments require Enterprise.
- On LangSmith Cloud, base traces have 14-day retention; longer investigation windows require extended retention.
- Agent traces need additional execution detail to explain failures inside external API calls.
4. Langfuse
Best for: Teams that want control over their tracing infrastructure while managing prompts and evaluations in the same platform.
Langfuse records traces, nested observations, and sessions, with prompt management and evaluation workflows. Sessions connect multiple conversation turns; metadata helps filter the records.
To connect those observations to API failures, attach customer and integration identifiers so you can find the corresponding execution logs. The details within each tool call depend on your instrumentation, including any spans sent via OpenTelemetry.
You can store and inspect these traces in Langfuse Cloud or operate the platform yourself. Self-hosting gives your team responsibility for the application and supporting storage services.
Pros
- Groups traces into conversation sessions.
- Offers cloud and self-hosted deployment options.
Cons
- Self-hosting requires operating the application services, PostgreSQL, ClickHouse, Redis or Valkey, and object storage.
- Cloud billing counts traces, observations, and scores, so one agent run can consume multiple units. Estimate usage against a representative workflow.
- Visibility inside custom integration logic depends on the spans your instrumentation emits.
5. Arize Phoenix
Best for: Engineers who want local or self-managed agent trace analysis and evaluation using OpenTelemetry and OpenInference.
Phoenix traces model, tool, and retrieval operations using OpenTelemetry and OpenInference. It supports local analysis, evaluations, datasets, and experiments.
In a local debugging workflow, you can inspect those spans, evaluate selected steps, and compare changes against saved examples. Moving that workflow into production adds requirements for persistent storage, authentication, backups, and availability.
When planning that deployment, distinguish Phoenix from Arize AX. They are separate products, so an Arize AX plan does not describe what a Phoenix installation includes.
Pros
- Supports OpenTelemetry and OpenInference instrumentation.
- Runs locally for trace inspection and evaluation.
Cons
- Your team operates storage, upgrades, and availability when self-hosting Phoenix.
- Its source-available code uses Elastic License 2.0, which matters if your requirement specifies a particular software license.
- Provider connection management and API execution remain separate responsibilities.
6. Braintrust
Best for: Teams that want production failures to feed directly into agent regression testing.
Braintrust connects production logs with evaluation datasets. Logs and experiments share a data structure, allowing teams to save a failed interaction and test changes against it.
To help identify interactions worth investigating, Braintrust’s September 2026 release added Patterns for recurring issues and Debugger for individual runs.
Once you identify a failure, turn it into a regression case. For integrations, that might be an agent that successfully calls an API but selects the wrong destination account. Preserve the input, tool arguments, and expected outcome, then use that case to compare changes to the agent’s instructions or tool schema.
Pros
- Turns production examples into evaluation cases.
- Supports scoring, feedback, and OpenTelemetry ingestion.
Cons
- Starter includes 14-day retention and Pro includes 30-day retention; plan for the history your team needs.
- Its Enterprise self-hosting architecture keeps the data plane in your infrastructure and the control plane hosted by Braintrust.
- Scores and processed data are separate usage dimensions to estimate.
How to choose your observability setup
The comparison comes down to the failure you need to explain. For customer-facing API integrations, start with Nango: its runtime and execution logs capture connection issues, provider requests, syncs, and webhooks that affect agent reliability.
Then identify what you need to investigate beyond those operations. Add Datadog for correlation with an existing APM setup, or an agent tracing and evaluation platform for model behavior and tool selection. Each addition should answer a specific question that your integration logs do not cover.
Test the division of responsibilities before committing. Run a small proof of concept in a test environment with three cases:
- A provider returns
429 Too Many Requests, and the tool succeeds after retrying. Can you inspect the failed attempt and the total delay? - A tool receives valid arguments for the wrong customer account. Can you find the account context and evaluate the mistake?
- A background sync fails before an agent uses its data. Can you connect the stale result to the earlier integration failure?
For each case, check how much custom instrumentation you need and whether an engineer can retrieve the evidence within your required retention window.
Frequently asked questions
What should you record when monitoring AI agent tool calls?
Record the tool name, duration, outcome, customer connection identifier, and a correlation identifier. For external API requests, include HTTP status codes, retry attempts, and provider errors. Capture arguments and results where appropriate, with credentials and sensitive customer data removed. This lets you distinguish incorrect tool use from failed API execution.
Does OpenTelemetry automatically connect agent and integration traces?
OpenTelemetry provides mechanisms for recording and transporting telemetry. Connecting operations across services also requires context propagation. Verify that your services preserve trace context and that your backend maps the incoming spans correctly. Where automatic linking is unavailable, use recorded connection identifiers, operation identifiers, and timestamps to correlate records.
Can LLM observability replace tool call monitoring?
LLM observability helps inspect model inputs, outputs, usage, and quality. Agent integrations also need visibility into authentication failures, HTTP requests, retries, syncs, and webhooks. A platform can display both when it receives the relevant telemetry; confirm that your instrumentation actually captures those operations.
Start observing your agent integrations with Nango
Put these checks into practice with a prebuilt Nango tool. Connect a test account, run the tool, and inspect its execution logs before extending the integration for your application. Follow the quickstart for setup and the observability guide for logs and telemetry export.
Top comments (0)