What 84,800 local spans taught me about my coding agents and my local AI services

When I run one coding agent in one terminal, I can scroll up to find out what failed. My setup stopped working that way a few months ago.

On my workstation I run two kinds of software at the same time. The first kind is coding agents: Claude Code with subagents, OpenAI Codex CLI, GitHub Copilot CLI, Aider, and Cline. The second kind is the local services those agents call: a LiteLLM router for model routing and fallbacks, Ollama for local models, SearXNG for search, Firecrawl for scraping, and a local memory service.

Failures in that setup rarely print a stack trace. A parent agent stops answering. A nightly job produces nothing. A scrape-and-summarize run dies in the middle. I could not tell whether the fault was in the agent, in the router, or in a service.

Tools like ccusage and CodeBurn answer how many tokens I spent. agentsview lets me read transcripts. I needed something different: one place to see which child session hung, which service was slow, and whether last night's jobs ran.

So I wrote agent-tap. It reads the logs, databases, and health endpoints that my agents and services already produce, and it stores them in one SQLite file. A dashboard at http://127.0.0.1:6007 shows the result.

What the first run showed

The local SQLite database indexed 21,032 traces and 84,866 spans across 21,030 sessions. I looked at the error and correlation tables and found five problems that I had not noticed before.

1. An MCP token refresh failed about 2,030 times. An MCP server returned HTTP 400 on every OAuth refresh (mcp_oauth_400_refresh_loop), and the harness retried in the background without showing an error. Grouping spans by error.class made the loop visible in one query.
2. The local router closed connections after 120 seconds. Long prompts took more than 120 seconds to reach the first token, so the router hit its first_byte_timeout and logged gateway_http_499 drops. I found it by joining harness spans with gateway spans.
3. LiteLLM 1.85.0 crashed in the middle of a stream. Delegated runs died with truncated tool calls (litellm_midstream_fallback_error). The parent-to-child link took me from the failed parent to the exact line in the gateway log.
4. Child agents hung for 600 seconds. A child session waited on an interactive prompt with no terminal attached, and the parent looked frozen until the 600 second ACP timeout (acp_prompt_timeout_600s).
5. Codex background jobs failed with HTTP 401. In a terminal, an expired credential fails loudly. In a job without a terminal, the 401 turned into an empty review. A query for Codex spans with status ERROR listed every affected job.

Agents and services in one model

An agent run fails in the agent, or in a service the agent called. To see both, agent-tap treats every target as a service with a kind.

• A harness is a coding agent such as Claude Code, Codex, Copilot CLI, Aider, or Cline. It appears in Harness Activity with sessions, turns, tool calls, skill calls, and parent-to-child links.
• A gateway is a model router or proxy such as LiteLLM or an OpenAI-compatible local gateway.
• A service is anything else, such as Ollama, SearXNG, or Firecrawl. A service can be an HTTP endpoint, a Docker container, a process, or a log file.

Gateways and services appear in Running Services & Gateways with their status, request count, error count, and p50 and p90 latency. All targets share the error classes, the trace drawer, and the schedule view. To add a service of your own, run agent-tap onboard --name my-api --kind service --health-url http://127.0.0.1:9000/health.

How it works

agent-tap does not wrap your agents or sit in front of them. It reads files and databases that already exist and remembers its position in each one, so every poll reads only new data. The storage is one SQLite file in WAL mode. The tables have the same shape as Arize Phoenix, and you can query them with agent-tap query.

Prompt text is off by default. Before agent-tap writes a span, it drops prompt and response text and tool arguments. It keeps sizes, token counts, tool names, durations, session links, and error classes. If you turn prompt storage on, a redaction pass still masks bearer tokens, sk- keys, GitHub tokens, and JSON Web Tokens.

If an adapter misreads your logs, run agent-tap bug-report. It prints a diagnostic bundle with secrets masked and your home directory replaced by ~, and --open starts a pre-filled GitHub issue.

Try it

agent-tap is MIT licensed and needs Python 3.11 or later. It runs on Linux, macOS, Windows, and WSL2.

pipx install agent-tap
agent-tap init
agent-tap run --once
agent-tap serve

Run agent-tap doctor first if you only want to see what it finds. The code and documentation are at https://github.com/nareshh74/agent-tap. If you run several agents or local services, I would like to know which one your logs break first.

Naresh Kumar