mech.app

Posted on Sep 2 Originally published at mech.app

Pi Agent Harness: What a Unified LLM API and Agent Loop Reveal About Tool-Calling Boundaries

#agents #api #architecture #llm

Pi hit 1.0 after nearly a year of development by the Gatsby team. It's trending at #8 on GitHub with 100K+ stars, positioned as a self-extensible coding agent with a unified multi-provider LLM API. The interesting part is not the coding agent itself. It's the runtime layer underneath: how Pi normalizes tool-calling across OpenAI, Anthropic, and Google, manages state across multi-step workflows, and explicitly punts on permission boundaries.

This is a case study in agent runtime design. Pi exposes the plumbing between reasoning (LLM calls) and execution (tool invocation), and it forces you to make a choice: convenience or isolation.

The Unified LLM API Problem

Every major LLM provider has a different tool-calling schema. OpenAI uses tools with function objects. Anthropic uses tools with input_schema. Google uses function_declarations. Pi's @earendil-works/pi-ai package abstracts this into a single interface.

Here's what that looks like:

import { createLLM } from '@earendil-works/pi-ai';

const llm = createLLM({
  provider: 'openai', // or 'anthropic', 'google', custom endpoint
  model: 'gpt-4',
  apiKey: process.env.OPENAI_API_KEY,
});

const response = await llm.chat({
  messages: [{ role: 'user', content: 'What is the weather in SF?' }],
  tools: [
    {
      name: 'get_weather',
      description: 'Get current weather for a location',
      parameters: {
        type: 'object',
        properties: {
          location: { type: 'string' },
        },
        required: ['location'],
      },
    },
  ],
});

The abstraction hides three things:

Schema translation: Pi converts your tool definition into the provider's native format at call time.
Response normalization: Tool calls come back in a consistent shape regardless of provider.
Streaming alignment: Providers handle streaming tool calls differently. Pi buffers and reconstructs them.

The impedance mismatch is real. OpenAI returns tool_calls as an array. Anthropic returns content blocks with tool_use types. Google returns functionCall objects. Pi's abstraction layer maps all of these to a single ToolCall type with name, arguments, and id.

Agent Runtime and State Management

The @earendil-works/pi-agent-core package sits on top of the LLM API. It manages the agent loop: prompt, tool call, tool execution, result injection, repeat.

The runtime tracks:

Conversation history: Messages accumulate across turns.
Tool call state: Which tools were invoked, with what arguments, and what they returned.
Execution context: Environment variables, working directory, available tools.

When a tool call happens, the runtime:

Receives the tool call from the LLM.
Looks up the tool handler in its registry.
Executes the handler with the parsed arguments.
Injects the result back into the conversation as a tool role message.
Sends the updated history back to the LLM.

This is a synchronous blocking loop. If a tool takes 30 seconds to run, the agent waits 30 seconds. If a tool fails, the error message goes back to the LLM, and the LLM decides what to do next (retry, skip, abort).

What Happens When a Tool Fails

Pi does not have built-in retry logic or circuit breakers. If a tool throws an exception, the runtime catches it, serializes the error message, and appends it to the conversation history as a tool result with an error flag.

The LLM sees:

{
  "role": "tool",
  "tool_call_id": "call_abc123",
  "content": "Error: ENOENT: no such file or directory, open '/tmp/missing.txt'"
}

The LLM can:

Ask the user for clarification.
Try a different tool.
Retry the same tool with different arguments.
Give up.

This is a design choice. Pi treats the LLM as the orchestrator. The runtime is just plumbing. If you want retries, timeouts, or fallback logic, you implement them in your tool handlers or wrap the agent loop.

The Permission Boundary Problem

Pi's documentation is explicit: "Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it."

This is not an oversight. It's a trade-off. Adding a permission system means:

Defining a policy language (what can this tool access?).
Enforcing boundaries at runtime (syscall interception, capability tokens, sandboxing).
Handling edge cases (what if a tool spawns a subprocess?).

Pi punts on this. If you need isolation, you containerize. The docs outline three patterns:

Pattern	Boundary	Overhead	Use Case
Gondolin extension	Browser extension sandbox	Low	Keep Pi and provider auth on host, isolate tool execution in browser
Docker Compose	Container network + volume mounts	Medium	Run Pi in a container, mount specific directories, restrict network access
Kubernetes	Pod security policies + network policies	High	Multi-tenant deployments, strict resource limits, audit logs

The Gondolin pattern is interesting. It runs Pi on the host but executes tools inside a browser extension sandbox. The extension has limited filesystem access and no direct network access. Tool results flow back to Pi over a message-passing bridge.

Docker Compose is the middle ground. You define a docker-compose.yml with volume mounts for the directories Pi needs to read/write, and you use Docker's network isolation to block outbound connections except to specific hosts (LLM APIs, internal services).

Kubernetes is the heavy option. You use pod security policies to drop capabilities, network policies to enforce egress rules, and resource quotas to prevent runaway tool execution.

Architecture: How the Pieces Fit

Pi is four packages:

@earendil-works/pi-ai: LLM API abstraction. Handles provider-specific schemas, streaming, and retries.
@earendil-works/pi-agent-core: Agent loop. Manages conversation state, tool registry, and execution flow.
@earendil-works/pi-coding-agent: Coding agent CLI. Pre-built tools for filesystem, shell, git, and code editing.
@earendil-works/pi-tui: Terminal UI with differential rendering for real-time agent output.

The flow:

User input → TUI → Agent Core → LLM API → Provider (OpenAI/Anthropic/Google)
                        ↓
                   Tool Registry
                        ↓
                   Tool Handlers (filesystem, shell, etc.)
                        ↓
                   Tool Results → Agent Core → LLM API → Provider

The agent core is stateful. It holds the conversation history in memory. If the process crashes, you lose the session. There is no built-in persistence layer. If you need durable state, you wrap the agent core and snapshot the conversation history to disk or a database after each turn.

Self-Extensible Coding Agent

The coding agent can modify its own tools. It has a create_tool tool that generates a new tool definition and registers it at runtime. The tool definition is TypeScript code. The agent writes it, saves it to disk, and dynamically imports it.

This is powerful and dangerous. The agent can:

Add a tool to fetch API keys from environment variables.
Add a tool to execute arbitrary shell commands.
Add a tool to modify its own tool registry.

There is no approval gate. If the LLM decides to create a tool, the tool gets created. If you're running Pi with your AWS credentials in the environment, the agent can create a tool that reads them and sends them to an external server.

The mitigation is containerization. If Pi runs in a container with no AWS credentials, no network access, and a read-only filesystem except for a scratch directory, the blast radius is limited.

Observability and Telemetry

Pi includes a @earendil-works/pi-telemetry package. It defines vendor-neutral telemetry contracts: structured logs, traces, and metrics. The reference adapter writes to stdout in JSON format.

You can plug in your own adapter to send telemetry to Datadog, Honeycomb, or an OpenTelemetry collector. The telemetry schema includes:

LLM call traces: Provider, model, token count, latency, cost estimate.
Tool call traces: Tool name, arguments, result, duration, error flag.
Agent loop traces: Turn count, total tokens, total cost, session ID.

This is useful for debugging multi-step workflows. You can trace a failed tool call back to the LLM response that triggered it, see the arguments that were passed, and inspect the error message.

Failure Modes

Pi's failure modes are predictable:

LLM rate limit: The LLM API returns a 429. Pi does not retry. The error bubbles up to the agent loop, which appends it to the conversation history. The LLM sees the error and may ask the user to wait.
Tool timeout: A tool runs for too long. Pi does not have built-in timeouts. The tool blocks the agent loop until it completes or crashes.
Tool crash: A tool throws an exception. The error message goes back to the LLM. The LLM may retry or give up.
State corruption: The conversation history grows too large. The LLM context window overflows. Pi does not truncate or summarize. The LLM API returns an error. The agent loop stops.
Permission denied: A tool tries to access a file or network resource it doesn't have permission for. The error message goes back to the LLM.

The common thread: Pi does not hide errors. It surfaces them to the LLM and lets the LLM decide what to do.

Trade-Offs

Aspect	Pi's Choice	Alternative	Implication
Permission system	None (user's permissions)	Built-in sandboxing	You must containerize for isolation
State persistence	In-memory only	Automatic snapshots	Session lost on crash
Tool retries	None (LLM decides)	Automatic retry with backoff	More code in tool handlers
Context window	No truncation	Sliding window or summarization	Agent stops when context overflows
Provider lock-in	Unified API	Provider-specific code	Easier to switch providers, harder to use provider-specific features

Technical Verdict

Use Pi when:

You need to support multiple LLM providers without rewriting tool-calling logic.
You want a lightweight agent runtime that doesn't impose opinions on state management or error handling.
You're comfortable containerizing for security boundaries.
You need a self-extensible coding agent that can modify its own tools.

Avoid Pi when:

You need built-in permission boundaries without containerization.
You need automatic state persistence across crashes.
You need automatic retry logic or circuit breakers for flaky tools.
You need to use provider-specific features (like OpenAI's structured outputs or Anthropic's prompt caching) that don't map cleanly to a unified API.

Pi is plumbing. It solves the provider abstraction problem and gives you a basic agent loop. Everything else (security, persistence, observability, error handling) is your responsibility. That's a feature, not a bug. It keeps the runtime small and forces you to think about the boundaries that matter for your deployment.

DEV Community