DEV Community: Paw from Oz

A small CLI to stop hitting Claude Code's "too many MCP tools loaded" warning

Paw from Oz — Sun, 26 Jul 2026 12:06:01 +0000

If you run more than a couple of MCP servers in Claude Code, you've probably hit
the loaded-tools warning (~40 tools). The usual fix is manually enabling/disabling
servers by hand every time your task changes, and remembering which tools depend
on which.

mcp-profile is a small, dependency-free CLI (+ a Claude Code skill) that lets
you define named profiles — e.g. "research", "coding", "deploy" — each listing
the MCP servers that task actually needs, then switch between them with one
command:

mcp-profile use research

It edits the same enabledMcpjsonServers / disabledMcpjsonServers fields
Claude Code already reads — no new config format, no daemon, no account, no
network calls. MIT licensed.

Repo: https://github.com/pozga/mcp-profile

Feedback wanted: does this match how you're actually hitting the tool-count
problem? What would make this worth paying for (team-shared profiles? a
dashboard?) vs. just a free utility?

Update: Thanks @alexshev for the comment below —
shipped exactly what you asked for. The default profile example now ships five
task-type profiles out of the box: coding, research, deploy, docs, data.
See the v0.2 commit.

How I caught a Claude prompt regression before it hit production

Paw from Oz — Sun, 26 Jul 2026 10:24:41 +0000

I changed one line in my system prompt last week. A single word — "concisely" became "briefly". I thought nothing of it.

Two hours later, a user complained that the support bot was cutting them off mid-explanation.

I had no tests. I had no diff. I had no idea what changed.

That's what pushed me to build AgentSpec — a testing framework for AI agents. This is how I actually use it.

The problem with testing AI agents

Traditional testing is deterministic: input X always produces output Y. AI agents don't work that way. Ask the same question twice and you get slightly different answers. That's not a bug — it's the whole point.

So you can't do expect(response).toBe("Reset your password by..."). You need flexible assertions that check intent, not exact output.

AgentSpec gives you that.

Setting up your first test

Install it:

npm install -g @ozperium/agentspec

AgentSpec tests any HTTP agent that accepts {input: "..."} and returns {output: "..."}. Wrap your existing agent in one route and you're done.

Create a test file tests/support-agent.yaml:

name: "support agent regression suite"
tests:
  - name: "handles password reset"
    input: "How do I reset my password?"
    expect:
      contains_any: ["reset", "password", "email", "link"]
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing to correct team"
    input: "I want a refund for last month"
    expect:
      contains_any: ["billing", "refund", "team", "connect"]
      tool_called: "transfer_to_billing"

  - name: "doesn't hallucinate policy details"
    input: "What's your return policy?"
    expect:
      not_contains_any: ["30 days", "60 days", "90 days"]
      contains: "policy"

Run it against your agent:

agentspec run --endpoint https://your-agent.example.com/chat --dir tests

The behavior diff — where the real value is

After the first passing run, AgentSpec stores the baseline output for each test. When something changes, you see exactly what shifted:

WARNING REGRESSION support agent > handles password reset
  Behavior REGRESSED — test was passing, now failing
  + added: briefly, concise, short
  - removed: step, by, step, here, is, how

That's the word diff between what the agent used to say and what it says now. In my case, adding "briefly" to the system prompt made the agent stop giving step-by-step instructions — which users actually wanted.

This diff is what I didn't have before. It would have caught the regression in 30 seconds.

LLM-as-judge for semantic checks

Some things are hard to check with string matching. "Is this response helpful?" isn't a regex problem.

AgentSpec supports LLM-as-judge using a local Ollama model — no API costs, runs in CI:

# Start Ollama locally
ollama pull qwen2.5:7b

# Run with judge
agentspec run --endpoint https://your-agent.example.com/chat \
  --judge-endpoint http://127.0.0.1:11434/v1/chat/completions \
  --judge-model qwen2.5:7b

tests:
  - name: "response is empathetic"
    input: "I've been waiting 3 weeks for my order"
    expect:
      llm_judge: "The response should acknowledge frustration and offer help, not dismiss the complaint"

  - name: "doesn't upsell when user is upset"
    input: "This product broke after one day"
    expect:
      llm_judge: "Focus on resolving the issue — no promotional language or upsell attempts"
      not_contains_any: ["upgrade", "premium", "offer"]

CI integration

Add this to your GitHub Actions workflow and you'll catch regressions before they merge:

- name: Test agent behavior
  run: |
    npm install -g @ozperium/agentspec
    agentspec run --endpoint ${{ secrets.AGENT_URL }} --dir tests --ci

--ci mode exits with code 1 on any failure. Combine it with branch protection and no prompt change ships without passing the test suite.

Start small

You don't need 50 tests. Start with three scenarios:

The happy path — does it handle the most common request correctly?
The edge case that burned you before — the thing that went wrong last time
The thing you'd be embarrassed by — hallucinated facts, wrong policy details, off-brand tone

Three tests, 15 minutes to write, run on every deploy. That's the baseline.

npm install -g @ozperium/agentspec
agentspec init
agentspec run --endpoint https://your-agent.example.com/chat

GitHub: github.com/Ozperium/agentspec

Stop manually logging your AI API calls — one-line auto-logging for OpenAI and Anthropic

Paw from Oz — Sat, 25 Jul 2026 04:23:00 +0000

Every time you call an AI API, you're spending money. But unless you're checking the dashboard after each session, you have no idea which project or model is driving the bill.

I solved this by wrapping my AI clients with a one-line middleware that logs every call automatically.

The problem with manual logging

AICostTracker lets you run aicost log myapp gpt-4o 1500 800 to record a call. Useful — but nobody does this consistently. You forget, you're in flow, you'll do it later.

The logs end up patchy and the summary is useless.

Auto-logging with `track()`

In v0.1.2, I added a track() function that wraps your OpenAI or Anthropic client:

npm install @ozperium/aicost-tracker

import OpenAI from 'openai';
import { track } from '@ozperium/aicost-tracker';

const openai = track(new OpenAI(), { project: 'myapp' });

// From here, every call is logged automatically
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this doc' }]
});

That's it. No changes to your existing code beyond the wrap.

It works the same way with Anthropic:

import Anthropic from '@anthropic-ai/sdk';
import { track } from '@ozperium/aicost-tracker';

const anthropic = track(new Anthropic(), { project: 'summarizer' });
const response = await anthropic.messages.create({ model: 'claude-3-5-sonnet-20241022', ... });

What gets logged

After a few sessions:

aicost summary

  AI Cost Tracker — Summary
  ══════════════════════════════════════════════════
  Total cost:       $1.2847
  Input tokens:     124,000
  Output tokens:    67,500

  By Project:
  ──────────────────────────────────────────────────
  myapp                $    0.9102  31 calls
  summarizer           $    0.3745  12 calls

  By Model:
  ──────────────────────────────────────────────────
  gpt-4o               $    0.9102  31 calls
  claude-3-5-sonnet    $    0.3745  12 calls

How it works

track() patches client.chat.completions.create (OpenAI) or client.messages.create (Anthropic) to intercept the response, extract usage from the usage field, and call logUsage() before returning. It's a thin synchronous wrapper — zero added latency.

If logging fails for any reason, the error is swallowed silently. Your API call always gets its response.

Per-project breakdown

The { project: 'myapp' } option lets you tag every call with a project name. If you're running multiple agents or tools in the same codebase, give each its own project tag:

const researchAgent = track(new OpenAI(), { project: 'research' });
const summaryAgent = track(new OpenAI(), { project: 'summary' });

Then aicost summary shows the breakdown by project automatically.

GitHub: https://github.com/Ozperium/aicost-tracker

Also: AgentSpec for testing AI agent behavior, quota for monitoring rate limits.

Testing AI agents is hard. I built a framework for it.

Paw from Oz — Fri, 24 Jul 2026 04:22:54 +0000

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.

You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect(output).toBe("transfer to billing") fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built AgentSpec to fix this — a testing framework designed specifically for non-deterministic AI output.

What it looks like

Define tests in YAML:

name: "billing-agent-tests"
tests:
  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

Run them:

npm install -g @ozperium/agentspec
agentspec init
agentspec run

Output:

AgentSpec v1.2.0

  ✓ routes billing question (312ms)
  ✓ handles expired token (891ms)
  ✗ response is on-topic
    Expected semantic similarity ≥ 0.5, got 0.31
    Input: "how do I reset my password?"
    Output: "Please contact support for account issues."

2 passed, 1 failed

Why assertions built for AI

Standard test assertions assume deterministic output. AI output isn't.

AgentSpec provides:

contains / not_contains — substring presence (handles paraphrasing)
contains_any / contains_all — flexible multi-keyword matching
semantically_similar — word-overlap similarity score, not exact match
regex — pattern matching for structured output
tool_called — verify the agent invoked the right tool
max_latency_ms — catch regressions in response time
llm_judge — use a local model (Ollama) to evaluate quality in natural language

CI integration

agentspec run --ci   # exits 1 on failure, JUnit XML output

Drop it in GitHub Actions:

- name: Run AgentSpec
  run: agentspec run --ci

Now every PR that changes a prompt, model, or tool gets behavior regression tests.

Behavior diff reports

Run with --diff to compare against a previous baseline:

agentspec run --diff baseline.json

Shows exactly what changed between runs — useful when you swap models or update prompts.

HTTP agents

Point it at any running agent endpoint:

agentspec run --endpoint http://localhost:3000/chat

No SDK integration required.

Install

npm install -g @ozperium/agentspec
agentspec init
agentspec run

GitHub: https://github.com/Ozperium/agentspec

Also in the stack: AICostTracker for tracking what you spend on AI APIs, and quota for monitoring rate limits before they stop you.

I built a terminal tool to see your AI coding limits before they stop you

Paw from Oz — Thu, 23 Jul 2026 22:23:44 +0000

The worst moment in a coding session is hitting a rate limit mid-task.

You're deep in a refactor. Claude Code stops. Or Codex resets. And you spend the next 20 minutes switching tools, losing context, or just waiting. The "Codex Resets" thread on Hacker News got 270 points and 175 comments. The pain is widespread.

The problem isn't the limit — it's that you don't know how close you are until you hit it.

I built quota to fix that.

What it does

$ quota

  quota — your ai usage, at a glance

  Claude Code   ████████████░░  3h 12m of 5h    reset in 2h 48m
  Codex         ███░░░░░░░░░░░  3 of 40 reqs    reset in 4h 12m
  Cursor        ──────────────  not connected

One command. Shows remaining usage for every AI coding tool you have connected, along with reset times.

You can also run it in watch mode:

quota watch 30   # re-check every 30 seconds

Or pipe it into scripts:

quota --json | jq '.providers.claude_code.percent_remaining'

How it works

quota reads provider state locally — no API calls, no accounts, no telemetry. It reads from ~/.claude and ~/.codex directly, the same files the tools themselves write. Everything stays on your machine.

npm install -g @ozperium/quota
quota

Why not just check the app?

Claude Code and Codex don't surface remaining quota prominently. You either have to open a browser dashboard or wait for the error. Neither is useful when you're mid-task in a terminal.

quota lives where you work — in the terminal, right next to your code.

What's next

OAuth login for Cursor and Grok (local reads aren't possible for these)
Shell prompt integration (show remaining % in starship/oh-my-posh)
CI check — fail a build if you're out of quota before a long test run

GitHub: https://github.com/Ozperium/quota
npm: https://www.npmjs.com/package/@ozperium/quota

Also in the stack: AgentSpec for testing AI agent behavior, and AICostTracker for tracking what you're spending on AI APIs.

I built a local-first CLI to track what I'm actually spending on AI APIs

Paw from Oz — Thu, 23 Jul 2026 22:23:17 +0000

The worst part of using AI APIs isn't the cost. It's not knowing what you're spending until the bill arrives.

I wanted per-project cost tracking — which project used the most tokens? Which model was the expensive one? The cloud dashboards don't tell you that.

So I built AICostTracker — a local CLI that logs every API call and breaks down costs by project and model.

Auto-logging (the easy way)

Install as a library and wrap your AI client — every call is logged automatically, no manual work:

npm install @ozperium/aicost-tracker

import OpenAI from 'openai';
import { track } from '@ozperium/aicost-tracker';

const openai = track(new OpenAI(), { project: 'myapp' });

// Nothing else to do — all calls now logged
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'hello' }]
});

Works with Anthropic too:

import Anthropic from '@anthropic-ai/sdk';
import { track } from '@ozperium/aicost-tracker';

const anthropic = track(new Anthropic(), { project: 'docs-bot' });
// All messages.create calls now logged automatically

The wrapper is transparent — it never surfaces logging errors to your code, and adds zero latency to the API call itself.

See your costs

aicost summary

  AI Cost Tracker — Summary
  ══════════════════════════════════════════════════
  Total entries:    12
  Total cost:       $0.4821
  Input tokens:     48,000
  Output tokens:    31,200

  By Project:
  ──────────────────────────────────────────────────
  myapp                $    0.3102  8 calls
  docs-bot             $    0.1719  4 calls

  By Model:
  ──────────────────────────────────────────────────
  gpt-4o               $    0.3102  8 calls
  claude-3-5-sonnet    $    0.1719  4 calls

Manual logging (CLI)

If you prefer not to install as a library, you can log calls manually:

npm install -g @ozperium/aicost-tracker
aicost log myapp gpt-4o 1500 800 "generated README"
aicost summary

Why local-first?

All data lives in ~/.aicost/usage.jsonl — one JSON line per call, greppable, portable, yours. No cloud account, no telemetry, no dashboard to maintain.

Pricing is baked in for OpenAI (gpt-4o, o1, gpt-4-turbo, gpt-3.5), Anthropic (claude-3.5-sonnet, claude-3-opus, haiku), Google (gemini-1.5), and local/Ollama (cost: $0).

GitHub: https://github.com/Ozperium/aicost-tracker

Also in the stack: AgentSpec for testing AI agent behavior, and quota for monitoring rate limits before they stop you.

Stoke: A kill switch for runaway AI agents

Paw from Oz — Mon, 20 Jul 2026 19:00:58 +0000

I've been building with AI coding agents daily — Claude Code, OpenCode, custom agents. They're productive, but they have a failure mode that costs real money: they retry, loop, and fan out on metered API keys. Billing alerts fire hours after the damage is done.

Observability tools (Langfuse, Helicone, Portkey) tell you what happened. None of them stop it from happening.

Stoke

Stoke is a single Rust binary (~5.5 MB, zero runtime dependencies) that sits between your AI agents and model providers. It enforces policy before any provider is called:

Hard budget caps — per-API-key spend limits in USD. Over the limit = 429 Budget exceeded, not an alert. Streamed responses accrue spend too.
Loop detection — exact prompt-hash matching plus optional semantic similarity. 5 similar requests from one key in 60 seconds = blocked for 120 seconds. Catches agents stuck in retry loops.
Rate limiting — per-key requests-per-minute over a sliding window.

The key insight: enforce, don't just observe.

What else it does

Node-aware routing — polls your Ollama nodes, places requests on warm machines, balances by live load, fails over automatically
Federation — one Stoke can front another across machines, with a hop guard that prevents cycles
Auto-routing — send "model": "auto" and Stoke scores every (model, node) candidate on cost, predicted latency, and your preferences. Every response carries a receipt explaining the decision.
Anthropic Messages API — point Claude Code at Stoke with ANTHROPIC_BASE_URL. Same enforcement, then passthrough.
Plugin system — built-in PII redaction, audit logging, prompt harness + HTTP webhooks in any language + JS/TS via deno_core
Fail-closed auth — no keys configured and no dev flag = every request rejected. There is no accidentally-open state.

Why Rust

Every Python proxy (LiteLLM, OpenRouter) adds latency and memory overhead — 500MB+ RAM, Postgres, Redis. Stoke is 5.5 MB, starts in milliseconds, uses ~10 MB RAM, and has zero runtime dependencies. It's a single binary you can curl-pipe-sh install.

Multi-machine setup

If you have Ollama running on two machines (say a MacBook and a Mac Studio), Stoke routes between them automatically — preferring the machine where the model is already loaded in RAM, balancing by in-flight count, and measuring real time-to-first-token per node.

Or use federation: run a Stoke on each machine, keep Ollama on 127.0.0.1, and front them with a type = "stoke" provider. Stoke is the only network-facing door, and it requires auth.

Quickstart

curl -sSf https://stokegate.com/install | sh -s -- --version nightly

Minimal config:

routing = "single"
default_model = "qwen3:8b"

[server]
host = "127.0.0.1"
port = 8787

[[providers]]
name = "local"
base_url = "http://127.0.0.1:11434/v1"
tier = "local"

Start it:

STOKE_API_KEYS=stk-mykey ./target/release/stoke

Point any OpenAI-compatible client at http://localhost:8787/v1 — OpenCode, Cursor, custom apps. Budget caps, rate limits, and loop detection apply to every request.

Status

Pre-release, MIT licensed, EU-based. Dogfooded daily on a real two-Mac + Ollama setup. Built in the open at github.com/Ozperium/stoke.

The smoke scripts (./scripts/smoke.sh, ./scripts/smoke_federation.sh) run the full system end-to-end with mock Ollama nodes — no Ollama required to verify the claims.

What's next

Anthropic to OpenAI translation (so Claude Code can run against local models)
Homebrew formula and Docker images
Per-key loop-detection thresholds
OpenTelemetry traces and Prometheus metrics

Feedback welcome. The repo is github.com/Ozperium/stoke.

Testing AI Agents with AgentSpec: Jest for Non-Deterministic Behavior

Paw from Oz — Sun, 19 Jul 2026 19:13:47 +0000

AI agents are non-deterministic. This is both their superpower and their biggest testing challenge.

When you change a prompt, swap a model, or update a tool definition, the agent's behavior shifts in ways you can't predict. Traditional test tools (Jest, Vitest, Playwright) are built for deterministic code — they expect exact string matching. AI agents don't work that way.

Last week, I changed a system prompt in a support agent. It still passed all our Jest tests (the function still returned a string). But the agent had stopped offering password reset help — a regression that only surfaced when a user complained.

That's why I built AgentSpec.

What is AgentSpec?

AgentSpec is a testing framework designed specifically for AI agents. It provides assertions that handle non-deterministic output, behavior diff reports that show what changed, and CI/CD integration out of the box.

Assertions for AI output

Instead of exact string matching, AgentSpec provides assertions that work with the variability of AI responses:

# agentspec.yaml
name: "support agent"
tests:
  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support team"]
      tool_called: "transfer_to_billing"

  - name: "response is on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

Available assertions:

contains / not_contains — substring checks
contains_any / contains_all — flexible multi-string matching
regex — pattern matching (error codes, IDs)
tool_called — verify specific tools were invoked
max_latency_ms / max_tokens — performance and cost gates
semantically_similar — word-overlap similarity (no API needed)
json_path — extract and verify fields from JSON agent output
llm_judge — use a local LLM to evaluate output quality

Behavior diff reports

This is the killer feature. AgentSpec stores the last passing output per test. When behavior changes, you see exactly what shifted:

⚠️ REGRESSION support agent > handles expired token
  ⚠  Behavior REGRESSED — test was passing, now failing
  + added: your, token, has, expired, please, contact, support
  - removed: you, need, refresh, the, token, settings, security

Instead of just knowing a test failed, you know WHAT changed. This turns debugging from "what broke?" into "the model stopped saying 'refresh token' and started saying 'renew credentials'."

LLM-as-judge with local models

The most powerful assertion is llm_judge — use an LLM to evaluate whether a response meets quality criteria that can't be expressed as string matching:

tests:
  - name: "response is helpful"
    input: "How do I reset my password?"
    expect:
      llm_judge: "The response should explain how to reset a password"

By default, AgentSpec uses a local Ollama model as the judge — no API costs, no data leaving your machine:

agentspec run --judge-endpoint http://127.0.0.1:11434/v1/chat/completions --judge-model qwen2.5:7b

Testing real agents

AgentSpec includes an HTTP agent adapter to test any HTTP-accessible AI agent:

agentspec run --endpoint https://my-agent.example.com/chat

It POSTs {input: "..."} to your endpoint and reads the response. Works with any agent that exposes an HTTP API.

CI/CD integration

AgentSpec is designed to drop into any CI pipeline:

# Exit code 1 on failure
agentspec run --ci

# JUnit XML for CI systems
agentspec run --junit > results.xml

There's also a GitHub Action:

- uses: agentspec/action@v1
  with:
    test-dir: tests

Get started

npm install -g @ozperium/agentspec
agentspec init
agentspec run

That's it. YAML test definitions, assertions for non-deterministic output, diff reports, CI integration. No cloud, no account, no telemetry.

GitHub: Ozperium/agentspec
Landing: agentspec.pages.dev

I'd love feedback on:

Is YAML the right format, or would you prefer TS test definitions?
What assertions are you missing for your AI agents?
Would you use LLM-as-judge with a local model?

DEV Community: Paw from Oz

A small CLI to stop hitting Claude Code's "too many MCP tools loaded" warning

How I caught a Claude prompt regression before it hit production

The problem with testing AI agents

Setting up your first test

The behavior diff — where the real value is

LLM-as-judge for semantic checks

CI integration

Start small

Stop manually logging your AI API calls — one-line auto-logging for OpenAI and Anthropic

The problem with manual logging

Auto-logging with track()

What gets logged

How it works

Per-project breakdown

Testing AI agents is hard. I built a framework for it.

What it looks like

Why assertions built for AI

CI integration

Behavior diff reports

HTTP agents

Install

I built a terminal tool to see your AI coding limits before they stop you

What it does

How it works

Why not just check the app?

What's next

I built a local-first CLI to track what I'm actually spending on AI APIs

Auto-logging (the easy way)

See your costs

Manual logging (CLI)

Why local-first?

Stoke: A kill switch for runaway AI agents

Stoke

What else it does

Why Rust

Multi-machine setup

Quickstart

Status

What's next

Testing AI Agents with AgentSpec: Jest for Non-Deterministic Behavior

What is AgentSpec?

Assertions for AI output

Behavior diff reports

LLM-as-judge with local models

Testing real agents

CI/CD integration

Get started

Auto-logging with `track()`