Ali Suleyman TOPUZ

Posted on Aug 16 • Originally published at topuzas.Medium on Aug 16

Four Tools I Bolted Onto My Claude Agent (and Why Each One Earned Its Place)

#llmobservability #mcpserver #developertools #agents

A few months ago I built an internal agent on top of Claude Code and the Claude Agent SDK for my team. Nothing exotic: it had MCP access to GitHub, Jira and Slack, and its job was to triage incoming bug reports, draft PRs for the boring fixes, and post a daily rundown so nobody had to dig through fifteen tickets before standup.

It worked. Then it worked too well, and I started letting it do more: close stale tickets on its own, comment on PRs, run semi-unattended overnight. That’s when four gaps showed up that the base Claude Code setup just doesn’t cover: I had no fast way to steer it without typing, no way to stop a bad decision before it executed, no real visibility into what it was doing across dozens of runs, and no memory that survived between a Claude Code session and a Claude Desktop session. I ended up wiring in four separate tools to close those gaps: SKI, Prefactor, LangWatch, and Memmy Agent. This is what each one actually did for me, warts included.

The starting point

Nothing fancy: Claude Code for the day-to-day, a couple of MCP servers for GitHub and Jira, and a Slack bot wired to the Claude Agent SDK for the “runs on its own” part. That setup is fine until the agent starts taking actions you didn’t watch happen in real time. That’s the moment you start wanting the four things below.

Talking instead of typing: SKI

SKI is a voice layer for Claude Code (and Codex). You hold a hotkey, talk, and the agent talks back. It’s built to feel like a teammate on a call rather than dictation software. It runs locally on your machine and is free, which made it an easy first thing to try.

I use it almost entirely for the “thinking out loud” part of a session: describing a bug, arguing through an approach, asking the agent to explain a diff back to me, while I’m doing something else with my hands. What it did not replace was precise editing. The moment I need to reference an exact line, a specific variable name, or paste a stack trace, I’m back on the keyboard. So it’s additive, not a replacement, more like a second input mode I switch into depending on what I’m doing. No config to speak of: install, hit the key, talk. That simplicity is the whole pitch and it delivers on it.

Somebody has to sign off: Prefactor

This is the one where my assumptions were wrong going in. I’d heard “agent identity” and expected an auth/access-control product. What Prefactor actually turned out to be, once I read past the landing page, is a production evaluation and enforcement layer: it scores every agent run for quality, drift and risk as it happens, and (this is the part that mattered to me) it can pause a risky action and hold it for a human to approve before it executes, not just log that it happened.

That’s exactly the problem I had. My agent closing a ticket automatically is low stakes. My agent posting a comment on a customer-facing PR, or writing to a shared doc, is not something I wanted happening without a human able to veto it in the window between decision and execution.

Setup was genuinely quick:

npm i @prefactor/sdk
prefactor init

// agent.ts
import { prefactor } from "@prefactor/sdk";
const pf = prefactor({ agent: "ticket-triage-agent" });
// Pull real context into the run so evals are grounded,
// not just judging the model's own output in isolation
const span = pf.customSpan("triage_ticket");
span.attach(await jira.getIssue(ticketId));
span.attach(await github.getRelatedPRs(ticketId));
// High-risk action: this gets held for a human, not auto-executed
await pf.gate("close_ticket", async () => {
  await jira.closeIssue(ticketId);
});

The first time it actually held a run (the agent wanted to close a ticket that a customer had just replied to, which my triage logic hadn’t caught) felt like the tool paying for itself in one afternoon.

The honest caveats: it’s cloud-only, no self-hosted option, which was a real hesitation for us since some of what flows through custom spans is customer data. When I checked, SOC 2 Type II was still “in progress” and RBAC was still on the roadmap rather than shipped, fine for an internal tool with a small team, but I’d think harder about it for anything regulated. It also overlaps conceptually with LangWatch (both trace and evaluate runs), so I had to be deliberate about which one owns what. More on that below.

Tracing everything: LangWatch

LangWatch is where I actually go to understand agent behavior across many runs: traces, evals, and agent testing, built on OpenTelemetry so it’s not a walled garden. Because some of what my agent touches is internal ticket and customer content, I didn’t want traces leaving our network by default, so I ran it self-hosted instead of the cloud version:

git clone https://github.com/langwatch/langwatch.git
cd langwatch
docker compose up -d

# requirements: pip install langwatch --break-system-packages
import langwatch
import anthropic

langwatch.setup(api_key="local-dev-key", endpoint="http://localhost:5560")
client = anthropic.Anthropic()
@langwatch.trace()
def triage(ticket_text: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": ticket_text}],
    )
    return response.content[0].text

Every model call, tool call, and decision shows up as a span, with cost and latency attached, which is what let me catch that our average triage run was burning way more tokens than expected. Turned out one tool was returning entire Jira comment histories instead of the last few comments.

Where LangWatch and Prefactor ended up dividing labor for me: LangWatch is where I go before shipping a change, for testing, evals, cost/latency debugging, poking at a specific trace. Prefactor is what’s watching production and holding the runs I actually care about stopping. Running both felt redundant for the first week; it stopped feeling redundant the first time one caught something the other didn’t.

Making the agent remember: Memmy Agent

The last gap was the dumbest one to still have in 2026: my agent forgot everything the moment I switched from a Claude Code session to Claude Desktop. Same project, same me, zero shared context. Memmy Agent is a local-first memory hub that sits underneath Claude Code, Cursor, Codex and a few others, and exposes that memory through a CLI, a desktop app, and an OpenAI-compatible API.

I run it as the CLI, mostly:

memmy onboard # sets up ~/.memmy/config.yaml and the workspace
memmy status # sanity check on model/provider config
memmy-memory init # wires memory into whatever agent you're running
memmy-memory search "how did we handle the last Jira rate-limit issue"

Minimal BYOK config, so it’s not tied to any one model provider:

# ~/.memmy/config.yaml
agents:
  defaults:
    model: openai/gpt-4.1
    provider: openai
    timezone: Europe/Istanbul
providers:
  openai:
    apiKey: ${OPENAI_API_KEY}

Everything is stored locally by default (SQLite under ~/.memmy/, no cloud round-trip required), which mattered to me for the same reason self-hosting LangWatch did. The part that actually saved time was the history import: it scanned my existing Claude Code sessions and built a working memory baseline instead of starting from zero.

Caveat, and it’s a real one: it’s young. 70 stars, 22 forks, a couple of open issues, features like team collaboration still on the roadmap rather than shipped. It did exactly what I needed, cross-session memory without a cloud dependency, but I’d check back on it in six months before betting anything critical on it.

Putting it together

Roughly, the stack looks like this:

you (voice, via SKI)
      |
      v
Claude Code / Claude Agent SDK <-- Memmy (shared memory, local)
      |
      v
MCP tools: GitHub, Jira, Slack
      |
      v
LangWatch (trace + eval, self-hosted) --- dev/test loop
      |
      v
production run
      |
      v
Prefactor (score + gate risky actions) --- prod enforcement
      |
      v
action executes (or waits for you to approve it)

And the four tools side by side, plain and simple:

TOOL LAYER PROBLEM IT SOLVES DEPLOY MODEL
---------- --------------- ----------------------------------- ------------------
SKI Input/interface Hands-free interaction with the Local, free
                                  agent while pairing
Prefactor Prod enforcement Catches + holds risky agent Cloud only
                                  actions before they execute
LangWatch Observability Traces, evals, cost/latency Cloud or
                                  debugging pre-production self-hosted
Memmy Agent Memory Shared context across Claude Local-first
                                  Code, Desktop, Cursor, etc.

Would I keep all four?

Yes, but not with equal confidence. SKI and the self-hosted LangWatch instance are staying: low risk, immediate payoff, no vendor lock-in I’m worried about. Prefactor earned its place the day it held a bad ticket closure, but I’m watching its compliance roadmap before I’d trust it with anything more sensitive than what I’m running today. Memmy is the one I’d call promising rather than settled: it solved a real annoyance, but it’s early enough that I’m treating it as an experiment, not infrastructure.

None of these came from Anthropic, and none of them are things Claude does out of the box. That’s sort of the point. The base agent is genuinely capable, but the moment it starts acting with real permissions in a real workflow, the gaps between “impressive demo” and “thing I trust unattended” are exactly what this stack ended up filling in.

DEV Community