I got tired of not knowing what my AI agents were doing, so I built a tiny observability tool

I build small LLM agents. Not the impressive kind you see in demos, just
practical little things that answer support questions or dig through some
docs and come back with an answer. And for months the most annoying part of
that wasn't the model or the prompt. It was that I had no idea what the thing
was actually doing.

You know the feeling. A run takes nine seconds and you don't know why. It
called a tool, then called it again, then a third time, and somewhere in
there it spent forty cents. The logs are a wall of JSON. You tweak a prompt,
ship it, and you genuinely can't tell if you made it better or worse until a
user complains. I'd been squinting at print() output like it was 2015.

So I went looking for something to just show me my agent runs. And the tools
exist — they're fine, some are really good — but they all wanted something I
didn't want to give.

The two things that kept bugging me

The hosted ones want your traces on their servers. Which means your prompts,
which for me means actual customer messages, sitting in someone else's
database so I can look at a dashboard. No thanks.

The self-hostable ones are built for companies, and it shows. The one
everybody recommends needs Postgres and ClickHouse and Redis and an S3
bucket to run. Four stateful services. I'm logging maybe a few thousand model
calls a day on a side project. Standing up a ClickHouse cluster to look at
that is like renting a forklift to carry a bag of groceries.

I sat with that for a bit and realized the mismatch was the whole problem.
This isn't company-scale data. It's my-laptop-scale data. And the moment you
accept that, the design gets really boring in a good way: it's just a web app
with a SQLite file. That's it. One process.

So I built that. It's called Otterscope.

What it actually is

It's one Go binary. You run it, it opens a SQLite file, and it listens for
OpenTelemetry traces on the standard port. You point your agent at it with
one environment variable — the same one every OpenTelemetry setup already
uses — and your runs start showing up. No account, no config file, no other
services to babysit.

The thing I cared most about getting right was the shape of the data.
Everyone else shows you spans, which is the raw OpenTelemetry unit, and it
reads like a stack trace. I wanted runs. One agent execution is one run, and
inside it you see the steps: this LLM call, that tool call, the loop where it
called the same tool three times. Click into a call and you read the real
messages that went in and came back out, with the token counts and the cost
sitting right there. When something goes sideways, you can actually see where.

There's a cost table that knows the current prices for the major providers,
so a run tells you what it cost instead of just how many tokens it burned. If
it hits a model I don't have a price for, it shows you the tokens and stays
honest about not knowing the dollar amount, which matters more than it
sounds — I'd rather see a blank than a made-up number.

And then there's the part I'm most attached to: the evals live in the same
place as the traces. You write a check ("the answer should never contain 'I
can't help with that'") or point an LLM at the run as a judge, and the result
gets stamped onto the actual production run. Most tools make evals a separate
product you run against a separate dataset. Here it's just a column on the
runs you already have. There's a compare view too, so "did my prompt change
make last week worse than the week before" is a question you can actually
answer by looking, instead of by vibes.

The genuinely annoying part

If you build one of these, here's the thing nobody warns you about: there is
no single format for AI traces. There are at least three, all live in the
wild right now, and they disagree.

OpenTelemetry has a "GenAI" convention, except it's mid-migration, so there's
an old attribute layout and a new one and both are being emitted by real
tools today. Then Arize's OpenInference is its own thing, and it's what the
OpenAI Agents SDK and a bunch of the LangChain world speak. Then the Vercel
AI SDK does its own ai.* attributes, and — this is the fun one — its spans
mix the old and new OpenTelemetry styles together in the same span, so if you
guess based on one attribute you get it wrong.

So a big chunk of Otterscope is just a translation layer that eats all of
these and turns them into one clean model. It's the least glamorous code in
the project and probably the most valuable, because it means you don't have
to care which dialect your framework speaks. And because I keep the raw
payload of everything that comes in, when I improve that translation later,
you can replay your old data through the new version and it just gets better
retroactively. That felt worth doing.

What it's not

It's single-user right now. There's no login, no teams, no roles. That's on
purpose — I built the thing I needed, and I'm one person. It binds to
localhost by default so you're not accidentally exposing it, and if you want
a team version with real auth, that's a "when someone actually asks" feature,
not a "build it on spec because a spreadsheet said B2B" feature.

It's also young. It'll have rough edges. But it does the thing I wanted, which
was to stop flying blind.

Trying it

The fastest way is Docker:

docker run -p 8317:8317 -p 4318:4318 -v otterscope:/data ghcr.io/otterscope/otterscope

Then point any OpenTelemetry-instrumented agent at http://localhost:4318
and open http://localhost:8317. There's a sample command that fills it
with fake runs if you just want to click around first. The repo has setup
snippets for Pydantic AI, the OpenAI Agents SDK, LangGraph, and the Vercel AI
SDK.

Repo's here: https://github.com/otterscope

If you'd rather run it on a server than on your laptop, I wrote up the whole
thing on a small VPS separately — creating the box, Docker vs a systemd
service, and how to keep it private with an SSH tunnel or lock it down with a
firewall: Self-hosting observability for your AI agents on a $6
Droplet.

Why DigitalOcean, since a few people asked

Because the tool's whole personality is "small and predictable," and that's
what I want from the box it runs on too.

A $6 Droplet runs this comfortably. There's no managed database bill on top,
because the database is a file — I back it up by copying the file or
snapshotting the Droplet, and I'm done. The pricing is flat and I can do the
math in my head, which matters when the whole point of the project is not
having a surprise infrastructure bill for a side project. I don't want to
find out at the end of the month that some egress line item ate the savings.

I've also just had a good time on it. The docs are readable, the Droplet
comes up in under a minute, and the Marketplace Docker image means "spin up a
box that can run a container" is a checkbox rather than an afternoon. For
something like this — one binary, one file, one small server sitting next to
your agents — it fits the shape of the thing. That's really the whole reason.

Anyway. If you've been squinting at agent logs, go self-host something and
stop. It doesn't have to be mine, but it turns out it really doesn't need to
be complicated.