I build small LLM agents. Not the impressive kind you see in demos, just
practical little things that answer support questions or dig through some
docs and come back with an answer. And for months the most annoying part of
that wasn't the model or the prompt. It was that I had no idea what the thing
was actually doing.
You know the feeling. A run takes nine seconds and you don't know why. It
called a tool, then called it again, then a third time, and somewhere in
there it spent forty cents. The logs are a wall of JSON. You tweak a prompt,
ship it, and you genuinely can't tell if you made it better or worse until a
user complains. I'd been squinting at print() output like it was 2015.
So I went looking for something to just show me my agent runs. And the tools
exist — they're fine, some are really good — but they all wanted something I
didn't want to give.
The two things that kept bugging me
The hosted ones want your traces on their servers. Which means your prompts,
which for me means actual customer messages, sitting in someone else's
database so I can look at a dashboard. No thanks.
The self-hostable ones are built for companies, and it shows. The one
everybody recommends needs Postgres and ClickHouse and Redis and an S3
bucket to run. Four stateful services. I'm logging maybe a few thousand model
calls a day on a side project. Standing up a ClickHouse cluster to look at
that is like renting a forklift to carry a bag of groceries.
I sat with that for a bit and realized the mismatch was the whole problem.
This isn't company-scale data. It's my-laptop-scale data. And the moment you
accept that, the design gets really boring in a good way: it's just a web app
with a SQLite file. That's it. One process.
So I built that. It's called Otterscope.
What it actually is
It's one Go binary. You run it, it opens a SQLite file, and it listens for
OpenTelemetry traces on the standard port. You point your agent at it with
one environment variable — the same one every OpenTelemetry setup already
uses — and your runs start showing up. No account, no config file, no other
services to babysit.
The thing I cared most about getting right was the shape of the data.
Everyone else shows you spans, which is the raw OpenTelemetry unit, and it
reads like a stack trace. I wanted runs. One agent execution is one run, and
inside it you see the steps: this LLM call, that tool call, the loop where it
called the same tool three times. Click into a call and you read the real
messages that went in and came back out, with the token counts and the cost
sitting right there. When something goes sideways, you can actually see where.
There's a cost table that knows the current prices for the major providers,
so a run tells you what it cost instead of just how many tokens it burned. If
it hits a model I don't have a price for, it shows you the tokens and stays
honest about not knowing the dollar amount, which matters more than it
sounds — I'd rather see a blank than a made-up number.
And then there's the part I'm most attached to: the evals live in the same
place as the traces. You write a check ("the answer should never contain 'I
can't help with that'") or point an LLM at the run as a judge, and the result
gets stamped onto the actual production run. Most tools make evals a separate
product you run against a separate dataset. Here it's just a column on the
runs you already have. There's a compare view too, so "did my prompt change
make last week worse than the week before" is a question you can actually
answer by looking, instead of by vibes.
The genuinely annoying part
If you build one of these, here's the thing nobody warns you about: there is
no single format for AI traces. There are at least three, all live in the
wild right now, and they disagree.
OpenTelemetry has a "GenAI" convention, except it's mid-migration, so there's
an old attribute layout and a new one and both are being emitted by real
tools today. Then Arize's OpenInference is its own thing, and it's what the
OpenAI Agents SDK and a bunch of the LangChain world speak. Then the Vercel
AI SDK does its own ai.* attributes, and — this is the fun one — its spans
mix the old and new OpenTelemetry styles together in the same span, so if you
guess based on one attribute you get it wrong.
So a big chunk of Otterscope is just a translation layer that eats all of
these and turns them into one clean model. It's the least glamorous code in
the project and probably the most valuable, because it means you don't have
to care which dialect your framework speaks. And because I keep the raw
payload of everything that comes in, when I improve that translation later,
you can replay your old data through the new version and it just gets better
retroactively. That felt worth doing.
What it's not
It's single-user right now. There's no login, no teams, no roles. That's on
purpose — I built the thing I needed, and I'm one person. It binds to
localhost by default so you're not accidentally exposing it, and if you want
a team version with real auth, that's a "when someone actually asks" feature,
not a "build it on spec because a spreadsheet said B2B" feature.
It's also young. It'll have rough edges. But it does the thing I wanted, which
was to stop flying blind.
Trying it
The fastest way is Docker:
docker run -p 8317:8317 -p 4318:4318 -v otterscope:/data ghcr.io/otterscope/otterscope
Then point any OpenTelemetry-instrumented agent at http://localhost:4318
and open http://localhost:8317. There's a sample command that fills it
with fake runs if you just want to click around first. The repo has setup
snippets for Pydantic AI, the OpenAI Agents SDK, LangGraph, and the Vercel AI
SDK.
Repo's here: https://github.com/otterscope
If you'd rather run it on a server than on your laptop, I wrote up the whole
thing on a small VPS separately — creating the box, Docker vs a systemd
service, and how to keep it private with an SSH tunnel or lock it down with a
firewall: Self-hosting observability for your AI agents on a $6
Droplet.
Why DigitalOcean, since a few people asked
Because the tool's whole personality is "small and predictable," and that's
what I want from the box it runs on too.
A $6 Droplet runs this comfortably. There's no managed database bill on top,
because the database is a file — I back it up by copying the file or
snapshotting the Droplet, and I'm done. The pricing is flat and I can do the
math in my head, which matters when the whole point of the project is not
having a surprise infrastructure bill for a side project. I don't want to
find out at the end of the month that some egress line item ate the savings.
I've also just had a good time on it. The docs are readable, the Droplet
comes up in under a minute, and the Marketplace Docker image means "spin up a
box that can run a container" is a checkbox rather than an afternoon. For
something like this — one binary, one file, one small server sitting next to
your agents — it fits the shape of the thing. That's really the whole reason.
Anyway. If you've been squinting at agent logs, go self-host something and
stop. It doesn't have to be mine, but it turns out it really doesn't need to
be complicated.
Top comments (7)
Keeping the raw payload so old data can be replayed through a better translator later is the decision I'd steal. It turns your normalization from something you have to get right on day one into something you can be wrong about and fix, which is the only realistic plan given three dialects that keep moving. The blank-instead-of-a-guessed-price call is the same instinct, and both are the kind of thing that only shows up when one person is the whole roadmap.
The translation-layer section is the part I'd underline, so I pulled the bundles before saying anything —
npm packon ai@7.0.30, @ai-sdk/otel@1.0.30 and ai@6.0.229, then read the shipped dist. Three things that might save your parser some pain.Your "mixes old and new in the same span" is real, and I can hand you the exact shape. In ai@6.0.229 the core still emits spans itself (
getTracerappears 9×), and 4 of its 34 attribute objects put vendorai.*right next togen_ai.system— which is the pre-rename name ofgen_ai.provider.name(0 occurrences in v6). One span, three vintages:{ai.prompt.messages, gen_ai.request.model, gen_ai.system}. You weren't imagining it.But the ground moved under that parser in v7. ai@7.0.30 has zero opentelemetry deps, and zero occurrences of
opentelemetry,getTracer,startSpanorgen_ai.in its bundle. Telemetry became a callback interface (registerTelemetry(),onStart/onStepStart) and span emission moved out to@ai-sdk/otel. Consequence for Otterscope: an agent that upgrades to v7 and doesn't add that package stops sending traces — no error, no empty span, the runs just never arrive, and it's indistinguishable from "nothing ran." (That's read from the bundle rather than measured on a live v7 agent — but there's no tracer left in core to measure.)The dialect is now a user choice, not a version fact.
@ai-sdk/otel@1.0.30exports two builders:OpenTelemetry(30gen_ai.*, no pre-rename keys) andLegacyOpenTelemetry(59ai.*+ 14gen_ai.*, still shippinggen_ai.system, 2 of its 11 attribute objects still mixed). Same SDK version, two shapes on the wire, depending on which class the user registered.Why single-attribute sniffing loses: 13
gen_ai.*keys show up in both builders —gen_ai.request.modelamong them. Only two are unambiguous:gen_ai.provider.name⇒ new builder,gen_ai.system⇒ Legacy. Key on those two and the guess goes away.Keeping the raw payload for replay is the decision that ages well here — it's what lets you fix the mapping after the dialect shifts under you again.
the my-laptop-scale framing is the most honest thing said in the observability space in a while - the forklift-for-groceries problem is real. the feature that would make this a daily driver follows from your own sentence: "you tweak a prompt, ship it, and you cannot tell if you made it better or worse". a side-by-side of two run groups - same inputs, before and after a prompt version - beats any dashboard at that job: token deltas, tool-call counts, which runs diverged and at what step. does otterscope group runs by prompt or config version yet, or is that roadmap territory?
The privacy point is the one I feel most, since agent traces are basically raw customer messages and shipping them to a hosted dashboard is a non-starter for support work. The four-stateful-service tax for a few thousand calls a day is real too. When you kept it tiny, did you still capture per-step token and tool-call spans, or trim down to just the run timeline?
I hit the same wall building small agent setups of my own. The agent's summary of what it did and what its diff or logs actually show can drift apart fast, and the summary is always the more flattering version. What convinced me observability was worth the setup cost was catching a case where the agent reported success but the actual tool call had silently failed and it just moved on. Does your tool surface that kind of mismatch directly, or is it still on the human to cross reference the log against the claim?
Great write-up. The translation layer handling all three OTel dialects is the part that would quietly save people hours of debugging and most wouldn't even notice it's there.
If your AI agents kept a diary, what would surprise you most (snacks included)?