DEV Community

Cover image for I built Obyflow. Try it, break it, and tell me what goes wrong
Anupam Kumar
Anupam Kumar

Posted on

I built Obyflow. Try it, break it, and tell me what goes wrong

#webdev #ai #javascript #opensource

A few months ago, I got tired of using console.log to debug LLM apps.

Sometimes a chain would fail without much information. Sometimes a vector search would return nothing. And when something stopped working, it was hard to figure out what had changed.

So I built Obyflow, an open-source tool for tracing and debugging LLM and other applications.

It can track things like LLM calls, vector searches, errors, logs, and LangChain steps. It runs locally using SQLite, so you don't need to set up another backend.

When something goes wrong, you can also use an LLM to look at the collected information and give you a possible reason for the failure. There are checks in place so the diagnosis is based on the available data instead of just making up an answer.

Live demo: https://obyflowapp.onrender.com

But this post isn't really about explaining every feature.

I want people to try it and tell me where it breaks.

What Obyflow does

Some of the main parts are:

  • A typed event model for traces, logs, metrics, errors, embeddings, vector operations, chains, tool calls, LLM calls, and custom events.
  • Trace support across services. The Node.js and Python SDKs can pass trace information through HTTP requests.
  • Basic anomaly detection using mean/stddev and median/MAD.
  • Diagnosis for LangChain, LangGraph, and LlamaIndex failures.
  • Checks for vector database problems such as empty results, low similarity, and slow queries.
  • A way to check what changed before an incident, such as a deployment, git commit, configuration change, feature flag, or model version.
  • Incident fingerprints so similar problems can be compared.
  • Token and cost tracking.
  • Redaction for sensitive fields.
  • Support for OpenAI, Anthropic, Gemini, Ollama, or no LLM at all.
  • Node.js and Python SDKs, with LangChain and several vector database integrations.

The TypeScript side is a pnpm/Turborepo monorepo, and the Python SDK is a separate package.

You can start it with:

npm install -g obyflow
npx obyflow init
npx obyflow start
npx obyflow investigate --since 1h
Enter fullscreen mode Exit fullscreen mode

Why I'm asking you to break it, not just star it

Stars are nice, but they don't really tell me anything. What actually helps is friction — a config option that doesn't behave the way the README implies, an edge case in trace correlation that only shows up once you have three services involved, a vector-DB adapter nobody's tried against a real cluster yet.

So if you have a few minutes, here's what would actually help:

  1. Open an issue. Even a small one. "The --since flag doesn't parse 30m the way I expected" is exactly the kind of report that makes this better. You don't need to know the codebase to file a good bug report.
  2. Send a PR. Anything from a typo fix to a new vector-DB adapter. CONTRIBUTING.md walks through the workflow and conventions.
  3. Just run it and tell me what happened. Point it at a small LangChain app or whatever you're already building, and let me know if the root-cause summary actually made sense. If it didn't, that's still useful — "I ran investigate and the diagnosis was wrong" tells me exactly where to look.

Why this might be a decent place to start (or continue) your OSS journey

If you've been meaning to contribute to open source but keep bouncing off big, intimidating repos, here's why I think a project at this stage can be a good place to start:

  • It's small enough to actually understand. You can read through the event model, the CLI, and one adapter in an afternoon. You don't need years of context to make sense of it.
  • There's a range of things to work on. TypeScript core, a CLI, Node and Python SDKs, LangChain adapters, vector-DB integrations, some anomaly-detection math, and the LLM-facing diagnosis logic. Whatever you're trying to get better at, there's probably a piece here that matches it.
  • It's early enough that your contribution actually matters. This isn't a repo where a PR disappears into a queue of thousands. Right now, a good issue or PR has a real, visible effect on where the project goes.
  • A bug report counts as a contribution. If PRs feel like a big first step, start with an issue instead. Writing a clear, reproducible bug report is its own skill, and it's how a lot of people get their start before they ever open a PR.
  • Cross-SDK parity is a real, ongoing problem here. The Node and Python SDKs need to stay behaviorally consistent even though they're versioned independently — if you're interested in SDK design, that's a genuine problem living in this repo right now, not a toy one.

Where to go

If you try it and it just works, that's good to know too. But if it breaks, that's the more useful message. Open an issue, send a PR, or just tell me what you ran and what happened. I'll be in the comments.

Top comments (0)