Something broke in your app last night. A checkout flow failed, an LLM call timed out, a vector search came back empty — you don't know which, and you're not going to find out by scrolling through a wall of logs across three different services.
That's the problem I kept running into while building AI-heavy applications, and it's why I built Obyflow — an AI-native, CLI-first observability platform for tracing, debugging, and understanding modern applications.
The gap in existing tooling
Traditional observability tools were built for a world of HTTP requests and database queries. They're good at that. But modern applications — the ones with LLM calls, vector-store lookups, and multi-step LangChain pipelines — have failure modes that are harder to explain with those tools alone:
Why did this chain step silently return empty results?
Was that a slow embedding call or a timed-out tool call?
Did the retriever get zero matches, or just low-similarity ones?
Is this incident related to yesterday's deploy, or a model version bump?
Answering these questions usually means manually correlating logs, metrics, and traces across dashboards that weren't designed to work together. Obyflow tries to close that gap.
What Obyflow actually does
Obyflow captures structured events locally — using SQLite, with no external backend required — correlates them into full traces, and uses an LLM of your choice to turn the raw evidence into a plain-English root-cause investigation.
npx obyflow start
npx obyflow investigate --since 1h
npx obyflow ask "why did checkout fail today?"
That last command isn't a gimmick. It runs a real evidence-backed investigation: Obyflow gathers the relevant traces, errors, and anomalies from the time window, builds an evidence graph, and asks your configured LLM to explain what happened.
The investigation also goes through grounding validation, which flags parts of the answer that don't match the underlying evidence. So an unsupported root cause isn't presented as if your traces actually proved it.
A quick tour of the features
A few things came out of actually building and debugging AI-heavy applications, rather than from a feature checklist written in advance.
Structured event model. Ten typed event kinds — trace, log, metric, error, embedding, vector_op, chain, tool_call, llm_call, custom — each carrying typed attributes. An llm_call event captures model, provider, token counts, and latency out of the box. A vector_op captures the database provider, similarity scores, and result counts.
Automatic trace correlation. The Node and Python SDKs auto-propagate x-obyflow-trace-id and x-obyflow-parent-span-id headers across outbound HTTP calls, so a trace stays linked across service boundaries without manual header wiring.
Purpose-built diagnosis engines. Instead of generic anomaly detection, Obyflow has diagnosis logic for LangChain, LangGraph, and LlamaIndex failures — including failed steps, tool-call timeouts, and empty retriever results — as well as vector database issues such as empty results, low similarity, and slow queries.
"What changed" correlation. Incidents can be correlated against deployments, git commits, config changes, feature flags, and model version bumps. Git correlation uses real commit metadata — including the author, changed files, and insertions/deletions — from your local repository.
Half of debugging is answering "what changed right before this broke," so Obyflow tries to answer that automatically.
Token usage and cost tracking. Every llm_call event tracks prompt and completion tokens, which can be rolled up by service into estimated USD cost using built-in pricing tables for Claude, GPT-4o/5, and Gemini.
Confidence scoring. Investigations come back labeled HIGH, MEDIUM, or LOW confidence based on evidence volume, anomaly severity, and how many services and deployments correlate. The goal is to give you a signal about when the investigation has strong support and when you should keep digging yourself.
Under the hood: a real monorepo, not a wrapper
Obyflow isn't one big SDK. It's a set of focused packages:
packages/
core/ event model, storage, evidence graph, anomaly detection
cli/ the `obyflow` CLI
node-sdk/ @obyflow/node — Node.js instrumentation
adapters/
adapter-framework/ LangChain callback handler
adapter-vectordb/ Pinecone, Qdrant, Weaviate, Chroma, pgvector, Milvus
llm/
llm-core/ shared adapter interface
llm-anthropic/ llm-openai/ llm-gemini/ llm-ollama/
python/
obyflow-python/ Python SDK (ASGI/WSGI middleware, LangChain callback)
LLM providers are pluggable — Anthropic, OpenAI, Gemini, Ollama, or none at all if you just want evidence-only mode without a summarization step.
Inbound HTTP tracing is handled differently per language by design. The Node SDK auto-instruments http-based servers such as Express, Koa, Fastify, and raw http.createServer through a runtime patch installed by start(), with zero extra instrumentation code.
Python doesn't have a safe equivalent that works consistently across its different sync and async server models, so it uses explicit middleware instead — ObyflowASGIMiddleware for FastAPI/Starlette, or ObyflowWSGIMiddleware for Flask/Django.
Getting started
npm install -g obyflow
npm install @obyflow/node
pip install obyflow-python
For Node.js:
import { start } from "@obyflow/node";
const handle = start({ service: "checkout-api" });
From there, handle.instrument is pre-bound to your service and store, so instrumenting a vector store can be as simple as:
handle.instrument.pinecone(index);
The full setup, framework-specific middleware wiring, and configuration options are covered in the README.
Where this is going
This is still an early, actively developed project. v1.1.1 just shipped across npm and PyPI.
Redaction, incident memory — including fingerprinting recurring incidents and learning from past resolutions — and OpenTelemetry export are already in. There's still a lot more surface area to cover: more vector databases, more frameworks, and better anomaly baselining as usage grows.
If any of this sounds familiar — especially if you've felt the pain of debugging an LLM pipeline with tools originally built around REST APIs — I'd genuinely appreciate you trying Obyflow and telling me what breaks, what's missing, or what's confusing.
Demo: obyflowapp.onrender.com
Issues, discussions, and PRs are all welcome. Feel free to contribute, report a bug, share feedback, or suggest an idea — every bit of feedback helps.
And if it saves you an afternoon of grepping through logs, a ⭐ on the repo goes a long way for a project this early.
Top comments (2)
Nice project! I like the CLI-first approach.
Thanks!