Langfuse is an open-source observability and evaluation platform for LLM applications. It handles trace ingestion, prompt versioning, dataset snapshots, and eval workflows in a single stack. For financial agents and regulated environments, this means audit-grade trace storage, reproducible eval datasets, and self-hosted deployment that keeps sensitive data inside your perimeter.
The architecture splits into three layers: ingestion (traces, spans, generations), management (prompts, datasets, scores), and evaluation (playground, LLM-as-judge). The boundary between logging and evaluation is explicit. Traces flow in via SDK or API, prompts are versioned independently, and datasets are frozen snapshots of production traces or synthetic examples.
Trace Schema and Ingestion
Langfuse uses a three-tier hierarchy:
- Trace: Top-level container for a user request or agent session.
- Span: A step within a trace (tool call, retrieval, reasoning loop).
- Generation: An LLM completion tied to a specific prompt version.
Each entity carries metadata (user ID, session ID, tags), timestamps, and optional scores. Generations link to prompt versions, so you can correlate eval results with the exact prompt text and parameters used in production.
Ingestion happens via:
- SDK decorators (Python, TypeScript, LangChain, LlamaIndex)
- REST API for custom integrations
- OpenTelemetry bridge for existing instrumentation
The SDK batches events and sends them asynchronously to avoid blocking agent execution. Traces land in PostgreSQL with JSONB columns for flexible metadata queries.
Prompt Versioning and Linking
Prompts are first-class objects with version history. Each version has:
- Name and label (e.g.,
fraud-detection-v3) - Template text with variable placeholders
- Model config (temperature, max tokens, stop sequences)
- Commit timestamp and author
When you log a generation, you pass the prompt name and version. Langfuse resolves the template, stores the rendered text, and links the generation to that prompt version. This makes it possible to:
- Roll back to a previous prompt without redeploying code
- A/B test prompts by routing traffic to different versions
- Audit which prompt version produced a specific output
The prompt registry lives in the database, not in code. You fetch the latest version at runtime or pin to a specific label. This decouples prompt iteration from application deployment.
Dataset Management for Reproducible Evals
Datasets are collections of input-output pairs or input-only examples. You create them by:
- Sampling production traces (filter by tag, score, or time range)
- Uploading CSV or JSON with expected outputs
- Cloning an existing dataset and modifying examples
Each dataset is versioned. When you run an eval, you snapshot the dataset state so results are reproducible even if you later add or remove examples.
Evals run in the Langfuse UI or via SDK. You define:
- Dataset to evaluate
- Prompt version or model config
- Scoring function (exact match, LLM-as-judge, custom code)
Results are stored as scores linked to the generation. You can compare prompt versions, model providers, or parameter sweeps in a single view.
Deployment Model for Regulated Environments
Langfuse offers cloud-hosted and self-hosted options. For financial or healthcare agents, self-hosting is the default:
- Docker Compose for single-node deployments
- Kubernetes Helm chart for multi-replica setups
- PostgreSQL as the only external dependency
The self-hosted stack includes:
- Web server (Next.js frontend, tRPC API)
- Worker for async jobs (dataset exports, batch scoring)
- PostgreSQL with row-level security for multi-tenant isolation
You control the network boundary. Traces never leave your VPC. You can encrypt the database at rest, rotate API keys per environment, and audit all access via PostgreSQL logs.
For high-throughput agents, the ingestion path is the bottleneck. Langfuse batches writes and uses upsert logic to handle duplicate trace IDs. If you exceed 10,000 generations per minute, consider:
- Sharding by tenant or project
- Running multiple worker replicas
- Using a read replica for dashboard queries
Observability and Failure Modes
Langfuse exposes metrics via the web UI and optional Prometheus endpoints:
- Trace count and latency percentiles
- Generation token usage and cost
- Error rate by model or prompt version
Common failure modes:
| Failure | Symptom | Mitigation |
|---|---|---|
| SDK batch timeout | Traces dropped silently | Increase flush interval or reduce batch size |
| Database connection pool exhaustion | 500 errors on ingestion | Scale worker replicas or tune pool size |
| Prompt version mismatch | Eval results link to wrong template | Pin prompt versions in code, not labels |
| Dataset drift | Eval scores degrade over time | Snapshot datasets before each eval run |
The SDK retries failed requests with exponential backoff. If the server is unreachable, traces are logged locally and uploaded when connectivity resumes. This works for short outages but can cause memory pressure in long-running agents.
Integration with Agent Frameworks
Langfuse integrates natively with LangChain, LlamaIndex, and Autogen. For LangChain:
from langfuse.callback import CallbackHandler
from langchain.chains import LLMChain
handler = CallbackHandler(
public_key="pk-...",
secret_key="sk-...",
trace_name="fraud-detection",
user_id="user-123"
)
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
result = chain.run(input="Check this transaction")
The callback handler automatically creates a trace, logs the prompt and completion as a generation, and links metadata. You can add custom scores or tags in the same callback.
For custom agents, use the SDK directly:
from langfuse import Langfuse
langfuse = Langfuse()
trace = langfuse.trace(name="agent-session", user_id="user-123")
span = trace.span(name="tool-call", input={"tool": "search"})
generation = trace.generation(
name="reasoning",
prompt="fraud-detection-v3",
model="gpt-4",
input=prompt_text,
output=completion_text
)
span.end(output={"results": search_results})
trace.score(name="accuracy", value=0.95)
This gives you full control over the trace structure and lets you instrument multi-step workflows without framework lock-in.
Security Boundaries and Access Control
Langfuse uses project-level API keys and role-based access. Each project is isolated:
- Traces, prompts, and datasets are scoped to a project
- API keys grant read or write access per project
- Users can belong to multiple projects with different roles
For financial agents, you typically run one project per environment (dev, staging, prod) and rotate keys quarterly. The database schema enforces project isolation via foreign keys, so a compromised key cannot leak traces from other projects.
Sensitive data in traces (PII, account numbers) should be redacted before ingestion. Langfuse does not auto-redact. You handle this in the SDK callback or via a proxy layer that strips fields before they reach the database.
Technical Verdict
Use Langfuse when:
- You need audit-grade trace storage with prompt version linking for compliance
- You want reproducible evals via dataset snapshots and frozen prompt versions
- You prefer self-hosted deployment to keep sensitive agent data in your VPC
- You integrate with LangChain, LlamaIndex, or Autogen and want zero-config observability
Avoid Langfuse when:
- Your agent throughput exceeds 50,000 generations per minute (consider streaming to a data lake instead)
- You need real-time anomaly detection or alerting (Langfuse is optimized for post-hoc analysis)
- You want automatic PII redaction or data masking (you must implement this upstream)
- Your eval workflow requires complex DAGs or conditional branching (Langfuse evals are linear prompt-dataset-score pipelines)
Langfuse is mature infrastructure for teams that treat evals as a first-class concern. The prompt versioning and dataset snapshot model makes it possible to reproduce eval results months later, which matters when regulators ask why your fraud detection agent approved a specific transaction. The self-hosted option and PostgreSQL backend give you full control over data residency and retention policies.
Top comments (1)
Everyone optimizes for speed. Almost nobody optimizes for being able to undo it.