Agent observability has a labeling problem. When an agent clicks the wrong button or cites the wrong document, every API call returns 200. The trace looks clean. The error is semantic, not syntactic. You need a human to read the entire execution log and decide whether the agent actually did what it was supposed to do.
Laminar built flow-1, a small open-source model trained with reinforcement learning to find these implicit failures in agent traces. It runs as a coding agent inside a harness called Signals, treating each span (LLM call, tool invocation, retrieval step) as a file in a repo. The model starts from a compressed string representation of the full trace and investigates like a developer debugging a failed CI run.
The interesting part is not that they use an LLM to analyze traces. The interesting part is how they trained a cheap model to do it without ground-truth labels.
The Implicit Failure Problem
Traditional observability assumes failures are explicit. An HTTP 500, a timeout, an exception. You set thresholds, write alerting rules, and aggregate error rates.
Agents fail differently:
- The agent completes successfully but books the wrong flight.
- The agent retrieves three documents but misses the most relevant one.
- The agent retries a tool call five times, succeeds on the sixth, but the retry loop burned 30 seconds and three dollars.
- The agent hallucinates a function argument that happens to parse as valid JSON.
These are not errors in the traditional sense. They are trace patterns that a human would flag as wrong after reading the full execution context. The challenge is teaching a model to recognize them without manually labeling thousands of traces.
How RL Training Works Without Ground Truth
Laminar's approach uses synthetic investigations of realistic agent workflows. They generate traces from agents running tasks in software engineering, legal research, customer support, and other domains. Then they use reinforcement learning to train the model on whether its analysis matches the expected outcome.
The training loop works like this:
- Generate a synthetic agent trace with a known success or failure condition.
- Feed the trace to flow-1 and ask it to identify errors.
- Reward the model when it correctly identifies the failure mode or confirms success.
- Penalize the model when it flags false positives or misses real errors.
The key insight is that you can generate ground truth for synthetic traces because you control the task definition. If the agent was supposed to book a flight to SFO and it booked one to OAK, you know that's wrong. You don't need a human to label it.
After training on thousands of these synthetic investigations, the model learns patterns that generalize to real production traces.
Architecture: Signals Harness and Trace-as-Repo
flow-1 runs inside Signals, a harness that structures the analysis as a coding task. Each span in the trace becomes a file. The model navigates the trace like a developer navigating a codebase.
Trace representation:
trace/
span_001_llm_call.json
span_002_tool_call.json
span_003_retrieval.json
span_004_llm_call.json
...
The model receives a compressed string representation of the full trace and a user prompt like "Find errors in this customer support agent trace." It can read individual spans, compare tool inputs to outputs, check whether retrieval results were actually used, and flag suspicious retry patterns.
This is different from feeding the entire trace into a context window and asking for a summary. The model actively investigates. It decides which spans to inspect, in which order, and how to correlate them.
Deployment model:
- Traces are collected via Laminar's open-source SDK (OpenTelemetry-compatible).
- Traces are stored in a backend (Laminar Cloud or self-hosted).
- flow-1 runs as a batch job or on-demand analysis, not inline during agent execution.
- The model outputs a structured report with flagged spans and error descriptions.
This means flow-1 does not add latency to agent execution. It analyzes traces after the fact, which is fine because most implicit failures are not time-critical. You want to know that your agent is booking the wrong flights, but you don't need to know in real time.
Cost and Performance Trade-offs
Laminar claims flow-1 matches GPT-6-sol in detection quality while being 23x cheaper. This makes it affordable to analyze every production trace without sampling.
| Model | Cost per trace | Detection quality | Deployment |
|---|---|---|---|
| GPT-6-sol | ~$0.23 | High (baseline) | API call |
| flow-1 | ~$0.01 | Matches GPT-6-sol | Self-hosted or cloud |
| GPT-4o-mini | ~$0.05 | Lower (misses subtle errors) | API call |
| Rule-based alerting | ~$0.00 | Very low (explicit errors only) | Self-hosted |
The cost difference matters at scale. If you run 10,000 agent executions per day and want to analyze every trace, GPT-6-sol costs $2,300 per day. flow-1 costs $100 per day. Rule-based alerting is free but catches almost nothing.
The quality claim is based on Laminar's own benchmark, which tests detection of implicit failures in synthetic traces. The benchmark is public, but it's not an industry standard. You should validate on your own traces before trusting the numbers.
What flow-1 Actually Detects
The model is trained to flag:
- Wrong tool calls: Agent invokes a function with incorrect arguments or at the wrong time.
- Missed retrievals: Agent skips a retrieval step that would have provided the correct answer.
- Hallucinated citations: Agent references a document that was not in the retrieval results.
- Inefficient retries: Agent retries a tool call multiple times when the first failure was deterministic.
- Context drift: Agent loses track of the original task and starts solving a different problem.
These are all patterns that a human would catch by reading the trace, but that traditional observability misses because there is no explicit error signal.
Versioning and Evaluation Challenges
Training an error-detection model is one thing. Evaluating it in production is harder. The ground truth is "this trace felt wrong," which is subjective and changes over time.
Laminar's approach:
- Maintain a benchmark of synthetic traces with known failure modes.
- Version the model and track detection rate on the benchmark.
- Collect feedback from users on false positives and false negatives.
- Retrain periodically with new synthetic traces and user feedback.
This is not a perfect solution. Synthetic traces may not cover all real-world failure modes. User feedback is sparse and biased toward obvious errors. But it's better than no evaluation at all.
The bigger risk is model drift. If you train on synthetic traces from 2024 and deploy in 2026, the model may miss new failure patterns introduced by new agent architectures or new tools. You need a continuous training loop.
Security and Privacy Boundaries
Running an LLM on production traces raises obvious questions:
- Data leakage: Traces may contain PII, API keys, or proprietary business logic.
- Model access: Who can query flow-1 and see the analysis results?
- Trace retention: How long do you store traces, and where?
Laminar offers both cloud and self-hosted deployment. If you self-host, the traces never leave your infrastructure. If you use Laminar Cloud, traces are encrypted in transit and at rest, but you are trusting Laminar with your data.
The model itself does not learn from your traces. It's a frozen checkpoint. But the analysis results may reveal sensitive information, so you need access controls on the Signals UI and API.
When to Use This
flow-1 makes sense if:
- You run agents in production and care about implicit failures.
- You have enough trace volume that manual inspection is not feasible.
- You are already collecting traces with OpenTelemetry or a similar framework.
- You want to catch errors that rule-based alerting misses.
flow-1 does not make sense if:
- Your agents are simple enough that you can read every trace by hand.
- Your failure modes are explicit (timeouts, exceptions, HTTP errors).
- You are not willing to trust an ML model to flag errors without human review.
- You need real-time error detection during agent execution.
The model is also not a replacement for unit tests, integration tests, or evals. It's a debugging tool, not a correctness proof.
Technical Verdict
Laminar's RL-trained error detector is a useful primitive for agent observability. It solves a real problem (implicit failures are hard to catch) with a practical approach (train a cheap model on synthetic traces). The cost savings over GPT-6-sol are significant, and the deployment model (batch analysis of stored traces) avoids adding latency to agent execution.
The main risks are model drift, evaluation challenges, and the need to trust an ML model to flag errors. If you are already collecting agent traces and struggling to make sense of them, flow-1 is worth testing. If you are not yet at the scale where manual inspection is painful, stick with simpler tools.
The broader lesson is that observability for agents requires learned anomaly detection, not just log aggregation. Rule-based alerting works for explicit failures. For implicit failures, you need a model that understands what the agent was trying to do.
Top comments (0)