<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: GermanGerken</title>
    <description>The latest articles on DEV Community by GermanGerken (@germangerken).</description>
    <link>https://dev.to/germangerken</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4257165%2Fe2383336-6ebe-4010-9aca-2ba9881af033.jpg</url>
      <title>DEV Community: GermanGerken</title>
      <link>https://dev.to/germangerken</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/germangerken"/>
    <language>en</language>
    <item>
      <title>Your agent's success rate barely moved. One tool's error rate tripled.</title>
      <dc:creator>GermanGerken</dc:creator>
      <pubDate>Sat, 21 Nov 2026 15:57:22 +0000</pubDate>
      <link>https://dev.to/germangerken/your-agents-success-rate-barely-moved-one-tools-error-rate-tripled-1k9l</link>
      <guid>https://dev.to/germangerken/your-agents-success-rate-barely-moved-one-tools-error-rate-tripled-1k9l</guid>
      <description>&lt;p&gt;&lt;em&gt;A reproducible experiment on why aggregate metrics can miss agent regressions, and what analyzing traces across the whole dataset reveals instead.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://germangerken.github.io/2026/11/18/agent-regression-case-study/" rel="noopener noreferrer"&gt;my blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;When releasing a new version of an agent, teams typically review a dashboard showing metrics such as success rate, mean cost, and p95 latency. If these numbers are similar to the previous week's, the release is considered acceptable.&lt;/p&gt;

&lt;p&gt;I tested how well this approach detects a regression I intentionally introduced. The results were unsatisfactory, and the reason is worth examining.&lt;/p&gt;

&lt;h2&gt;
  
  
  Experiment setup
&lt;/h2&gt;

&lt;p&gt;I developed a small, seeded simulator for a customer-support agent. Each run produces a trace that includes a router, LLM planner calls, retrieval, tool calls, and a final guard. The simulator supports three intents: order status (45% of traffic), refunds (30%), and product questions (25%).&lt;/p&gt;

&lt;p&gt;I then created a second version with one intentional fault in the refund flow, which shows up in two ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the planner repeatedly checks the refund policy, adding one or two extra rounds;&lt;/li&gt;
&lt;li&gt;the refund API times out more often, and retries often fail.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both versions generate 600 traces. This synthetic data has a known outcome, which is intentional: I know exactly what failed, so I can evaluate what each method detects.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dashboard results
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Baseline&lt;/th&gt;
&lt;th&gt;Candidate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Success rate&lt;/td&gt;
&lt;td&gt;94.8%&lt;/td&gt;
&lt;td&gt;92.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mean cost per trace&lt;/td&gt;
&lt;td&gt;$0.0168&lt;/td&gt;
&lt;td&gt;$0.0181 (+7.5%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Error events&lt;/td&gt;
&lt;td&gt;0.6%&lt;/td&gt;
&lt;td&gt;1.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Success rate dropped by two points. With 600 traces per version, the 95% confidence interval for this difference ranges from −4.8 to +0.7 points. Since this interval includes zero, a careful reviewer would consider the change noise. By the numbers, they would be right to.&lt;/p&gt;

&lt;p&gt;Cost increased by 7.5%, which most teams would not consider a release blocker.&lt;/p&gt;

&lt;p&gt;The regression is real and large, but it is not visible in these metrics: refunds account for only 30% of traffic, and most refund runs still succeed after a retry. The average masks the issue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Path analysis results
&lt;/h2&gt;

&lt;p&gt;I analyzed both exports using &lt;a href="https://github.com/GermanGerken/journeygraph" rel="noopener noreferrer"&gt;JourneyGraph&lt;/a&gt;, an open-source tool I maintain. It does not collect traces: it reads existing exports, aggregates them into a single graph of steps and transitions, and then compares two such graphs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pipx install journeygraph
journeygraph diff baseline.jsonl candidate.jsonl --output-dir diff
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first three reported changes are as follows (step labels are shortened for clarity):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;change: tool:refund_api error-event rate moved from 9% to 30% (+20 percentage points).
change: Path variant request → router → planner → order_lookup → planner → policy_check
        → planner → refund_api → planner → answer_guard covers 10% of candidate traces
        versus 28% (-18 percentage points).
change: Path variant … → (llm:planner → tool:policy_check)+ → … covers 12% of candidate
        traces versus 0% (+12 percentage points).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnie5rp6x7qkls566l4q1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnie5rp6x7qkls566l4q1.png" alt="JourneyGraph comparison report: notable changes" width="800" height="600"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both symptoms of the planted fault are identified, and in the order I would want to read them:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One tool's error rate tripled&lt;/strong&gt;, from 9% to 30% (62 of 210 calls).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The main refund path decreased&lt;/strong&gt; from 28% of all traces to 10%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A new looped path appeared.&lt;/strong&gt; The &lt;code&gt;(planner → policy_check)+&lt;/code&gt; notation indicates that this pair repeats. This path now covers 12% of traces and did not exist previously.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The single-version report for the candidate provides supporting counts: traces with a repeated &lt;code&gt;refund_api&lt;/code&gt; call increased from 9 to 41, and the &lt;code&gt;policy_check → planner → policy_check&lt;/code&gt; loop appears in 91 traces compared to zero previously.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4yk6he65irq9mvtawrcz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4yk6he65irq9mvtawrcz.png" alt="Aggregate graph of the candidate version: refund_api is highlighted with a 30% error rate" width="800" height="455"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the two perspectives differ
&lt;/h2&gt;

&lt;p&gt;Success rate is a single value per trace, averaged across all traces. A regression limited to one intent and partially masked by retries has minimal impact on this metric.&lt;/p&gt;

&lt;p&gt;A path represents a structural change. For example, "this sequence of steps did not exist last week and now covers 12% of runs" is not averaged away. Similarly, a per-step error rate isn't diluted, since its denominator is the number of calls to that step rather than all traffic.&lt;/p&gt;

&lt;p&gt;Trace viewers such as Langfuse, Phoenix, or LangSmith are built to inspect individual traces in detail, and they are good at it. The relevant question here is different: what patterns emerge across 600 runs, and how have these patterns changed between versions? Answering it requires aggregation across traces before using a viewer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Integrating into CI
&lt;/h2&gt;

&lt;p&gt;The same comparison can be used to gate a pipeline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# rules.toml
&lt;/span&gt;&lt;span class="k"&gt;[regressions]&lt;/span&gt;
&lt;span class="na"&gt;max_success_rate_drop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.03&lt;/span&gt;
&lt;span class="na"&gt;max_error_event_rate_increase&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt;
&lt;span class="na"&gt;max_mean_cost_per_trace_increase&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.15&lt;/span&gt;
&lt;span class="na"&gt;forbid_new_categories&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;journeygraph check candidate.jsonl --rules rules.toml --baseline baseline.jsonl
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PASS max_success_rate_drop: success among known outcomes dropped by 0.02 (limit 0.03)
FAIL max_error_event_rate_increase: error-event rate increased by 0.0119583 (limit 0.01)
PASS max_mean_cost_per_trace_increase: mean cost per trace increased by (relative) 0.0747896 (limit 0.15)
check: 5 of 6 rule(s) passed        # exit code 5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The build fails, which is the correct outcome. However, the margin is narrow: it failed one global rule by 0.002. With a slightly higher threshold, this regression would have passed. Global thresholds have the same averaging limitations as the dashboard. Rules scoped to a single step or a single cohort are the next thing I am building, and this experiment is the reason.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The data is simulated.&lt;/strong&gt; It demonstrates that the method can detect a known, planted fault. It does not indicate how often real regressions resemble this scenario.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Associations do not imply causation.&lt;/strong&gt; The report states "traces that include X had a lower success rate". It does not claim that X caused the failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clear step names are important.&lt;/strong&gt; This simulation uses eight step types. A raw OpenTelemetry export from a microservice system contains dozens of low-level spans per trace, which can make the graph noisy. Improved grouping for such cases is in progress.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scalability.&lt;/strong&gt; On my laptop, processing 240,000 events takes 16 seconds and uses about 830 MB of memory. Growth is linear, so one million events would need an estimated 3.5 GB. I have not measured that yet; I plan to benchmark larger datasets and will update this post with the results.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Reproducing the experiment
&lt;/h2&gt;

&lt;p&gt;The simulator, the rules file, and the expected output are in &lt;a href="https://github.com/GermanGerken/journeygraph/tree/main/examples/agent-regression" rel="noopener noreferrer"&gt;examples/agent-regression&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python simulate_agent.py baseline baseline.jsonl
python simulate_agent.py candidate candidate.jsonl
journeygraph diff baseline.jsonl candidate.jsonl --output-dir diff
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;JourneyGraph is Apache-2.0, runs locally, has no runtime dependencies, and sends no telemetry. It reads JSONL, CSV, Parquet, and OTLP exports, with field mappings for Langfuse, Phoenix, and LangSmith exports.&lt;/p&gt;

&lt;p&gt;If you run it on your own traces and the report is inaccurate or unhelpful, I welcome your feedback: please &lt;a href="https://github.com/GermanGerken/journeygraph/issues" rel="noopener noreferrer"&gt;open a GitHub issue&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>observability</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
