When I started building agent-exec-trace, I thought I knew where the difficulty would be. OSS agent-exec-trace is an observability layer for AI age...
For further actions, you may consider blocking this person and/or reporting abuse
Green gate for the wrong reason — yes, and mine had no cancelling bugs to blame. The gate was the system coming back up. Autonomous system of mine crash-looped, I shipped a commit, it recovered, I closed the incident. Checked it properly later: the incident was an authentication failure and my commit had corrected clock synchronisation, a different failure class entirely. The two were never connected. Verdict against myself: falsified. Recovery is the greenest signal there is and it proves nothing about cause.
One thing I'd push on from your own numbers: empty_response firing on 100% and 28 detectors firing on 0% are the same measurement. Neither carries any information about the runs. You treat the first as the bug and the second as not-yet-exercised, but your own closing line applies to both. And what would finish them isn't a fire on real data — it's a fire and a documented silence on real data, on cases where you know in advance which one should happen. A detector that fires on a real corpus has only proven it can fire.
Juan, that's a better answer than I would have given. The question of "does it tell you the run was clean, or does it tell you it couldn't read the run?" is exactly the axis I wish more comparison pages included. Most tooling renders those two states identically, and the 42.4% / 99.2% numbers are measuring exactly that gap. Thank you for articulating it better than I did.
Thanks — though the axis is yours, I only named it. It exists because you published the 42.4% instead of the number that looked fine.
One concrete thing, if you want it to stick: put that number in the repo, not just in a post. Structural compatibility against a corpus you didn't produce, with the corpus named and the date it was measured. In a post it's a story about your build. In a README it's a figure the next tool has to answer. Nobody publishes one right now, and whoever does it first sets the axis instead of arguing for it.
The 42.4% compatibility number after normalization is the honest core of this piece. Going from "looked decent" to a much uglier real figure is exactly the moment where most writeups would quietly stop reporting the metric, publishing the drop, not just the fix, is what makes the rest of the lessons credible.
The green-gate-from-two-cancelling-bugs story is the sharpest one here. A failing test tells you something true. A milestone marked done because a Docker port misconfiguration and an SDK default happened to produce a demo that looked fine is a failure mode that's actively worse than an honest red X, because it launders "never actually verified" into "verified." That's a strong argument for insisting on one real end-to-end path per milestone rather than trusting a checklist of sub-checks that can each pass in isolation while the thing they compose into never runs for real.
The synthetic-vs-real distinction generalizes past observability tooling too: synthetic data proving a detector can fire and real data proving it should fire are different claims, and conflating them is how a hallucination detector ends up 98% "accurate" against traces engineered to make its job trivial. The three-layer validation sequence (unit tests → synthetic → real) is a good default anywhere a detection system's output depends on messy real-world input shape rather than logic correctness alone.
Talha, thank you for reading so carefully. You caught exactly what I was trying to convey — the 42.4% number is the honest version, and publishing the drop rather than just the fix was important to me. The green-gate-from-two-cancelling-bugs story still bothers me more than any single detector failure. A red test is honest. Two bugs high-fiving each other into a green checkmark is a special kind of dishonesty that looks disciplined. The three-layer validation sequence you identified (unit → synthetic → real) is the framework I'm now using across everything I build. Appreciate the thoughtful analysis.
This really resonated @debashish_ghosal, especially the point that structure without enough evidence can leave observability blind.
I’m running into a similar distinction from the decision side: knowing that evidence is
presentis not the same as knowing that it actually supports the fact or conclusion being asserted.For example, a required contract may exist, but the relevant section may be missing. Structurally the evidence is present, yet a decision should probably remain unresolved rather than proceed.
It makes me wonder whether agent observability eventually needs to capture not only what evidence was retrieved, but also the claim-to-evidence relationship the agent relied on.
Have you thought about representing that explicitly in the trace model?
Brian, thank you — your point about evidence presence vs. evidence sufficiency hits a distinction I hadn't fully articulated. The example is perfect: a required contract may exist structurally, but if the relevant section is missing, the decision should remain unresolved. The claim-to-evidence relationship you're describing is exactly the direction I think agent observability needs to go. I haven't modeled that explicitly in the trace format yet, but your framing makes me think it belongs as a first-class span annotation rather than something the detector reconstructs after the fact. Really appreciate you raising this — it's going into my design notes.
Great write-up! The point about "the traces were the wrong shape" really resonated with me. It's a lesson that's easy to underestimate until you start working with production data.
I've had a similar experience building infrastructure tooling—everything looks great with synthetic or controlled data, but real-world inputs quickly expose assumptions you didn't even realize you were making.
I also liked your point about green gates. A passing CI pipeline can create a false sense of confidence if the end-to-end path hasn't actually been exercised.
Thanks for sharing such an honest account of the engineering process.
Thank you for reading and for the kind words. "The traces were the wrong shape" was the moment the whole project pivoted for me. Everything looks great with synthetic or controlled data — and then real-world inputs expose assumptions you didn't even know you were making. That's the pattern I keep running into across different tools. The green gate point you raised is the one I now find hardest to trust without an end-to-end smoke path. Appreciate you sharing your own experience with infrastructure tooling — it's the same lesson, just a different domain.
"mock agents are polite and real traces are not" is the line that lands hardest here. we hit the same wall with a RAG tracing setup — detectors worked perfectly against synthetic traces, then fell apart on live calls where field names shifted between providers.
the trace schema drift is what makes agent observability harder than service observability. a service has a contract. an agent execution path is shaped by whatever context it got this run.
curious how you're handling schema evolution: if the HuggingFace corpus taught you one shape, what happens when a framework ships a major version that reorganizes span fields?
The framing shift from "detect the anomaly" to "understand the execution graph" is the real unlock. Once you're dealing with multi-step agents, the interesting failures aren't the ones that throw errors — they're the ones that complete successfully but reasoned wrong somewhere in the middle, and you only see it downstream.
THANK YOU - I will definitely add this to the learning. You nailed it - this is exactly, what happened and I am re-learning this as I am working on a related agent related project. The execution graph, its dependencies, who calls what tool and so on and forth is just consuming so much time to address either with shims or providing LLM endpoint, or a postres db it wants to talk to and so on. Troubleshooting the failure is now getting into understanding how the agent works. It does not scale when I have aspirationally chosen 10 framework to support x 10 agents on each. Sometimes I will scale back, sometimes I have removed agents from field test that turned out to be either too many steps or dependencies too hard to setup. I would imagine this would be a problem others are facing too in this space.
The empty_response detector firing on 100% of traces is such a clean failure mode. The rules were not lying. The spans never carried the shape those rules assumed, so every detector either screamed or stayed silent. I have hit the same wall instrumenting coding agent runs: if tool calls, retries, and cost live in freeform logs instead of consistent span names and attributes, you end up tuning thresholds on noise. Getting the trace contract right before the detector catalog is the unglamorous work that makes the rest of the system mean something.
Edu, thank you for reading. You described the failure mode perfectly: the rules weren't lying, the spans just never carried the shape those rules assumed. Every detector either screamed or stayed silent. The "trace contract before detector catalog" framing is exactly right — it's the unglamorous foundation work that determines whether the rest of the system has anything meaningful to work with. Your experience with coding agent instrumentation mirrors what I found: freeform logs vs. consistent span attributes is where the real battle is. Appreciate you sharing that.
how would you compare it with langsmith, langfuse , acruxcore etc
Not the OP, but one axis worth adding to that comparison, taken from the post itself: the interesting question isn't which of them ships the deeper detector catalogue — it's what each one assumes about trace shape, and what it does when the assumption doesn't hold.
That's what the 42.4% and the 99.2% here are actually measuring, and I've never seen that number published for any of the tools you listed. Point any of them at a corpus you didn't produce and the same question applies: does it tell you the run was clean, or does it tell you it couldn't read the run? Those are different answers, and most tooling renders them identically.
Talha, thank you for the question. The honest answer is that agent-exec-trace is lower-level and more instrumentation-focused than LangSmith or LangFuse. Those are full platforms with their own storage, UI, and evaluation workflows. agent-exec-trace is closer to an OpenTelemetry-style SDK that focuses specifically on behavioral anomaly detection (loops, retry storms, cost spikes, hallucination patterns) rather than general-purpose tracing. The key differentiator right now is the detector catalog being open and extensible, and the whole thing running against your existing observability stack. Appreciate you asking.
Thanks