DEV Community

Cover image for I built agent-inspect to debug TypeScript AI agent trajectories
Raju Dandigam
Raju Dandigam

Posted on

I built agent-inspect to debug TypeScript AI agent trajectories

#typescript #ai #agents #testing

Your AI agent failed.

Again.

The final answer is wrong, but the logs look fine:

tool call started
model call started
tool call completed
model call completed
fallback used
error: timeout
Enter fullscreen mode Exit fullscreen mode

Which tool caused the timeout? Did the model answer before retrieval finished? Was the fallback expected? Did the agent call the same tool twice?

This is where console.log stops feeling like debugging and starts feeling like archaeology.

I kept hitting this problem while building TypeScript AI agents. Once the flow moved beyond a single model call, the debugging loop became a system:

plan → retrieve → rank → generate → validate → maybe retry → maybe hand off
Enter fullscreen mode Exit fullscreen mode

Flat logs lost the structure. Output only tests could miss a bad path that happened to produce a plausible answer. Model graded evals helped with semantic quality, but they were a poor fit for every deterministic CI rule. And raw traces were too risky to paste casually into issues or pull requests.

So I built agent-inspect.

AgentInspect is a local evidence debugger and trajectory-test toolkit for TypeScript AI agents.

It turns one local trace into three things: a readable execution tree, a deterministic regression gate, and a derived Evidence v2 bundle that you can review before sharing.

No account. No collector. No default upload. Metadata only by default.

one local JSONL trace
├─ Debug   → view · report · explain
├─ Prevent → check · contract · CI
└─ Share   → redact · bundle · verify
Enter fullscreen mode Exit fullscreen mode

The bug is often in the trajectory

A support agent can return a plausible answer after doing almost everything wrong.

The healthy path might be:

plan-request
└─ retrieve_policy
   └─ rank-results
      └─ generate_answer
         └─ policyShown: passed
Enter fullscreen mode Exit fullscreen mode

The regression might be:

generate_answer          <- answered before retrieval
retrieve_policy
retrieve_policy          <- duplicate call
search_docs              <- wrong tool, failed
policyShown: failed
Enter fullscreen mode Exit fullscreen mode

An output only test may pass. A flat log may contain every event. Neither makes the wrong path obvious.

The final answer is only one fact about the run. Tool choice, ordering, repetition, completion, duration, token usage, and observed outcomes are facts too. Together, those facts form the agent's trajectory.

That trajectory should be inspectable. It should also be testable.

Make execution boundaries explicit

You can start with manual instrumentation:

import { inspectRun, observeOutcome, step } from "agent-inspect";

const answer = await inspectRun(
  "support-agent",
  async () => {
    const policy = await step(
      "retrieve_policy",
      () => retrievePolicy(),
      {
        type: "tool",
        metadata: { toolName: "retrieve_policy" },
      },
    );

    const result = await step(
      "generate_answer",
      () => draftAnswer(policy),
      {
        type: "llm",
        metadata: { model: "your-model" },
      },
    );

    await observeOutcome("policyShown", {
      expectation: "The answer cites a retrieved policy",
      status: "passed",
      method: "custom",
    });

    return result;
  },
  { traceDir: ".agent-inspect" },
);
Enter fullscreen mode Exit fullscreen mode

The wrapper records those boundaries as local JSONL while preserving the application's return value and errors. Raw prompts and model outputs are not required for the core workflow.

If your application already emits structured logs or uses AI SDK, OpenAI Agents JS, LangChain, or LangGraph you can use an adapter or reader instead of wrapping every step manually.

Get a useful result without an API key

The shortest path uses a generated synthetic demo:

npm install agent-inspect
npx agent-inspect init --yes
node examples/agent-inspect-demo.mjs
npx agent-inspect list --dir .agent-inspect
Enter fullscreen mode Exit fullscreen mode

init writes a small config and demo into your project. The demo does not call a model or upload a trace.

Copy the run ID printed by list, then use the same local artifact for the three jobs below.

1. Debug: read the execution tree 

npx agent-inspect view <run-id> --dir .agent-inspect --summary
npx agent-inspect report <run-id> --dir .agent-inspect
npx agent-inspect explain <run-id> --dir .agent-inspect
Enter fullscreen mode Exit fullscreen mode

A terminal lists a local AgentInspect run and renders its execution tree with step types, durations, model metadata, and token counts

The tree restores the structure that flat logs lose: nested steps, tool and model calls, durations, safe metadata, errors, and observed outcomes. explain summarizes local trace facts deterministically; its default path makes no provider call.

The useful question changes from:

Did the run fail?
Enter fullscreen mode Exit fullscreen mode

to:

Where did the passing and failing trajectories first diverge?
Enter fullscreen mode Exit fullscreen mode

That distinction matters when the visible answer looks fine but the agent skipped a required retrieval, safety, or validation step.

2. Prevent: turn the path into a CI gate

Some agent quality questions are subjective. Helpfulness, tone, and open ended answer quality can benefit from model graded evaluation.

But many regressions are structural:

  • Was retrieve_policy called?
  • Did the forbidden search_docs tool appear?
  • Did generation happen before retrieval?
  • Did the run complete?
  • Did an observed outcome fail?
  • Did the agent exceed a duration or token budget?

Those checks do not need another model. They can be deterministic:

npx agent-inspect check <run-id> --dir .agent-inspect \
  --preset trajectory \
  --required-tool retrieve_policy \
  --forbidden-tool search_docs \
  --fail-on-observation failed
Enter fullscreen mode Exit fullscreen mode

The preset and explicit shorthand rules are additive. A healthy run exits 0; a trajectory-rule failure exits 1.

For the committed regression fixture, the result is concrete:

Check status: fail
Summary: 2 failed, 0 warning(s), 0 error(s)

- outcome.status: Observed outcome count 1 matched [failed].
- tool.usage: Forbidden tool search_docs appeared.
Enter fullscreen mode Exit fullscreen mode

The same deterministic trajectory check exits zero for the good run and one for the regression

Same trace. Same rules. Same verdict. No model judge and no provider call in the check path.

That makes it suitable for a normal CI step. If your test fixture writes a trace to a stable path:

- name: Run deterministic agent fixture
  run: node run-agent-fixture.mjs

- name: Check agent trajectory
  run: |
    npx agent-inspect check .agent-inspect/ci-run.jsonl \
      --preset trajectory \
      --required-tool retrieve_policy \
      --fail-on-observation failed \
      --evidence-on fail
Enter fullscreen mode Exit fullscreen mode

--evidence-on fail writes local Evidence for triage when the check fails. It does not upload the artifact.

When CLI flags outgrow one command, the Beta TraceContract API expresses the same expectations in TypeScript:

import { openTraceFile } from "agent-inspect/readers";
import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const read = await openTraceFile("./.agent-inspect/demo-regression.jsonl");

const contract = defineTraceContract({
  run: { requireCompleted: true },
  tools: {
    required: ["retrieve_policy"],
    forbidden: ["search_docs"],
  },
  observations: { failOn: ["failed"] },
});

const result = evaluateTraceContractRead(read, contract);
if (result.status !== "pass") process.exitCode = 1;
Enter fullscreen mode Exit fullscreen mode

The principle is simple: use deterministic trace facts for structural CI rules, and reserve model grading for semantic quality.

3. Share: derive reviewable Evidence v2

A failing trace is often the best debugging artifact. It can also contain prompts, tool arguments, retrieved documents, customer identifiers, error messages, or secrets.

The collaboration strategy should not be “paste the raw trace into Slack.”

AgentInspect keeps the source trace read-only and creates a derived bundle:

npx agent-inspect verify-safe <run-id> --dir .agent-inspect
npx agent-inspect bundle <run-id> --dir .agent-inspect \
  --profile share \
  --out ./evidence
npx agent-inspect bundle verify ./evidence
Enter fullscreen mode Exit fullscreen mode

A terminal creates a share-checked Evidence v2 bundle and verifies its listed file hashes offline

The bundle can include:

evidence.html          self-contained offline review surface
evidence.json          versioned manifest and SHA-256 file hashes
trace.jsonl            redacted derived trace
check-results.json     deterministic findings
redaction-report.json  detector summary without secret values
summary.md             human-readable overview
Enter fullscreen mode Exit fullscreen mode

bundle verify checks the manifest, listed files, hashes, assessment, and provenance offline.

This is an integrity check. It is not a signature or a compliance certificate.

The wording matters: the artifact is share-checked, not “certified safe.” verify-safe and redaction are best-effort controls. Review the generated HTML and safety results before attaching a bundle to a pull request, incident, or public issue.

Use the capture path that matches your stack

The local evidence model is not tied to one agent framework.

Your stack Capture path
Custom TypeScript functions or classes inspectRun, step, observe, or createInspector
Vercel AI SDK @agent-inspect/ai-sdk
OpenAI Agents JS @agent-inspect/openai-agents
LangChain or LangGraph @agent-inspect/langchain
Existing structured logs agent-inspect logs readers
OpenInference or OTLP JSON local standards readers
Vitest or Jest reporters plus experimental trace matchers

The root package is enough for custom capture, the CLI, deterministic checks, and Evidence. Optional packages add only the integration you need.

Optional: give a coding assistant the same facts

The Preview MCP path exposes configured local evidence through bounded, read-only tools:

npx agent-inspect mcp configure --client cursor
Enter fullscreen mode Exit fullscreen mode

The command is a dry run by default, so you can review the generated configuration before enabling it.

A connected coding assistant can then investigate the same TraceFacts used by the CLI: What failed first? Which required tool was missing? What changed between the passing and failing runs?

This is not replay, an auto fix engine, or a hidden upload path. It is optional read-only access to explicitly configured local evidence.

Where it fits—and where it does not

AgentInspect owns the laptop to pull request evidence loop:

capture locally
→ understand the path
→ fail CI on structural drift
→ derive reviewable evidence
Enter fullscreen mode Exit fullscreen mode

It complements hosted observability and evaluation platforms. Use hosted tools when you need production dashboards, long term retention, fleet wide alerting, team wide trace search, hosted datasets, or prompt management.

Use AgentInspect when you need to inspect one TypeScript agent run immediately, enforce deterministic trajectory expectations in CI, compare a passing and failing local run, or hand off a redacted, hash-verifiable artifact.

The boundary is intentional. AgentInspect is not:

  • A maintainer hosted SaaS or production APM replacement
  • A hosted trace retention or prompt-management service
  • An LLM as judge or dataset platform by default
  • A replay or automatic remediation engine
  • A chain of thought recorder
  • A compliance certification tool

Try the complete loop

The current release is 6.17.2, requires Node.js 20 or newer, uses persisted schema 1.0, and is MIT licensed. Legacy v0.1 and v0.2 traces remain readable.

npm install agent-inspect
npx agent-inspect init --yes
node examples/agent-inspect-demo.mjs
npx agent-inspect list --dir .agent-inspect
Enter fullscreen mode Exit fullscreen mode

Then inspect, check, and derive Evidence from the run:

npx agent-inspect view <run-id> --dir .agent-inspect --summary
npx agent-inspect check <run-id> --dir .agent-inspect --preset trajectory
npx agent-inspect bundle <run-id> --dir .agent-inspect \
  --profile share \
  --out ./evidence
npx agent-inspect bundle verify ./evidence
Enter fullscreen mode Exit fullscreen mode

One local trace should be able to tell you what the agent did, prove that the regression stays fixed, and give a teammate evidence they can review without making upload the price of admission.

How you can help

AgentInspect is open source and MIT licensed. If this workflow is useful to you:

  • Star the repository so more TypeScript agent developers can find it.
  • Try the keyless demo and open an issue if anything feels confusing.
  • Add a starter or recipe for your stack.
  • Share a real debugging workflow you want the project to support.
  • Pick up a good first issue or contribute documentation.

Most of all, leave a comment below.
How do you debug agent runs today?
Which trajectory rule would you put in CI first?
And if AgentInspect does not fit your workflow, tell me why that feedback is just as useful.

Top comments (29)

Collapse
 
mudassirworks profile image
Mudassir Khan •

the healthy vs regression trajectory comparison is the clearest framing of this problem I've seen. we had the exact generate_answer before retrieve_policy failure in prod for about six weeks — output quality stayed high enough to pass eval because the model guessed well from context. routing bug was invisible.

the part of agent-inspect I'm most curious about is the deterministic contract layer. are the contracts structural (tool A before tool B) or temporal (A within X ms of B)? the temporal version is where most interesting failures live in practice.

does trajectory diff work across retries, or does each attempt get its own trace root?

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, that six-week example is exactly why I prefer contracts over output-only evals. The useful assertions are primarily structural—required or forbidden operations, causal order, completion, and outcomes—and budget-oriented, such as run or step duration and token ceilings. I’m wary of tight “A within X ms of B” rules because CI jitter can make them brittle; duration SLOs are safer when the threshold has operational meaning. Retries remain under one trace root as distinct step occurrences when they are visible to the instrumentation, so trajectory comparison can reason about the retry shape rather than treating every attempt as a separate run. Internal client retries still need to be surfaced by the adapter.

Collapse
 
mudassirworks profile image
Mudassir Khan •

the duration SLO framing makes way more sense than timeout assertions — we burned a couple sprints on flaky tests where the threshold was "fast enough on dev hardware" rather than "signals a real regression."

the retry as distinct step occurrence model is the piece I haven't fully reasoned through. if an adapter surfaces retries, do contract violations during the retry (correct final outcome, wrong causal path) count as failures in your experience?

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

Yes—when the contract is protecting a causal or safety property, I would count the retry-path violation as a failure even if the final answer is correct. A caught retry can leave the overall run successful, but it should not erase the failed tool occurrence or an invalid sequence such as a side effect happening before approval.

The important caveat in the current 6.17.6 behavior is that the default requiredOrder rule uses first-occurrence ordering. It does not automatically mean “this must hold for every retry.” Failure/retry limits and forbidden/max-call rules can evaluate the separate finished tool occurrences, but an every-occurrence ordering guarantee needs a stricter rule.

That gap is exactly what the open causal-ordering contribution is exploring: opt-in happens-before and all-occurrences modes while preserving the compatible default: github.com/rajudandigam/agent-insp...

My practical split is: tolerate and report an expected transient retry, but fail the gate when any attempt crosses an authorization, idempotency, or required-order boundary. Does that match the sort of retry failures that cost you those sprints, or were yours mostly timing variance without a semantic path change?

Thread Thread
 
mudassirworks profile image
Mudassir Khan •

mostly timing variance in our case, but the semantic line is exactly the split we missed. we had retries that looked like transient failures but were masking an approval step that had quietly been removed from the standard path a sprint earlier. retry succeeded, output correct, path wrong.

the all occurrences mode sounds like the right default for authorization steps. how’s the opt in designed — at the rule level or the contract level?

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, I’d put the opt-in at the individual rule level, then allow a contract-level default only as shorthand. Authorization ordering usually needs all-occurrences semantics, while another rule in the same contract may intentionally care only about the first successful occurrence. A rule such as { order: [...], occurrenceMode: "all" } keeps that intent reviewable; the contract can set defaultOccurrenceMode for safety-heavy suites without removing per-rule overrides. Your removed-approval example is exactly the case that should fail even when the retry recovers.

Thread Thread
 
mudassirworks profile image
Mudassir Khan •

the per rule occurrenceMode makes sense. we ran into exactly the split you're describing: authorization ordering needed all occurrences semantics while a different rule in the same contract only cared about first success, and a contract level default forced us to duplicate rules to express that.

the occurrenceMode: "all" shape keeps it readable without leaking the semantic intent into the contract name. did you consider making the default configurable at the contract level as a fallback, or intentionally left it as a shorthand to keep contracts portable?

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, I considered the contract-level default as authoring shorthand: evaluation should normalize it into an explicit mode for every rule, with the per-rule value winning. That lets safety-heavy suites default to all without repeating it everywhere, while the resolved contract stays portable and reviewable because downstream checks do not depend on an implicit global setting. I’d also include the resolved mode in the check result and evidence bundle so a reviewer never has to infer which default applied. Did your duplicate-rule workaround create separate findings, or only extra configuration?

Thread Thread
 
mudassirworks profile image
Mudassir Khan •

separate findings — each instance got its own evidence chain, which made the reviewer workflow cleaner even if the raw count looked inflated. we added a dedup pass at report render to surface the union.

the resolved mode in check result is the right call. having to infer which default applied is exactly where review fatigue sets in on bigger suites. does the bundle surface the full resolved contract or just the individual rule result?

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, today the bundle carries the individual deterministic findings in check-results.json; it does not yet emit the fully normalized contract as a separate first-class artifact. I think the safer shape is to add a resolved-contract.json (and its hash to the manifest), have every finding reference a stable resolved rule ID, and keep deduplication in the report renderer so the underlying evidence chains remain intact. That would let a reviewer see both the union and exactly which default or override produced each result. In your union view, did you preserve the member finding IDs so a reviewer could drill back into each evidence chain?

Thread Thread
 
mudassirworks profile image
Mudassir Khan •

yes — member finding IDs stayed in the union, each row carries its source finding ref so drill back works. what we lost was the override chain: we knew the finding existed but not which default or rule level produced it. your resolved-contract.json shape fixes exactly that gap.

does the hash go into the manifest before or after the renderer reads it? trying to figure out whether the audit trail for a changed default propagates forward cleanly.

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, I’d have the renderer consume the resolved contract first, write every derived file, and then build the manifest last from the final byte content—including the hashes for resolved-contract.json and the rendered report. That makes a changed default propagate forward into the resolved-contract hash, report hash, and manifest without a renderer depending on a digest added afterward. Verification should recompute every listed hash without re-rendering; otherwise the audit step could normalize away the change it is supposed to detect. Your source finding references are a useful model for carrying the override-chain provenance into each result.

Thread Thread
 
mudassirworks profile image
Mudassir Khan •

the "build manifest last from final byte content" order is the clean version — we had the reverse for a while and the audit was comparing renderer output against itself, which meant a normalization bug in the renderer cleared itself on verify.

recomputing hashes independent of the renderer is the harder engineering constraint. our verification step lazy calls the same serializer the renderer uses, which defeats the point. your explicit path split is cleaner. any footguns you've seen in the hash implementation when byte content includes timestamps or non deterministic output?

Thread Thread
 
raju_dandigam profile image
Raju Dandigam •

@mudassirworks, the biggest footgun is letting verification regenerate any bytes. I’d assign volatile values such as generatedAt, IDs, and collection order once in a build context, serialize each artifact once, hash those exact UTF-8 bytes, and have verification read files verbatim. Stable JSON key ordering and newline rules help reproducible builds, but verification should still fail if a timestamp changes after the manifest is written. For truly nondeterministic previews, I’d keep the preview outside the integrity set or derive it from a hashed canonical artifact and label that boundary. Has your nondeterminism come mainly from time/IDs or from collection ordering?

Collapse
 
reidmarlow profile image
Reid Marlow •

This is the level where agent debugging starts to feel sane. A flat log tells you calls happened. A trajectory tree tells you which branch lied to you. I like the Evidence bundle angle too, since CI needs artifacts a reviewer can read without replaying the whole run.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

Thanks @reidmarlow really appreciate this. That “calls happened” vs “which branch actually went wrong” distinction is exactly the problem AgentInspect is trying to solve.

And yes, the Evidence bundle is meant to make the trace useful beyond local debugging too something you can hand to a reviewer in CI or a PR without replaying the run or sharing the raw trace.

Glad that part resonated.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

Another thank-you round for people whose writing kept pushing me toward better answers around state drift, evaluation, observability, and trustworthy agent behavior.

Shout-out to @dovzhikova @engtwindev @ev3lynx727 @fazal_mansuri_ @gde @googleai @happynood @kgjohnson @kirandeepjassalcrypto @marcossouzadotdev

A lot of the thinking behind agent-inspect came from seeing similar problems show up across different stacks. If you get a chance to read the launch post, I’d really value your thoughts.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

A big thank-you to a group of folks whose posts and discussions helped sharpen how I think about agent reliability, tool boundaries, observability, and human-debuggable workflows while building agent-inspect.

Really appreciate the ideas and pressure-testing from @aidam @akilahngqueen @anthonymax @bokuwalily @coder11 @curi0us_dev @dailycontenthub @davidloibner @debashish_ghosal @dipankar_sarkar

If agent trajectories, tool-call receipts, and local-first debugging are themes you care about too, I’d genuinely love your feedback on the article and the direction of the project.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

I also wanted to thank more builders who’ve been exploring the hard parts of agent systems in public: tool selection, test reliability, memory, failure paths, receipts, and production-safe automation.

Appreciate the signal from @qawalah @sapph1re @shogun_the_grt @swapnanilsaha @trknhr @tsvetang2 @tznthou @vezzu_ruthvik @waxell @webpro255 @wyndev

If this article resonates, I’d love for you to poke holes in it, disagree with it, or suggest where agent-inspect should go next.

Collapse
 
pravesh_sudha_3c2b0c2b5e0 profile image
Pravesh Sudha •

Great guide

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@pravesh_sudha_3c2b0c2b5e0, thanks for reading. If you try the keyless demo, I’d be especially interested in whether the execution tree, deterministic CI gate, or reviewable evidence bundle is the most useful part of the workflow for you.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Great article, Raju!

My Key Takeaways:
Spotting the right answer for the wrong reasons: you have nailed a huge pain point in agent development—an agent can spit out a totally plausible answer while doing horrifying things under the hood (like skipping retrieval, calling the same tool twice, or taking an absurdly expensive fallback path).

Local-first privacy is a huge win: In a space flooded with heavy SaaS observability tools asking for API keys and telemetry subscriptions, keeping this lightweight, local-first, and redacted by default makes it infinitely easier to drop traces directly into PRs without leaking data.

I was thinking about this - how about you add zero-friction wrappers or auto-instrumentation. Having to manually wrap execution blocks in inspectRun and step() adds noticeable boilerplate to codebase routines. Building auto-middleware hooks for standard TypeScript setups—like Vercel AI SDK or standard OpenTelemetry spans—would make the adoption low barrier.

How does it handle legitimate model creativity vs. actual test flakiness in CI? Since non-deterministic LLMs will naturally swap tool order or take alternative valid paths depending on minor prompt tweaks, how do you set tight contract assertions without making the build pipeline fragile?

Thanks again! Looking forward to great things!

Collapse
 
raju_dandigam profile image
Raju Dandigam •

Thanks @debashish_ghosal both are important points.

On instrumentation, I agree that asking developers to manually wrap every agent step won’t scale. AgentInspect already supports framework adapters for common paths, and reducing setup friction further is definitely an area I want to keep improving.

For CI, I don’t think the goal should be to force an agent to take the exact same path every time. The better approach is to test behavioral invariants for example, a required tool was used, a forbidden tool was not used, the run completed, or an expected outcome was observed.

That way, different valid trajectories are still allowed, while real regressions remain deterministic to catch.

Thanks for raising these, they’re exactly the kinds of questions that matter for practical agent debugging and testing.

Collapse
 
richard_smith_154156d471ef profile image
Richard Smith •

The "right answer for wrong reasons" problem is so real. I've spent way too long assuming an agent worked correctly just because the output looked good.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@richard_smith_154156d471ef, same here—that false confidence is what pushed me toward trajectory checks. The output tells us what the user saw, while the trace can prove whether retrieval, validation, and required tools actually happened before the answer. A plausible result should not erase a broken path.

Collapse
 
rohitnirban profile image
Rohit Yadav •

How does AgentInspect handle valid variations in an agent’s trajectory without making CI tests too strict or flaky?

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@rohitnirban, the key is to assert invariants rather than one golden path. For example: retrieval happened before generation, no forbidden write ran after a policy block, the run completed, retries stayed within budget, and one of several allowed tool branches was used. Different wording, ordering among independent siblings, or valid alternative branches can still pass. In agent-inspect, I’d use deterministic checks for those structural facts and leave semantic answer quality to a separate evaluation layer.

Collapse
 
trknhr profile image
Teruo Kunihiro •

This feels like a very good direction. Testing only the final answer misses a lot of important agent failures, so making tool usage, ordering, retries, and validation steps testable in CI makes a lot of sense.

I especially like that the contracts focus on invariants rather than requiring one exact trajectory that seems like the right balance for nondeterministic agents. The local-first design and evidence bundles are nice touches too. Curious how it handles parallel calls and nested agents, but this looks genuinely useful.

Collapse
 
raju_dandigam profile image
Raju Dandigam •

@trknhr, parallel calls are represented as sibling steps under the same parent, while nested agents or tools keep explicit parent-child relationships. Each step retains its own lifecycle and duration, so sibling durations are not incorrectly added as though the work were sequential. The contract layer can assert parentage and required dependencies without forcing one completion order.

View full discussion (29 comments)