DEV Community: Teller

Trace-to-Training: how agent runs become learning data

Teller — Fri, 26 Jun 2026 01:50:39 +0000

Trace-to-Training: how agent runs become learning data

Every agent run is a data point. Most frameworks throw it away.

WasmAgent keeps it — evaluated by the compliance engine, ranked by outcome, exported as a typed ComplianceEvalRecord ready for SFT or DPO training. No human labeling.

Three repair modes

import { ComplianceRun } from "@wasmagent/compliance";

const run = new ComplianceRun({
  mode: "full_pcl",   // "direct" | "prompt_retry" | "full_pcl"
  taskSpec: {
    instruction: "Write a summary in exactly 3 bullet points.",
    constraints: [{ type: "format", rule: "bullet_count", value: 3 }],
  },
});

const result = await run.execute(agent, input);
// result.complianceEvalRecord → typed, versioned, schema-validated

direct — one shot, record pass/fail.

prompt_retry — retry once with a rephrased prompt.

full_pcl — full repair loop: run → evaluate → patch/regenerate → re-evaluate → record the entire trace.

What the numbers show

IFEval × Qwen2.5-1.5B-Q4 (3 seeds × 50 samples):

Mode	Pass rate	Std dev
prompt_retry	46.0%	±2.0pp
full_pcl	54.7%	±1.2pp

+8.7pp. The variance drop (±2.0 → ±1.2) matters for production reliability.

Reproduce: bun packages/compliance/benchmarks/ifeval/run.ts --limit=50 --seed=42

The repair trace is the training data

When full_pcl repairs a failing output, RepairPlanner records every attempt:

// Inside ComplianceEvalRecord
attempts: [
  { strategy: "direct",     output: "...", passed: false },
  { strategy: "patch",      output: "...", passed: false },
  { strategy: "regenerate", output: "...", passed: true  },
]

The full sequence — what failed, what was tried, what worked — is what feeds DPO training. The model learns from failure traces, not just final outputs.

Parallel rollouts for preference pairs

import { RolloutForkRunner, RolloutRanker } from "@wasmagent/core";

const runner = new RolloutForkRunner({ forks: 4 });
const rollouts = await runner.run(agent, input, taskSpec);

const ranked = new RolloutRanker().rank(rollouts);
// ranked[0] → chosen (SFT)
// ranked[1..] → rejected (DPO pairs)

The compliance verifier is the reward signal. No human annotation.

Try it

git clone https://github.com/WasmAgent/wasmagent-js
bun test packages/compliance/   # 113 pass / 0 fail

Code: packages/compliance · RolloutForkRunner · RolloutRanker

Series: AEP (part 1) · MCP Trust Pack (part 2) · Trace-to-Training (part 3)

MCP Trust Pack: a security layer for MCP tool calls

Teller — Fri, 26 Jun 2026 01:50:09 +0000

MCP Trust Pack: a security layer for MCP tool calls

MCP makes it easy for agents to call tools. Too easy.

When your agent calls fs_write or shell_exec, something needs to answer: is this allowed? Is this state-changing? Who authorized it? By default, MCP has no answer.

Here's how to add that layer in ~20 lines.

MCPGateway: drop-in security layer

import {
  MCPGateway,
  buildServerCard,
  createRequestIdentity,
  isStateChangingTool,
} from "@wasmagent/mcp-firewall";

// Register the server at startup
const card = buildServerCard({
  serverId: "filesystem",
  tools: await mcpClient.listTools(),
  operatorVerified: true,
});

const gateway = new MCPGateway({ serverCards: [card] });
const identity = createRequestIdentity({
  principal: "agent:run-abc123",
  sessionId: "sess-xyz",
});

// Before every tool call:
const decision = gateway.evaluate({ identity, serverId: "filesystem", tool, args });

if (decision.invocation.decision !== "allow") {
  throw new Error(`Blocked: ${decision.invocation.reason}`);
}

const result = await mcpClient.callTool(tool.name, args);
const obs = gateway.wrapResult(tool.name, result, decision); // marks trust level

Four layers run in evaluate(): vetting → policy → consent → taint. One call, full coverage.

State-changing tools are classified automatically

isStateChangingTool({ name: "fs_write",   description: "write a file" }) // true
isStateChangingTool({ name: "fs_read",    description: "read a file"  }) // false
isStateChangingTool({ name: "send_email", description: "send email"   }) // true

State-changing tools can be gated behind a ScopeLease — a time-bounded grant that expires:

import { createScopeLease, isScopeLeaseValid } from "@wasmagent/mcp-firewall";

const lease = createScopeLease({
  principalHash: identity.principalHash,
  serverId: "filesystem",
  grantedTools: ["fs_write"],
  ttlSeconds: 300,      // 5 min
  maxInvocations: 10,
  stateChanging: true,
});

if (!isScopeLeaseValid(lease)) throw new Error("Lease expired");

GatewayDecision feeds AEP directly

The decision's evidenceRef slots straight into AEPEmitter — no manual wiring:

emitter.addAction({
  tool_name: decision.invocation.toolName,
  state_changing: decision.stateChanging,
  capability_decision: {
    decision: decision.invocation.decision,
    reason_code: decision.evidenceRef.policyDecision,
  },
  tool_descriptor_digest: decision.evidenceRef.toolManifestDigest,
});

Try it

git clone https://github.com/WasmAgent/wasmagent-js
bun test packages/mcp-firewall/

Code: packages/mcp-firewall · packages/mcp-gateway

Series: AEP (part 1) · MCP Trust Pack (part 2) · Trace-to-Training (part 3)

Your AI agent called a tool. Can you prove it followed the rules?

Teller — Fri, 26 Jun 2026 01:26:57 +0000

Your AI agent called a tool. Can you prove it followed the rules?

Your agent just wrote a file. You have logs. But can you answer this:

Was the policy gate applied before the tool ran — or after?

Logs can't tell you that. Here's how we solved it.

The gap in current agent frameworks

Most frameworks give you a log line like:

[2026-07-07T09:00:01Z] tool:fs_write path=/tmp/report.txt status=ok

That tells you the tool ran. It doesn't tell you:

Whether a policy evaluated the call first
What the pre-state looked like before the write
Whether the agent was within its token and risk budget
Which agent in a delegation chain authorized this

For a hobby project, that's fine. For anything touching real data, it's not.

AEP: structured proof, not a log stream

WasmAgent's @wasmagent/aep package records every tool call as an ActionEvidence object — Zod-validated, schema-versioned, with pre/post state digests baked in.

import { AEPEmitter } from "@wasmagent/aep";

const emitter = new AEPEmitter({
  run_id: "run-abc123",
  repo_commit: "5c1102f",
  model_id: "claude-sonnet-4-6",
});

// Before tool execution:
emitter.addAction({
  tool_name: "fs_write",
  state_changing: true,
  capability_decision: {
    capability: "fs_write",
    subject: "agent:run-abc123",
    resource: "/tmp/report.txt",
    decision: "allow",
    reason_code: "policy:default-v1",
  },
  precondition_digest: "sha256:a1b2c3...",
  input_taint_labels: ["user_provided"],
});

// After tool execution:
emitter.addAction({
  tool_name: "fs_write",
  state_changing: true,
  post_state_digest: "sha256:d4e5f6...",
});

emitter.setBudgetLedger({
  token_budget: { limit: 4000, spent: 142 },
  risk_budget:  { limit: 1.0,  spent: 0.2 },
});

const record = emitter.build();

The capability_decision is part of the same record as the action — not a separate log entry that could be reordered or dropped.

OTel spans for everything else

For real-time observability, AEP also emits named OpenTelemetry spans:

import { AEP_SPAN_NAMES } from "@wasmagent/otel-exporter";

// Plug into any OTel collector:
AEP_SPAN_NAMES.TOOL_CALL       // "tool.call"
AEP_SPAN_NAMES.POLICY_CHECK    // "policy.check"
AEP_SPAN_NAMES.SANDBOX_EXEC    // "sandbox.exec"
AEP_SPAN_NAMES.VERIFIER_CHECK  // "verifier.check"
AEP_SPAN_NAMES.LLM_GENERATE    // "llm.generate"
AEP_SPAN_NAMES.MCP_REQUEST     // "mcp.request"
// + 3 more

The spans go to Grafana/Jaeger/etc. The AEPRecord is what you keep for audit and training data.

Multi-agent: delegation chain

In a single-agent setup, this is useful. In a multi-agent setup — orchestrator delegates to a subagent — it becomes essential:

run_context: {
  agent_id: "orchestrator",
  subagent_id: "coder-agent",
  delegation_chain: ["orchestrator", "planner", "coder-agent"],
  scope_lease_id: "lease-xyz",  // ← subagent can only do what parent granted
}

Try it

git clone https://github.com/WasmAgent/wasmagent-js
bun test packages/aep/src/

Next in this series: MCP Trust Pack — the gateway layer that enforces policy before tools execute.

Code: packages/aep · packages/otel-exporter