DEV Community: Raju Dandigam

Docker Sandboxes Changed the Trust Boundary for AI Coding Agents

Raju Dandigam — Sun, 20 Sep 2026 16:42:38 +0000

A coding agent that can only suggest a patch is one kind of risk. An agent with a shell, package manager, Docker daemon, credentials, and network access is another.

The useful security question is not “Is the agent inside a container?” It is:

Which resources cross the isolation boundary, in which direction, and with what authority?

Docker Sandboxes make that question concrete. A local sandbox runs the agent inside a microVM with its own kernel and Docker Engine. The agent has broad control inside that VM—including sudo—while access to the host is mediated through explicit workspace, credential, network, skills, and MCP boundaries.

That is a stronger model than starting an agent directly on a developer laptop. It is not the same as making the agent harmless.

Start with the workspace mode

There are three materially different file boundaries:

Mode	Host repository	Agent writes
Direct mount	Shared read-write	Appear immediately in the working tree
Clone mode	Host repository mounted read-only	Private in-VM clone
Mountless	Not shared	Sandbox filesystem only

Direct mode is convenient for an interactive edit-review loop. It also means the agent can change package.json scripts, CI configuration, editor tasks, AI configuration, and other files that may execute later on the host.

Clone mode changes the review boundary. The agent works on a private clone while the original host repository remains read-only. That is a better default for broad exploratory tasks or unfamiliar repositories. Mountless mode is strongest when the task does not require host files.

The choice should follow the task, not developer habit.

Isolation does not end at the filesystem

Docker's local sandbox model separates several capabilities:

Docker Engine

The sandbox receives a private Docker Engine rather than access to the host daemon. This matters because mounting the host Docker socket into an ordinary container can effectively grant host-level control.

Credentials

Provider credentials can be injected by a host-side proxy into permitted outbound requests, so raw key values do not need to enter the VM. This reduces credential exposure, but the agent can still exercise whatever authority those proxied credentials grant.

Network

Outbound TCP traffic passes through a host proxy and a deny-by-default policy. Review the active allowlist. Broad domains may permit more services than the task needs.

“Deny by default” does not necessarily mean “nothing is reachable.” Docker's Balanced preset begins with a baseline allowlist for common model providers, package managers, code hosts, registries, and cloud services; Open and Locked Down differ materially. Inspect the effective rules with sbx policy ls and narrow them for the task instead of inferring the policy from the product name.

Shared skills

A shared skill store is a deliberate exception: one sandbox can modify instructions or scripts later consumed by another. If several sandboxes share it read-write, treat them as participating in the same trust boundary.

MCP servers

The MCP gateway is another explicit bridge. Remote servers remain outside the VM. Local stdio MCP servers run on the host, not inside the sandbox. A host-side MCP tool can therefore have authority the sandbox itself does not.

Threat-model the whole action path

For every capability, write down four facts:

type CapabilityBoundary = {
  resource: "workspace" | "network" | "credential" | "mcp" | "skill";
  direction: "into_sandbox" | "out_of_sandbox" | "both";
  authority: string;
  reviewBeforeUse: boolean;
};

For example:

const policy: CapabilityBoundary[] = [
  {
    resource: "workspace",
    direction: "both",
    authority: "private clone only",
    reviewBeforeUse: true,
  },
  {
    resource: "network",
    direction: "out_of_sandbox",
    authority: "registry and model API only",
    reviewBeforeUse: false,
  },
  {
    resource: "mcp",
    direction: "both",
    authority: "read-only issue tracker",
    reviewBeforeUse: true,
  },
];

This forces “the agent has MCP” into a specific statement about a specific server and tool set.

A practical review sequence

Before running an autonomous coding task:

Prefer clone or mountless mode unless live host edits are required.
Remove network destinations the task does not need.
Provide task-scoped credentials with the least useful authority.
Review every local MCP server as host code.
Disable shared skills when cross-sandbox mutation is unnecessary.
Inspect changes before executing modified hooks, scripts, or CI files.
Keep human approval around publishing, deployment, and other irreversible actions.

The last step matters because isolation controls where code runs. It does not decide whether a proposed business action is appropriate.

The discussion above is about Docker's documented local sandbox model. Cloud sandboxes have their own lifecycle, credential, and connectivity behavior. Record which environment executed the task before treating a sandbox result as security evidence.

A sandbox is a boundary, not a verdict

MicroVM isolation, a private Docker daemon, proxied credentials, and explicit network policy significantly improve the execution boundary for coding agents. The remaining risk travels through the resources intentionally shared across it.

That is the design lesson worth carrying to any agent environment: grant the smallest workspace, network, credential, and tool surface that can complete the task—and treat every bridge back to the host as part of the security model.

References

Your Cypress Retry Passed - But the Test Is Still Telling You Something

Raju Dandigam — Sat, 19 Sep 2026 17:36:31 +0000

Attempt one fails. Attempt two passes. CI turns green.

Did the system recover from harmless noise, or did the retry hide an intermittent product defect?

Cypress retries are useful because browser tests meet networks, animations, databases, and asynchronous UI state. The mistake is treating a passing retry as if the first failure never happened.

A result has two dimensions

Store the final verdict and the attempt history separately:

Attempts	Final result	Reliability signal
Pass	Pass	Stable in this run
Fail, Pass	Pass	Flaky evidence
Fail, Fail, Fail	Fail	Consistent failure
Pass, Fail	Depends on strategy	Non-repeatable behavior

A pipeline may reasonably allow Fail, Pass while a team investigates. It should not classify it as equivalent to Pass.

The first failed attempt preserves evidence about the unstable boundary: an element appeared late, a request escaped the stub, state leaked from a previous test, or a backend was unavailable.

Choose retry semantics explicitly

Standard Cypress retries stop when a test passes:

import { defineConfig } from "cypress";

export default defineConfig({
  retries: {
    runMode: 2,
    openMode: 0,
  },
});

That is an availability-oriented policy: give the test more chances before failing the build.

For a high-signal suite, Cypress also provides experimental retry strategies. This configuration fails the final result if any attempt failed, while stopping as soon as a later pass proves flakiness:

export default defineConfig({
  retries: {
    experimentalStrategy: "detect-flake-but-always-fail",
    experimentalOptions: {
      maxRetries: 2,
      stopIfAnyPassed: true,
    },
    openMode: true,
    runMode: true,
  },
});

That policy is intentionally strict. Use it for release-critical journeys where intermittent behavior is itself a failure. Experimental retries are global, use boolean openMode and runMode, and can change in future Cypress releases, so pin and review the version.

Preserve the failed attempt

Cypress gives failed retry screenshots attempt-number suffixes. If you use Cypress Cloud, retry history, screenshots, video, and failure details are available in the test result history.

For local CI artifacts, retain evidence when any attempt failed—not only when the final verdict failed. In the Node event lifecycle, each test result contains attempts:

function hadFailedAttempt(results: CypressCommandLine.CypressRunResult) {
  if (results.status !== "finished") return true;

  return results.runs.some((run) =>
    run.tests.some((test) =>
      test.attempts.some((attempt) => attempt.state === "failed"),
    ),
  );
}

The exact reporting integration depends on your runner, but the policy should be clear: a flaky pass may unblock the job, yet it must still create a triage record with its failed-attempt artifacts.

Diagnose the class of flake

Do not begin by increasing the retry count. Classify the failure first.

Application race

The UI exposes an intermediate state as actionable. Fix the application state model or wait on a user-visible invariant.

Test race

The test uses arbitrary sleeps or reads before the page is ready. Replace time guesses with retryable queries and assertions.

State leakage

The outcome depends on order, cached data, or an uncleared session. Run the spec repeatedly in isolation and in shuffled suite order.

Dependency instability

A real network or service participates in a test intended to be deterministic. Stub it, or move the test to a separately classified integration suite.

Genuine environmental noise

Occasional browser or infrastructure failures do happen. Track their rate so “temporary” does not become permanent background failure.

Treat flake rate as owned work

Use a rolling window, not one alarming run:

flake rate = tests with mixed attempts / tests executed

Segment by spec, browser, operating system, and failure signature. Do not put unbounded error text into metric labels; keep detailed evidence in logs and artifacts.

Set an ownership rule: quarantine only with an issue, an owner, an expiry, and a reason. A quarantined test without an exit condition is a deleted test with extra steps.

Give flakiness its own budget

A release pipeline can separate product failure from suite instability without ignoring either:

type ReliabilityGate = {
  failedTests: number;
  flakyTests: number;
  maxFlakyTests: number;
};

function mayRelease(gate: ReliabilityGate) {
  return gate.failedTests === 0 && gate.flakyTests <= gate.maxFlakyTests;
}

That example is a policy shape, not a recommended threshold. A payment or authentication journey may allow zero flaky tests; a lower-risk suite may temporarily tolerate a small, owned backlog. Tighten the budget as causes are removed, and fail immediately when a known critical spec exhibits mixed attempts.

Trend the first-failure signature separately from the final verdict. If ten unrelated tests start failing first on the same network wait, the shared dependency—not ten selectors—is probably the better investigation target.

Retries answer a narrow question: can this test pass on another attempt? They do not answer whether the application is reliable. Preserve the first failure, choose the final-status policy deliberately, and make flaky green different from stable green.

References

Testing Streaming AI Interfaces with Cypress Without Asserting Every Token

Raju Dandigam — Fri, 18 Sep 2026 16:02:27 +0000

A streaming AI test becomes brittle the moment it expects this exact sequence:

Hel → Hello → Hello, I → Hello, I can help

Chunk boundaries are transport details. They can change with buffering, provider behavior, network timing, or a client-library upgrade while the visible result remains correct.

The UI still has deterministic responsibilities. It must show progress, distinguish a tool call from ordinary text, ignore stale events, support cancellation, and end in a coherent state. Test those contracts instead of every token.

Model the stream as events

Do not let the component treat an incoming byte stream as an undocumented collection of callbacks. Normalize it first:

type AgentEvent =
  | { type: "started"; runId: string }
  | { type: "text_delta"; runId: string; text: string }
  | { type: "tool_started"; runId: string; name: string }
  | { type: "tool_completed"; runId: string; name: string }
  | { type: "completed"; runId: string }
  | { type: "failed"; runId: string; code: string };

The renderer can derive a small set of user-visible states:

idle → connecting → streaming → using_tool → streaming → complete
                                  └──────────────→ failed

That state machine is the stable testing surface. “Two chunks arrived instead of three” usually is not.

Inject a controllable stream in tests

For component or end-to-end tests, put the transport behind an interface. In test builds, expose a synthetic event source:

declare global {
  interface Window {
    agentTestStream?: { emit(event: AgentEvent): void };
  }
}

Now Cypress controls causality without calling a model:

it("shows a tool phase and completes the answer", () => {
  cy.visit("/assistant?stream=test");
  cy.get("[data-testid=ask]").click();

  cy.window().then(({ agentTestStream }) => {
    agentTestStream!.emit({ type: "started", runId: "run-1" });
    agentTestStream!.emit({
      type: "text_delta",
      runId: "run-1",
      text: "Checking the policy. ",
    });
    agentTestStream!.emit({
      type: "tool_started",
      runId: "run-1",
      name: "search_policy",
    });
  });

  cy.get("[data-testid=status]").should("contain", "Searching policy");

  cy.window().then(({ agentTestStream }) => {
    agentTestStream!.emit({
      type: "tool_completed",
      runId: "run-1",
      name: "search_policy",
    });
    agentTestStream!.emit({
      type: "text_delta",
      runId: "run-1",
      text: "Returns are accepted within 30 days.",
    });
    agentTestStream!.emit({ type: "completed", runId: "run-1" });
  });

  cy.get("[data-testid=answer]")
    .should("contain", "Checking the policy")
    .and("contain", "30 days");
  cy.get("[data-testid=status]").should("contain", "Complete");
});

This verifies progressive rendering and the final semantic landmarks. It does not care how the provider would divide the sentence.

Test the failures streaming makes easy to miss

The highest-value cases are usually not the happy path.

A stale run writes into a newer answer

Start run-1, then start run-2. Emit a late delta from run-1 and assert that it is ignored. Every event should carry a run identifier, and the renderer should accept only the active one.

Cancellation changes the terminal state

After the user clicks Stop, emit another delta. The UI must remain cancelled and must not append the text. Also assert that the transport's abort function was called.

A tool fails after text has appeared

Partial prose should not leave the interface looking complete. Emit tool_started, then failed, and assert that the error is visible while the incomplete answer is clearly marked.

Duplicate events arrive

Reconnects can replay events. Give events stable IDs or sequence numbers and assert that a duplicate delta is not rendered twice.

The stream ends without a terminal event

Close the transport after several deltas but before completed or failed. The UI should move to an explicit interrupted state instead of presenting partial prose as a finished answer. This catches a class of bugs that an HTTP status assertion cannot see.

Use network interception at the right layer

cy.intercept() is useful for controlling the initial request, authentication errors, HTTP status, and response delay. It is less useful when the test needs precise control over a long series of application-level stream events. An injected adapter keeps the browser behavior real while making the event schedule deterministic.

Keep one integration test against the actual streaming endpoint to verify framing and parsing. Keep most UI state tests provider-free and synthetic. That split makes failures easier to classify: protocol problem or rendering problem.

At the protocol layer, include fixtures where one logical event is split across network chunks and several events arrive in one chunk. Your parser must reconstruct frames before the UI reducer sees them. The component test can then remain deliberately unaware of TCP, SSE, or fetch buffering.

Assert responsibilities, not prose

A robust streaming test usually checks:

the visible state transition;
ordering around tool activity;
cancellation and stale-event handling;
accessibility announcements;
the final structured outcome or a few semantic landmarks;
absence of duplicated content.

It should rarely check every intermediate string.

For accessibility, assert that status changes are announced once and that rapidly arriving deltas do not flood a live region. The visual answer may update continuously while assistive technology receives milestone events such as “searching,” “approval required,” and “complete.”

Streaming interfaces are asynchronous state machines wearing a chat UI. Once tests target that state machine, they become both stricter about real bugs and less sensitive to irrelevant token timing.

References

Build an Agent UI That Explains Its State with Angular Signals

Raju Dandigam — Fri, 18 Sep 2026 00:09:49 +0000

Most chat interfaces have three states: empty, loading, and finished. Agent workflows have many more.

An agent may be planning, waiting for a tool, requesting approval, retrying a recoverable error, or suppressing an unsafe action. If the UI represents all of those as one spinner, users cannot tell whether the system is working, blocked, or about to change something.

The better model is not “messages plus loading.” It is an explicit state machine.

Define states the product can explain

Start with observable runtime states—not the model's hidden reasoning:

type AgentUiState =
  | { kind: "idle" }
  | { kind: "planning" }
  | { kind: "using_tool"; tool: string }
  | { kind: "waiting_for_approval"; proposalId: string }
  | { kind: "recovering"; attempt: number }
  | { kind: "blocked"; reasonCode: string }
  | { kind: "completed"; outcome: string }
  | { kind: "failed"; message: string };

This vocabulary should come from actual runtime events. Do not fabricate a “thinking” narrative that implies access to private chain-of-thought.

Store facts; compute presentation

Angular signals work well when the event stream is the source of truth and presentation is derived from it:

import { computed, signal } from "@angular/core";

type AgentEvent = {
  type: "run_started" | "tool_started" | "approval_required" |
        "retry_started" | "run_completed" | "run_failed";
  tool?: string;
  proposalId?: string;
  attempt?: number;
  outcome?: string;
  message?: string;
};

const events = signal<AgentEvent[]>([]);

const state = computed<AgentUiState>(() => {
  const event = events().at(-1);
  if (!event) return { kind: "idle" };

  switch (event.type) {
    case "run_started": return { kind: "planning" };
    case "tool_started":
      return { kind: "using_tool", tool: event.tool! };
    case "approval_required":
      return { kind: "waiting_for_approval", proposalId: event.proposalId! };
    case "retry_started":
      return { kind: "recovering", attempt: event.attempt! };
    case "run_completed":
      return { kind: "completed", outcome: event.outcome! };
    case "run_failed":
      return { kind: "failed", message: event.message! };
  }
});

Angular's signals guide recommends computed() for derived state and warns against using effects to propagate state changes. That distinction matters here. The event list is state; the current label, available actions, and accessibility message are derivations.

Use effect() only for a real side effect such as analytics or persistence—and keep it independent from the state transition itself.

Treat approval as a real state

Approval is not a modal layered over “loading.” It pauses one proposal and creates a new user decision.

@switch (state().kind) {
  @case ('using_tool') {
    <p aria-live="polite">Using {{ state().tool }}</p>
  }
  @case ('waiting_for_approval') {
    <app-action-review
      [proposalId]="state().proposalId"
      (approved)="approve($event)"
      (rejected)="reject($event)" />
  }
  @case ('recovering') {
    <p aria-live="polite">Recovering, attempt {{ state().attempt }}</p>
  }
  @case ('blocked') {
    <p role="alert">Action blocked: {{ state().reasonCode }}</p>
  }
}

The review component should show the proposed tool, bounded arguments, evidence freshness, and scope of approval. “Continue?” is not enough for a consequential action.

Prevent stale streams from winning

A common UI bug occurs when request A is cancelled, request B starts, and a late event from A overwrites B's state. Give each event a run ID and ignore events for inactive runs:

const activeRunId = signal<string | null>(null);

function acceptEvent(runId: string, event: AgentEvent) {
  if (runId !== activeRunId()) return;
  events.update((current) => [...current, event]);
}

Cancellation should be a runtime operation as well as a visual one. Removing a spinner does not stop a network request or tool call.

For reconnectable streams, add an event ID or monotonically increasing sequence number. Ignore duplicates, detect gaps, and request a snapshot when the UI cannot safely reconstruct state. A signal will faithfully render bad ordering if the transport contract never defined ordering.

Use immutable updates such as events.update(...). Angular's readonly signal surface does not prevent deep mutation of an object or array, and mutating a retained value in place can make state changes harder to reason about.

Make the state accessible and actionable

Expose aria-busy="true" only while work is genuinely progressing. An approval state is not busy: focus should move to the review controls, the proposed effect should be described, and Approve and Reject should remain keyboard accessible. Announce concise state changes through an aria-live region, but do not stream every token into it.

Test transitions, not animation timing

Component tests can drive events deterministically:

it("shows approval after a tool proposal", () => {
  mount(AgentPanelComponent);

  emit({ type: "run_started" });
  emit({
    type: "approval_required",
    proposalId: "synthetic-proposal",
  });

  cy.findByRole("button", { name: /approve/i }).should("be.visible");
  cy.findByText(/synthetic-proposal/i).should("exist");
});

Cypress documents current Angular component-testing support in its Angular guide. Keep model and tool calls stubbed for these state tests; use a smaller number of integration tests for the real event protocol.

Explain what the system knows

An agent UI should answer three questions:

What observable stage is active?
Does the user need to act?
What outcome was verified?

Angular signals make those answers easy to derive from a structured event stream. The hard part is defining honest states. Do that first, and the UI stops being a chat box with a spinner—it becomes a trustworthy view of the workflow.

References

Structured Output Is an API: Evolving Gemini Schemas Without Breaking Your Agent

Raju Dandigam — Thu, 17 Sep 2026 16:58:46 +0000

Structured output solves an important model problem: instead of asking Gemini for “JSON-like” prose, you provide a schema and receive syntactically structured data.

Then the software-engineering problem begins.

Once a tool, UI, downstream agent, database, or audit process depends on that JSON, the model output is an API. Renaming a field can break consumers. Adding an enum value can send an older UI into an impossible state. Valid JSON can still describe an invalid business decision.

Version the contract inside the data

Suppose a travel assistant initially returns:

type TravelDecisionV1 = {
  schemaVersion: "1";
  action: "book" | "ask" | "wait";
  reason: string;
};

Later, the product needs suppression and evidence references. Quietly changing the original shape is risky. Introduce a new discriminated version:

type TravelDecisionV2 = {
  schemaVersion: "2";
  decision: "act" | "ask" | "wait" | "suppress";
  reasonCode: string;
  evidenceIds: string[];
  confidence?: {
    value: number;
    source: "classifier" | "model" | "rule";
  };
};

type TravelDecision = TravelDecisionV1 | TravelDecisionV2;

The version belongs in the persisted value, not only in the deployment configuration. Stored outputs can outlive the model call that created them.

Generate against a schema—and validate again

Gemini supports structured output through a provided JSON Schema. Its JavaScript SDK can use a JSON Schema together with Zod validation. The abbreviated example below keeps the model identifier in configuration:

import { GoogleGenAI } from "@google/genai";
import * as z from "zod";

const decisionSchema = z.object({
  schemaVersion: z.literal("2"),
  decision: z.enum(["act", "ask", "wait", "suppress"]),
  reasonCode: z.string().min(1),
  evidenceIds: z.array(z.string()),
});

const jsonSchema = z.toJSONSchema(decisionSchema);
const client = new GoogleGenAI({});

const response = await client.interactions.create({
  model: process.env.GEMINI_MODEL!,
  input: "Evaluate the synthetic travel-change fixture.",
  response_format: {
    type: "text",
    mime_type: "application/json",
    schema: jsonSchema,
  },
});

const decision = decisionSchema.parse(
  JSON.parse(response.output_text),
);

The Gemini structured-output documentation explicitly recommends application validation. Gemini implements a subset of JSON Schema, and very large or deeply nested schemas may be rejected.

For a public contract, make the schema as closed and bounded as the supported Gemini subset allows. Limit array sizes, constrain string formats where meaningful, and reject unexpected properties at the application boundary. A permissive schema followed by a strict consumer only moves the failure downstream.

Separate four kinds of validity

“The response was valid” can mean several different things:

Check	Example
Syntax	The response parses as JSON
Shape	Required properties and enum values match the schema
Semantics	Referenced evidence IDs exist
Policy	`act` is permitted for this user and current state

Structured generation helps primarily with the first two. Your application still owns the last two.

function validateSemantics(d: TravelDecisionV2, knownIds: Set<string>) {
  if (d.decision === "act" && d.evidenceIds.length === 0) {
    throw new Error("ACT_REQUIRES_EVIDENCE");
  }

  if (d.evidenceIds.some((id) => !knownIds.has(id))) {
    throw new Error("UNKNOWN_EVIDENCE_REFERENCE");
  }
}

Do not retry every validation failure with “please fix your JSON.” A syntax failure, an unsupported enum, and a policy violation require different handling. A policy violation should normally be blocked, not reprompted until the model finds a permissive answer.

Prefer additive evolution

Changes are usually safer when they are additive:

add an optional field;
introduce a new version before removing the old one;
keep readers tolerant of known historical versions;
migrate persisted records explicitly;
reject unknown versions rather than guessing.

Enum additions deserve special care. TypeScript consumers often use exhaustive switches:

function renderDecision(d: TravelDecisionV2) {
  switch (d.decision) {
    case "act": return "Executing";
    case "ask": return "Needs information";
    case "wait": return "Monitoring";
    case "suppress": return "No action";
    default: return "Unsupported decision";
  }
}

That fallback protects the UI, but it should also emit a bounded compatibility error.

Contract-test model and schema changes

For each supported model configuration, test:

a normal V2 result;
every enum branch;
missing evidence;
an old persisted V1 record;
an unknown future version;
schema-compliant but policy-invalid data.

Do not assert exact prose in reasonCode explanations. Assert the stable schema, references, and business constraints.

Persist a small generation envelope beside the output:

type GenerationEnvelope = {
  schemaVersion: "2";
  schemaHash: string;
  modelId: string;
  generatedAt: string;
  value: TravelDecisionV2;
};

schemaHash is not a replacement for the human-readable version. It detects drift when two deployments both claim to produce V2 but were built from different schema documents. The model identifier supports diagnosis; it should not become the only migration key.

JSON is the start of the contract

Structured output makes agent integration substantially cleaner, but it does not remove API design. Version the value, validate it at runtime, distinguish semantic and policy failures, and preserve migration tests.

If another component can break when the model's JSON changes, you already have an API—even if there is no HTTP endpoint.

References

Give Your Coding Agent Eyes: Debug Cypress Through a Live Browser Session

Raju Dandigam — Wed, 16 Sep 2026 17:31:00 +0000

A coding agent can run cypress run and read the exit code. That tells it whether a spec passed. It does not show the Command Log, the DOM at the failed command, the accessibility tree, or the browser state a human sees in Cypress open mode.

This gap matters when the failure is visual or stateful. “Element not found” could mean the selector is wrong, the page never loaded, an overlay covered the control, or the test inspected the wrong retry attempt.

cypress tap adds a terminal interface to a live Cypress open-mode session. The interesting part is not that an AI can execute another command. It is that the agent can inspect the same evidence a developer would use before proposing a fix.

The loop is run, wait, inspect, then edit

Start Cypress and select a Chromium-based browser:

npx cypress open --e2e --browser=chrome

In another terminal:

npx cypress tap specs --json
npx cypress tap run cypress/e2e/login.cy.ts --json
npx cypress tap status --json

The run command requests a run and returns immediately. It does not mean the spec has started or finished. A reliable agent must poll status, set its own timeout, and wait for a terminal passed or failed state.

There is a subtle stale-state trap: the previous run's verdict remains readable until the next run starts. The JSON status includes startedAt; compare it with the run you requested before trusting the verdict.

That produces a bounded protocol:

type TapStatus = {
  status:
    | "not connected"
    | "browser not selected"
    | "spec not selected"
    | "loading"
    | "running"
    | "passed"
    | "failed";
  startedAt: string | null;
};

async function waitForVerdict(requestedAfter: number) {
  for (let attempt = 0; attempt < 60; attempt++) {
    const status = await runJson<TapStatus>("npx cypress tap status --json");
    const started = status.startedAt ? Date.parse(status.startedAt) : 0;

    if (started >= requestedAfter &&
        (status.status === "passed" || status.status === "failed")) {
      return status;
    }

    await delay(500);
  }

  throw new Error("Cypress session did not produce a fresh verdict");
}

The timeout is part of the correctness model. A build failure can leave the session in loading, so “poll until success” can otherwise become an agent loop.

There is another important contract: cypress tap status exits successfully for a determinable stage, including a reported test failure. Automation must branch on the JSON status, not treat shell exit code zero as a passing spec. Prefer --json, pin Cypress in the lockfile, and let the agent inspect cypress tap --help so its command assumptions match the attached session.

Inspect the failing attempt, not only the latest screen

Once the run fails, get the reporter view:

npx cypress tap reporter --json
npx cypress tap reporter --test-id r3 --attempt 1 --json

The test-level report includes routes, hooks, the Command Log, and failure details. If retries occurred, inspect the failed attempt instead of assuming the final attempt represents the original failure.

Then narrow the investigation:

npx cypress tap command \
  --test-id r3 \
  --command-id 7 \
  --attempt 1 \
  --json

npx cypress tap pin \
  --test-id r3 \
  --command-id 7 \
  --attempt 1 \
  --at after

npx cypress tap aria --selector "main"
npx cypress tap inspect --selector "[data-testid=submit]"

pin restores a command snapshot into the app-under-test frame. dom, aria, and inspect can then read that historical state. This is far stronger evidence than asking an agent to infer browser state from an error string.

Put limits around the agent

Browser access should not become an unbounded “fix until green” loop. I use a small policy:

Run one named spec.
Poll for a fresh verdict with a deadline.
Inspect only the failing test and relevant command.
Allow one focused code change.
Rerun the same spec.
Stop and request human review if the second run fails differently.

The agent should also report what it observed and why the proposed edit addresses that evidence. A green rerun is necessary, but it does not prove the change preserved the intended assertion.

Know the current boundary

cypress tap is beta and requires Cypress 15.21.0 or later. It works with cypress open, not headless cypress run, and currently requires a Chromium-based browser. Its commands and output may still change between releases.

Beta also changes the evidence contract. Store the Cypress version, browser, testing type, session identifier, spec path, attempt, and startedAt beside any captured report. A JSON field that changes in a later release should fail your adapter clearly instead of being interpreted as a new test result.

It also exposes page structure to the terminal. Treat DOM text, routes, and console properties as potentially sensitive, especially when a coding agent sends context to a remote model.

The useful mental model is simple: the agent is not replacing the test runner or the reviewer. It is gaining a structured window into the live evidence between “failed” and “I know why.”

References

Your Agent Eval Set Is Rotting: Build a Failure-Mining Loop for Google ADK

Raju Dandigam — Tue, 15 Sep 2026 19:18:29 +0000

An agent evaluation set starts healthy. It contains the obvious intents, a few tool failures, and the happy paths used during development.

Six months later, production has changed. New tools exist. Users phrase requests differently. A fallback introduced last quarter now handles 30% of traffic. Yet CI still runs the same twelve examples and reports green.

The problem is not only stale prompts. The eval set itself is rotting.

A durable evaluation program needs a controlled path from observed failures back into tests:

observed run -> candidate -> human review -> sanitization
             -> eval case -> CI -> release comparison

First, define a failure candidate

Do not copy complete production conversations into a test folder. Create a bounded intermediate record:

type FailureCandidate = {
  candidateId: string;
  detectedAt: string;
  intent: string;
  terminalOutcome: "failed" | "escalated" | "wrong_outcome";
  firstFailureRole?: "model" | "tool" | "handoff" | "policy";
  toolSequence: string[];
  reasonCode: string;
  sourceReference: string; // restricted internal pointer
};

This record is for triage. It is not automatically an eval case, and it should not contain raw customer content unless your approved process requires it.

Mine patterns, not anecdotes

The loudest incident is not necessarily the most representative. Select candidates across several dimensions:

intent and workflow;
tool and failure role;
successful recovery versus escalation;
common and rare trajectory shapes;
model, prompt, policy, and dataset version;
outcome severity.

A simple trajectory signature helps deduplicate repeated symptoms:

import { createHash } from "node:crypto";

function signature(c: FailureCandidate): string {
  return createHash("sha256")
    .update([
      c.intent,
      c.terminalOutcome,
      c.reasonCode,
      c.toolSequence.join(">"),
    ].join("|"))
    .digest("hex");
}

The hash does not prove two failures have the same cause. It creates a grouping candidate for human review. Keep a few examples from a large cluster, but do not allow one outage to dominate the suite.

Convert only after review

A useful promotion checklist is:

confirm the failure is real rather than an instrumentation or harness defect;
identify the smallest reproducible input and environment state;
remove personal, proprietary, and transient data;
label the expected tool behavior and business outcome;
record why the case deserves long-term coverage;
assign an owner and review date.

The resulting case should be synthetic or safely transformed:

{
  "caseId": "refund-policy-stale-cache",
  "input": "Can this synthetic order still be refunded?",
  "fixture": "order-after-policy-window.json",
  "expected": {
    "requiredTools": ["lookup_order", "lookup_policy"],
    "forbiddenTools": ["issue_refund"],
    "outcome": "ask_or_decline"
  }
}

Evaluate more than the final answer

Google's Agents CLI evaluation model distinguishes dimensions such as tool-use quality, multi-turn tool use, trajectory quality, task success, and final-response quality. That separation is important: a fluent answer can follow the wrong path, while an awkward answer may have used tools correctly. See the official evaluation guide.

Use deterministic checks for stable facts:

a protected tool was absent;
authorization preceded execution;
a handoff included required fields;
the run completed within a controlled call budget.

Use semantic graders for questions that genuinely require judgment, such as helpfulness or groundedness. Keep their reports separate so engineers know whether CI found a structural violation or a probabilistic quality regression.

Where local traces help

I maintain AgentInspect; it is one concrete implementation of this broader pattern. With agent-inspect@6.19.0, a TypeScript team can turn supported local traces into bounded facts and deterministic trajectory checks. It is not a production sampling or dataset-management platform. Production selection, privacy review, and ADK evaluation remain separate responsibilities.

That boundary is useful: the same failure can move through distinct trust zones instead of silently becoming training or test data.

Add retirement, not only promotion

Eval suites also need deletion rules. Retire or revise a case when:

its feature no longer exists;
its fixture depends on an obsolete schema;
several cases test the same invariant;
the original expected behavior is no longer correct.

Track suite composition by intent and failure role. A growing case count is not evidence of improving coverage.

Measure the health of the suite itself

Give each case provenance and a review clock:

type EvalCaseMetadata = {
  caseId: string;
  promotedFrom: string;
  owner: string;
  addedAt: string;
  reviewAfter: string;
  intent: string;
  firstFailureRole: "model" | "tool" | "handoff" | "policy";
};

A small dashboard can then expose coverage by intent, age, failure role, and last meaningful failure. Flag cases that have not been reviewed after a tool-schema or policy revision. The point is not to delete old cases automatically; it is to make stale expectations visible before they become release authority.

Watch the promotion funnel as well: candidates detected, candidates reviewed, cases accepted, duplicates rejected, and cases retired. A sudden drop between detection and review often signals an ownership problem rather than an agent-quality improvement.

Close the loop deliberately

The goal is not to replay production. It is to convert reviewed failures into small, safe, owned regression cases. Start with the last ten meaningful incidents. Group them, select representatives, sanitize them, and label both trajectory and outcome expectations.

An eval set stays useful when production teaches it—through a controlled review boundary, not an automatic data pipe.

References

Google ADK Callbacks Are a Policy Plane, Not Just Hooks

Raju Dandigam — Mon, 14 Sep 2026 17:43:33 +0000

An agent proposes refund_order. The tool exists, its arguments are valid JSON, and the model sounds confident. None of that answers the question that matters: is this action allowed now?

Google's Agent Development Kit gives us interception points before and after agents, models, and tools. The tempting implementation is to add a log statement to each callback. The more useful design is to treat those callbacks as a small policy plane.

That does not mean placing every business rule in a callback. It means using the boundary to make four decisions explicit:

may this operation proceed?
should its inputs be normalized?
which budget or authorization applies?
what evidence should be recorded?

A callback can change control flow

ADK callbacks are not passive listeners. A beforeModelCallback can return a response and skip the model call. A beforeToolCallback can return a tool result and skip the tool. After-callbacks can replace results. That makes callback return values part of application behavior, not merely observability. The ADK callback documentation explains these interception semantics for TypeScript and the other supported SDKs.

Callback order is part of that behavior. ADK can run callback lists and plugin callbacks in sequence, with earlier callbacks able to modify inputs seen by later ones or short-circuit the remaining chain. Treat the registered order as reviewed configuration: authentication and tenant resolution should not accidentally run after a callback that can approve or execute an operation.

Start with a small decision vocabulary:

type PolicyDecision = {
  outcome: "allow" | "block";
  reasonCode: string;
  policyVersion: string;
};

function authorizeTool(
  toolName: string,
  args: Record<string, unknown>,
): PolicyDecision {
  if (toolName === "refund_order" && args.approved !== true) {
    return {
      outcome: "block",
      reasonCode: "APPROVAL_REQUIRED",
      policyVersion: "refund-policy-3",
    };
  }

  return {
    outcome: "allow",
    reasonCode: "POLICY_SATISFIED",
    policyVersion: "refund-policy-3",
  };
}

This decision is deterministic and reviewable. It does not ask the model whether its own proposed action is safe.

Keep the callback thin

The callback should delegate to policy code rather than growing into an untestable collection of if statements. The following is deliberately schematic because exact tool callback types can change between ADK releases:

async function beforeTool({ tool, args }: {
  tool: { name: string };
  args: Record<string, unknown>;
}) {
  const decision = authorizeTool(tool.name, args);

  await policyEvidence.write({
    tool: tool.name,
    outcome: decision.outcome,
    reasonCode: decision.reasonCode,
    policyVersion: decision.policyVersion,
  });

  if (decision.outcome === "block") {
    return {
      error: "Tool execution blocked by policy",
      reasonCode: decision.reasonCode,
    };
  }

  return undefined; // continue with normal tool execution
}

Register that function through the agent's beforeToolCallback. Use beforeModelCallback for model-input limits or serving an approved cache entry, and afterToolCallback for result normalization and outcome evidence.

The boundary becomes easier to review when responsibilities stay distinct:

Boundary	Suitable responsibility
Before model	Input limits, budget checks, approved cache lookup
Before tool	Authorization, argument validation, freshness checks
After tool	Result normalization, outcome classification
After agent	Final evidence and response-level validation

Test absence, not only the error message

A guardrail test is incomplete if it checks only the returned text. The callback might return “blocked” after the real tool already ran.

it("does not execute a refund without approval", async () => {
  const refund = vi.fn();
  const decision = authorizeTool("refund_order", {
    orderId: "synthetic-123",
    approved: false,
  });

  if (decision.outcome === "allow") await refund();

  expect(decision.reasonCode).toBe("APPROVAL_REQUIRED");
  expect(refund).not.toHaveBeenCalled();
});

At integration level, preserve evidence that the policy callback ran and the protected tool did not. That is stronger than matching a model-generated refusal sentence.

Also test the installed callback chain, not only the pure policy function. A unit test can prove authorizeTool() works while one runner quietly omits the plugin. A useful integration assertion records the callback name, policy version, decision, proposal ID, and whether execution began. For a blocked proposal, tool.execution.started must be absent.

Callbacks are not the entire security architecture

A callback runs inside the application process. It can contain a bug, be omitted from another agent, or be registered in the wrong order. High-risk authorization should also be enforced at the tool or service boundary. The application still needs least-privilege credentials, idempotency, audit records, and tests.

Google explicitly recommends ADK Plugins for modular security guardrails instead of scattering individual callbacks across agents. That is the right graduation path when the same policy must apply consistently to several agents.

Plugins still run inside your application. They improve reuse and consistency; they do not replace authorization at the resource server. Think of the callback or plugin as the orchestration decision point and the downstream service as the final enforcement point.

Make policy visible

Callbacks are most valuable when they expose a control point that already exists conceptually. Give each decision a stable reason code and policy version. Record bounded facts, not raw prompts or hidden reasoning. Test both the allowed path and the absence of a forbidden side effect.

Used that way, an ADK callback is more than a hook. It is where an agent proposal becomes an application decision.

References

Prompts Are Code. Genkit Makes the Runtime Reviewable.

Raju Dandigam — Sun, 13 Sep 2026 16:40:15 +0000

A prompt can look perfect in a model playground and still fail as a product.

The production input arrives in a different shape. Authentication data leaks into the prompt. The model returns prose where the UI expects JSON. A three-line wording change improves two examples and breaks eight others. Nobody can explain which version generated yesterday’s result.

The lesson is not that prompts are unimportant.

It is that a prompt is one artifact inside a larger runtime.

Genkit is useful for TypeScript teams because it connects prompts, flows, schemas, tools, context, traces, and evaluations in one application model. The result is not automatically reliable—but the important boundaries become reviewable.

Series note: This is Part 8 of Reliable Google AI Agents in TypeScript. The examples were checked against Genkit 1.42.0 and @genkit-ai/google-genai 1.42.0 in September 2026.

Put the prompt in a versioned artifact

Genkit’s Dotprompt format stores template content, model configuration, schemas, and tool-loop settings in .prompt files.

---
model: googleai/gemini-flash-latest
config:
  temperature: 0.3
input:
  schema:
    hotelName: string
    oldPrice: number
    newPrice: number
output:
  schema:
    title: "string"
    body: string
---
Write a concise price-drop alert for {{hotelName}}.
The price changed from {{oldPrice}} to {{newPrice}}.
Do not invent amenities, policies, or availability.

Keeping this in prompts/price-alert.prompt produces several practical benefits:

prompt changes appear in Git review;
model and generation settings travel with the template;
input and output expectations are visible;
the Developer UI can iterate on the prompt without burying it in service code;
variants can be evaluated against the same examples.

But a .prompt file should not own authorization, retries, billing rules, or irreversible side effects.

Wrap the prompt in a typed flow

A Genkit flow creates the application boundary around generation. It has a stable name, validated input and output, trace identity, and a deployment surface.

import { genkit, z } from "genkit";
import { googleAI } from "@genkit-ai/google-genai";

const ai = genkit({
  plugins: [googleAI()],
});

const AlertInput = z.object({
  hotelName: z.string().min(1),
  oldPrice: z.number().positive(),
  newPrice: z.number().positive(),
});

const AlertCopy = z.object({
  title: z.string().min(1),
  body: z.string().min(1),
});

const AlertOutput = AlertCopy.extend({
  shouldSend: z.boolean(),
});

const priceAlertPrompt = ai.prompt("price-alert");

export const priceAlertFlow = ai.defineFlow(
  {
    name: "priceAlertFlow",
    inputSchema: AlertInput,
    outputSchema: AlertOutput,
  },
  async (input) => {
    if (input.newPrice >= input.oldPrice) {
      return { title: "", body: "", shouldSend: false };
    }

    const { output } = await priceAlertPrompt(input);
    const copy = AlertCopy.parse(output);

    return { ...copy, shouldSend: true };
  },
);

The price rule runs before the model. Zod validates the generated structure after it. Gemini is responsible for bounded language generation—not for redefining whether a price actually dropped.

That separation is the difference between a prompt demo and an application workflow.

Keep execution context out of model input

AI applications often have data the code needs but the model does not.

An auth token may be needed to call a user-scoped service. A tenant ID may be required for a database query. Neither should be interpolated into the prompt merely because a tool needs it.

Genkit’s context object provides a side channel that propagates through flows, prompts, and tools.

const TripSummary = z.object({
  tripId: z.string(),
  destination: z.string(),
});

const searchUserTrips = ai.defineTool(
  {
    name: "searchUserTrips",
    description: "Search trips owned by the signed-in user",
    inputSchema: z.object({ query: z.string() }),
    outputSchema: z.array(TripSummary),
  },
  async ({ query }, { context }) => {
    const userId = context.auth?.uid;

    if (!userId) {
      throw new Error("Authentication required");
    }

    return tripStore.search({ userId, query });
  },
);

The model can decide to call searchUserTrips. The tool derives ownership from trusted execution context rather than a model-generated user ID.

That is a small design decision with a large security effect.

Context is still not automatically trusted. The HTTP or function binding that invokes the flow must authenticate the request and populate context.auth. Genkit propagates the object; your boundary establishes its integrity.

Treat model output as untrusted data

An output schema improves the boundary, but structured output is not business authorization.

For example, the model could generate a valid alert body that mentions an unavailable amenity. Or it could return a valid title for a user who opted out of notifications.

Use different controls for different failure classes:

Failure	Appropriate control
Wrong field type	Zod/output schema
User opted out	Deterministic policy before generation
Unsupported factual claim	Grounding plus evidence validation
Poor tone	Evaluator or human review
Duplicate delivery	Idempotent side-effect boundary

A schema answers “Can downstream code safely parse this?” It does not answer “Should the business act on it?”

Observe the whole flow

Genkit traces flows, model calls, tools, and intermediate operations in the Developer UI. Its telemetry is based on OpenTelemetry, while production export depends on the telemetry and observability integration you configure.

That creates two complementary views:

The native Genkit trace shows the actual flow, prompt/model operation, tools, latency, and schema failures.
A framework-neutral evidence layer can express a stable cross-framework contract for pull-request review or CI.

For example:

priceAlertFlow
├─ validate_input
├─ business_rule          price_drop=true
├─ prompt.price-alert     version=git:8f3c2a1
├─ validate_output        schema=AlertCopy
└─ return                 shouldSend=true

AgentInspect can complement the native trace when the application maps these important operations into a run and wants deterministic required/forbidden checks or a redacted review artifact. It should not duplicate every span, and this article does not imply a first-class Genkit adapter already exists.

The stable contract is more important than owning every telemetry signal.

Evaluate changes against a dataset

Prompt review by intuition is not enough.

Genkit supports evaluation datasets and evaluators through its tooling. A useful price-alert dataset should include:

[
  {
    "input": {
      "hotelName": "Example Hotel",
      "oldPrice": 240,
      "newPrice": 190
    },
    "reference": {
      "shouldSend": true
    }
  },
  {
    "input": {
      "hotelName": "Example Hotel",
      "oldPrice": 190,
      "newPrice": 240
    },
    "reference": {
      "shouldSend": false
    }
  }
]

The real suite should also include:

equal prices;
missing or malformed input;
unusually large values;
prompt injection inside hotelName;
unsupported amenity claims;
an unauthenticated tool request;
a consent-policy block;
a duplicate delivery attempt.

Some checks should remain deterministic: schema validity, correct shouldSend, no raw auth data, and no notification for a price increase. Other qualities—clarity, tone, or faithfulness to retrieved evidence—may use an evaluator or human review.

A useful evaluation suite does not require every sentence to match. It asks whether the workflow still satisfies its product contract.

Deploy the flow, not a loose prompt

Genkit flows can be served from a Node.js application with the supported Express integration and deployed to a platform such as Cloud Run.

import { startFlowServer } from "@genkit-ai/express";

startFlowServer({
  flows: [priceAlertFlow],
});

The deployed unit is the workflow with its schema, context handling, tools, and telemetry—not a text template floating independently from the code that constrains it.

That matters operationally:

a rollback can restore compatible prompt and code together;
a trace identifies the named flow;
an evaluation compares versions using the same dataset;
a security review can see where trusted context enters;
a release gate can distinguish structural failures from subjective quality changes.

The runtime is the product

“Prompts are code” is a useful starting point because it moves prompt work into version control.

Production engineering has to go further.

The prompt is the instruction artifact. The flow is the application contract. Context carries trusted execution data. Tools provide bounded capabilities. Schemas protect downstream code. Traces explain the run. Evaluations protect behavior as the system changes.

Genkit brings those pieces into one TypeScript development model.

That is the level at which a Gemini application becomes maintainable—not when its prompt sounds impressive, but when its complete workflow can be reviewed, tested, observed, and safely changed.

References

Earlier in the series: Gemini Function Calling Is Not an Agent Runtime · Testing Google ADK TypeScript Agents Without Chasing Sentences · From Local Traces to Production Observability for Google AI Agents

A Screenshot Is Not an Agent Failure Artifact

Raju Dandigam — Sat, 12 Sep 2026 19:13:05 +0000

A screenshot of an execution tree looks useful in a pull request.

It is also missing almost everything a skeptical reviewer should ask:

Which trace produced it?
Was the trace changed after capture?
Which redaction policy was applied?
Did the crop remove a warning?
Can I inspect the failing step without asking the author to reproduce it?

Raw JSONL answers some of those questions and creates another problem: it may contain identifiers, attributes, or bounded previews that should not leave the developer's machine.

The practical handoff unit is neither screenshot nor raw trace. It is a derived, reviewable evidence artifact.

A picture is a view, not the evidence chain

A screenshot preserves pixels. It does not preserve provenance.

Reviewer question	Screenshot	Evidence package
What path ran?	Sometimes	Yes
Which source produced it?	No	Source run IDs and hashes
What was redacted?	No	Redaction report and policy
Did a file change later?	No	Packaged-file hashes
Can I inspect offline?	Only the crop	HTML, JSON, summary, trace copy

This is the model I want instead:

private source trace
        |
        | derive + redact
        v
reviewable package
  ├─ evidence.html
  ├─ summary.md
  ├─ redacted trace
  ├─ check results
  ├─ redaction report
  └─ evidence.json ----> hashes + provenance
                              |
                              v
                       offline verification

The source remains private. The reviewer gets a bounded object with enough context to question the result.

Build a bundle from one run or a session

AgentInspect's Evidence v2 bundle is one implementation of this pattern. These commands were verified against agent-inspect@6.17.6.

Create a package for one local run:

npx agent-inspect bundle <run-id> \
  --dir .agent-inspect \
  --profile share \
  --out ./agent-evidence

Or package a multi-run session with stricter redaction:

npx agent-inspect bundle \
  --session sess-support-042 \
  --dir .agent-inspect \
  --profile strict \
  --out ./agent-evidence

The source traces are read-only. The output contains derived copies and review surfaces such as:

agent-evidence/
├─ evidence.html
├─ evidence.json
├─ trace.html
├─ trace.jsonl
├─ summary.md
├─ metadata.json
├─ check-results.json
├─ redaction-report.json
├─ eval-results.json
├─ performance-summary.json
└─ assets/runs/...

evidence.html is the convenient entry point. evidence.json is what makes the folder inspectable as a package.

What the manifest records

The Evidence v2 manifest separates four facts:

{
  "evidenceFormatVersion": "1.0",
  "generator": {
    "name": "agent-inspect",
    "version": "6.17.6"
  },
  "source": {
    "runIds": ["run_example"],
    "traceSchemaVersions": ["1.0"],
    "sourceHashes": [
      {
        "runId": "run_example",
        "algorithm": "sha256",
        "hash": "..."
      }
    ]
  },
  "policy": {
    "redactionProfile": "share",
    "verificationPolicy": "share"
  },
  "files": [
    {
      "path": "trace.jsonl",
      "sha256": "...",
      "role": "redacted-trace"
    }
  ]
}

The important separation is between sourceHashes and files[].sha256:

source hashes identify the pre-redaction bytes used to derive the artifact;
file hashes protect the exact bytes inside the finalized package.

The source hash does not reveal the redacted content. It also does not preserve the original for you; retention remains a separate policy decision.

Verify before review

Run the offline verifier:

npx agent-inspect bundle verify ./agent-evidence

It checks:

the Evidence manifest shape;
required listed files;
SHA-256 hashes for packaged files;
unexpected files, which fail by default;
the recorded assessment; and
generator and source provenance.

If someone edits summary.md, replaces the redacted trace, or drops an unlisted file into the finalized directory, verification should stop passing.

That makes a precise claim: the finalized package is internally consistent with its manifest.

Imagine a reviewer receives two folders named agent-evidence. One contains the original bundle. In the other, someone corrected a sentence in summary.md after generation because the explanation was unclear. The edit may be innocent, but the artifact now describes bytes different from the ones recorded in the manifest.

The correct workflow is to fix the source or approved summary input, regenerate the bundle, and verify again. Hash failure does not tell you whether a change was malicious. It tells you that the finalized review object is no longer the same object.

Integrity is not authenticity

This boundary is easy to overstate, so it deserves blunt language.

bundle verify is an integrity check. Evidence v2 is not cryptographically signed by a trusted identity. Anyone who can replace the entire folder could create a different valid bundle.

It is also not:

a live hash-chained write-ahead log;
proof that every event was durable before a side effect;
proof of replayability or exactly-once execution;
nonrepudiation; or
a compliance or audit certification.

If authorship or trusted delivery matters, add signing, access control, and trusted storage in your approved pipeline. Do not rename a local hash check into a stronger security property.

“Share-checked” still means “review it”

The default share profile is intended for pull requests and internal support. strict is a better starting point for public or external sharing.

The bundle command assesses the redacted artifact and refuses UNSAFE or UNKNOWN output unless --allow-unsafe is explicitly supplied. That is a useful fail-closed guard. It is not a guarantee that arbitrary free text contains no sensitive information.

Before attaching a package:

open evidence.html and summary.md;
review redaction-report.json and check-results.json;
run bundle verify;
confirm the destination is approved; and
apply your organization's data-retention and sharing policy.

Capture less in the first place. Redaction should not be the only protection between a raw customer payload and a public issue.

For pull requests, I would also record the verified bundle path or CI artifact name in a small behavior-evidence section. That gives reviewers one canonical object instead of a screenshot in Slack, a trace in local storage, and a summary pasted into the PR with no durable relationship among them.

Better evidence changes the review conversation

A good artifact does not decide whether an agent change is acceptable. It lets everyone inspect the same bounded object:

the first explicit failure;
the observed outcome;
the contract finding;
the sanitized execution path;
the applied policy; and
the package's integrity status.

That moves the discussion from “trust my screenshot” to “here is the evidence, here is what was removed, and here is what the verifier can—and cannot—prove.”

The pinned bundle documentation and Evidence format document the current layout and trust boundaries.

What would your reviewer need before they could safely inspect—and challenge—an agent failure artifact?

One Passing Agent Run Is Not a Release Signal

Raju Dandigam — Fri, 11 Sep 2026 18:58:48 +0000

I can make almost any agent change look good with one carefully chosen prompt.

Run it once. Watch the right tool fire. Read a polished answer. Record the demo.

Ship it?

Not yet.

A single trace is excellent evidence for one execution: what ran, what failed, how long it took, and whether declared invariants held. A release decision asks a different question:

Does this change behave acceptably across the representative cases we care about?

That shift—from run to set—is where agent testing starts to resemble engineering instead of demonstration.

Give each evidence layer one job

In agent-inspect@6.17.6, I use three local, deterministic layers:

Layer	Question
Suite	Did each named case satisfy its expected checks?
Cohort	What moved between baseline and candidate groups?
Gate	Should CI accept the recorded evidence?

representative traces
        |
        v
      suite --------> named case results
        |
        v
baseline vs candidate -> cohort deltas
        |
        v
       gate --------> CI exit code + evidence

These commands read persisted traces. They do not rerun the agent, invoke a model, or decide whether prose is semantically good.

Start with named cases, not random traffic

A compact suite makes the review set explicit:

{
  "name": "refund-release",
  "traces": "./.agent-inspect",
  "cases": [
    {
      "id": "eligible-order",
      "runId": "refund-eligible",
      "requireTools": ["lookup_order", "refund_order"]
    },
    {
      "id": "unknown-order",
      "runId": "refund-unknown",
      "forbidTools": ["refund_order"]
    },
    {
      "id": "approval-required",
      "runId": "refund-approval",
      "requireTools": ["request_approval"]
    }
  ]
}

Validate the configuration and run it:

npx agent-inspect suite validate \
  --config agent-inspect.suite.json

npx agent-inspect suite run \
  --config agent-inspect.suite.json \
  --markdown

The suite distinguishes a violated expectation from missing evidence. If refund-unknown does not exist, the case is skipped with a diagnostic; it is not silently treated as a pass. An all-skipped suite cannot become a green release signal.

That distinction sounds small until CI says “passed” for a case that never ran.

The set is part of the test

The most important suite question is not “Did it pass?” It is “Why are these the representative cases?”

For a refund agent, I would begin with:

an ordinary eligible order;
an unknown order that must not trigger a refund;
an amount requiring approval;
a missing-data path;
a retryable dependency failure; and
a non-retryable tool failure.

Version the suite beside the application. When product behavior intentionally changes, update the evidence set and expectations in the same review.

A useful review matrix keeps scenario coverage and evidence type visible:

Scenario	Structural check	Outcome check	Semantic check
Eligible order	lookup before refund	refund record exists	explanation is accurate
Unknown order	refund tool forbidden	no mutation occurred	refusal is useful
Approval required	approval tool present	request queued	message explains next step

No single column can substitute for the other two. A structurally correct run can still answer badly; a fluent answer can still hide an unauthorized mutation.

This is also where “prove the suite can say no” becomes useful. Keep at least one known-bad trace or mutation that must fail. A check that has never rejected evidence and a check that silently stopped running can produce the same green icon.

Compare behavior as a cohort

For a prompt, model, tool-schema, or orchestration change, label captured runs:

import { inspectRun } from "agent-inspect";

await inspectRun("refund-agent", runScenario, {
  traceDir: ".agent-inspect",
  metadata: {
    cohort: "candidate",
    scenario: "eligible-order",
    model: "approved-model",
  },
});

Then compare baseline and candidate traces:

npx agent-inspect cohort \
  --dir .agent-inspect \
  --baseline before \
  --candidate candidate \
  --cohort-key cohort \
  --group-by model \
  --metric errorRate,duration,toolChoice,observationFailure

A cohort report preserves the run count and the actual aggregate values. Imagine this synthetic result:

before / approved-model (10 runs)
  error rate: 0%
  average duration: 820ms
  dominant tool: lookup_order

candidate / approved-model (10 runs)
  error rate: 10%
  average duration: 970ms
  dominant tool: refund_order

That is a reason to investigate, not a statistically universal conclusion. The CLI can describe the recorded groups; it cannot make a small, biased fixture set representative.

A positive delta is not automatically bad

Suppose the candidate makes one extra tool call. That might be waste—or a newly required authorization check.

Suppose duration improves by 20%. That might be a better path—or the candidate skipped retrieval.

Suppose the dominant tool changed. That might be a regression—or the exact migration you intended.

Structural metrics describe what changed. Product assertions and semantic evaluation decide whether the change is acceptable. Keep those judgments separate so the report does not pretend to know your domain.

Turn credible evidence into a fail-closed gate

Once cases and thresholds reflect actual release policy, add a gate:

npx agent-inspect gate \
  --suite agent-inspect.suite.json \
  --format github \
  --output ./agent-gate-artifacts \
  --evidence-on fail \
  --evidence-profile share

Or gate a directory directly:

npx agent-inspect gate \
  --dir .agent-inspect \
  --max-error-rate 5 \
  --forbid-tool delete_account

Gate exit codes distinguish a policy failure from invalid configuration, unreadable traces, and unsupported output formats. Evidence generation does not suppress the original gate result.

That behavior matters because agent tooling is especially vulnerable to accidental fail-open configurations. Version 6.17.5 strengthened empty-check handling, rule-execution evidence, observed-outcome requirements, and requiredOrder semantics; 6.17.6 preserves those behaviors while hardening packaging and ingest boundaries.

What a green gate actually means

A passing gate means the encoded checks passed for the inspected evidence set.

It does not prove:

the cases represent production traffic;
the final answers are correct;
the model will repeat the behavior;
the thresholds are good; or
the agent is safe in production.

That is not a weakness unique to agent tests. It is the normal boundary of evidence-based release decisions. The remedy is not one giant magical score; it is several checks with clear jobs.

My smallest credible release loop

capture named cases
      -> run deterministic suite
      -> compare baseline/candidate cohorts
      -> inspect semantic quality separately
      -> make CI fail closed
      -> preserve the evidence

The pinned suites, cohorts, and gates documentation labels these surfaces Beta, and the cohort recipe includes a deliberately regressing fixture.

One good run can prove that a path exists. It cannot prove that a release is ready.

What is the smallest set of agent cases you would require before approving a change?

Three Agent Runs Passed. One Missing Handoff Broke the Workflow

Raju Dandigam — Thu, 10 Sep 2026 14:45:48 +0000

My triage agent extracted a refund request and its order reference.

The refund specialist said the reference was missing.

The escalation agent then opened a generic support ticket.

All three runs completed successfully.

Viewed separately, each trace told a plausible story. Viewed as one user journey, they contradicted one another. The failure lived between the green runs.

That is the uncomfortable part of multi-agent debugging: a handoff can lose the one field that mattered while every component reports success.

The timestamp trap

The tempting investigation is:

find the triage trace;
search for a specialist trace a few milliseconds later;
assume the next escalation belongs to the same request.

That works in a demo with one user. It breaks under queues, retries, parallel sessions, worker concurrency, and clock differences.

Time proximity is useful for discovery. It is weak evidence of causality.

10:00:00.100 triage completed
10:00:00.102 triage completed
10:00:00.110 specialist started
10:00:00.111 specialist started

Which triage run produced which specialist run?
The timestamps cannot answer.

The relationship needs an identity that survives the boundary.

Give the journey an explicit spine

For a cross-run workflow, I want four small pieces of metadata:

sessionId: the user journey or workflow instance;
workflowName: the reusable workflow type;
handoffFrom and handoffTo: the declared edge; and
retryOf plus attempt when a run is a retry.

Here is a synthetic TypeScript example verified against agent-inspect@6.17.6:

import { inspectRun, step } from "agent-inspect";

const traceDir = ".agent-inspect";
const sessionId = "sess-support-042";

const triage = await inspectRun(
  "triage-agent",
  async () => {
    return step("extract-request", async () => ({
      category: "refund",
      orderRef: "internal-value",
    }));
  },
  {
    traceDir,
    metadata: {
      sessionId,
      workflowName: "support-refund",
      handoffFrom: "triage-agent",
      handoffTo: "refund-specialist",
    },
  },
);

// The bug: orderRef disappears while building the handoff payload.
const specialistInput = { category: triage.category };

const specialist = await inspectRun(
  "refund-specialist",
  async () => {
    return step(
      "read-handoff",
      async () => ({
        needsEscalation: !("orderRef" in specialistInput),
      }),
      {
        metadata: {
          receivedFields: Object.keys(specialistInput),
          orderRefPresent: "orderRef" in specialistInput,
        },
      },
    );
  },
  {
    traceDir,
    metadata: {
      sessionId,
      workflowName: "support-refund",
      handoffFrom: "refund-specialist",
      handoffTo: "escalation-agent",
    },
  },
);

if (specialist.needsEscalation) {
  await inspectRun(
    "escalation-agent",
    async () => {
      await step("open-ticket", async () => undefined);
    },
    {
      traceDir,
      metadata: { sessionId, workflowName: "support-refund" },
    },
  );
}

Notice what the trace does not store: the order reference itself. The field name and presence bit are enough to locate the broken boundary.

Inspect one session instead of three files

List grouped workflow activity:

npx agent-inspect sessions --dir .agent-inspect

Then inspect the declared handoffs and each run's timeline:

npx agent-inspect session sess-support-042 \
  --dir .agent-inspect \
  --timeline \
  --diagnostics

The session view first gives the investigation a bounded shape:

session: sess-support-042

triage-agent
  handoff: triage-agent -> refund-specialist
                     |
                     v
refund-specialist
  handoff: refund-specialist -> escalation-agent
                     |
                     v
escalation-agent

three runs: completed

Then inspect the specialist run's bounded step metadata:

npx agent-inspect view <refund-specialist-run-id> \
  --dir .agent-inspect \
  --verbose

read-handoff
  receivedFields: category
  orderRefPresent: false

The escalation was not the original failure. The field disappeared when specialistInput was constructed.

That sounds obvious after the fact. Before the runs are correlated, it is easy to blame the last agent because it produced the visible symptom.

Correlation metadata is reviewable architecture

Explicit fields improve more than the trace viewer. They expose intended topology in code review.

A reviewer can ask:

Should these runs share a session?
Is the handoff target correct?
Which fields are required at this boundary?
Is this a retry of a specific run or merely another attempt?

AgentInspect refuses to manufacture a causal link from timestamps alone. If attempt is greater than one without retryOf, the relationship can be labeled correlated rather than explicit, with a diagnostic explaining the ambiguity.

That distinction matters. A convincing diagram with an invented edge is worse than an incomplete diagram that admits uncertainty.

Trace the boundary, but validate it too

Observability explains what executed. It should not be the first place a missing required field is discovered.

Add runtime validation before sending and after receiving:

type RefundHandoff = {
  category: "refund";
  orderRef: string;
};

function assertRefundHandoff(
  value: Record<string, unknown>,
): asserts value is RefundHandoff {
  if (value.category !== "refund") {
    throw new Error("Invalid refund category");
  }

  if (typeof value.orderRef !== "string" || value.orderRef.length === 0) {
    throw new Error("Missing orderRef at refund handoff");
  }
}

The schema check prevents the bad handoff. The session trace shows which boundary and version actually executed when behavior still surprises you. Those are complementary jobs.

Record the contract version, not the payload

Multi-agent workflows evolve. The triage agent may emit orderRef today while a newer specialist expects orderId. Even when both fields are strings, that is a protocol change.

Add a bounded contract identifier to the step or run metadata:

metadata: {
  sessionId,
  workflowName: "support-refund",
  handoffFrom: "triage-agent",
  handoffTo: "refund-specialist",
  handoffContract: "refund-request/v2",
  receivedFields: ["category", "orderRef"],
}

The identifier lets a reviewer ask whether producer and consumer used the same boundary definition without storing the business value itself. It also separates two failure classes that otherwise look identical:

missing field
  ├─ producer never emitted it
  ├─ transport dropped it
  ├─ mapper renamed it
  └─ consumer expected another contract version

AgentInspect treats these values as metadata; it does not enforce refund-request/v2. Your schema validator or message layer still owns that guarantee.

What sessions do not solve

A session index is not a workflow engine. It does not deliver messages, guarantee field consumption, or validate every handoff schema. Current handoff-related TraceContract support is not a complete policy language.

Session identifiers can also become sensitive correlation data. Use random identifiers rather than business values, keep metadata bounded, and apply an appropriate redaction profile before sharing traces outside the team.

The pinned sessions and outcomes documentation describes the current boundary.

The first question is not “Which agent failed?”

For multi-agent systems, start with:

At which boundary did the state stop matching the workflow's promise?

One explicit session ID, two handoff endpoints, and a few safe field-presence checks can turn three green traces into one explainable failure.

Where does your multi-agent system validate a handoff: before sending, after receiving, or both?