DEV Community

kosei
kosei

Posted on Originally published at qiita.com

Design Notes for the Claude Agent SDK

Design Notes for the Claude Agent SDK

I'm building an agent with the Claude Agent SDK for the first time, so I'm collecting what I learn and the design decisions I need to think through. I'll update these notes as I work on the implementation, learn more, or find something that needs correcting.

Key design considerations

These notes cover termination conditions, permissions, delegation, failure handling, and evaluation when building agents with the Claude Agent SDK. Explanations about implementing your own loop with the Messages API are included separately.

  • Verify success: Don't treat a successful SDK result as sufficient. Check the actual deliverables and the application's acceptance criteria.
  • Make stopping possible: Distinguish success, stopping without success, and human handoff. Set limits on time, turns, cost, and retries.
  • Enforce permissions at execution time: Combine tool configuration, runtime authorization, and backend permissions. Decide what happens when a permission check itself fails.
  • Delegate independent work: Define each subagent's scope and response format. Validate its results before starting dependent work.
  • Isolate data: Separate each user's workspace, credentials, conversations, and memory. Don't treat external content as authorization.
  • Compare changes: Use a fixed set of evaluation cases to measure success rates, prohibited actions, cost, and duration. Record the versions of the execution environment.

Start by defining success and stopping conditions

First, define what makes the agent loop succeed or fail. Then distinguish the conditions for success, stopping without success, and handing off to a human.

Instructions such as "handle this properly" or "improve this" leave both the target outcome and the stopping point unclear.

What the official documentation says

The Claude Agent SDK repeats a cycle: the model decides what to do, a tool runs, and the model reassesses the result. The loop normally ends when the model responds without a tool call. You can inspect the final state through ResultMessage.subtype.

Treat success as successful SDK completion, and verify the application requirements separately. Turn limits can be configured with max_turns / maxTurns, and budget limits with max_budget_usd / maxBudgetUsd. The budget limit is not a strict spending limit enforced before each charge; it can be exceeded, so leave a margin.

Source: How the agent loop works

Applying this in practice

The following are design proposals based on the official documentation. They should be distinguished from behavior the SDK automatically guarantees.

1. Define three kinds of termination conditions

Type What to define Example: a customer support agent
Success Deliverables and verifiable acceptance criteria Identify the applicable policy and generate a draft response with all required information
Stop without success Conditions that end processing even if the task is incomplete Time limit reached, the same error recurring, or retry limit exceeded
Handoff Conditions that require a human instead of further autonomous decisions Insufficient evidence, conflicting policies, or an operation outside the agent's authority

An outcome such as "out of scope" or "cannot be fulfilled" can still be a valid business result if it is correctly identified and explained. Don't confuse that with a system failure.

2. Don't accept the agent's own success report as proof

Write acceptance criteria in the prompt, and check those criteria in the calling code as well.

For document extraction, receiving JSON is not enough. Separately validate required fields, correspondence with the source document, and consistency of monetary amounts. For code changes, verify that the changes actually exist and that the required tests have run with the expected results.

One possible success rule: successful SDK completion + required output present + application validation passed.

Structured output can be validated against a format such as JSON Schema. However, valid structure does not guarantee correct content. The official documentation also explains that a success result without the expected structured_output should be treated as a failure.

Source: Get structured output from agents

3. Enforce limits in the execution environment, not just the prompt

Don't rely only on an instruction such as "stop after a few attempts." Control turns, budget, execution time, and retries.

In particular, define a stopping condition for repeated attempts with the same input and error when no progress is being made. Time limits and progress detection are application-level controls proposed here.

4. Define approval boundaries and when updates may be retried

Hooks such as PreToolUse can control actions before execution, while PostToolUse can support monitoring and processing afterward.

Source: Intercept and control agent behavior with hooks

For reading, drafting, and committing changes to external systems, explicitly define which operations may run automatically and which require prior approval. Also define when updates may be retried. Section 8 covers checking external state after a timeout and preventing duplicate execution.

5. Make the next action clear even when processing stops early

Record the termination reason, validation results, completed changes, unfinished work, and information needed to resume.

For example, the application could return:

  • status: completed / blocked / needs_human / failed
  • evidence: supporting evidence, deliverables, and verification results
  • remaining_work: unfinished work
  • next_action: retry, ask a human, or another next step

These are application-defined fields, not standard SDK termination states.

6. Define when to delegate to subagents

Decide whether to delegate based on whether a task can be completed independently with little information sharing.

The official guidance recommends delegation for parallelizable work, work that benefits from separate context, and tasks that can proceed independently without shared state. It favors direct handling by the parent for simple tasks, sequential work, edits to a single file, and tasks that need continuous context throughout the process. Avoid overusing subagents for things as small as a simple code search.

Source: Claude prompting best practices — Subagent orchestration

The following examples translate that guidance into practical design choices.

Consideration Better suited to delegation Better handled by the parent
Dependencies Research AWS, Azure, and GCP independently The scope of research B depends on the result of A
Context The task can be completed with the supplied documents and instructions The task requires a detailed understanding of the ongoing conversation
Work target Investigate separate modules independently Make closely related changes to the same file
Workload Read large logs and return evidence and key findings Change a few lines or perform a single search
Evaluation Review from another perspective against clear criteria Make the final decision based on the overall intent

Delegation adds the work of writing instructions, passing information, checking results, and integrating them. Use it when the benefits of parallelism or context isolation outweigh that overhead. If the task is simply to read several files, the parent may be able to run independent tool calls concurrently. Parallel tool execution and delegation of work that involves judgment are different choices.

What to specify when delegating

  • Purpose and scope: what to investigate and where responsibility ends
  • Inputs and assumptions: required documents, constraints, and decision criteria
  • Completion, stopping, and handoff conditions: what must be confirmed before finishing, and when to return control to the parent
  • Response format: conclusions, evidence, unresolved questions, and changes made
  • Permissions: research only, or permission to edit and execute as well

Even after a subagent finishes its work, the parent should validate the evidence and deliverables before accepting the result.

"The child finished its investigation" and "the overall objective was achieved" are different statements. The parent should check evidence, deliverables, and consistency with other results, then perform the necessary application-level validation before accepting the output.

7. Design tool availability separately from execution permissions

Give each role only the tools it needs, and separately validate which operations those tools may perform at runtime.

Giving every subagent the same large tool catalog forces each one to consider options outside its role. Make tool names and descriptions clear, and remove unnecessary tools from the configuration. An instruction saying "don't use this" should not be the only boundary.

How the SDK settings differ

Setting What it controls Important distinction
AgentDefinition.tools Tools available to a subagent Explicitly list the tools to provide. If omitted, the subagent inherits tools available to it
allowedTools (allowed_tools in Python) Automatic approval of tool execution This does not restrict which tools are available. Tools outside the list may still execute under other permission settings
permissionMode, allow/deny rules, and canUseTool Allowing, denying, or requesting confirmation for execution Design this separately from tool availability. Define both automatic approval and decisions based on arguments and targets

Sources: Subagents in the SDK — Tool restrictions, Configure permissions

Example tool sets by role

Role Example tools
Researcher Web search and page retrieval
Code reviewer Read, Grep, and Glob
Implementer Reading, editing, and running required tests
Integrator Read the supplied research results; return to the parent or researcher if more investigation is needed
Refund handler Order lookup and a dedicated refund tool; validate amounts and target orders separately

For a code reviewer, for example, explicitly set tools: ["Read", "Grep", "Glob"]. Avoid providing unnecessary editing or search tools, and define where the agent should hand off when its tools are insufficient.

A proposed three-layer control model

  1. Tool configuration: expose only the tools needed for the role.
  2. Runtime checks: validate arguments, target resources, amounts, and approval status in code or hooks.
  3. Backend permissions: restrict operations through AWS IAM, read-only database users, API authorization, and similar controls.

Consider alternate paths through general-purpose tools. If you provide Bash, an agent may be able to perform an operation through a CLI or HTTP request even when the dedicated tool for that operation is absent. Not providing a particular tool does not necessarily make the operation impossible. Enforce prohibited operations in the execution environment and backend as well.

8. Recover locally when possible, and return decisions to the parent when necessary

A subagent should handle temporary failures it can recover from within its scope. When it reaches a processing limit or needs a change in approach, it should return partial results and ask the parent to take over the decision.

These are application design proposals, not behaviors the SDK automatically guarantees in every case.

Local recovery versus escalation

Situation Example response
Temporary search or read timeout Retry with backoff within attempt and time limits
Invalid arguments that can be corrected within the assigned scope Correct the arguments and retry
Permission or authentication failure Report the required intervention to the parent instead of repeating the same request
Retry limit or time budget reached Return partial results and unfinished work
A change in research scope, source, or overall approach is needed Return enough information for the parent to decide

Waiting 1 second, then 2 seconds, then 4 seconds, with a maximum of three retries, is only an example configuration. A timeout is not guaranteed to resolve quickly. Set limits on total time and cost as well as retries, and account for automatic retries in the SDK or HTTP client so that the total number of attempts does not multiply unexpectedly.

For billing, sending, or updating operations, the external operation may already have completed when a timeout occurs. Before retrying, prevent duplicate execution through state checks, idempotency keys, or similar mechanisms.

Why local recovery helps, and what context isolation changes

Ordinary subagents run in separate conversations. Intermediate tool calls and results remain with the subagent, and the parent normally receives its final message. Starting a new worker for every temporary failure can therefore introduce repeated investigation and additional handoff work.

However, delegating again does not always mean starting from scratch. Supported subagents can resume with their history, and a fork may inherit the parent's conversation. Distinguish a new invocation, resuming an existing agent, and forking.

Sources: Subagents in the SDK, Resume subagents

Return a structured report that supports the next decision, even on failure

Reporting incomplete work as success can cause the parent to miss gaps in research or validation. Retrying indefinitely, on the other hand, blocks the overall workflow. When a limit is reached, distinguish complete, partial, and failed outcomes, and return both the findings and their limitations.

The following is an application-specific output example, not a standard SDK termination state:

{
  "status": "partial",
  "error_type": "timeout",
  "attempts": 3,
  "attempted_queries": ["The search conditions actually attempted"],
  "completed": ["Research into sources A and B"],
  "missing": ["Verification of source C"],
  "partial_results": [
    {"finding": "A verified fact", "source": "Source URL"}
  ],
  "next_action": "Investigate an alternative to source C"
}
Enter fullscreen mode Exit fullscreen mode

The parent can request only the unfinished work, resume the existing worker, use another source, or provide a partial answer that explicitly states the gaps. It should verify coverage and evidence instead of accepting the child's report at face value.

9. Finish normalization in SDK hooks before the model reads the result

This section covers behavior relevant to developers implementing agents with the Claude Agent SDK. Design the callbacks registered in options.hooks (hooks in the Python options) and their return values.

A hook firing, or normalized data appearing in a log, does not prove that the result passed to the model was replaced.

Distinguish normal waiting behavior from asynchronous output

By default, the SDK waits for a hook's return value. A PostToolUse hook can transform a tool result and return a replacement before the model reads it.

Returning { async: true } in TypeScript, or {"async_": True} in Python, instead tells the SDK to continue without waiting. This is intended for side effects such as logging, metrics, and notifications. It cannot block the action, modify the result, or inject context.

A JavaScript async function or Python async def is not itself the problem. If the callback awaits the transformation and returns a normal output object, the SDK waits for that return value. This is separate from the SDK's asynchronous output flag.

Input changes, additional context, and result replacement

Setting or return field Purpose Normalization consideration
matcher Select the target tool If the hook already fires, changing the matcher alone will not fix an unapplied result
PreToolUse.updatedInput Change arguments passed to a tool Does not directly replace the result returned by an external server
PostToolUse.additionalContext Add information to the result The original data remains; this is not result replacement
PostToolUse.updatedToolOutput Replace the tool result passed to the model Return it after transformation completes, using normal waiting behavior
updatedMCPToolOutput Earlier field for replacing MCP tool results Check support in your version. The current documentation also describes updatedToolOutput for all tools
{ async: true } Continue without waiting Do not use it for normalization or result replacement

A PostToolUse response containing decision: "block" and a reason is also different from replacing the tool result. It does not undo side effects from a tool that has already executed.

A schema mismatch can leave the original result visible to the model

In the Claude Agent SDK, if a replacement for a built-in tool does not match its output schema, the replacement is ignored and the original tool result is used. This is behavior of the Claude Code runtime used by the SDK, and it is also documented in the Python SDK type definitions. It is not a general rule of the Claude API or MCP protocol.

The sequence is: execute the tool, obtain its original result, return a replacement from PostToolUse, validate the schema, then use the replacement if valid or the original result if invalid. A failed replacement does not discard the original result. The official sources referenced here do not explain the rationale behind this design.

Tool type How updatedToolOutput is handled
Built-in tools such as Bash, Read, and Edit The replacement must match the tool's output structure; otherwise, the original result is used
MCP tools The replacement is passed through without the schema validation used for built-in tools

For Bash, preserve the structure containing fields such as stdout, stderr, interrupted, and isImage, rather than returning only a replacement string. The following example follows the official documentation:

{
  "hookSpecificOutput": {
    "hookEventName": "PostToolUse",
    "updatedToolOutput": {
      "stdout": "Connected to [HOST]",
      "stderr": "",
      "interrupted": false,
      "isImage": false
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

In the implementation, copy the original tool_response and transform only the required fields. Match each tool's schema rather than reusing the Bash structure for other tools. A PostToolUse response with decision: "block" adds a reason alongside the original result; it cannot hide the original information. Likewise, additionalContext only adds information.

Do not rely solely on PostToolUse replacement to redact sensitive information. Because a schema mismatch can expose the original result to the model, information that must be hidden should be removed in the tool or a wrapper. If redaction fails, return an error rather than raw data. This is an application-level design proposal.

updatedToolOutput changes the result shown to the model. It does not undo completed file changes or network requests. The official documentation also explains that OpenTelemetry tool spans and analytics events record the original output before the hook runs.

Test valid replacements, deliberately invalid replacements, and cases where sensitive information appears in both stdout and stderr.

Sources: Agent SDK — Hooks, Python SDK — PostToolUseHookSpecificOutput

Conceptual normalization example

Use code to normalize timestamps to ISO 8601 or map status codes to labels, rather than asking the model to do it. Define units, time zones, and how unknown codes are handled.

// Conceptual callback registered for PostToolUse.
// Implement normalizeToolResponse yourself, preserving the tool's output format.
const normalizeHook = async (input) => {
  if (input.hook_event_name !== "PostToolUse") return {};

  const normalized = await normalizeToolResponse(input.tool_response);

  return {
    hookSpecificOutput: {
      hookEventName: "PostToolUse",
      updatedToolOutput: normalized
    }
  };
};
Enter fullscreen mode Exit fullscreen mode

This async function returns after the transformation completes; it does not return { async: true }. Preserve required output structures such as MCP's content. For built-in tools, also match the output schema required by that tool.

Verify the transformed values in the tool result passed to the model, not just in the hook's execution log.

Source: Claude Agent SDK — Hooks: Outputs / Asynchronous output

10. Make the boundary between autonomous work and prior confirmation concrete

Instead of only saying "ask before making dangerous changes," define what the agent may do autonomously and what requires human confirmation, using conditions and contrasting examples.

If an agent frequently asks about trivial changes but proceeds without asking about major ones, the decision boundary may be unclear. Consider reversibility, public interfaces, dependencies, and effects on shared systems, rather than only the number of changed lines or task difficulty. The following are example team rules, not criteria automatically guaranteed by the SDK.

Examples of proceeding versus asking

Change Response Reason
Rename a variable used only inside a function Proceed autonomously No effect on external usage or behavior
Reformat code without changing behavior Proceed autonomously A local, reversible change
Change a public API name, required argument, or return contract Ask before changing it Affects compatibility with callers
Change behavior that other modules depend on Ask before changing it Effects extend beyond the edited location
Delete a shared file or update shared infrastructure Ask before execution Affects other people's work or shared state

Contrast an internal variable rename with a public API rename and include a short explanation. A few contrasting examples can communicate a decision pattern that applies to unfamiliar cases. Take existing explicit authorization into account, and avoid asking for the same approval repeatedly.

Source: Claude prompting best practices — Use examples effectively / Balancing autonomy and safety

Give prompts and runtime controls different responsibilities

Mechanism Responsibility
Prompt Explain criteria that depend on the meaning of a change and its dependencies, such as whether it breaks compatibility
Permission settings and a PreToolUse hook implemented in code Deny actions or request confirmation based on conditions that can be evaluated, such as tool names, paths, commands, and arguments
API diff checks, static analysis, and similar tools Detect changes such as public API modifications and provide evidence for hooks or CI

A prompt guides model behavior but is not an execution control that guarantees compliance. Hooks implemented in code can inspect execution requests independently of the model's intent. However, their enforcement is limited to the checks that have actually been implemented.

Formatting and a behavior change can affect the same file through the same editing tool. Paths and tool names alone cannot distinguish them. State the semantic boundary in the prompt, then reinforce machine-detectable conditions with permissions, hooks, and analysis. This does not mean hooks can never incorporate semantic judgments.

Sources: Agent SDK — Hooks, Agent SDK — Configure permissions

Explain exactly when confirmation is required

  • Adding "IMPORTANT" to an ambiguous instruction does not define what makes a change dangerous.
  • When prior confirmation is required, briefly state the proposed change, its impact, and why confirmation is needed.

11. Design parallel execution and sequencing around dependencies and start conditions

Let agents choose flexible research procedures, and enforce mandatory dependencies in application code.

The following are proposals based on the official documentation. Distinguish prompt-based guidance from execution order guaranteed by code.

Start only independent tasks at the same time

Task Required input Start condition
A: Find uses of deprecated APIs Source code Can start in parallel with B
B: Investigate module dependencies Source code Can start in parallel with A
C: Create a migration plan Results from A and B Start after both final results arrive and pass acceptance checks

Listing multiple tool calls in the same response does not pass an earlier call's result to a later one. Even if the runtime executes them sequentially, the results of A and B are not automatically inserted into the already-generated arguments for C.

Explicitly pass ordinary subagents the results, assumptions, evidence, and storage locations they need. They may obtain additional information through tools or inherit context through a fork, but they cannot access results that do not exist yet.

When to use prompts and when to use code

Condition Control method
Research procedures or delegates may change during the task Tell the parent agent the dependencies and start conditions
A fixed workflow requires C to start only after A and B finish Separate the SDK calls and control sequencing with await in code
Updates or messages may happen only after validation passes Validate results in code before starting the next stage

For example, instruct the parent to:

  • Run A and B in parallel because they are independent.
  • Do not start C until both final reports have arrived.
  • Pass C the results, evidence, and unresolved questions from A and B.
  • If one fails, retry within the limits. If required inputs are missing, do not proceed to C; report partial results and the reason for stopping.

A notification that background work has started does not mean the investigation is complete. Acceptance of a background invocation is not its final report. Check the waiting and completion-notification behavior of your version, and wait for final results before starting dependent work.

Source: Subagents in the SDK

Controlling a fixed workflow in code: pseudocode

# run_agent is an application-defined function.
# It consumes Agent SDK query() to completion,
# checks successful completion and the required final output, then returns.
api_report, dependency_report = await asyncio.gather(
    run_agent("Find uses of deprecated APIs"),
    run_agent("Investigate dependencies"),
)

# Validate required fields, evidence, and unfinished work.
# Do not proceed if validation fails.
validate_reports(api_report, dependency_report)

plan = await run_agent(
    make_plan_prompt(api_report, dependency_report)
)
Enter fullscreen mode Exit fullscreen mode

Here, the application coordinates multiple SDK sessions, rather than a parent agent internally delegating to subagents. Each agent can choose how to conduct its investigation, while code fixes the dependencies between stages.

Define timeouts, concurrency limits, how to retry only failed investigations, and when partial results are sufficient to proceed. One exception in asyncio.gather does not necessarily stop all other work, so also decide whether the remaining tasks should continue or be cancelled. For tasks that change the same file or external state, serialize them or isolate their workspaces to avoid conflicts.

Sources: Claude prompting best practices — Chain complex prompts, Building effective agents

Distinguish Messages API result formatting from the Agent SDK's responsibilities

If you implement the loop yourself with the Messages API, return the tool_result for each tool_use in the same assistant response together in the next user message. Match them using tool_use_id, and put result blocks before explanatory text. Do not omit calls that failed or were not executed; return is_error: true and a reason.

With the Agent SDK, the SDK normally executes tools and returns their results. You do not need to construct tool_result blocks manually each time. Also distinguish the events emitted by the SDK from the conversation-history format sent to the Messages API.

tool_choice: any requires at least one tool call in the Messages API. It does not guarantee a particular degree of parallelism or execution order.

Sources: Parallel tool use, How the agent loop works

12. Plan for failures in permission-checking hooks

Implementing a permission check does not automatically mean the operation will be denied if the check itself fails.

The current official documentation explains that, for many events, processing continues with a warning when a callback raises an exception, its output fails validation, or a return value is missing. If a PreToolUse check fails, processing moves on to the normal permission evaluation. Registering the hook alone therefore does not guarantee denial.

For a PreToolUse hook protecting important actions, set onFailure: "block" in TypeScript or on_failure="block" in Python. The documented minimum versions are TypeScript Agent SDK v0.3.317 and Python Agent SDK v0.2.177. Support is also required in Claude Code; if it is unavailable, startup fails with an error.

# check_policy is an application-defined permission-checking callback.
# Do not treat an indeterminate result or internal failure as permission.
hooks = {
    "PreToolUse": [
        HookMatcher(
            matcher="Bash",
            hooks=[check_policy],
            on_failure="block",
        )
    ]
}
Enter fullscreen mode Exit fullscreen mode

This is not a uniform guarantee across all events. PostToolUse cannot undo side effects that have already occurred, and events such as Stop ignore this setting. Check timeout behavior for each event as well.

Deliberately trigger exceptions, invalid return values, and timeouts, then verify that protected actions do not execute. If the check catches an exception and returns {}, that is a normal response, so make sure failures are not treated as permission to proceed.

Source: Agent SDK — Block the action when a callback fails

13. Define user isolation and trust boundaries for external content

Separate sessions or subagent conversations do not, by themselves, isolate files, credentials, or access to external services.

The official documentation warns against relying on the defaults of query() for multi-user isolation, because some host settings and memory are loaded outside the scope of settingSources. Separate filesystems by user, and combine settingSources: [] (setting_sources=[] in Python) with CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 in env. This does not remove server-managed settings obtained through organization credentials.

Source: Use Claude Code features — Settings and isolation

The following are application-level design proposals.

Area What to define
Workspace Per-user filesystems, mount boundaries, and cleanup. Changing cwd alone is not isolation
Conversations and resumption Map session IDs to users and authorize resumption so users cannot resume someone else's session
Memory and settings Explicitly define what is shared and what is isolated between users
Credentials Use minimum necessary permissions and lifetimes; where possible, attach credentials through a proxy outside the execution environment
Network access and tools Validate destinations, target resources, and user IDs at runtime
Logs and deliverables Define access permissions, retention, and removal of secrets

Search results, documents, and MCP tool results can contain malicious instructions directed at the agent. Treat them as data to process, not as evidence of user approval or a grant of authority. Limit the impact through network restrictions, filesystem permissions, and API authorization rather than relying only on prompts.

Source: Securely deploying AI agents

14. Run regression evaluations with fixed cases and record applicable versions

Don't judge the system by a single successful run. Compare the same evaluation cases before and after changing the model, prompt, tools, or SDK.

This section is a design proposal, not an automatic SDK guarantee. For each case, define inputs, required environment, expected results, allowed and prohibited operations, and acceptance criteria. Check execution traces and external state as well as the wording of the response.

Evaluation case What to check
Normal operation Requirements, deliverables, and supporting evidence are satisfied
Insufficient or conflicting evidence The agent does not present guesses as facts and follows the defined stopping or handoff procedure
Unauthorized actions or malicious tool results Prohibited actions are not executed, and external content does not expand permissions
Timeouts and retries Processing stops within limits and updates are not executed twice
Hook failures Protected actions do not execute when a critical pre-execution check fails
User isolation Other users' files, conversations, and credentials are inaccessible

Record business success rates, the number of prohibited actions executed, cost, duration, and the frequency of unnecessary confirmation requests. Model outputs vary, so repeat important cases and inspect failures. Define acceptance thresholds and tolerable variation for each use case.

Include the SDK version, the actual Claude Code runtime version, model ID, versions of the prompt, tool definitions, and settings, and the execution date in evaluation records. Also record the verification date and applicable versions in specification notes, and revalidate them when updating.

Pre-build checklist

  • [ ] Have you defined the scope and deliverables, and distinguished success, stopping without success, and human handoff?
  • [ ] Can you verify deliverable content and evidence, rather than relying only on successful SDK completion?
  • [ ] Have you set turn, cost, time, and retry limits, with a way to stop when no progress is being made?
  • [ ] Have you illustrated the boundary between automatic execution and prior approval, and enforced required restrictions through code or permissions?
  • [ ] Have you provided only the tools needed for each role and checked alternate paths through tools such as Bash and backend permissions?
  • [ ] Have you specified each delegated task's scope, inputs, permissions, and completion criteria, with the parent validating results before acceptance?
  • [ ] Are only independent tasks parallelized, with dependent stages starting after all required results arrive and pass validation?
  • [ ] Have you distinguished retries for temporary failures from human handoff and prevented duplicate updates?
  • [ ] Have you verified both that the model receives normalized results and that protected actions are denied when critical pre-execution checks fail?
  • [ ] Have you isolated user data, credentials, and conversations, and ensured that instructions in external documents or tool results cannot expand permissions?
  • [ ] Can you record partial results and stopping reasons, and compare success rates, prohibited actions, cost, and time across fixed cases? Do you also record the SDK, model, and settings versions used for evaluation?

Top comments (0)