DEV Community

Cover image for How to Compare Agent Frameworks on State, Tooling, and Recovery
Harry Gilbert
Harry Gilbert

Posted on

How to Compare Agent Frameworks on State, Tooling, and Recovery

Choosing an AI agent framework is relatively straightforward when building a prototype. The decision becomes more complicated when agents interact with enterprise APIs, maintain workflow state, execute sensitive operations, and recover from infrastructure failures.

Frameworks such as LangGraph, CrewAI, Google Agent Development Kit (ADK), Microsoft Agent Framework, Pydantic AI, and the OpenAI Agents SDK offer different approaches to agent development. However, their abstractions, persistence mechanisms, tool execution models, and recovery capabilities are not interchangeable.

For AI engineers and software architects, three technical dimensions deserve particular attention:

  • State management: How the framework stores, updates, and restores execution context.
  • Tooling: How agents discover, validate, authorize, and invoke external operations.
  • Recovery: How workflows respond to interruptions, timeouts, partial failures, and repeated execution.

These capabilities influence whether an agent remains reliable after moving into production.

This article presents a framework-independent comparison methodology, practical Python examples, failure injection scenarios, and a technical scorecard for evaluating agent frameworks.

1. Why Framework Selection Matters in Production

Consider an AI agent responsible for processing customer refund requests.

The workflow needs to:

  1. Retrieve a customer record.
  2. Retrieve the associated transaction.
  3. Evaluate refund eligibility.
  4. Prepare a refund operation.
  5. Obtain approval when required.
  6. Execute the refund through a payment API.
  7. Update the customer support system.

During a demonstration, this workflow may execute successfully within seconds.

In production, several things can go wrong.

The payment API might time out after processing the refund. A worker could crash before saving the execution result. An approval request might remain pending for several hours. A model could generate invalid tool arguments.

These are not exclusively language model problems. They are distributed systems and software engineering problems.

An agent framework should be evaluated according to how effectively it allows developers to manage these conditions.


2. Understand the Different Types of Agent State

One of the biggest mistakes when comparing agent frameworks is treating conversation memory and workflow state as the same capability.

Production agents typically require several types of state.

Conversation State

Conversation state contains information from previous interactions.

Examples include:

  • User messages.
  • Model responses.
  • Previous tool results.
  • Conversation summaries.
  • Relevant contextual information.

This state helps the model maintain continuity, but it does not necessarily identify which business operations have been completed.

Execution State

Execution state represents the current progress of a workflow.

It may contain:

  • Workflow identifier.
  • Current execution step.
  • Completed operations.
  • Pending tool calls.
  • Retry counters.
  • Execution status.
  • Approval decisions.
  • Error information.

Execution state is particularly important when workflows need to resume after interruptions.

Business State

Business state represents information owned by external applications.

Examples include payment records, customer accounts, invoices, support tickets, and inventory levels.

An agent framework may store references to these records, but the authoritative business state should remain in the appropriate system of record.

Long-Term Memory

Long-term memory allows an agent to retrieve information across separate sessions.

This may include user preferences, previous task outcomes, or organizational knowledge.

It should not replace durable execution state.

3. Evaluate How Frameworks Represent State

Different frameworks expose state through graphs, sessions, flows, message histories, or runtime contexts.

The important question is whether developers can represent and recover the information required by their application.

Consider this framework-independent Python state model:

from typing import Literal, TypedDict


class RefundWorkflowState(TypedDict):
    workflow_id: str
    customer_id: str
    transaction_id: str
    refund_amount_cents: int
    operation_id: str

    status: Literal[
        "pending",
        "validated",
        "awaiting_approval",
        "approved",
        "executing",
        "completed",
        "failed",
        "reconciliation_required"
    ]

    approval_id: str | None
    refund_reference: str | None
    retry_count: int
    last_error: str | None
Enter fullscreen mode Exit fullscreen mode

This state structure provides information about both the current workflow position and the proposed business operation.

However, defining a state object is only the beginning.

The framework must also provide a mechanism for persisting, retrieving, and updating it consistently.

Questions to Ask When Comparing State Management

  • Is state explicitly defined or reconstructed from conversation history?
  • Can developers use typed state schemas?
  • Is state automatically persisted?
  • Which storage backends are supported?
  • Can execution resume after a process restart?
  • Are state changes versioned?
  • How are concurrent updates handled?
  • Can individual workflow executions be isolated?
  • How are state schemas migrated after application updates?

A framework that provides flexible state representation but weak persistence may still require substantial application-level infrastructure.


4. Compare Checkpointing and Persistence Mechanisms

Checkpointing allows a workflow to save its progress at specific execution boundaries.

When a process fails, the application may resume from the most recent valid checkpoint.

However, checkpointing behavior differs across frameworks.

Some persist state after individual graph steps. Others save conversation history or serialize an interrupted execution.

These differences affect recovery behavior.

A Common Failure Scenario

Consider this sequence:

Step 1: Retrieve customer
Step 2: Validate refund eligibility
Step 3: Execute refund
Step 4: Save execution result
Step 5: Update support ticket
Enter fullscreen mode Exit fullscreen mode

Suppose the worker crashes after step three but before step four.

The external payment system may already have processed the refund, even though the orchestration framework has not recorded its completion.

Resuming the workflow from an earlier checkpoint could repeat the refund operation.

This is why persistence alone is insufficient.

What to Verify

During framework evaluation, determine:

  • Where checkpoints are created.
  • Whether incomplete steps are restarted.
  • Whether completed steps can be replayed.
  • How pending tool results are recorded.
  • What happens when the state database becomes unavailable.
  • Whether interrupted workflows survive application upgrades.

Checkpointing preserves execution progress. It does not independently guarantee exactly-once execution of external business operations.


5. Evaluate Tool Definition and Schema Validation

Tooling determines how an agent interacts with external applications.

A framework may allow developers to expose Python functions, REST APIs, MCP servers, database operations, or other application components as tools.

However, tool calling should involve more than generating function arguments.

Important Tooling Capabilities

Evaluate whether the framework supports:

  • Typed input schemas.
  • Structured tool outputs.
  • Runtime argument validation.
  • Dynamic tool availability.
  • Asynchronous execution.
  • Tool-specific timeouts.
  • Error handling.
  • Tool-level authorization.
  • Human approval.
  • Execution tracing.

Example: Typed Tool Arguments

Pydantic can provide a framework-independent validation layer.

from pydantic import BaseModel, Field


class RefundRequest(BaseModel):
    transaction_id: str = Field(min_length=1)

    amount_cents: int = Field(
        gt=0,
        le=100_000
    )

    reason: str = Field(
        min_length=5,
        max_length=500
    )
Enter fullscreen mode Exit fullscreen mode

An agent-generated tool request can be validated before execution.

def validate_refund_request(
    raw_arguments: dict
) -> RefundRequest:

    return RefundRequest.model_validate(
        raw_arguments
    )
Enter fullscreen mode Exit fullscreen mode

Invalid arguments are rejected before the underlying payment service is called.

But schema validation does not establish whether the user is authorized to perform the operation.

A valid refund request may still violate business rules.


6. Separate Tool Availability From Tool Authorization

Some frameworks allow developers to dynamically expose or hide tools based on workflow context.

For example, an agent may have access to a refund tool only when processing payment-related requests.

This reduces unnecessary tool selection, but it should not be treated as a complete security mechanism.

Authorization must evaluate the specific operation and requesting identity.

Example: Application-Controlled Authorization

def execute_refund(
    user_id: str,
    raw_arguments: dict,
    policy_service,
    payment_service,
    operation_id: str
):
    request = RefundRequest.model_validate(
        raw_arguments
    )

    allowed = policy_service.authorize(
        user_id=user_id,
        action="refund:create",
        resource_id=request.transaction_id,
        amount_cents=request.amount_cents
    )

    if not allowed:
        raise PermissionError(
            "Refund operation not authorized"
        )

    return payment_service.refund(
        transaction_id=request.transaction_id,
        amount_cents=request.amount_cents,
        reason=request.reason,
        idempotency_key=operation_id
    )
Enter fullscreen mode Exit fullscreen mode

The authenticated identity and operation identifier should come from trusted application context, not from model-generated arguments.

This example illustrates three separate responsibilities:

  • The framework coordinates tool selection.
  • The validation layer checks argument structure.
  • The policy service determines whether the operation is allowed.

Production implementations also require approval checks, secure credential handling, audit records, and reconciliation.

The OpenAI Agents SDK documentation makes a similar distinction between dynamic tool availability and resource-specific authorization.

Reference: OpenAI Agents SDK Tool Documentation


7. Compare MCP Integration Carefully

The Model Context Protocol (MCP) provides a standardized mechanism for connecting AI applications to external tools and resources.

Several agent frameworks support MCP integrations.

However, MCP compatibility does not automatically make a framework's tool execution secure or reliable.

What Developers Should Verify

  • How MCP servers are registered.
  • How tool definitions are discovered.
  • Whether tool schemas are validated.
  • How server authentication is configured.
  • Whether tool availability can be restricted.
  • How credentials are isolated.
  • How server timeouts are handled.
  • Whether tool calls appear in execution traces.
  • How failed or interrupted calls are recovered.

Consider an MCP server exposing operations for a CRM.

The agent framework may discover functions for reading and updating customer records.

The MCP server must still enforce access permissions for each protected operation.

A model should not receive unrestricted access simply because a tool is available through a standardized protocol.

Evaluation criterion: MCP support should reduce integration complexity without weakening application security boundaries.

Official documentation: https://modelcontextprotocol.io/


8. Evaluate Retry Policies and Error Handling

AI agents frequently interact with services that can fail independently.

Examples include:

  • Model provider timeouts.
  • API rate limits.
  • Database connectivity failures.
  • Invalid tool arguments.
  • Authentication errors.
  • External service outages.

A framework should allow developers to distinguish between failures that can be retried and failures requiring intervention.

Transient Failures

These may include temporary network problems and certain service availability errors.

A bounded retry policy with exponential backoff may be appropriate.

Permanent Failures

Examples include invalid arguments, missing resources, and unauthorized operations.

Repeating the same request is unlikely to resolve these problems.

Uncertain Execution Outcomes

These occur when an external operation might have succeeded, but the application did not receive confirmation.

For example, a payment service may process a transaction before the network connection drops.

The application should reconcile the operation's status rather than blindly retrying it.

Example: Error Classification

class RetryableToolError(Exception):
    pass


class PermanentToolError(Exception):
    pass


class UncertainOutcomeError(Exception):
    pass


def classify_error(error: Exception) -> str:
    if isinstance(error, UncertainOutcomeError):
        return "reconcile"

    if isinstance(error, RetryableToolError):
        return "retry"

    if isinstance(error, PermanentToolError):
        return "stop"

    return "escalate"
Enter fullscreen mode Exit fullscreen mode

The classifications should reflect the external service's actual guarantees.

A timeout on a read-only API may be safe to retry. A timeout during a financial transaction may require reconciliation.

Framework Evaluation Questions

  • Can retry policies be configured per tool?
  • Can developers define retryable exceptions?
  • Are maximum attempts configurable?
  • Is retry information persisted?
  • Can failed workflows be manually resumed?
  • Can uncertain outcomes trigger reconciliation?
  • Are retry attempts visible in traces?

9. Test Idempotency Under Realistic Failure Conditions

Idempotency is essential when agents execute operations that modify external systems.

A logical operation should not produce duplicate business effects simply because the request is retried.

Examples include:

  • Creating invoices.
  • Processing refunds.
  • Sending notifications.
  • Updating subscriptions.
  • Creating support tickets.
  • Placing purchase orders.

Stable Operation Identifiers

An operation identifier must remain consistent across retries of the same logical action.

from uuid import uuid4


def prepare_refund_operation(state: dict) -> dict:
    if not state.get("operation_id"):
        state["operation_id"] = str(uuid4())

    return state
Enter fullscreen mode Exit fullscreen mode

The operation identifier must be durably persisted before the external API is invoked.

Generating a new identifier for every retry would defeat the purpose.

Simulating a Lost API Response

The following example demonstrates why idempotency matters.

class MockPaymentService:

    def __init__(self):
        self.completed_operations = {}
        self.committed_refunds = 0

    def refund(
        self,
        operation_id: str,
        simulate_timeout: bool = False
    ):
        if operation_id in self.completed_operations:
            return self.completed_operations[
                operation_id
            ]

        result = {
            "refund_id": f"refund-{operation_id}",
            "status": "completed"
        }

        self.completed_operations[
            operation_id
        ] = result

        self.committed_refunds += 1

        if simulate_timeout:
            raise TimeoutError(
                "Response lost after commit"
            )

        return result
Enter fullscreen mode Exit fullscreen mode

Now test retry behavior:

def test_refund_retry_is_idempotent():
    service = MockPaymentService()
    operation_id = "workflow-42-refund-1"

    try:
        service.refund(
            operation_id,
            simulate_timeout=True
        )
    except TimeoutError:
        pass

    result = service.refund(operation_id)

    assert result["status"] == "completed"
    assert service.committed_refunds == 1
Enter fullscreen mode Exit fullscreen mode

The first request commits the refund but fails to return a response.

The second request uses the same operation identifier and receives the existing result.

Only one refund is committed.

This mock demonstrates the principle. It does not provide durable storage or concurrency protection.

A production service must enforce idempotency atomically and retain operation records for an appropriate period.


10. Evaluate Human Approval and Workflow Resumption

A production agent may need to pause while waiting for human authorization.

For example, a refund above a predefined threshold may require approval from a finance manager.

The framework should support controlled interruption and resumption.

Approval Lifecycle

Agent proposes operation
          |
          v
Validate tool arguments
          |
          v
Check authorization policy
          |
          v
Persist pending operation
          |
          v
Request human approval
          |
     +----+-----+
     |          |
  Approved    Rejected
     |          |
     v          v
Revalidate    Terminate
     |
     v
Execute operation
     |
     v
Persist result
Enter fullscreen mode Exit fullscreen mode

Approval should be tied to the exact operation being proposed.

If the refund amount changes after approval, a new review should be required.

What to Verify

  • Can workflows pause without keeping a worker active?
  • Is the pending operation stored durably?
  • Can execution resume after a restart?
  • Is the reviewer's identity recorded?
  • Can approvals expire?
  • Can rejected operations be terminated?
  • Are modified arguments detected?
  • Can approved operations be replayed?

LangGraph documents graph interrupts supported by its persistence mechanisms. The OpenAI Agents SDK also supports approval interruptions and serializable run state for later continuation.

References:


11. Comparing Six Agent Frameworks

The following comparison focuses on documented architectural approaches rather than popularity, benchmark rankings, or the number of available integrations.

1. LangGraph

Architecture: Graph-based agent orchestration.

LangGraph provides explicit workflow graphs, state transitions, persistence, and interruption mechanisms.

State management:

  • Explicit graph state.
  • Configurable checkpoint storage.
  • Workflow execution associated with thread identifiers.
  • Support for inspecting and restoring saved execution state.

Tooling:

  • Integration with model and tool abstractions.
  • Custom Python functions and execution nodes.
  • Flexible application-level validation and authorization.
  • Conditional routing and tool execution.

Recovery:

  • Checkpoint-based resumption.
  • Interrupt and resume mechanisms.
  • Durable execution patterns with appropriate persistence configuration.
  • Application-level responsibility for protecting external side effects.

Best suited for: Complex workflows that require explicit execution paths, persistent state, branching, and human approval.

What to test: Node replay behavior, checkpoint durability, concurrent state updates, and idempotent tool execution.

Official documentation: https://docs.langchain.com/oss/python/langgraph/overview

2. CrewAI

Architecture: Role-oriented multi-agent collaboration and flow-based orchestration.

CrewAI provides abstractions for agents, tasks, crews, and flows.

State management:

  • Flow state.
  • Structured workflow data.
  • Persistence mechanisms for supported flow configurations.
  • Agent collaboration context.

Tooling:

  • Custom agent tools.
  • Framework-provided integrations.
  • MCP connectivity.
  • Task-oriented tool execution.

Recovery:

  • Flow persistence and execution controls.
  • Error handling and retry configuration.
  • Recovery behavior dependent on the selected execution and persistence architecture.

Best suited for: Workflows involving specialized agents with clearly defined responsibilities.

What to test: Recovery after interrupted tasks, coordination between agents, repeated tool calls, and failure propagation across crews.

Official documentation: https://docs.crewai.com/

3. Google Agent Development Kit (ADK)

Architecture: Modular agent development with workflow and multi-agent capabilities.

Google ADK provides agent composition, session management, tools, and integrations for building AI-powered applications.

State management:

  • Sessions and session state.
  • Execution events.
  • Application and user-level contextual information.
  • Persistence options depending on deployment configuration.

Tooling:

  • Function-based tools.
  • Agent delegation.
  • MCP integrations.
  • Google ecosystem integrations.

Recovery:

  • Session and workflow continuation mechanisms.
  • Execution events for investigating workflow behavior.
  • Recovery capabilities that should be validated against the selected ADK version and runtime.

Best suited for: Teams developing modular agents and integrating with Google Cloud or related AI infrastructure.

What to test: Session persistence, interrupted execution, tool confirmation, and behavior across worker restarts.

Official documentation: https://google.github.io/adk-docs/

4. Microsoft Agent Framework

Architecture: Agent abstractions combined with structured workflow orchestration.

Microsoft Agent Framework provides tools for building agents and coordinating multi-agent workflows.

State management:

  • Agent sessions.
  • Workflow state and checkpoint-related capabilities.
  • Durable state options through supported runtime integrations.

Tooling:

  • Function tools.
  • Agent orchestration.
  • Human approval patterns.
  • Enterprise application integrations.

Recovery:

  • Workflow recovery mechanisms.
  • Durable execution through the Durable Task extension where appropriate.
  • Support for long-running workflows and delayed approvals using that extension.

Best suited for: Enterprise environments that require structured orchestration and integration with Microsoft technologies.

What to test: Which durability guarantees belong to the base framework and which depend on additional infrastructure.

Official documentation: https://learn.microsoft.com/en-us/agent-framework/

Durable execution reference: https://learn.microsoft.com/en-us/azure/durable-task/sdks/durable-agents-microsoft-agent-framework

5. Pydantic AI

Architecture: Python-first, type-safe agent development.

Pydantic AI emphasizes structured data, validation, dependency injection, and integration with Python application architecture.

State management:

  • Message history.
  • Typed application data.
  • Graph state through related graph abstractions.
  • Durable execution through supported integrations.

Tooling:

  • Typed function tools.
  • Pydantic-powered validation.
  • Dependency injection.
  • Structured model outputs.
  • MCP integrations.

Recovery:

  • Deferred tool execution and approval workflows.
  • Retry and validation mechanisms.
  • Durable workflow support through runtime integrations.

Best suited for: Python engineering teams that prioritize type safety, structured outputs, and explicit application-level controls.

What to test: Persistence ownership, runtime integration complexity, deferred execution, and recovery after uncertain tool outcomes.

Official documentation: https://ai.pydantic.dev/

6. OpenAI Agents SDK

Architecture: Agent-centric execution loops with tools, handoffs, guardrails, and tracing.

The OpenAI Agents SDK provides a comparatively focused set of abstractions for coordinating model-driven workflows.

State management:

  • Sessions for working conversation context.
  • Serializable run state for supported pause and resume scenarios.
  • Application-controlled persistence and runtime integrations.

Tooling:

  • Python function tools.
  • Structured argument generation and validation.
  • MCP integrations.
  • Agent handoffs.
  • Tool guardrails.

Recovery:

  • Approval interruptions.
  • Resumption from saved run state.
  • Configurable workflow controls.
  • Additional application or runtime mechanisms for workload-specific durability requirements.

Best suited for: Tool-connected agents that benefit from a relatively lightweight, Python-first execution model.

What to test: Recovery after unexpected process termination, session consistency, guardrail coverage, and repeated external operations.

Official documentation: https://openai.github.io/openai-agents-python/


12. Where Durable Workflow Engines Fit

An agent framework does not necessarily need to own the entire workflow lifecycle.

In some architectures, a durable workflow engine manages business execution while the agent framework handles model interactions.

Temporal is an example of infrastructure designed for durable workflow execution.

A combined architecture may look like this:

Business Application
          |
          v
Durable Workflow Engine
          |
    +-----+------+
    |            |
    v            v
Agent Runtime   State Store
    |
    v
LLM Decision
    |
    v
Policy and Validation Layer
    |
    v
Human Approval (if required)
    |
    v
Idempotent Activity Execution
    |
    v
Enterprise APIs
Enter fullscreen mode Exit fullscreen mode

In this arrangement:

  • The durable workflow engine owns business execution progress.
  • The agent framework handles model interactions and proposed actions.
  • The policy layer enforces authorization.
  • External services implement business operations.
  • Persistent identifiers protect against duplicate execution.

This approach can be useful when workflows last for extended periods, involve multiple external systems, or require strong operational guarantees.

It also introduces additional infrastructure and integration complexity.

At CodingCops, AI agent development and enterprise software integration are offered as complementary engineering services. For projects of this type, framework selection is only one architecture decision. The surrounding persistence, authorization, integration, and recovery mechanisms must also be evaluated against the intended business workflow.

13. Build a Framework-Independent Benchmark

Comparing agent frameworks through separate demonstrations is rarely useful.

Each candidate should execute the same workload under comparable conditions.

Step 1: Define a Representative Workflow

Use a workflow that includes:

  1. Structured user input.
  2. An LLM-based decision.
  3. A read-only tool call.
  4. A state update.
  5. A conditional branch.
  6. A human approval gate.
  7. A side-effecting tool call.
  8. Result validation.
  9. Workflow completion.

Step 2: Standardize Dependencies

Keep the following consistent across frameworks:

  • Language model and version.
  • Prompt instructions.
  • Tool definitions.
  • Input dataset.
  • External service mocks.
  • Infrastructure capacity.
  • Timeout requirements.
  • Business validation rules.

For recovery tests, use scripted model responses where possible.

This helps separate framework behavior from nondeterministic model behavior.

Step 3: Introduce Controlled Failures

Test each framework against the following scenarios.

Scenario A: Model Timeout

The model fails to respond within the allowed time.

Expected behavior: Bounded retry or controlled workflow failure.

Scenario B: Invalid Tool Arguments

The model generates a malformed request.

Expected behavior: Validation rejects the request before execution.

Scenario C: Worker Termination

The application process stops during an active workflow.

Expected behavior: Execution recovers according to the defined durability requirements.

Scenario D: Lost Tool Response

An external service commits an operation, but its response is lost.

Expected behavior: Reconciliation or idempotent retry without duplicate effects.

Scenario E: Delayed Approval

A workflow remains paused while the application restarts.

Expected behavior: The pending operation remains available and can resume after valid approval.

Scenario F: Unauthorized Tool Call

The model requests access to a protected resource.

Expected behavior: The authorization layer denies the operation.

Scenario G: Concurrent Resumption

Two workers attempt to resume the same workflow.

Expected behavior: Concurrency controls prevent conflicting updates and duplicate business operations.

Scenario H: Workflow Version Change

A new application version is deployed while earlier workflows remain unfinished.

Expected behavior: Existing workflows either resume compatibly or enter an explicit migration or recovery process.


14. Create a Common Testing Interface

A simple adapter interface can make benchmark implementations comparable.

from typing import Protocol


class AgentFrameworkAdapter(Protocol):

    def start(self, request: dict) -> str:
        ...

    def get_state(
        self,
        workflow_id: str
    ) -> dict:
        ...

    def approve(
        self,
        workflow_id: str,
        approval_id: str
    ) -> None:
        ...

    def resume(
        self,
        workflow_id: str
    ) -> dict:
        ...
Enter fullscreen mode Exit fullscreen mode

Each framework implementation exposes the same operations through an adapter.

This allows the benchmark to run equivalent tests without depending directly on framework-specific APIs.

Example Recovery Test

def test_pending_approval_survives_restart(
    adapter_factory
):
    adapter = adapter_factory()

    workflow_id = adapter.start({
        "customer_id": "customer-10",
        "transaction_id": "txn-42",
        "amount_cents": 2500
    })

    state = adapter.get_state(workflow_id)

    assert state["status"] == "awaiting_approval"

    approval_id = state["approval_id"]
    operation_id = state["operation_id"]

    restarted_adapter = adapter_factory()

    recovered = restarted_adapter.get_state(
        workflow_id
    )

    assert recovered["status"] == "awaiting_approval"

    assert recovered["operation_id"] == operation_id

    restarted_adapter.approve(
        workflow_id,
        approval_id
    )

    result = restarted_adapter.resume(
        workflow_id
    )

    assert result["status"] == "completed"
Enter fullscreen mode Exit fullscreen mode

This test assumes both adapters connect to the same persistent backend.

For a genuine crash-recovery benchmark, run the adapters in separate processes and forcibly terminate the first worker.

Also verify the external service's committed operations, rather than relying only on the framework's reported final state.


15. Measure the Right Performance and Reliability Metrics

A framework comparison should include measurable outcomes.

Workflow Completion Rate

The percentage of valid test workflows that reach the expected final state.

Completion Rate =
Successful Workflows / Total Workflows × 100
Enter fullscreen mode Exit fullscreen mode

Recovery Success Rate

The percentage of interrupted workflows that recover correctly.

Recovery Rate =
Successfully Recovered Workflows
/
Interrupted Workflows × 100
Enter fullscreen mode Exit fullscreen mode

Duplicate Operation Rate

The percentage of tested logical operations that produce unintended duplicate business effects.

For sensitive operations, the acceptable observed result should generally be zero in the test suite.

Zero observed duplicates do not prove that duplicates are impossible under all production conditions.

Unauthorized Execution Rate

The percentage of intentionally unauthorized test actions that are incorrectly executed.

Recovery Latency

The time required to restore an interrupted workflow and return it to a valid execution state.

Workflow Latency

Measure median and P95 end-to-end execution time.

Separate:

  • Model inference latency.
  • External tool latency.
  • Queue wait time.
  • Persistence overhead.
  • Orchestration overhead.

Cost per Successful Workflow

Include model usage, infrastructure, storage, retries, and relevant external service costs.

Cost per Successful Workflow =
Total Measured Operating Cost
/
Successfully Completed Workflows
Enter fullscreen mode Exit fullscreen mode

A framework that completes workflows reliably may be more economical than one with lower individual execution costs but frequent failures.


16. A Weighted Framework Evaluation Scorecard

The following scoring model is designed for production enterprise agents that interact with external applications.

The weights are illustrative and should be adjusted according to workload requirements.

State Management: 30%

Evaluate:

  • Explicit state structures.
  • Persistent execution context.
  • Checkpointing.
  • State isolation.
  • Concurrency handling.
  • Version compatibility.

Tooling and Security: 30%

Evaluate:

  • Typed tool definitions.
  • Argument validation.
  • Tool authorization.
  • MCP integration.
  • Human approval.
  • Credential isolation.
  • Tool-level observability.

Failure Recovery: 30%

Evaluate:

  • Retry policies.
  • Crash recovery.
  • Workflow resumption.
  • Idempotency integration.
  • Reconciliation.
  • Partial failure handling.

Developer Experience and Operations: 10%

Evaluate:

  • Documentation.
  • Testing support.
  • Debugging.
  • Deployment requirements.
  • Maintenance effort.
  • Infrastructure complexity.

Scoring Method

Assign each category a score from zero to five.

  • 0: Capability absent or not demonstrated.
  • 1: Minimal support.
  • 2: Partial support requiring substantial customization.
  • 3: Adequate support for the defined workload.
  • 4: Strong support demonstrated through testing.
  • 5: Comprehensive support validated under representative conditions.

Calculate the weighted score as follows:

weights = {
    "state": 30,
    "tooling": 30,
    "recovery": 30,
    "operations": 10
}

scores = {
    "state": 4,
    "tooling": 5,
    "recovery": 3,
    "operations": 4
}

total_score = sum(
    weights[category] * scores[category] / 5
    for category in weights
)

print(total_score)
Enter fullscreen mode Exit fullscreen mode

Output:

80.0
Enter fullscreen mode Exit fullscreen mode

These scores are hypothetical and do not represent benchmark results for any named framework.

A high weighted score should not compensate for a critical authorization, persistence, or recovery failure.

Mandatory security and reliability requirements should be evaluated separately.


17. Common Mistakes When Comparing Agent Frameworks

Confusing Conversation Memory With Durable State

A framework may remember previous conversations without knowing which business operations were committed.

Assuming Checkpointing Prevents Duplicate Actions

Checkpoints can restore execution progress, but external side effects require separate protection.

Treating Schema Validation as Authorization

Correctly structured arguments can still describe an unauthorized operation.

Choosing Based on the Number of Integrations

A large tool ecosystem does not establish reliable execution or secure access.

Comparing Only Successful Workflows

Failures reveal important differences in persistence, recovery, and operational behavior.

Ignoring Infrastructure Dependencies

Some capabilities require additional databases, managed runtimes, or durable execution services.

Assuming Multi-Agent Architecture Is Necessary

A deterministic workflow with one model-driven decision may be easier to operate than a collection of autonomous agents.

Ignoring Framework Version Changes

State serialization formats and runtime behavior may change during upgrades.

Production teams should test migration and backward compatibility before deployment.


18. Which Agent Framework Should You Choose?

The appropriate framework depends on the workload.

Consider LangGraph when explicit state graphs, branching, checkpoints, and controlled interruptions are central requirements.

Consider CrewAI when the workload benefits from specialized agents with distinct roles and task responsibilities.

Consider Google ADK when modular agent composition and Google ecosystem integration align with the application architecture.

Consider Microsoft Agent Framework when structured orchestration and Microsoft enterprise infrastructure are important.

Consider Pydantic AI when Python type safety, structured outputs, and application-controlled validation are priorities.

Consider the OpenAI Agents SDK when a focused agent execution model with tools, handoffs, guardrails, and tracing meets the application requirements.

Consider combining an agent framework with a durable workflow engine when execution reliability, long-running workflows, and business-critical side effects require stronger orchestration infrastructure.

These are architectural starting points, not universal recommendations.


Frequently Asked Questions

What is the most important factor when choosing an AI agent framework?

For production applications involving external systems, persistent state, secure tool execution, and predictable recovery are particularly important.

The priorities depend on the workload's complexity and risk.

What is the difference between agent state and memory?

Agent memory provides contextual information from previous interactions.

Execution state records workflow progress, completed operations, pending actions, and recovery information.

Is LangGraph better than CrewAI for production agents?

Neither is universally better.

LangGraph emphasizes explicit graph-based execution and persistence, while CrewAI provides abstractions for role-oriented agents and flows.

The appropriate choice depends on the workflow.

Does MCP make agent tools more reliable?

MCP standardizes aspects of tool connectivity.

It does not independently guarantee authorization, safe retries, idempotency, or reliable external service behavior.

Can agent frameworks guarantee exactly-once tool execution?

Framework persistence alone cannot guarantee exactly-once business effects across arbitrary external services.

Applications generally need idempotency, transaction controls, reconciliation, or equivalent mechanisms.

Do production AI agents need persistent state?

Workflows involving multiple steps, approvals, external operations, or recovery generally benefit from persistent execution state.

Short-lived, read-only interactions may have simpler requirements.

How should developers test agent recovery?

Introduce controlled failures such as worker termination, model timeouts, missing tool responses, concurrent resumption, and delayed approvals.

Verify the resulting workflow state and actual external business effects.

Can Temporal replace an AI agent framework?

Temporal can coordinate durable business workflows, but agent-specific reasoning and model integration may require separate libraries or application components.

The technologies can complement each other.

Are human approval controls necessary for every AI agent?

No. Read-only and low-risk workflows may not require approval.

Sensitive, irreversible, financial, or otherwise high-impact operations may require additional authorization and review.

How can engineering teams reduce framework lock-in?

Keep business logic, tool implementations, authorization policies, and evaluation datasets separate from framework-specific abstractions where practical.

Use adapters and documented data formats for important integrations.


Final Thoughts

Agent frameworks should be evaluated based on how they behave when production conditions become unpredictable.

Model responses can be invalid. External APIs can fail. Workers can restart. Approvals can take hours. Business operations can succeed even when their responses are lost.

State management determines whether the system knows what happened.

Tooling determines whether the system can interact with business applications within appropriate boundaries.

Recovery determines whether the workflow can continue without creating inconsistent or unauthorized outcomes.

Rather than selecting a framework based on popularity, engineering teams should implement a representative workflow, introduce controlled failures, and compare measurable results.

The most suitable AI agent framework is the one that provides the required execution guarantees without introducing unnecessary architectural and operational complexity.

Top comments (0)