DEV Community: Arif Sefa

AI Evals: Stop Trusting Your Agent's “Done”

Arif Sefa — Tue, 03 Nov 2026 12:42:15 +0000

AI agents are getting pretty good at doing things.

But an agent can complete a task without actually doing it correctly.

Imagine this scenario.

You give your coding agent a Jira ticket:

Add a credit card limit update feature to the payment service.

The agent reads the ticket, searches the repository, finds the relevant service, implements the changes, writes tests, and reports:

"Implementation completed. All tests passed."

Looks good, right?

But let's look a little closer.

Did it find the correct payment service?
Did it follow the existing architecture?
Did it validate the user's authorization?
Did it accidentally modify unrelated files?
Did it actually execute the tests?
Did it use the correct MCP tools?
Could it have completed the task with fewer steps?

The agent says it's done. But how do we know?

This is where Evals (Evaluations) come in.

I think Evals are becoming one of the most important parts of AI engineering, especially as we move from simple LLM applications to autonomous agents and multi-agent workflows.

Let's look at how we can actually implement them.

1. Evals Are Not Just Another Type of Test

Traditional software testing is built around predictable behavior.

For example:

const result = calculateCreditCardLimit(10000, 5000);

expect(result).toBe(5000);

We know the input, we know the expected output, and we can verify the result.

AI systems are different.

The same task can produce different outputs depending on the model, prompt, context, available tools, and execution environment.

Consider asking an AI agent:

"Find the credit card limit update implementation and explain how it works."

There might be several valid ways to investigate the codebase.

One agent might search by endpoint name. Another might start from the controller. Both could eventually find the correct implementation.

Requiring exactly the same execution sequence would be unnecessarily restrictive.

Instead, we need to evaluate whether the agent reached the correct result and whether its execution was reasonable.

I find it useful to think about Evals as another layer in our engineering workflow.

Traditional testing	AI evaluation
Does the API return the expected response?	Did the agent implement the correct API behavior?
Do unit tests pass?	Did the agent write meaningful tests and execute them?
Does the function handle invalid input?	Did the agent understand and handle the edge cases?
Does the application meet performance requirements?	Did the agent complete the task within an acceptable budget?
Does the integration work?	Did the agent select and use the right tools?

These approaches complement each other.

We still need unit tests, integration tests, and end-to-end tests. Evals help us understand whether the AI system can reliably produce the changes those tests are supposed to verify.

2. Start With Real Tasks, Not Random Prompts

Before building an evaluation system, we need a collection of tasks to evaluate.

Let's continue with our credit card example.

Imagine we have a coding agent connected to Jira, GitLab, and an internal codebase through MCP.

I would start with a small dataset containing different types of engineering tasks.

[
  {
    "id": "CC-001",
    "category": "code-discovery",
    "input": "Find the credit card limit update implementation.",
    "expected": {
      "behavior": "Identify the correct controller and service.",
      "mustNotModifyFiles": true
    }
  },
  {
    "id": "CC-002",
    "category": "feature-implementation",
    "input": "Add authorization validation to credit card limit updates.",
    "expected": {
      "behavior": "Validate card ownership before updating the limit.",
      "testsRequired": true
    }
  },
  {
    "id": "CC-003",
    "category": "bug-fixing",
    "input": "Fix the issue where invalid credit card limits are accepted.",
    "expected": {
      "behavior": "Reject invalid limits and preserve existing behavior.",
      "testsRequired": true
    }
  },
  {
    "id": "CC-004",
    "category": "architecture-analysis",
    "input": "Explain the credit card limit update flow.",
    "expected": {
      "behavior": "Explain the BFF, API and service interactions.",
      "mustNotModifyFiles": true
    }
  }
]

These four scenarios already test different capabilities:

Understanding an existing codebase.
Implementing a new feature.
Fixing a bug.
Understanding system architecture.

But don't stop at happy paths.

Add scenarios such as:

The Jira ticket has incomplete requirements.
The relevant API documentation is missing.
An MCP server returns a timeout.
The repository contains multiple implementations of the same feature.
The agent encounters an existing test failure.
Two agents attempt to modify the same file.
The user requests an operation without sufficient authorization.

A good evaluation dataset should represent the problems your agent will encounter in real life.

One practice I particularly like is turning production failures into permanent evaluation cases.

If an agent makes a mistake today, we should be able to detect the same mistake in tomorrow's evaluation run.

3. Deterministic Evals: Let Code Check the Result

Not every evaluation needs another AI model.

In fact, if we can verify something with ordinary code, we should.

Let's say our coding agent implements the credit card limit update feature.

We want to check:

Was the correct service modified?
Was authorization validation added?
Were tests created?
Did the tests actually pass?
Were unrelated files modified?

We can write a simple evaluator.

type AgentResult = {
  modifiedFiles: string[];
  toolCalls: string[];
  testsPassed: boolean;
};

function evaluateCreditCardFeature(result: AgentResult) {
  return {
    correctServiceModified: result.modifiedFiles.some(
      file => file.includes("payment-service")
    ),

    testsAdded: result.modifiedFiles.some(
      file => file.endsWith(".spec.ts")
    ),

    testsExecuted: result.toolCalls.includes("run_tests"),

    testsPassed: result.testsPassed
  };
}

Now we can evaluate the actual result instead of trusting the agent's final message.

For example:

{
  "correctServiceModified": true,
  "testsAdded": true,
  "testsExecuted": true,
  "testsPassed": false
}

The agent did most of the work, but the implementation isn't ready.

This is a much more useful result than simply receiving "Task completed."

Where can deterministic checks help?

What we want to evaluate	How we can check it
API contract	OpenAPI / JSON Schema
Type correctness	TypeScript compiler
Unit tests	Jest / Vitest
Code quality	ESLint
Security issues	Semgrep / CodeQL
Modified files	Git diff
Tool usage	Agent execution trace
Database changes	SQL assertions
API behavior	Integration tests

For example, after the agent finishes:

yarn nx test payment-service
yarn nx run payment-service:lint
yarn nx run payment-service:build

The important detail is that the evaluator executes these commands independently.

We shouldn't accept an agent's statement that tests passed as proof that they actually passed.

4. Tool Selection: Did the Agent Use the Right Tools?

This becomes especially interesting when we start using MCP.

Imagine our agent has access to these tools:

const tools = [
  "search_jira",
  "search_repository",
  "read_file",
  "get_api_documentation",
  "run_tests",
  "create_merge_request"
];

Now give it this task:

"Find the current credit card limit update implementation and explain its behavior."

A reasonable execution might look like:

1. search_repository
2. read_file
3. get_api_documentation
4. Generate explanation

But imagine this:

1. search_jira
2. create_merge_request
3. search_repository
4. read_file

The agent has performed unnecessary and potentially dangerous actions.

We can inspect its tool calls and compare them with the expected behavior.

function evaluateToolUsage(
  actualTools: string[],
  requiredTools: string[],
  forbiddenTools: string[]
) {
  const missing = requiredTools.filter(
    tool => !actualTools.includes(tool)
  );

  const forbidden = forbiddenTools.filter(
    tool => actualTools.includes(tool)
  );

  return {
    passed: missing.length === 0 && forbidden.length === 0,
    missing,
    forbidden
  };
}

const result = evaluateToolUsage(
  [
    "search_repository",
    "read_file",
    "get_api_documentation"
  ],
  [
    "search_repository",
    "read_file"
  ],
  [
    "create_merge_request"
  ]
);

console.log(result);

The expected result is a passing evaluation.

Of course, there can be multiple valid execution paths. We don't need to force every agent to call tools in exactly the same order.

What matters is whether it used the necessary tools, avoided forbidden actions, and achieved the intended result.

This is particularly important when building an MCP router with hundreds of available endpoints.

A model might be capable of using tools, but that doesn't mean it will always select the right one.

Tool selection itself should be evaluated.

5. Trajectory Evals: The Final Answer Isn't the Whole Story

A trajectory is the sequence of actions an agent takes while completing a task.

Consider two agents working on the same credit card feature.

Agent A

1. Read Jira task
2. Search repository
3. Identify payment service
4. Read existing implementation
5. Check API documentation
6. Implement changes
7. Run unit tests
8. Fix failing test
9. Report result

Agent B

1. Read Jira task
2. Search entire repository
3. Read 35 unrelated files
4. Search the same code again
5. Call MCP tool 6 times
6. Generate implementation
7. Skip tests
8. Report success

Both might produce similar final responses.

But their execution quality is very different.

We should evaluate:

Number of tool calls.
Unnecessary file reads.
Repeated operations.
Token consumption.
Recovery from failures.
Whether the agent verified its changes.

A simple trace evaluator might look like this:

type AgentTrace = {
  toolCalls: number;
  filesRead: number;
  retries: number;
  testsExecuted: number;
  durationMs: number;
  totalTokens: number;
};

function evaluateTrajectory(trace: AgentTrace) {
  return {
    toolEfficiency: trace.toolCalls <= 20,
    contextEfficiency: trace.filesRead <= 25,
    retryStability: trace.retries <= 3,
    verification: trace.testsExecuted > 0,
    timeBudget: trace.durationMs <= 120_000,
    tokenBudget: trace.totalTokens <= 30_000
  };
}

These numbers are just examples. A code-discovery task and a multi-file feature implementation shouldn't have the same execution budget.

The goal isn't to make agents use as few tools as possible.

It's to understand whether the tools and context they consume actually contribute to solving the task.

6. LLM-as-a-Judge: When We Need Another Model to Evaluate the Result

Some things are difficult to verify with deterministic tests.

For example:

"Explain how credit card limit updates work in this codebase."

There can be multiple correct explanations, so exact string matching isn't particularly useful.

We can use another LLM as an evaluator.

The process is straightforward:

The agent completes the task.
We collect its response.
A judge model receives the task, reference information, and response.
The judge evaluates the response against a defined rubric.
We store the scores and explanation.

Example judge prompt:

You are evaluating an AI coding agent.

Task:
Explain how credit card limit updates work.

Reference information:
- The request enters through the BFF.
- The payment API handles the request.
- The service validates card ownership.
- The business layer validates the requested limit.
- The repository persists the updated limit.

Evaluate using these criteria:

1. Technical correctness: 0-5
2. Completeness: 0-5
3. Relevance: 0-5
4. Unsupported assumptions: 0-5

Do not reward confident language.
Penalize invented implementation details.
Highlight missing authorization checks.

Return structured JSON.

Possible result:

{
  "technical_correctness": 4,
  "completeness": 3,
  "relevance": 5,
  "unsupported_assumptions": 1,
  "reason": "The answer explains the BFF and service layers but misses the ownership validation."
}

This is useful when comparing different models or prompt configurations.

For example, we could compare a local Qwen model with another coding model using the same tasks and the same evaluation criteria.

However, an LLM judge can also be wrong.

It may prefer a confident but inaccurate answer, overlook a technical detail, or score the same answer differently across runs.

For critical checks, combine LLM judging with deterministic assertions and human review.

7. RAG Evals: Did We Retrieve the Right Information?

Evals are not limited to coding agents.

Consider a RAG system that searches internal engineering documentation.

Question:

"How does the credit card limit update process work?"

The retrieval system returns five documents.

But what if the correct API documentation isn't among them?

The model might still generate a convincing explanation using unrelated documents.

We need to evaluate two different things:

Retrieval quality: Did we find the right information?

Answer quality: Did the model use that information correctly?

Let's define a small dataset:

[
  {
    "question": "How is a credit card limit updated?",
    "expectedDocuments": [
      "credit-card-api.md",
      "payment-service.md"
    ],
    "expectedFacts": [
      "ownership validation",
      "limit validation",
      "database update"
    ]
  },
  {
    "question": "What happens when a limit update fails?",
    "expectedDocuments": [
      "credit-card-error-handling.md"
    ],
    "expectedFacts": [
      "error response",
      "validation failure",
      "retry behavior"
    ]
  }
]

There are several useful retrieval metrics.

Recall@K

Did the correct documents appear in the top K results?

Suppose there are three expected documents and the retriever returns only one of them in its top five results.

[
Recall@5 = \frac{1}{3}
]

Precision@K

How many retrieved documents were relevant?

If only two of the five returned documents are relevant:

[
Precision@5 = \frac{2}{5}
]

MRR (Mean Reciprocal Rank)

How high does the first relevant document appear?

If the first relevant document is at position two:

[
RR = \frac{1}{2}
]

These metrics help us identify whether the problem is in retrieval or generation.

For example, a model might produce the correct answer even though the retriever returned the wrong documents.

That doesn't mean the RAG system is reliable. The model might simply have guessed correctly.

8. Multi-Agent Evals: Individual Agents Can Work, But the System Can Still Fail

Let's take a step further.

Imagine we have a multi-agent development workflow.

A Jira ticket enters the system, and several agents work together.

                   Jira Ticket
                        |
                        v
                  Planner Agent
                        |
                 Task Decomposition
                        |
           +------------+------------+
           |            |            |
           v            v            v
       Backend       Frontend       Tester
        Agent          Agent         Agent
           |            |            |
           v            v            v
        API Code      UI Code     Test Results
           |            |            |
           +------------+------------+
                        |
                        v
                   Review Agent
                        |
                        v
                   Final Result

How do we evaluate this?

We should evaluate each agent separately.

Agent	What should we evaluate?
Planner	Did it identify all requirements and dependencies?
Backend	Did it implement the correct business rules?
Frontend	Did it integrate the API correctly?
Tester	Did it create meaningful test scenarios?
Reviewer	Did it identify security and architectural problems?

But individual performance isn't enough.

We also need to evaluate coordination.

For example:

Did the frontend agent start before the API contract was ready?
Did two agents modify the same file?
Did the planner forget an important requirement?
Did the reviewer inspect the latest implementation?
Did the orchestrator recover when one agent failed?

This is where workflow-level evaluation becomes important.

An agent might be excellent at writing code but unreliable when coordinating with other agents.

The same principle applies to distributed agent architectures: communication, task ownership, shared state, and failure handling are all part of the evaluation.

9. Regression Evals: Don't Let Improvements Break Existing Behavior

Here's a situation that will probably look familiar to anyone experimenting with prompts and models.

You update your system prompt to improve backend code generation.

Your backend agent performs better.

But suddenly, it starts modifying more files than necessary during frontend tasks.

Without regression evaluations, you might not notice until production.

A simple workflow:

             Prompt / Model Change
                       |
                       v
                   Run Evals
                       |
                       v
                 Compare Results
                       |
                 +-----+-----+
                 |           |
              Improved    Degraded
                 |           |
                 v           v
               Accept     Investigate
                 |           |
                 v           v
               Deploy     Fix / Revert

For example:

Metric	Before	After
Backend success	78%	91%
Frontend success	92%	76%
Tool accuracy	89%	94%
Average tokens	28K	35K

The new prompt improved backend performance but damaged frontend performance and increased token consumption.

This is why every meaningful change to a model, prompt, tool, context strategy, or orchestration logic should trigger relevant evaluations.

I would treat this as part of the normal development lifecycle, just like running unit and integration tests before merging a change.

10. Let's Build a Small Evaluation Runner With TypeScript

So far, we've discussed different evaluation techniques.

Let's put some of them together.

We don't need a complex platform to get started. A simple evaluation runner is enough to demonstrate the idea.

Our runner should:

Load evaluation scenarios.
Execute the agent.
Collect the execution result.
Run deterministic checks.
Record latency and token usage.
Generate a report.

First, define the contracts.

type EvalCase = {
  id: string;
  input: string;
  expected: {
    requiredTools?: string[];
    forbiddenTools?: string[];
    mustNotModifyFiles?: boolean;
  };
};

type AgentExecution = {
  answer: string;
  modifiedFiles: string[];
  toolCalls: string[];
  testsPassed: boolean;
  durationMs: number;
  totalTokens: number;
};

type EvalResult = {
  id: string;
  passed: boolean;
  failures: string[];
  durationMs: number;
  totalTokens: number;
};

Now let's implement the evaluator.

async function evaluateCase(
  testCase: EvalCase,
  executeAgent: (input: string) => Promise<AgentExecution>
): Promise<EvalResult> {

  const execution = await executeAgent(testCase.input);
  const failures: string[] = [];

  for (const tool of testCase.expected.requiredTools ?? []) {
    if (!execution.toolCalls.includes(tool)) {
      failures.push(`Missing tool: ${tool}`);
    }
  }

  for (const tool of testCase.expected.forbiddenTools ?? []) {
    if (execution.toolCalls.includes(tool)) {
      failures.push(`Forbidden tool used: ${tool}`);
    }
  }

  if (
    testCase.expected.mustNotModifyFiles &&
    execution.modifiedFiles.length > 0
  ) {
    failures.push("Unexpected file modifications");
  }

  return {
    id: testCase.id,
    passed: failures.length === 0,
    failures,
    durationMs: execution.durationMs,
    totalTokens: execution.totalTokens
  };
}

Finally, run the dataset.

async function runEvaluation(
  dataset: EvalCase[],
  executeAgent: (input: string) => Promise<AgentExecution>
) {
  const results: EvalResult[] = [];

  for (const testCase of dataset) {
    results.push(
      await evaluateCase(testCase, executeAgent)
    );
  }

  const passed = results.filter(
    result => result.passed
  ).length;

  return {
    total: results.length,
    passed,
    failed: results.length - passed,
    successRate: passed / results.length,
    averageTokens:
      results.reduce((sum, r) => sum + r.totalTokens, 0) /
      results.length,
    results
  };
}

This is a deliberately small implementation. A production runner would also need isolated workspaces, test-result assertions, execution timeouts, structured traces, repeated runs, and error classification.

Also notice that the evaluator doesn't trust the agent's final response. It checks the actual execution data.

That's the idea.

Evaluation should be an executable part of your engineering process, not a manual review of a successful demo.

11. Open-Source Projects Worth Exploring

You don't have to build everything from scratch. These three projects are good starting points.

1. Promptfoo

https://github.com/promptfoo/promptfoo

Probably one of the easiest ways to start experimenting with Evals.

It supports:

Prompt and model comparisons.
Custom JavaScript assertions.
LLM-as-a-Judge.
RAG evaluations.
Coding agent evaluations.
Cost and latency checks.
Regression testing.

What makes it particularly interesting for coding agents is that it can evaluate actual agent execution, inspect traces, and run tasks in separate workspaces.

A good starting point:

https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/

2. OpenAI Evals

https://github.com/openai/evals

An open-source framework for evaluating LLMs and LLM-based systems.

It provides a framework for defining evaluation tasks, creating custom evaluators, and running repeatable experiments.

It's useful when you want to build evaluation datasets around your own use cases rather than relying only on generic benchmarks.

3. Inspect AI

https://github.com/UKGovernmentBEIS/inspect_ai

An open-source evaluation framework developed by the UK AI Security Institute.

What makes it interesting is its support for more complex agent evaluations.

It provides building blocks for:

Datasets.
Agents and tools.
Scorers.
Sandboxed execution.
Multi-step tasks.
Tool-using and multi-agent workflows.

It also supports custom and MCP tools, making it worth exploring if you're building systems that interact with external services.

Final Thoughts

I think we're entering a different phase of AI development.

We're no longer just building applications that generate text. We're building systems that can make decisions, modify codebases, interact with infrastructure, and execute entire workflows.

That changes how we should think about quality.

A successful demo tells us that an agent can complete a task under certain conditions.

An evaluation suite tells us how reliably it performs across different tasks and conditions.

For me, this is the important distinction:

Don't measure your agent by its best execution. Measure it by how consistently it gets things right.

Building an agent is one challenge.

Knowing when to trust it is another.

Your AI Doesn’t Have a Model Problem. It Has a Data Problem.

Arif Sefa — Thu, 29 Oct 2026 20:18:16 +0000

Why building reliable AI systems requires more than choosing a better LLM.

A book I read a while ago has been on my mind again recently: Designing Data-Intensive Applications by Martin Kleppmann.

I found myself going back to it, and this time, I started looking at some of its ideas through a different lens: AI systems.

We spend a lot of time discussing which model to use, how to improve prompts, and how to build better agent workflows. But many of the problems I keep running into aren't really about the model itself.

They're about data. How it's retrieved, how it stays up to date, how state is managed, and how different parts of a system stay consistent.

These are familiar engineering problems. We've been dealing with them long before LLMs became part of our daily work.

And I think many of the lessons from Kleppmann's book are just as relevant to AI agents as they are to traditional distributed systems.

Let's look at a few examples.

1. RAG Is a Data Engineering Problem, Too

Retrieval-Augmented Generation (RAG) is often presented as a straightforward solution: index your documents, retrieve relevant chunks, and send them to an LLM.

In practice, getting retrieval right is much harder.

Imagine building an AI assistant that understands a large banking application. A developer asks:

Where is the credit card detail screen, and which backend endpoint provides its data?

The answer might involve several parts of a monorepo:

A React Native screen
A BFF implemented with NestJS
A GraphQL query or REST endpoint
A .NET microservice
Shared UI components and business logic

A basic vector search might retrieve a file containing the words “credit card details.” But that doesn't necessarily tell us how the data flows through the application.

The relevant information may be spread across several files, and the relationships between them matter as much as the files themselves.

This is where data architecture starts to matter.

A more useful retrieval system could combine:

Semantic search: Find code and documents related to the question, even when they use different terminology.
Keyword search: Find exact endpoint names, symbols, identifiers, and error codes.
Code relationships: Follow imports, function calls, API contracts, and dependencies.
Metadata filters: Restrict results to the relevant application, language, or project.

The goal isn't simply to retrieve text that looks relevant. It's to reconstruct enough of the underlying system to answer the question correctly.

This is closely related to a broader idea in Designing Data-Intensive Applications: the way we organize, index, and derive data has a direct impact on what our systems can do with it.

A better model can't reliably compensate for information that your retrieval system never found.

2. Your Data Can Be Correct and Your Answer Can Still Be Wrong

Consider another scenario.

An API endpoint changes during a refactoring. The code is updated, the tests pass, and the application works as expected.

But the AI assistant still uses an old index.

A developer asks about the endpoint, and the assistant confidently returns the previous implementation.

The model might reason perfectly from the information it received. The problem is that the information is no longer current.

This is a data freshness problem.

Traditional data systems have dealt with similar challenges for years: replication lag, stale caches, delayed processing, and derived data that falls behind its source.

AI systems introduce another layer of derived data. Embeddings, code summaries, knowledge graphs, and retrieval indexes are all representations of an underlying source.

When that source changes, these representations need to be updated.

A practical indexing pipeline might look like this:

Source code changes
       |
       v
Detect changed files
       |
       v
Rebuild affected chunks and relationships
       |
       v
Update search indexes
       |
       v
Make updated knowledge available to agents

For a large repository, rebuilding everything after every change may be too expensive. Incremental indexing, dependency tracking, and background processing can make the pipeline more efficient.

The important part is having a clear strategy for keeping derived data in sync with its source.

This is also why an AI assistant should be able to identify where its information came from and, when necessary, check the current source before making a claim.

Retrieval quality isn't just about relevance. It's also about freshness.

3. Multi-Agent Systems Need Reliable State, Not Just Memory

Multi-agent systems are often described in terms of roles: a planner, a designer, a frontend developer, a backend developer, and a tester.

That sounds reasonable. But what happens when one agent fails halfway through a workflow?

Imagine a system that creates a GitLab merge request, runs tests, and asks a reviewer agent to inspect the changes.

The review request times out. The orchestrator doesn't know whether the operation failed or whether the result was simply delayed.

It retries the operation.

Depending on how the workflow is implemented, the system might create duplicate work, lose track of the original task, or report a result that doesn't reflect the actual state of the merge request.

Adding another agent won't solve this problem.

The workflow needs explicit state.

For example:

type TaskStatus =
  | "pending"
  | "running"
  | "completed"
  | "failed"
  | "retrying";

interface AgentTask {
  id: string;
  status: TaskStatus;
  attempts: number;
  result?: unknown;
  error?: string;
}

This is only a starting point. A production implementation would also need persistent storage, task ownership, timestamps, retry policies, and a way to recover after a process restarts.

For operations with side effects, idempotency is important too. Retrying a task should not blindly repeat an action that has already succeeded.

A reliable workflow should be able to answer questions such as:

Which tasks have completed?
Which agent currently owns a task?
What happens if an agent crashes?
Can a task be safely retried?
How do we recover after the orchestrator restarts?

These are familiar distributed-systems questions.

An LLM can decide what to do next, but the surrounding system must reliably track what has already happened.

Memory helps an agent remember context. State helps a workflow remain correct. They are not the same thing.

4. A Larger Context Window Doesn't Replace Good Data Architecture

When an AI system struggles with a large codebase, one tempting solution is to give the model more context.

Sometimes that helps. But it doesn't remove the need to decide which information matters.

Suppose your repository contains hundreds of projects and more than a million lines of code.

Sending thousands of files into a model's context window isn't necessarily useful, even if the model technically supports it. Much of that information will be irrelevant to the current task.

And if the relevant relationship between two files is missing, adding more unrelated files won't automatically make that relationship clear.

A more useful approach is to build a pipeline that selects and organizes information before it reaches the model:

User question
     |
     v
Understand the task
     |
     v
Retrieve relevant files and symbols
     |
     v
Follow dependencies and references
     |
     v
Rank and filter the results
     |
     v
Build focused context
     |
     v
LLM generates an answer

This approach can reduce unnecessary context, improve relevance, and make the system easier to debug.

It also creates opportunities to cache results, reuse indexes, and apply access controls before sensitive information reaches a model.

The context window is a resource. The data pipeline determines how effectively we use it.

5. AI Agents Need Observability Beyond Logs

Traditional application observability helps us understand what happened inside a system through logs, metrics, and traces.

Agentic systems need much of the same infrastructure, but there are additional questions to answer.

Suppose an agent returns an incorrect answer. Looking only at the final response won't tell us why.

Was the source data outdated? Did retrieval return the wrong documents? Did a tool call fail? Did the model misinterpret the result? Did an earlier agent pass incomplete information to the next one?

To investigate the problem, we need visibility into the entire execution.

For an agent workflow, useful signals include:

Signal	What it helps us understand
Retrieved documents	What information the agent actually saw
Tool calls and results	Which external operations were performed
State transitions	How the workflow progressed
Retries and failures	Where execution became unreliable
Model and token usage	Cost, latency, and resource consumption
Final output and evaluation	Whether the result met the expected criteria

Consider a coding agent that claims a bug is fixed.

A useful trace should help us determine which files it inspected, what changes it proposed, whether tests actually ran, and whether the test results support its conclusion.

This is more than logging the model's prompt and response. It's observing the behavior of a system that combines probabilistic reasoning with deterministic software components.

Without that visibility, debugging becomes guesswork.

And without evaluations, we may not even notice that a change to the retrieval pipeline has made the system worse.

6. The Common Thread: Reliable Systems Still Need Reliable Foundations

Looking at AI agents through the lens of data-intensive applications changes how we approach their design.

Instead of asking only, “Which model should we use?”, we can ask a broader set of engineering questions:

Is the information complete and up to date?
Can we trace where an answer came from?
Is workflow state persisted and recoverable?
Can we retry failed operations safely?
Can we observe and evaluate the entire execution?
Can the system scale without retrieving unnecessary information?

These questions don't replace model selection, prompt engineering, or agent design. They complement them.

A powerful model is still important. But it operates within a larger system, and the quality of that system places real limits on what the model can accomplish.

Final Thoughts

Revisiting Designing Data-Intensive Applications reminded me that many of the difficult problems in AI engineering aren't entirely new.

We are building new kinds of applications, but we still need to deal with data consistency, derived indexes, distributed execution, state management, fault tolerance, and observability.

The difference is that we now have LLMs making decisions and interacting with these systems.

That makes the engineering around them more important, not less.

If you're building AI agents, don't focus exclusively on making the model smarter. Pay attention to the data it sees, the state it maintains, and the systems it depends on.

Because sometimes, the problem isn't that your AI agent can't reason.

It's that the system hasn't given it the right information to reason about.

Further reading

Martin Kleppmann, Designing Data-Intensive Applications, O'Reilly Media, 2017.

The book explores the principles behind reliable, scalable, and maintainable data systems, including storage and retrieval, replication, consistency, batch processing, and stream processing. Many of those principles remain useful when designing the infrastructure behind modern AI agents.

Your MCP Server Needs Better Design for Better Agents

Arif Sefa — Sat, 24 Oct 2026 17:01:31 +0000

I’ve been thinking about MCP design a lot lately, especially the gap between making a tool available and making an agent use it well.

It’s easy to wrap an API in an MCP server, expose a bunch of tools, and call it done. Then the agent picks the wrong tool, guesses a parameter, retries a few times, and burns context just trying to figure out what we meant. The protocol may be working perfectly; the experience still isn’t great.

So what does a good MCP server actually look like? I think it comes down to a few practical choices: how we name and scope tools, how much context we expose, where Skills fit, and how we keep the whole thing safe in production.

Simple architecture visual: Coder Legion — The End of Glue Code.

1. Don’t start by turning everything into MCP

MCP is useful when a capability needs to be reused by different clients, maintained independently, or kept behind a clear security and execution boundary. But it isn’t the right answer for every function an agent calls.

flowchart LR
    U[User request] --> A[Agent]
    A --> D{Shared capability?}
    D -- No --> F[Local function]
    D -- Yes --> M[MCP server]
    M --> B[Business system]

Use a local function for a small, private operation that only one application needs. Consider MCP when multiple compatible clients need the same integration, the integration needs independent deployment or credentials, or you want a governed boundary between an agent and a system.

For example, a single application formatting a date does not need an MCP server. A company-wide, read-only service that lets approved agents search Jira issues, retrieve service documentation, or inspect deployment status may benefit from one.

My rule of thumb: don’t ask “Can I wrap this in MCP?” Ask “What do I gain by putting this capability behind a shared, independently governed interface?”

2. Design tools for the way agents actually use them

A common first attempt is to expose an existing REST API almost one-to-one:

GET /customers/{id}
GET /customers/{id}/accounts
GET /accounts/{id}/transactions
GET /transactions/{id}

That can be appropriate for predictable, low-level operations. But an agent may not know which endpoint to call first, which identifiers are required, or which of the 40 returned fields matter.

Instead, design a small set of task-oriented tools around the questions users actually ask:

find_customer(query)
list_customer_accounts(customer_id)
get_account_summary(account_id, period)
get_transaction_details(transaction_id)

The tool names should describe an action in domain language. Avoid vague names such as process_data, execute, or get_info. Avoid exposing internal database column names if the agent has no reason to know them.

Example: a tool schema that guides the agent

Too vague

{
  "name": "search",
  "description": "Search records",
  "inputSchema": {
    "type": "object",
    "properties": {
      "type": { "type": "string" },
      "q": { "type": "string" }
    }
  }
}

More useful

{
  "name": "search_service_docs",
  "description": "Search approved engineering documentation. Use for questions about service behavior, ownership, and integration steps. Does not search source code or production logs.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "A concise question or set of technical keywords"
      },
      "service": {
        "type": "string",
        "enum": ["cards", "payments", "accounts", "identity"]
      },
      "limit": {
        "type": "integer",
        "minimum": 1,
        "maximum": 10,
        "default": 5
      }
    },
    "required": ["query"],
    "additionalProperties": false
  }
}

The second schema tells the model what the tool is for, what it should not be used for, and which values are valid. Defaults and limits reduce the amount of guessing. That’s the goal: not a clever description, but fewer avoidable mistakes.

Practical guidelines

Use domain words the user and model both understand.
Make required fields genuinely required; default common optional fields.
Use enums and numeric bounds where the domain is finite.
Keep parameter counts manageable. Around eight or fewer is a useful heuristic, not a protocol limit.
Explain when not to use a tool if overlap with another tool is likely.
Keep descriptions informative but not encyclopedic: every token competes for context.

Another view of the MCP building blocks: Splunk — Securing AI agents with MCP.

3. Reduce tool confusion and context bloat

More tools don’t automatically make an agent more capable. If a client loads hundreds of similar tool definitions, the model has more to read and more chances to choose the wrong one. This is one of the most important trade-offs in MCP design: too little guidance creates confusion, but too much guidance also eats into the context window.

flowchart TD
    Q[User request] --> R[Find relevant tool]
    R --> T[Call focused tool]
    T --> S[Return concise result]
    S --> D[Fetch details if needed]

A useful pattern is progressive disclosure:

Keep the always-visible tool catalog small and clear.
Offer a lightweight discovery/search tool when the catalog is large.
Load detailed schemas, domain vocabulary, or instructions only when needed.
Return a compact summary by default and provide a separate detail operation.

For example, search_transactions might return transaction ID, date, merchant, and amount. A separate get_transaction_details tool can return status, fees, and settlement information when needed.

This is often better than returning 50 fields on every call.

Choosing a strategy

Strategy	Good when	Main trade-off
Clearer descriptions	You need a quick improvement without backend refactoring	Longer descriptions consume more context
Better schemas and defaults	Inputs have predictable types and valid values	Schemas need maintenance as the domain changes
Split tools by task	One tool currently does too many things	More tools can increase discovery complexity
On-demand discovery	The catalog is large or changes often	Adds a discovery call and some latency
Server-side interpretation	Requests use complex business language and must behave consistently	Adds inference cost, latency, and another component
One agent-backed tool	You need to hide a complex workflow behind a stable interface	Less transparency; harder debugging and higher operational cost

Do not start with a server-side LLM just to compensate for poor names and schemas. First try clearer tool boundaries, validation, defaults, and compact results. Add model-based interpretation only when evaluation shows it is needed.

Visual comparison of Skills and MCP: Daily Dose of Data Science.

4. Tools, resources, prompts, and skills are not interchangeable

MCP exposes different primitives for different jobs:

Tools perform actions or computations: create_ticket, search_logs, get_payment_status.
Resources expose context or data: service://payments/runbook, schema://customer.
Prompts provide reusable prompt templates or workflows a client can offer to a user.
Skills are usually instruction packages that explain how to approach a task, often with supporting files, examples, or scripts. Their behavior depends on the host that loads them.

A skill can tell an agent how to investigate a failed payment. MCP tools can provide the live capabilities it needs to query a payment status or inspect an approved log source.

flowchart LR
    U[Investigate failed payment] --> SK[Skill: playbook]
    SK --> A[Agent]
    A --> M[MCP tools and resources]
    M --> O[Evidence-based summary]

MCP vs. skills: a practical comparison

Question	MCP	Skill
Primary job	Standardized access to capabilities and context	Instructions for performing a task well
Can execute an external operation?	Yes, through a tool handler	Not inherently; may direct the agent to tools or scripts
Can explain a multi-step method?	Possible, but not its main purpose	Yes—this is a core strength
Portability	Across compatible MCP clients, subject to supported features and auth	Depends on host skill format and supported loading behavior
Typical failure mode	Tool ambiguity, unsafe access, poor schemas, latency	Instructions may not load, may be stale, or may reference unavailable tools

The “MCP is dead; Skills won” debate gets less useful when it assumes we have to choose one. A Skill can tell an agent when and how to investigate a problem; MCP can provide the actual capability to query a service or retrieve data. There is overlap, of course. A small workflow might be perfectly fine as a Skill that uses tools the host already has. I’d choose based on the runtime, reuse, and governance needs—not the latest hot take.

5. Don’t turn your whole Swagger file into a tool catalog

Imagine you have a Swagger/OpenAPI file with 500 endpoints. Exposing all 500 as MCP tools might feel like the quickest route, but it usually pushes the hard part onto the model: figuring out which endpoint matters, what sequence to call, and which fields are valid.

A discovery layer can help, but it should narrow the options to approved operations—not decide permissions.

A better design adds a discovery and policy layer:

flowchart TD
    O[Swagger / OpenAPI] --> I[Index and group endpoints]
    I --> G[Allowlist and authorization]
    G --> M[Discover focused MCP tool]
    M --> V[Validate and call API]
    V --> R[Normalize result and errors]

For example, index endpoint metadata such as operation ID, method, path, summary, tags, request schema, response schema, and required scopes. Search that index using keywords and semantic similarity if useful. But retrieval should only help select from an approved catalog; it must not grant permissions.

Expose focused operations such as get_card_summary or list_payment_methods when they map to meaningful user tasks. Keep raw endpoint access behind a deliberate, restricted tool only if advanced workflows truly need it.

JSON index or vector database?

Start with a generated JSON index when the catalog is manageable and exact keyword/tag search works.
Add embeddings or hybrid search when users describe operations in natural language and vocabulary differs from API terminology.
Keep the source of truth in OpenAPI; regenerate the index and test it in CI.
Never treat a vector match as authorization or proof that an operation is safe.

6. Make failure useful

A tool call will fail sometimes. The design question is whether the failure helps the agent recover or leaves it guessing.

An MCP tool is a contract between an uncertain caller and a real system. Validate every argument server-side—even if the schema is strict—because clients, models, and callers can still be wrong.

A useful result should be compact, structured when appropriate, and explicit about what happened.

Weak error

{ "error": "Request failed" }

Actionable error

{
  "error": {
    "code": "INVALID_PERIOD",
    "message": "The period must be one of: 7d, 30d, 90d.",
    "retryable": false
  }
}

A good error tells the agent what can be corrected without exposing stack traces, secrets, internal hostnames, or sensitive data. In other words: be specific about the fix, not about your internal implementation.

Design for:

Bounded output: pagination, result limits, field selection, summaries, and detail-on-demand.
Predictable errors: stable error codes and useful recovery hints.
Timeouts and cancellation: downstream calls must not hang indefinitely.
Retries: retry only transient failures; avoid blindly retrying non-idempotent actions.
Idempotency: for operations such as creating a ticket or initiating a payment, support an idempotency key or equivalent deduplication mechanism.
Explicit state: if a multi-step operation needs a durable handle, return an opaque workflow or resource ID and validate it on every subsequent call.

For long or multi-stage tasks, avoid pretending one giant tool call is always the best interface. Use explicit stages, durable state, and status checks where they improve observability and recovery.

7. Security can’t live in the tool description

A description saying “only use this carefully” is not a security control. The server has to enforce the rules even when the model gets the call wrong.

MCP does not make an unsafe backend safe. It can make a capability easier for an agent to reach, so access control and execution boundaries matter.

A safer request path

flowchart LR
    A[Agent] --> P[Authenticate and authorize]
    P --> V[Validate request]
    V --> H[Run tool safely]
    H --> L[Audit and monitor]

Minimum production checklist

Apply least privilege to tools, service accounts, database roles, and filesystem access.
Enforce authorization on the server for every call; never rely on a tool description or model judgment.
Separate read-only tools from write or destructive tools.
Require explicit user confirmation for consequential actions, such as deleting records, sending messages externally, or moving money.
Validate paths, URLs, identifiers, enum values, and payload sizes. Prevent path traversal, SSRF, SQL injection, and command injection.
Use parameterized queries and safe process APIs; do not concatenate model-generated text into SQL or shell commands.
Keep secrets out of prompts, tool descriptions, errors, and logs.
Treat tool outputs, retrieved documents, and remote content as untrusted input: they may contain prompt-injection instructions.
Pin and review server dependencies, scan code, sandbox where appropriate, and monitor changes to tool definitions and behavior.
Use authenticated transport for remote servers, and verify identity, token audience, scopes, and authorization boundaries.
Record who called what, with which authorization context, and what outcome occurred—without logging unnecessary sensitive payloads.

A tool called delete_customer is not safe merely because its description says “be careful.” Enforce the rules in code. For high-impact actions, use a two-phase design:

prepare_transfer(amount, destination)
  -> transfer summary + policy checks + confirmation token

confirm_transfer(confirmation_token)
  -> execute only if token is valid, unexpired, and bound to the approved request

The token should be server-generated and bound to the exact action, actor, and relevant parameters. Confirmation must not become a generic approval for a different action.

8. A few patterns I’d choose—and a few I’d avoid

Prefer	Avoid	Why
`get_payment_status(payment_id)`	`do_operation(type, payload)`	Intent is clearer and arguments are easier to validate
Small, domain-focused servers	One giant server for every company system	Limits blast radius and simplifies ownership
Short default responses + detail tools	Returning every field by default	Reduces token cost and noise
Server-side authorization	“The model knows not to do that”	Models are not security boundaries
Read/write separation	One tool with hidden destructive modes	Makes risk and approval requirements explicit
Stable error codes + recovery hints	Raw stack traces or “failed”	Supports reliable recovery without leaking internals
Evaluation fixtures and telemetry	“It worked in my demo”	Makes regressions measurable
MCP where reuse/isolation matter	MCP for every helper function	Avoids needless infrastructure and latency

The trade-offs you should acknowledge

MCP brings a common interface and a useful integration boundary, but it also adds serialization, process/network hops, configuration, version compatibility, and another component to operate. Tool discovery and rich schemas can consume context. A standardized protocol does not guarantee standardized business semantics. And security risks still exist at the tool, identity, and downstream-system layers.

The answer isn’t to dismiss MCP or put everything behind it. It’s to measure whether the boundary earns its keep: does it improve reuse, safety, consistency, or operations enough to justify the extra moving parts?

9. Test whether the agent can use the tools—not just whether the code runs

Unit tests can prove that a handler works for valid input. They cannot prove that an agent will select it correctly.

Build an evaluation set with realistic requests:

“Show my latest card payment” (clear intent)
“What happened to the payment from yesterday?” (ambiguous identifiers)
“Delete all test records” (potentially destructive)
“Find the card endpoint” (domain-language mismatch)
“Show me everything” (output explosion risk)

Track at least:

Correct tool selection
Argument validity
Task completion and factual correctness
Number of tool calls and retries
Latency, errors, and token usage
Unauthorized-action rejection
Regression rate after schema or description changes

Test against more than one model if your server is meant to support multiple clients. Use deterministic fixtures for business rules and separate model-dependent evaluations from ordinary unit tests. Include adversarial inputs and malicious content in retrieved data.

10. How I’d build a first version

For a new MCP server, build in this order:

Define the boundary: users, systems, data, permissions, and operations in scope.
Pick a few high-value tasks: avoid exposing the entire backend on day one.
Design schemas: clear names, tight constraints, safe defaults, and bounded output.
Separate primitives: tools for actions, resources for context, prompts for reusable templates; use skills for task instructions when the host supports them.
Implement policy in code: authentication, authorization, validation, approval, timeouts, and idempotency.
Add observability: correlation IDs, structured logs, metrics, traces, and audit events.
Create evals before broad rollout: include happy paths, ambiguous prompts, edge cases, and attack scenarios.
Measure and iterate: improve tool discovery and descriptions using actual failure data.
Version and govern: review changes to tool names, schemas, permissions, and semantics as API changes.

The point isn’t to expose more tools

The best MCP server isn’t the one with the longest tool list. It’s the one an agent can use without guessing, retrying needlessly, or being given more access than it needs.

Use MCP when a shared, governed interface is worth the extra boundary. Use Skills to explain a repeatable way of working when the host supports them. Use a direct function when that’s all you need. And combine the approaches when each has a clear job.

For me, the design goal is simple: make the right action easy to choose, hard to misuse, and straightforward to debug.

ACP: AI Agents Need a Common Language

Arif Sefa — Wed, 21 Oct 2026 16:58:33 +0000

From isolated AI assistants to interoperable agent systems, with practical examples in software development, banking, and distributed systems.

What happens when your AI agents are good at their jobs but terrible at working together?

Your backend agent is written in Python. Your frontend agent runs on TypeScript. Your testing agent uses a different framework, and your incident analysis agent lives in another service.

Individually, everything works. But the moment you try to connect them, you start writing adapters, translating payloads, handling different execution models, and maintaining a growing collection of custom integrations.

I've been thinking about this problem while exploring multi-agent development workflows. The interesting challenge isn't just getting an agent to complete a task. It's making independently developed agents collaborate without coupling the entire system to one framework or runtime.

That's where Agent Communication Protocol (ACP) becomes interesting.

In this article, we'll explore the original IBM/BeeAI ACP approach, walk through practical examples, compare ACP with MCP and A2A, and discuss what these protocols mean when you're building a real multi-agent system.

One important note before we begin: ACP can refer to different projects. This article focuses on the original IBM/BeeAI Agent Communication Protocol, whose development has since converged with the A2A ecosystem.

TL;DR

ACP was designed to standardize communication between independently implemented AI agents.
MCP focuses on how AI applications connect to tools, resources, and external capabilities.
A2A provides a standardized approach to communication and task delegation between agents.
These protocols address different integration boundaries and can complement one another.
A protocol doesn't replace orchestration, security, observability, or reliable error handling.
For new implementations, check the current A2A specification and official ACP migration guidance before choosing a protocol or SDK.

1. The problem isn't building agents. It's connecting them.

Building a single AI agent is becoming relatively straightforward. Frameworks provide model integrations, tool calling, memory, planning, and workflow execution.

The harder problem appears when you want multiple agents to collaborate.

Consider three independently developed agents:

A research agent that uses Python.
A reporting agent that runs on Node.js.
A visualization agent that uses a separate service.

Each agent might work perfectly on its own. However, they may expose completely different interfaces.

One expects:

{
  "query": "Analyze monthly revenue"
}

Another expects:

{
  "task_type": "report",
  "parameters": {
    "metric": "revenue",
    "period": "monthly"
  }
}

And the third might require a CSV file uploaded to a particular endpoint.

Now the orchestrator needs to understand all three interfaces, translate requests, handle errors, and interpret the results.

Add five more agents, and the integration layer becomes another software product you have to maintain.

This is the actual problem: agents can be intelligent individually but still be incompatible as a system.

A communication protocol aims to standardize the interaction layer without requiring every agent to use the same internal implementation.

2. What exactly is ACP?

The Agent Communication Protocol developed by IBM Research and the BeeAI ecosystem is an open protocol intended to make AI agents interoperable.

The core idea is simple:

An agent should be able to communicate through a predictable interface without requiring the caller to understand its internal implementation.

For example, the caller shouldn't need to know whether the receiving agent uses Python or TypeScript, LangGraph or a custom execution engine, a cloud-hosted LLM or a locally deployed model.

The caller needs to understand the agent's capabilities, how to send a request, and how to interpret the response.

Illustration: An orchestrator coordinates specialized agents through a communication layer. Source: CogitX — Multi-Agent AI Systems Architecture Guide.

Conceptually, the architecture looks like this:

                  User Request
                       |
                       v
                Orchestrator Agent
                       |
              Standardized Interface
                       |
           +-----------+-----------+
           |           |           |
           v           v           v
      Research      Coding      Testing
       Agent        Agent        Agent
       Python      TypeScript    Python
           |           |           |
           v           v           v
       Research      Code        Test
        Result      Changes      Report
           |           |           |
           +-----------+-----------+
                       |
                       v
                 Final Response

The agents remain independent. The communication layer provides a common way for them to exchange requests and results.

This distinction matters. ACP is a communication protocol, not a magic system that automatically makes agents intelligent, coordinates every decision, or guarantees correct results.

You still need application logic to decide which agent should do what, when to invoke it, and what to do if something goes wrong.

3. A practical example: building a feature with multiple agents

Let's use a realistic software development scenario.

A developer enters this request:

Add a virtual card creation screen, implement the corresponding backend endpoint, and write tests for the new flow.

The orchestrator breaks the request into tasks.

Step 1: The planner identifies the work

{
  "task": "Implement virtual card creation",
  "subtasks": [
    "Inspect the existing card creation flow",
    "Implement the backend endpoint",
    "Build the mobile screen",
    "Write integration tests",
    "Review security and validation"
  ]
}

This is an illustrative application-level task description, not a prescribed ACP wire format.

Step 2: The orchestrator delegates to specialized agents

The backend agent receives the API task. The frontend agent receives the screen requirements. The testing agent receives the expected behavior and acceptance criteria.

Each agent can use its own framework and tools.

Step 3: Agents return structured results

For example, the backend agent might produce:

{
  "status": "completed",
  "summary": "Implemented virtual card creation endpoint",
  "artifacts": [
    "VirtualCardController.cs",
    "CreateVirtualCardRequest.cs",
    "VirtualCardService.cs"
  ],
  "tests": {
    "passed": 12,
    "failed": 0
  }
}

The frontend agent could return a similar result containing its changed files, UI behavior, and test status.

Step 4: The orchestrator combines the results

It checks dependencies, determines whether the frontend can integrate with the endpoint, and sends any failures back to the relevant agent.

The testing agent might discover that the backend accepts a missing cardholder identifier. The orchestrator can route that finding back to the backend agent for correction.

The important point is that the agents don't have to share the same runtime or implementation framework to participate in the same workflow.

4. What happens when an agent needs another agent?

Suppose a customer support agent receives this request:

My virtual card was created successfully, but I can't see it in the mobile application.

The support agent doesn't necessarily have the expertise or permissions to diagnose the entire issue.

It may need help from several specialized agents:

A card-service agent checks whether the virtual card exists.
An API diagnostics agent investigates the backend response.
A mobile diagnostics agent checks the card-list rendering flow.
A reporting agent combines the findings.

An illustrative interaction might look like this:

Support Agent
     |
     | Diagnose missing virtual card
     v
Orchestrator
     |
     +----> Card Agent
     |          |
     |          +----> Card exists
     |
     +----> API Agent
     |          |
     |          +----> Response is successful
     |
     +----> Mobile Agent
                |
                +----> Card list uses stale cached data

     |
     v
Orchestrator combines evidence
     |
     v
Support Agent returns diagnosis

Notice that the agents don't need access to one another's private prompts, internal reasoning, or entire memory stores.

They need a well-defined way to exchange the information required for the task.

Collaboration doesn't have to mean sharing everything.

In a real implementation, the orchestrator would also need to validate permissions, ensure that sensitive customer information is handled appropriately, and distinguish confirmed findings from hypotheses.

5. ACP's design: why HTTP matters

One of the notable characteristics of the IBM/BeeAI ACP design is its HTTP-native, REST-oriented approach.

That is useful because most production systems already understand HTTP.

You can inspect requests, test endpoints, apply standard networking infrastructure, and integrate with existing authentication and observability systems.

The original protocol supports synchronous and asynchronous interactions, streaming through Server-Sent Events (SSE), and messages containing different kinds of content.

Let's look at what these concepts mean for developers.

Synchronous communication

A caller sends a request and waits for the result.

Client
  |
  | Request
  v
Classification Agent
  |
  | Result
  v
Client

An illustrative request might be:

{
  "task": "classify_support_ticket",
  "input": {
    "message": "My card payment was declined"
  }
}

The corresponding application-level response could be:

{
  "category": "card_payment",
  "confidence": 0.94
}

These JSON objects illustrate the task and result; they are not intended to define ACP's exact wire schema.

Asynchronous communication

Now imagine asking an agent to analyze a repository containing hundreds of projects.

That job might take several minutes. Keeping an HTTP request open for the entire execution may be inconvenient or unreliable.

Instead, the caller can initiate a task, receive an identifier, and check its progress later.

Orchestrator
     |
     | Start repository analysis
     v
Analysis Agent
     |
     | Return task identifier
     v
Orchestrator
     |
     | Continue other work
     |
     | Check status later
     v
Analysis Agent
     |
     | Completed result
     v
Orchestrator

An illustrative task record could be:

{
  "task_id": "repo-analysis-42",
  "status": "running",
  "progress": {
    "completed": 3,
    "total": 8
  }
}

Later, the agent might return:

{
  "task_id": "repo-analysis-42",
  "status": "completed",
  "result": {
    "services_analyzed": 128,
    "dependencies_found": 46,
    "issues_identified": 7
  }
}

These are conceptual application-level examples rather than canonical ACP payloads.

The architectural benefit is that the caller doesn't need to block while the agent performs a long-running task.

Streaming communication

Sometimes waiting for the complete response isn't ideal either.

Imagine a code review agent analyzing a large merge request. The user wants to see findings as they become available.

A streaming interaction can expose progress and incremental output:

Review started
    |
    +--> Inspecting changed files
    |
    +--> Checking dependency changes
    |
    +--> Found a possible authorization issue
    |
    +--> Reviewing test coverage
    |
    +--> Final report available

This is useful for developer tools, interactive assistants, monitoring dashboards, and long-running research workflows.

The exact streaming events and payloads depend on the protocol implementation and the application's contract.

6. Agent discovery: how do you know which agent to call?

Suppose your organization has 50 agents.

Some analyze logs, some review code, others investigate payment failures, and a few generate reports.

Hardcoding every agent into the orchestrator creates a maintenance problem. Every time a new agent is added, the orchestrator may need another integration.

Discovery addresses a related problem: identifying available agents and understanding their declared capabilities.

Imagine an agent description like this:

{
  "name": "log-analysis-agent",
  "description": "Analyzes application logs and identifies likely causes of failures",
  "capabilities": [
    "error_correlation",
    "request_trace_analysis",
    "failure_summary"
  ],
  "input_types": [
    "text/plain",
    "application/json"
  ],
  "output_types": [
    "application/json"
  ]
}

This is an illustrative agent manifest, not an exact ACP specification.

The orchestrator can use this metadata to determine that the log analysis agent is a candidate for a request involving an HTTP 500 error.

For example:

Find the likely reason why virtual card creation failed for a specific request.

The orchestrator identifies the task as log analysis, selects an agent that advertises the relevant capability, and delegates the work.

A different request, such as generating a chart from the findings, could be routed to a visualization agent.

Discovery becomes particularly valuable when agents are developed by different teams or deployed independently.

One important limitation: declared capabilities are not proof that an agent can solve a task correctly. A production system still needs validation, access control, health monitoring, and fallback behavior.

7. A real-world example: investigating a failed payment

Let's move from the development team to a banking environment.

A payment request failed. The user wants to know why.

Several systems may be involved:

The mobile application.
A backend-for-frontend (BFF).
The payment service.
An authorization service.
An external payment provider.
Centralized logs and tracing.

A traditional diagnostic script might follow a fixed sequence of API calls. An agent-based system can instead delegate specialized investigations and combine their evidence.

                    Incident Request
                           |
                           v
                    Triage Agent
                           |
             +-------------+-------------+
             |             |             |
             v             v             v
         Log Agent      API Agent     Payment Agent
             |             |             |
             v             v             v
        HTTP 500       Request DTO    Provider timeout
             |             |             |
             +-------------+-------------+
                           |
                           v
                     Evidence Review
                           |
                           v
                     Final Diagnosis

The log agent might find a correlated error. The API agent could confirm that the request reached the correct endpoint. The payment agent might identify a timeout from the external provider.

An example of a structured finding:

{
  "finding": "Payment provider timeout",
  "evidence": [
    {
      "source": "payment-service",
      "observation": "Provider request exceeded the configured timeout"
    },
    {
      "source": "application-logs",
      "observation": "The payment request returned HTTP 500"
    }
  ],
  "confidence": "medium",
  "recommended_next_step": "Check provider availability and retry policy"
}

This example deliberately separates evidence from the recommendation.

The agent should not claim that the provider caused the failure merely because it found an HTTP 500. The evidence must support that conclusion.

This is where a robust agent architecture goes beyond connecting endpoints: it needs traceability, structured results, and a way to distinguish observed facts from inferred explanations.

ACP can help standardize agent communication. It does not, by itself, guarantee accurate diagnoses or provide the entire incident orchestration system.

8. ACP vs. MCP vs. A2A: what is the difference?

These names are easy to confuse because they all appear in discussions about agentic AI.

The easiest way to understand them is to ask what kind of interaction you're trying to standardize.

MCP: connecting an AI application to tools and data

The Model Context Protocol (MCP) provides a standard way for AI applications to access tools, resources, and prompts exposed by MCP servers.

For example, a coding assistant might use MCP to access GitLab merge requests, an Elasticsearch query tool, a database schema, a filesystem, or an internal API.

AI Assistant
     |
     | Call search_logs tool
     v
MCP Server
     |
     v
Elasticsearch
     |
     v
Search Results

The main question is: How does the AI application access an external capability through a standard interface?

ACP: standardizing communication between agents

The IBM/BeeAI ACP design focuses on enabling agents implemented in different environments to exchange messages through a common interface.

For example, a support agent can request a diagnosis from a specialized incident agent without needing to know how that agent performs its internal analysis.

The main question is: How can independently developed agents communicate with one another?

A2A: interoperable agent-to-agent communication

Google introduced the Agent2Agent (A2A) protocol to enable agents to discover and collaborate with one another across different platforms.

It provides a standardized way to describe agent capabilities and interact with remote agents, including task-oriented workflows.

The main question is: How can agents built by different teams and organizations work together through a common protocol?

Illustration: MCP and A2A address different communication boundaries. Source: Jimmy Song — Agentic Design Patterns, Inter-Agent Communication.

Here's a simplified comparison:

Aspect	MCP	IBM/BeeAI ACP	A2A
Primary focus	AI applications accessing tools and context	Communication between agents	Communication and task delegation between agents
Typical interaction	Assistant → tool	Agent → agent	Client/agent → remote agent
Standardized interface	Tools, resources, prompts	Agent messaging and interaction	Agent discovery and task interaction
Common example	Query application logs	Ask a diagnostic agent to investigate	Delegate a task to a remote specialist
Orchestration	Usually handled by the application	Not the complete orchestration layer	Application logic still coordinates the overall workflow

This is a conceptual comparison, not a complete feature-by-feature specification.

The important distinction is that MCP and agent-to-agent protocols address different interaction boundaries. They are not necessarily competing alternatives.

How they can work together

Consider an incident investigation assistant.

The orchestrator delegates a task to a remote diagnostic agent through an agent-to-agent interface. The diagnostic agent then uses MCP to query internal logs and access approved tools.

User
 |
 v
Incident Orchestrator
 |
 | Agent-to-agent request
 v
Diagnostic Agent
 |
 | MCP tool call
 v
Log Search Server
 |
 v
Elasticsearch
 |
 v
Diagnostic Agent
 |
 | Structured result
 v
Incident Orchestrator
 |
 v
User

This is a useful architectural pattern because each protocol serves a different purpose.

The agent-to-agent layer handles collaboration. The tool-access layer handles access to external capabilities.

You don't need to expose the diagnostic agent's internal tool implementation to every agent that requests its help.

9. What does agent-to-agent communication look like?

A useful way to understand A2A is to look at how one agent can discover and delegate work to another.

For example, a research agent might need a separate data analysis agent to process a spreadsheet.

The client first discovers the remote agent's declared capabilities, then sends a task using the protocol's defined interaction model.

Illustration: An A2A interaction using JSON-RPC over HTTP, with optional SSE streaming and task states. Source: Peter Rosemann — A Developer's Deep-Dive into Google's A2A Protocol.

Conceptually, the workflow looks like this:

Research Agent
      |
      | Discover available capabilities
      v
Remote Agent Description
      |
      | Select suitable agent
      v
Data Analysis Agent
      |
      | Execute task
      v
Structured Result
      |
      v
Research Agent

The key idea is that the calling agent does not need to know the internal implementation of the remote agent.

A2A defines the communication contract; the remote agent remains responsible for executing its task.

For exact message fields, task states, and discovery behavior, consult the official A2A documentation.

10. An important update: ACP and A2A are converging

There is a detail worth understanding before choosing a protocol for a new project.

The original IBM/BeeAI Agent Communication Protocol should not be treated as an entirely separate, equally active standard alongside A2A without qualification.

IBM Research's ACP project page states that ACP is now part of A2A under the Linux Foundation and directs developers toward migration guidance.

That changes how I would approach a new implementation.

If you're building a new multi-agent system today, I would:

Understand the original ACP design because it provides useful architectural lessons about interoperability and lightweight agent communication.
Review the current A2A specification and official migration documentation before selecting an implementation.
Avoid committing to an older ACP-specific SDK or endpoint contract without checking its current maintenance status.
Keep the application-level task model independent of the transport protocol where practical, making future migrations easier.

Why does this matter?

Protocols evolve. A system that tightly couples business logic to a particular communication SDK can become difficult to migrate when standards and ecosystems change.

A clean separation between business logic, agent capabilities, and transport adapters can reduce that risk.

This is not an argument against learning ACP. Understanding why it was designed helps explain the problems that agent interoperability standards are trying to solve.

11. What ACP does not solve

It's tempting to think that adopting a common protocol makes a multi-agent system interoperable, secure, and reliable automatically.

It doesn't.

A protocol can standardize communication, but several engineering problems remain.

Authentication and authorization

An agent that can investigate a payment incident should not automatically be able to initiate a payment or access every customer's data.

You still need identity, permissions, scoped credentials, and policy enforcement.

Reliable execution

What happens when an agent times out after starting a task? Can the caller retry safely? Could the operation run twice?

These questions require deliberate handling of timeouts, retries, idempotency, and task state.

Semantic compatibility

Two agents may both accept JSON and still disagree about what a field means.

For example, one agent might interpret amount as a value in major currency units while another expects minor units.

A common transport format doesn't eliminate the need for clear schemas and domain contracts.

Observability

When a workflow spans six agents, you need to know which agent ran, what task it received, how long it took, and where a failure occurred.

Correlation IDs, distributed tracing, structured logs, and audit events are still essential.

Trust and untrusted output

An agent's response is not automatically correct or safe. It could contain a mistaken diagnosis, malformed data, or instructions that should not be trusted.

Validate results before using them in sensitive operations.

For enterprise systems, these concerns are not optional details. They're part of the architecture.

12. How I would design a multi-agent system today

If I were building an agent platform from scratch, I would separate it into four layers.

+------------------------------------------------+
|                  User Interface                |
|              Chat / Dashboard / API             |
+------------------------------------------------+
                        |
                        v
+------------------------------------------------+
|              Orchestration Layer               |
| Planning | Routing | Dependencies | Policies    |
+------------------------------------------------+
                        |
                        v
+------------------------------------------------+
|                Agent Layer                     |
|                                                |
| Research | Backend | Frontend | Test | Review  |
+------------------------------------------------+
                        |
                        v
+------------------------------------------------+
|             Integration Layer                  |
| Agent Protocol | MCP | Internal REST APIs       |
+------------------------------------------------+
                        |
                        v
+------------------------------------------------+
|           Enterprise Infrastructure            |
| GitLab | Logs | Databases | Services | CI/CD    |
+------------------------------------------------+

Each layer has a different responsibility.

The UI shows the task, execution progress, results, and approvals.
The orchestrator plans the workflow, selects agents, manages dependencies, and decides when human input is required.
The agents perform specialized work and return structured results.
The integration layer connects agents to one another and to external capabilities.
The infrastructure contains the actual business systems and data.

For example, a testing agent might discover that a backend task has completed and begin integration testing. If the tests fail, the orchestrator can send the failure report back to the backend agent.

The protocol helps the agents exchange information. The orchestrator determines what happens next.

That distinction is worth keeping clear when designing a system with multiple autonomous components.

13. The bigger idea: agents should be replaceable components

One of the strongest arguments for agent interoperability is that you shouldn't have to rebuild your entire system every time you replace one agent.

Suppose your original code review agent uses one framework and model provider.

Six months later, you find a better implementation that is faster, cheaper, or more accurate.

If the rest of the system depends on the agent's private APIs, memory format, and internal execution model, replacing it can be expensive.

If the agent exposes a stable, well-defined interface, the replacement becomes much more manageable.

The same principle applies to organizations.

A logistics provider could expose an agent that estimates delivery times. A retailer could use that capability without needing to understand the provider's internal algorithms or infrastructure.

The integration still needs appropriate authentication, contracts, and compatibility checks. But the architectural goal is clear: make the interaction contract stable while allowing implementations to evolve.

That's a familiar idea in software engineering. We already rely on interfaces, service contracts, and abstractions to decouple systems.

Agent communication protocols apply similar principles to a world in which the services being connected can reason, delegate tasks, and produce dynamic outputs.

Conclusion

The next challenge in agentic AI isn't simply creating agents that can perform increasingly complex tasks.

It's creating systems in which independently developed agents can collaborate reliably.

ACP was an important step toward that goal. Its emphasis on lightweight, HTTP-native communication, asynchronous execution, and interoperability offers useful lessons for anyone building multi-agent systems.

MCP and agent-to-agent protocols address different parts of the integration problem, and they can complement one another within the same architecture.

At the same time, the evolution of these standards matters. IBM Research now directs developers toward A2A for the continued evolution of the original ACP work.

For developers, I think the most useful takeaway is this:

Don't build a multi-agent system around the assumption that every agent will always use the same framework, model, or runtime. Build around clear capabilities and stable communication contracts instead.

The more independently your agents can evolve, the easier it becomes to experiment, replace components, and scale the system without rewriting everything around them.

And that is what makes agent interoperability an engineering problem worth paying attention to.

References and further reading

Note: The arXiv paper proposes a separate research approach also called ACP. It should be read as a research proposal, not as the official specification of the original IBM/BeeAI protocol. The JSON examples in this article are illustrative application-level payloads, not canonical ACP or A2A wire formats.

Why Most Multi-Agent Systems Don't Ship Software?

Arif Sefa — Fri, 16 Oct 2026 18:18:24 +0000

Building AI agents is getting easier. Getting them to work together and deliver reliable software is a different story.

I've been spending quite a bit of time thinking about how AI agents can work together to build software.

At first, the idea seems pretty straightforward. Instead of asking one AI model to handle everything, we can split the work between specialized agents. One plans the tasks, another writes the frontend, another handles the backend, and another tests the result.

Everyone has a role. Work can happen in parallel. In theory, we should be able to build software faster.

But there's a catch.

Getting several agents to generate code is relatively easy. Getting them to produce software that actually works together is much harder.

Imagine asking a frontend agent and a backend agent to build a task management application.

The frontend agent creates a beautiful task list and expects the API to return an assignedUser field. Meanwhile, the backend agent implements the endpoint using a field called assignee.

Both agents finish their tasks. Both report success. But the application doesn't work.

This is a small example, but it illustrates a much bigger problem. Each agent can do a reasonable job independently while the overall system still fails.

And these problems don't disappear just because we use a better model or write a more detailed prompt.

The real challenge is coordinating the work.

Start with the workflow, not the agents

When designing a multi-agent system, it's tempting to start by defining different roles:

A planner that breaks down requirements.
A frontend agent that writes React code.
A backend agent that builds APIs.
A tester that checks the implementation.
A reviewer that looks for bugs.

This is a reasonable starting point, but I think there's a more important question to answer first:

How does work move from one stage to the next?

Before deciding how many agents we need, we should understand what needs to happen for a feature to go from an idea to working software.

For example, a simple development workflow could look like this:

Requirement
    |
    v
Planning
    |
    v
Define API Contracts
    |
    +----------+
    |          |
    v          v
 Frontend    Backend
    |          |
    +-----+----+
          |
          v
       Testing
          |
          v
        Review
          |
          v
       Delivery

Notice that the frontend and backend can work in parallel, but only after the API contract is defined.

That small detail makes a big difference.

Without a shared contract, each agent has to make assumptions. The more assumptions they make independently, the more work we may need to redo later.

This is why I prefer to think about the workflow first and the agents second.

Let the task dependencies decide what runs in parallel

Let's say we're building a task management application.

We have five tasks:

const tasks = [
  {
    id: "requirements",
    dependencies: [],
  },
  {
    id: "api-contract",
    dependencies: ["requirements"],
  },
  {
    id: "frontend",
    dependencies: ["api-contract"],
  },
  {
    id: "backend",
    dependencies: ["api-contract"],
  },
  {
    id: "integration-tests",
    dependencies: ["frontend", "backend"],
  },
];

The frontend and backend don't need to wait for each other. They only need to wait for the API contract.

Once that contract is ready, both tasks can start.

A simple scheduler can determine which tasks are ready to run:

function getReadyTasks(tasks, completed) {
  return tasks.filter((task) => {
    if (completed.has(task.id)) {
      return false;
    }

    return task.dependencies.every((dependency) =>
      completed.has(dependency),
    );
  });
}

This is a simplified example, but the idea is straightforward: a task becomes eligible to run when its dependencies are complete.

In a real implementation, we'd also track running and failed tasks, handle retries, and prevent multiple workers from claiming the same task.

The important part is that we don't need to tell agents when to start by manually coordinating every step. We can let the dependency graph determine that.

Parallelism should come from the structure of the work, not from simply having multiple agents available.

An agent saying "done" doesn't mean the task is done

This is another problem worth thinking about.

Imagine a backend agent returns the following result:

{
  "status": "completed",
  "summary": "Implemented the task assignment API",
  "tests": "passed"
}

Looks good, right?

But what if the agent never actually ran the tests?

What if it ran the wrong command? Or what if the tests failed, but the agent misunderstood the output?

We shouldn't rely entirely on an agent's description of its own work.

Instead, the system should run the required checks and collect the results itself.

For example:

async function verifyProject(runner, workspace) {
  const checks = [
    ["npm", ["run", "typecheck"]],
    ["npm", ["test", "--", "--runInBand"]],
    ["npm", ["run", "build"]],
  ];

  const results = [];

  for (const [command, args] of checks) {
    const result = await runner.run(
      command,
      args,
      workspace,
    );

    results.push(result);

    if (result.exitCode !== 0) {
      return {
        passed: false,
        results,
      };
    }
  }

  return {
    passed: true,
    results,
  };
}

Here, runner represents a controlled command-execution service, and the example assumes the project has the corresponding npm scripts.

The distinction is simple: the agent writes the code, but the verification system checks whether it works.

This doesn't mean tests can prove that software is completely correct. They can't. But they give us independent evidence instead of relying on a model's confidence.

What happens when something fails?

Let's say the frontend expects this response:

{
  "items": [],
  "nextCursor": null
}

But the backend returns:

{
  "data": [],
  "total": 0
}

The integration test fails.

We could simply send the error back to the frontend agent and ask it to fix the problem. But that might be the wrong solution. Maybe the backend is the part that doesn't follow the agreed contract.

Before asking an agent to make another change, the workflow should identify what went wrong.

A useful failure report might look like this:

interface FailureReport {
  taskId: string;

  category:
    | "implementation_defect"
    | "contract_mismatch"
    | "test_defect"
    | "environment_failure";

  summary: string;
  evidence: string[];
}

This gives the orchestrator something more useful than a generic error message.

It can decide whether to retry the implementation, revisit the contract, fix the test, or ask a developer for help.

I'd also put a limit on automatic retries. If the same problem persists after a couple of attempts, the system should stop and explain what happened rather than keep generating code indefinitely.

Sometimes the best thing an agent can do is report that it is blocked.

The part I find most interesting

For me, the interesting part of multi-agent development isn't simply making agents more autonomous.

It's building a system where each piece of work has a clear purpose, where agents share the same assumptions, and where we can understand why a task succeeded or failed.

That requires a few things working together:

A planner that creates useful tasks rather than vague instructions.
A scheduler that respects dependencies.
Shared contracts that keep implementations consistent.
A controlled environment for making changes.
Automated checks that produce real execution results.
A review process that catches problems before delivery.

None of these ideas is particularly new in software engineering. We already use dependency graphs, CI pipelines, API contracts, isolated environments, and code reviews.

What's changing is that AI agents are now participating in these processes, and we need to design the surrounding system accordingly.

I wouldn't start by trying to make every agent fully autonomous. I'd start with a small workflow, make it observable, and see where it breaks.

Then I'd improve the parts that actually cause problems.

Final thoughts

Multi-agent systems can be useful, especially when a task has clear boundaries and several independent pieces of work.

But adding more agents doesn't automatically make software development faster or better.

If the agents don't share contracts, if their changes aren't integrated properly, or if nobody verifies the results, we may simply end up generating mistakes faster.

I think the more useful question is not how many agents we can run at once.

It's this:

How do we turn the work of several AI agents into a software change we can actually trust?

That's the problem I'd focus on when building an AI engineering workflow.

CodeAct: Why AI Agents Should Write Code, Not Just Call Tools

Arif Sefa — Tue, 13 Oct 2026 18:32:57 +0000

Executable Code Actions Elicit Better LLM Agents — Exploring a different way to build more capable AI agents.

Introduction: Are We Building AI Agents the Hard Way?

When building AI agents, we often focus on adding more tools, integrating more APIs, and connecting more MCP servers.

The assumption is straightforward: the more tools an agent has, the more capable it becomes.

But what if the real limitation isn't the number of tools available to an agent, but how the agent uses those tools?

This is the question explored by Executable Code Actions Elicit Better LLM Agents, a research paper presented at ICML 2024.

The paper introduces CodeAct, an approach in which an LLM can express actions through executable code instead of relying exclusively on conventional tool-call formats.

Rather than making isolated tool calls, an agent can write code that combines operations, processes intermediate results, controls execution flow, and interacts with an execution environment.

The idea is simple, but its implications for agentic AI are significant.

📄 Read the original research paper

1. What Is CodeAct?

Traditional LLM agents commonly interact with tools through predefined interfaces.

The model selects a tool, generates arguments, receives a result, and decides what to do next.

For example, an agent might need to retrieve customer records, filter them, and generate a report.

A tool-calling workflow could involve several separate calls:

{
  "tool": "get_customers",
  "arguments": {}
}

The model then receives the results and generates another tool call:

{
  "tool": "filter_customers",
  "arguments": {
    "status": "active"
  }
}

Finally, it might call another tool to generate the report.

This approach works well for many tasks. However, coordinating several dependent operations can require repeated model decisions, intermediate messages, and orchestration logic.

CodeAct offers another possibility.

The agent can express the workflow as executable code:

customers = get_customers()

active_customers = [
    customer
    for customer in customers
    if customer["status"] == "active"
]

report = generate_report(active_customers)

print(report)

The functions above represent tools exposed to the agent by its environment.

The important distinction is that the agent can express the relationships between operations directly in code.

It can use variables, loops, conditions, functions, and exception handling to coordinate its actions.

CodeAct treats executable code as an action format, not merely as text to be displayed to the user.

2. From Individual Tool Calls to Executable Workflows

Consider a task that requires collecting information from three services.

A conventional orchestration layer might explicitly define the sequence:

accounts = call_tool("get_accounts")
cards = call_tool("get_cards")
transactions = call_tool("get_transactions")

result = call_tool(
    "combine_financial_data",
    {
        "accounts": accounts,
        "cards": cards,
        "transactions": transactions
    }
)

A code-based approach can combine the same operations and perform additional processing locally within the execution environment:

accounts = get_accounts()
cards = get_cards()
transactions = get_transactions()

summary = {
    "account_count": len(accounts),
    "card_count": len(cards),
    "transaction_count": len(transactions),
    "total_spending": sum(
        transaction["amount"]
        for transaction in transactions
        if transaction["amount"] > 0
    )
}

print(summary)

This example assumes that the tools return data in the expected format and that positive transaction amounts represent spending.

The advantage is not simply fewer lines of code. It is the ability to define data flow and control flow dynamically.

An agent can decide how to process results without requiring a separate tool for every possible combination of operations.

Visualizing the difference

flowchart TD
    A[User Task] --> B[Traditional Agent]
    B --> C[Tool Call 1]
    C --> D[Observe Result]
    D --> E[Tool Call 2]
    E --> F[Observe Result]
    F --> G[Tool Call 3]
    G --> H[Final Answer]

    A --> I[CodeAct-Inspired Agent]
    I --> J[Generate Executable Code]
    J --> K[Execute Multiple Operations]
    K --> L[Observe Execution Result]
    L --> M{Task Complete?}
    M -- No --> J
    M -- Yes --> N[Final Answer]

The diagram illustrates the conceptual difference. Actual architectures vary, and conventional agents can also support batching, loops, and multi-step execution.

3. Why Executable Code Is a Powerful Action Space

3.1. Dynamic control flow

Not every task follows the same path.

Imagine an agent that retrieves customer data. If the request succeeds, it analyzes the result. If it fails, it investigates the error.

response = fetch_customer_data()

if response["status"] == "success":
    customer = response["data"]

    analysis = analyze_customer(customer)

    print(analysis)

else:
    error_details = response.get(
        "error",
        "Unknown error"
    )

    investigation = investigate_error(error_details)

    print(investigation)

In a rigid workflow, this branching logic might need to be defined in the orchestration layer.

With executable code, the agent can express the branch directly in the action it generates.

3.2. Repeated operations

Suppose an agent must inspect several repositories or services.

repositories = [
    "frontend",
    "backend",
    "shared-libraries"
]

results = {}

for repository in repositories:
    results[repository] = search_repository(
        repository,
        query="payment"
    )

for repository, matches in results.items():
    print(repository, len(matches))

A loop can coordinate repeated operations while preserving their results for later analysis.

The tools still need to be available to the execution environment. CodeAct does not automatically create new external capabilities.

3.3. Reusing intermediate results

An agent may need to process data from one operation before passing it to another.

transactions = get_transactions()

high_value_transactions = [
    transaction
    for transaction in transactions
    if transaction["amount"] >= 10000
]

customer_ids = {
    transaction["customer_id"]
    for transaction in high_value_transactions
}

customers = [
    get_customer(customer_id)
    for customer_id in customer_ids
]

print({
    "high_value_transactions": len(
        high_value_transactions
    ),
    "related_customers": len(customers)
})

Here, the agent can filter records, extract identifiers, and retrieve related entities using a single executable workflow.

In a real financial application, operations would need appropriate authorization, privacy controls, and clearly defined amount semantics.

3.4. Composing operations into reusable functions

Code can also make an action easier to understand and reuse.

def investigate_customer(customer_id):
    customer = get_customer(customer_id)

    transactions = get_transactions(customer_id)

    flagged_transactions = [
        transaction
        for transaction in transactions
        if transaction.get("flagged", False)
    ]

    return {
        "customer": customer,
        "transaction_count": len(transactions),
        "flagged_transactions": flagged_transactions
    }


result = investigate_customer("customer-123")

print(result)

This is useful when several operations form a logical unit of work.

It also introduces a new responsibility: the environment must control which functions the generated code is allowed to invoke.

4. The Execution Feedback Loop

One of the most interesting aspects of executable actions is the possibility of observing runtime feedback.

An agent does not have to stop after generating code. It can execute the code, inspect the result, and use that observation to decide what to do next.

The basic cycle is:

Generate → Execute → Observe → Revise

Consider this example.

The first attempt assumes a field exists:

response = fetch_customer()

print(response["customer_name"])

Suppose execution returns a KeyError.

The agent can inspect the returned structure:

response = fetch_customer()

print("Available fields:", list(response.keys()))

It might discover that the field is named name instead.

The next attempt can account for the actual structure:

response = fetch_customer()

customer_name = response.get(
    "customer_name",
    response.get("name", "Unknown")
)

print(customer_name)

This demonstrates how execution feedback can guide subsequent actions.

It does not mean that the model permanently learns from every error. Rather, the error becomes an observation that can inform the next step in the current task.

A simplified agent loop

task = "Retrieve and analyze customer data"

while not task_is_complete(task):
    code = llm.generate_code(
        task=task,
        observations=get_observations()
    )

    result = sandbox.execute(code)

    record_observation(result)

    if result.error:
        task = update_task_with_error(
            task,
            result.error
        )

print(get_final_answer())

This is conceptual pseudocode, not the exact implementation described in the research paper.

A production system needs explicit termination conditions, execution limits, observation handling, and protection against endless retries.

5. What Did the Researchers Discover?

The paper introduces M³ToolEval, a benchmark designed to evaluate tool use in tasks involving multiple tools and multiple interaction turns.

The researchers evaluate 17 different LLMs to investigate how executable code actions affect agent performance.

The paper reports improvements of up to:

20% in task success rate
30% fewer actions in reported experiments

These are maximum reported improvements, not guaranteed gains across all models, benchmarks, or tasks.

The findings suggest that how an agent represents and executes actions can have a significant impact on its ability to complete complex tasks.

The lesson is not simply that Python is better than JSON.

The more important point is that executable code offers a flexible action space in which an agent can coordinate operations, reuse intermediate results, and express conditional behavior.

For tasks that require several dependent operations, this flexibility can reduce the amount of explicit orchestration required.

6. CodeActAgent and CodeActInstruct

The research also introduces resources to support the approach.

CodeActAgent

CodeActAgent demonstrates an agent architecture that uses executable code to interact with tools and an execution environment.

CodeActInstruct

CodeActInstruct is a dataset of approximately 7,000 multi-turn interactions designed to support training models to use code as an action format.

The research includes models based on Llama 2 and Mistral, exploring how models can be trained to generate executable actions instead of relying exclusively on conventional tool-call formats.

This matters because the work investigates both the architecture and the model behavior needed to use it.

Explore the official CodeAct implementation on GitHub.

7. CodeAct and MCP: A Natural Combination?

MCP provides a standardized way for AI applications to discover and invoke tools exposed by external servers.

But having access to many tools does not automatically mean an agent can combine them efficiently.

Imagine an agent with access to these tools:

search_codebase(query)
read_file(path)
find_references(symbol)
get_file_structure(path)

A conventional agent could call these tools one at a time.

A CodeAct-inspired agent could coordinate them in a program:

query = "GET_CREDIT_CARD_DETAIL_DOCUMENT"

matches = search_codebase(query)

investigation = []

for match in matches:
    file_path = match["file"]

    source = read_file(file_path)

    investigation.append({
        "file": file_path,
        "match": match,
        "source_excerpt": source
    })

print(investigation)

The code above assumes the tools expose compatible return structures.

The distinction is important:

MCP standardizes how tools are exposed and invoked.
CodeAct explores using executable code as the format for agent actions.
An orchestration framework manages execution, state, permissions, and the broader agent lifecycle.

These ideas can complement one another, but they are not interchangeable.

A CodeAct-inspired architecture could use MCP tools as building blocks and generate code to coordinate them.

8. A Practical Developer-Agent Example: Trace an Endpoint to a Frontend Screen

Consider a banking application built around a frontend, GraphQL, a backend-for-frontend (BFF), and several microservices.

The task is:

Given a service endpoint, identify which frontend screens depend on it.

A useful investigation might follow this chain:

REST endpoint
    ↓
Backend service
    ↓
BFF service or resolver
    ↓
GraphQL operation
    ↓
Frontend hook or data layer
    ↓
Component
    ↓
Screen

A CodeAct-inspired agent could combine repository search tools to trace these relationships.

endpoint = "/creditcards"

service_matches = search_codebase(endpoint)

graphql_operations = set()

for match in service_matches:
    operations = find_related_graphql_operations(
        match["file"],
        match["line"]
    )

    graphql_operations.update(operations)

screens = set()

for operation in graphql_operations:
    consumers = find_frontend_consumers(operation)

    screens.update(consumers)

print({
    "endpoint": endpoint,
    "graphql_operations": sorted(graphql_operations),
    "screens": sorted(screens)
})

The search and tracing functions are hypothetical tools. They are not built-in CodeAct functions.

The agent can extend the investigation when it encounters incomplete results.

For example, it might search for an operation's name, inspect the matching file, and then follow imports into the frontend.

operation = "GET_CREDIT_CARD_DETAIL_DOCUMENT"

matches = search_codebase(operation)

frontend_files = []

for match in matches:
    if match["path"].startswith("apps/"):
        frontend_files.append(match["path"])

    elif match["path"].startswith("libs/"):
        frontend_files.append(match["path"])

unique_files = sorted(set(frontend_files))

print(unique_files)

This simplified example only filters search results by path. A real implementation would need to understand the repository structure, distinguish declarations from usages, and follow imports and data dependencies.

A possible investigation workflow

flowchart TD
    A[User provides endpoint] --> B[Search repository]
    B --> C[Find service implementation]
    C --> D[Trace BFF and resolver]
    D --> E[Find GraphQL operation]
    E --> F[Find frontend consumers]
    F --> G{Evidence complete?}
    G -- No --> H[Generate another search action]
    H --> B
    G -- Yes --> I[Return evidence-backed report]

This illustrates a possible CodeAct-inspired architecture, not a workflow implemented or evaluated in the paper.

The value is that the agent can dynamically compose existing search capabilities instead of requiring a separate hardcoded workflow for every endpoint.

However, CodeAct does not replace code indexing, semantic search, or reliable repository navigation. Those capabilities remain essential for large codebases.

9. Parallel Operations and Data Aggregation

Some investigations contain independent operations that can run in parallel.

For example, an agent might search the frontend, BFF, and shared libraries for the same operation.

Conceptually:

import asyncio

async def investigate_repository(repository, query):
    return await search_repository_async(
        repository,
        query
    )


async def investigate_all(query):
    repositories = [
        "frontend",
        "bff",
        "shared-libraries"
    ]

    tasks = [
        investigate_repository(repository, query)
        for repository in repositories
    ]

    results = await asyncio.gather(*tasks)

    return {
        repository: result
        for repository, result in zip(
            repositories,
            results
        )
    }


results = await investigate_all(
    "GET_CREDIT_CARD_DETAIL_DOCUMENT"
)

print(results)

This is an illustrative example. It assumes asynchronous tool functions and an execution environment that supports top-level await.

CodeAct does not automatically make tool calls parallel. The generated program, runtime, and available interfaces must support concurrency.

Parallel execution can improve latency when operations are independent, but it requires sensible concurrency limits, error handling, and resource management.

10. Does CodeAct Replace Traditional Tool Calling?

Not necessarily.

Traditional tool calling remains useful when operations are simple, well-defined, and require strict argument validation.

Executable code becomes more attractive when tasks involve:

Multiple dependent operations
Conditional or repetitive execution
Data transformation between tools
Dynamic workflows
Complex multi-step investigations

A hybrid design can use structured tool calls for well-defined operations and code-based orchestration when several operations need to be composed.

For example:

# A structured tool provides a trusted operation.
account = get_account(account_id)

# Generated code handles task-specific processing.
if account["status"] == "active":
    transactions = get_transactions(account_id)

    result = summarize_transactions(
        transactions
    )
else:
    result = {
        "message": "Account is not active"
    }

print(result)

The tool layer can preserve access controls and input validation while the agent uses code to coordinate the task.

The best architecture depends on the task, the available tools, the model, and the required security guarantees.

11. The Security Challenge

Giving an agent the ability to execute code increases its flexibility, but it also introduces risks.

Generated code must not receive unrestricted access to a host machine, production database, or sensitive filesystem.

A production implementation should consider:

Sandboxed execution
Execution time and memory limits
Restricted filesystem and network access
Explicit tool permissions
Validation of sensitive operations
Logging and auditing
Approval requirements for high-impact actions

A hypothetical sandbox interface might look like this:

result = sandbox.execute(
    code=generated_code,
    timeout_seconds=10,
    network_access=False,
    filesystem_access="restricted"
)

This is an illustrative API, not a real CodeAct function. Actual security controls depend on the execution platform.

The core principle is straightforward:

Give the agent flexibility over how it performs a task, but keep strict boundaries around what it is allowed to do.

Code execution should be isolated, observable, and governed by explicit permissions.

12. What CodeAct Does—and Does Not—Solve

CodeAct is not a complete solution to every agent problem.

It does not automatically provide:

Accurate reasoning
Reliable codebase indexing
Correct tool implementations
Persistent learning from previous failures
Guaranteed task completion
Safe execution by default

The approach changes the action space available to an agent. The quality of the outcome still depends on the model, the tools, the execution environment, and the way the task is evaluated.

This is an important distinction when deciding whether to adopt CodeAct in a real application.

Conclusion: Better Agents May Need Better Ways to Act

What makes CodeAct interesting is not simply that it allows an LLM to write Python. Models have been generating code for years.

The more significant idea is treating executable code as a general-purpose action format for AI agents.

Instead of designing every possible sequence of tool calls in advance, we can give agents a set of capabilities and let them compose those capabilities into executable workflows.

The research provides experimental evidence that this approach can improve performance on certain multi-tool tasks while reducing the number of actions required.

For developers building MCP integrations, agent orchestration platforms, and AI-powered developer tools, CodeAct offers a valuable architectural perspective.

Perhaps the next step in building more capable agents isn't simply adding more tools.

Perhaps it's giving agents better ways to use the tools they already have.