TriCall Harness: Separating LLM Planning, Execution Control, and Evidence Evaluation in Python Agents

kylin — Thu, 05 Nov 2026 07:02:43 +0000

An early-alpha, open-source Python implementation of a domain-driven Agent Harness design.

GitHub: https://github.com/jnukylinliu/tricall-harness

License: MIT

Language: Python

Status: Early Alpha

Introduction

Building an AI agent is relatively straightforward when the task is simple: interpret a request, select a tool, execute it, and return a response.

The challenge becomes more interesting when an agent must handle complex, domain-specific tasks.

Consider an agent that analyzes inventory, modifies Excel workbooks, or evaluates industrial equipment records.

In these applications, several responsibilities must be handled carefully:

Understanding what the user actually wants.
Identifying which registered capabilities can fulfill the request.
Executing those capabilities within defined permissions.
Determining whether the resulting evidence is sufficient.
Performing additional work when necessary.
Producing an answer that accurately reflects what was executed and what remains uncertain.

A successful tool invocation does not necessarily mean the user's request has been satisfied.

This distinction motivates TriCall Harness.

TriCall is built around a foundational design philosophy for general-purpose Agent Harnesses:

LLMs handle semantic uncertainty. Domain experts define professional intelligence. Python controls execution. An independent Evaluator reviews actual evidence before the final Answer.

The goal is not to make the language model responsible for everything, but to establish clear boundaries between reasoning, execution, professional knowledge, and evidence evaluation.

1. Three Questions, Three Responsibilities

TriCall organizes an Agent Harness around three semantic questions.

Question 1 — Planner: What should be done?

The Planner is an LLM-driven component responsible for understanding the user's request and selecting appropriate capabilities.

It works with an authorized catalog containing registered Tools, Skills, and Retrievers.

For example, a user might ask:

"Given our current inventory and recent sales, will we run out of stock before the next shipment?"

The Planner must understand that this request may require inventory information, sales history, replenishment information, and a stockout-risk assessment.

However, the Planner does not invent business algorithms or create unauthorized capabilities.

It selects and combines capabilities already provided by the application developer.

Execution — Python controls what actually happens

After planning, Python admission logic validates proposed steps before they reach the Runtime.

This includes checking registered capabilities, access scopes, arguments, duplicate proposals, and execution budgets.

The Runtime then invokes the authorized handlers and records their actual outcomes.

The important distinction is that the LLM proposes actions, while Python controls their admission and execution.

Question 2 — Evaluator: Is the evidence sufficient?

After execution, an independent Evaluator reviews the actual evidence.

It examines information such as:

The original request and bounded conversation context.
Which steps were executed.
The actual admitted arguments and returned results.
Execution statuses.
Domain-specific quality indicators.
Whether the evidence adequately supports the requested conclusion.

This phase is distinct from planning.

A handler may return successfully while its data is stale, incomplete, inconsistent, or insufficient for the user's question.

The Evaluator is responsible for identifying those gaps according to the Domain's defined criteria.

It can also propose additional capability calls.

Supplemental execution — Python Replanner

When more evidence is needed, TriCall uses a Python-controlled Replanner to admit additional steps proposed by the Evaluator.

The Replanner is not a fourth LLM planner.

It applies deterministic execution rules to supplemental proposals.

Those steps are executed, and the Evaluator reviews the updated evidence again.

Supplemental execution is not necessarily a correction of a planning mistake. Some tasks naturally require information discovered during an earlier execution step.

Question 3 — Answer: What can the evidence support?

Finally, an LLM Answer component produces the user-facing response.

Its responsibility is to communicate the supported result, including relevant uncertainty and missing evidence.

The distinction matters:

A completed tool call is not proof of a completed business task, and an Evaluator's approval is not proof of mathematical correctness.

2. The Execution Lifecycle

The basic lifecycle can be represented as:

User Request + Context
         |
         v
    Planner (LLM)
         |
         v
  Python Admission
         |
         v
   Runtime (Python)
         |
         v
   Evaluator (LLM)
         |
         +------ Evidence insufficient?
         |               |
         |               v
         |       Python Replanner
         |               |
         |               v
         |         Runtime
         |               |
         |               +----> Evaluator
         |
         v
     Answer (LLM)
         |
         v
       RunResult

Despite the name, TriCall does not mean exactly three model API calls.

It refers to three semantic responsibilities: planning, evaluation, and answering.

In the current implementation, normally completed runs retain all three roles, including simple requests with no executable steps.

The Evaluator may run multiple times when supplemental execution occurs.

The same underlying model can implement the semantic phases, while their responsibilities remain distinct.

3. Core and Domain: A Critical Architectural Boundary

One of TriCall's central design decisions is the separation between the reusable Harness and application-specific Domain logic.

The Core is responsible for the execution lifecycle.

It manages capability registration, admission, Runtime execution, bounded context, policies, execution results, stop conditions, and tracing.

The Domain defines professional capabilities and criteria.

It supplies:

Business Tools and Skills.
Data sources and Retrievers.
Professional algorithms and calculations.
Domain-specific prompts.
Data freshness and quality requirements.
Completion criteria.

Consider an inventory analysis application.

The Domain developer might define a Skill that calculates stockout risk using available stock, recent sales, and expected replenishment dates.

The Harness does not need to understand the mathematical details of that calculation.

Its job is to make the authorized capability available, execute it correctly, preserve the evidence, and organize subsequent evaluation.

A Skill can perform complex internal operations while exposing a single capability to the Harness.

This separation allows the same execution architecture to support different application domains without embedding industry-specific rules into the Core.

4. Why Evaluate Evidence After Execution?

Many agent workflows focus on selecting the correct tool.

That is necessary, but not always sufficient.

Suppose an inventory Tool returns a stock snapshot.

The call may have succeeded technically. However, the snapshot might be outdated.

Similarly, an Excel editing operation may successfully write a formula that is semantically incorrect.

These are different forms of success and failure:

Execution success: Did the handler complete?

Evidence quality: Is the returned information usable for the business task?

Task sufficiency: Does the available evidence support what the user requested?

TriCall keeps these concepts separate.

The Evaluator can inspect the execution evidence and domain-defined quality indicators before deciding whether to answer or request more work.

This does not eliminate model errors.

The Evaluator itself can make incorrect judgments. Domain algorithms, explicit quality metadata, testing, and application-level verification remain important.

The purpose is to make evidence review an explicit responsibility rather than assuming that successful execution automatically means sufficient evidence.

5. Registered Capabilities and Controlled Side Effects

TriCall exposes registered Tools, Skills, and Retrievers through a capability registry.

The application determines which capabilities exist and how they may be used.

For operations that modify external state, the current implementation requires explicit side-effect authorization.

For example, a write-capable operation must be registered accordingly, and the host application must enable side effects and expose the necessary scope.

An LLM cannot grant itself permission merely by requesting an operation.

However, these permissions are execution gates, not a substitute for a security sandbox, user authentication, or operating-system-level isolation.

Handlers run with the privileges of the hosting process.

This distinction is particularly important when building agents that modify files or interact with external systems.

6. Existing Domain Examples

The repository currently includes two primary demonstration Domains.

Retail Inventory Agent

The Retail example provides three read-only Tools and a stockout-risk Skill.

It demonstrates:

Natural-language capability selection.
Inventory and sales evidence.
Domain-defined data quality.
Compound requests.
Supplemental execution.
Evidence-based responses.

Its dataset is synthetic and uses a fixed scenario time.

The example is intended to demonstrate how a professional Domain integrates with the reusable Harness, not to provide a production inventory system.

Excel Agent

The Excel example supports reading, searching, summarizing, formatting, and editing authorized .xlsx workbooks.

Its current implementation includes five read Tools and two write Skills.

The application supports free-form English instructions and interactive sessions with bounded conversation history.

Write operations require explicit host authorization and save new workbook revisions.

Excel-specific file restrictions, workbook handling, formula semantics, and calculations remain within the Domain rather than being embedded in TriCall Core.

The repository also records both successful runs and a historical formula error that an Evaluator incorrectly judged sufficient.

This is an important limitation: adding an evaluation phase does not guarantee that every business error will be detected.

7. Getting Started

TriCall Harness is currently an early-alpha Python project requiring Python 3.10 or newer.

Clone the repository:

git clone https://github.com/jnukylinliu/tricall-harness.git
cd tricall-harness

Install the project and the Excel example dependencies:

python -m pip install -e .
python -m pip install -r examples/excel_agent/requirements.txt

Run the basic demonstration:

python examples/basic_demo.py

Explore supplemental execution with the scripted Retail example:

python -m examples.retail_inventory.run --mode scripted --case replan --details

Run the test suite:

python -m unittest discover -s tests -v

The scripted examples use predetermined model responses. They validate the execution and integration mechanisms without proving real-model reasoning quality.

The project also provides an adapter for compatible Chat Completions APIs.

For independent applications, developers can install the package into their own environment and implement Domain modules outside the TriCall source repository.

See the installation guide for packaging and independent integration details.

8. Current Validation and Limitations

TriCall has undergone initial packaging, integration, and execution testing.

The available validation record includes:

A 125-test offline regression suite.
Successful Ubuntu CI runs covering Python 3.10 through 3.13.
Packaging and ordinary wheel-installation checks.
Scripted supplemental execution.
Selected real-model observations for Retail and Excel tasks.
A separate, maintainer-tested third-party Domain integration.

These results support the feasibility of reusing the same Harness structure across different Domains.

They do not establish a general accuracy rate, production readiness, or superiority over other frameworks.

The current implementation also has deliberate boundaries.

It does not yet provide general-purpose durable execution recovery, distributed scheduling, parallel DAG execution, arbitrary synchronous handler cancellation, or automatic transaction rollback.

Execution is currently sequential.

Additional evaluation rounds can increase model latency and token consumption.

These are meaningful engineering trade-offs, especially for applications with tight response-time requirements.

TriCall is not intended to replace domain expertise with unconstrained model reasoning.

Instead, it gives developers a reusable lifecycle within which they can define and test their own professional capabilities.

9. A Design Philosophy, Not a Claim of Universal Superiority

Modern agent ecosystems already contain capable frameworks, graph execution systems, SDKs, and orchestration platforms.

TriCall does not claim to have invented planning loops, tool calling, evidence evaluation, or Agent Harnesses.

Its contribution is a particular organization of these established ideas:

Place LLMs where semantic interpretation is valuable.
Keep business intelligence and professional rules in developer-owned Domains.
Use Python for deterministic execution control and admission.
Evaluate actual evidence after execution.
Separate the final response from the execution and verification responsibilities.

This structure is intended to help developers avoid redesigning the entire agent execution lifecycle for every new business application.

Whether that approach improves reliability, development efficiency, or cost in a particular environment must be established through real application testing.

Conclusion

TriCall Harness explores a specific question:

What if an Agent Harness treated planning, execution control, professional intelligence, evidence evaluation, and answering as separate engineering responsibilities?

The current Python implementation offers an initial answer through a reusable execution lifecycle and application-defined Domains.

It is still early, and its limitations are documented openly.

Developers interested in domain-driven AI agents, evidence-based execution, and reusable Harness architectures are welcome to explore the repository, inspect the examples, open issues, and contribute.

GitHub Repository: https://github.com/jnukylinliu/tricall-harness

Architecture Documentation: https://github.com/jnukylinliu/tricall-harness/blob/main/docs/ARCHITECTURE.md

License: MIT

TriCall Harness — A foundational design philosophy for general-purpose Agent Harnesses.

DEV Community: kylin