By Harshith Vaddiparthy. Originally published in my AI agent harness guide.
An AI agent harness is the software around a model that lets it do work: supply the right information, offer tools, enforce permission rules, return results, and decide when to continue or stop. The model helps choose the next action. The harness connects that choice to an actual environment.
The walkthrough uses real software footage, fictional expense data, and my AI avatar and cloned voice. The transcript is available below without signing in.
That distinction matters when you are choosing an AI tool. A model name does not tell you whether the application can read your files, preserve the originals, ask before saving, recover from a failed action, or prove that the final result is correct. Those are properties of the complete system.
I recorded the walkthrough above to make those moving parts visible. The task is deliberately ordinary: read a small file of fictional expenses, calculate spending by category, flag a missing category, and save a short summary. There is no real financial account or customer data in this example. A small, inspectable task makes the boundary between a convincing answer and completed work much easier to see.
Model, tool, harness: three different jobs
The pieces of the expense demonstration
| Piece | Its job | What to inspect |
|---|---|---|
| Model | Interprets the request and proposes the next response or action. | Did it understand the missing-category rule? |
| Tool | Performs a specific operation outside the model. | Which file did it read, or which calculation did it run? |
| Harness | Connects instructions, context, tools, permissions, and results. | Which actions are available, which are blocked, and what evidence returns? |
The recorded application is DeepSeek Harness. In the interface, a conversation sits beside file tools and permission controls. You do not need that particular application to use the evaluation method. What matters is being able to inspect the input, the action, its result, and the boundary around it.
This general pattern also appears in Anthropic's guide to building effective agents: the model uses tools, receives feedback from the environment, and continues until it completes the task or reaches a stopping condition. The useful part is the feedback. Each step can give the system new evidence, rather than requiring it to finish everything from its first guess.
Start with a task whose answer you can check
My input contains 12 expense records. Eleven have a category. One does not. The instruction is explicit: retain that record, group it separately, and flag the missing category. Do not invent a label, delete the row, or quietly change the original file. The amounts are fictional and have no assigned currency.
Expected totals from the original 12 records
| Category | Amount |
|---|---|
| Software | 78.00 |
| Supplies | 46.00 |
| Travel | 86.00 |
| Uncategorized | 17.00 |
| Grand total | 227.00 |
The uncategorized amount is part of the 227.00 total. It is not an extra amount to add afterward. This is a small detail with a large lesson: handling uncertainty is part of the task definition. If I only asked for a tidy report, a plausible-looking classification could conceal the very exception I need to review.
Before running an agent, I want three things written down: the input it should use, the output I expect, and the rule for anything uncertain. For this task, that means the expense file, a category summary, and an explicit uncategorized group. A stronger prompt helps, but the application must still load the right files and enforce the allowed actions.
Follow the activity between the prompt and the answer
After submitting the task, I open the tool details. The first useful question is whether the agent read the intended input. A polished summary is weak evidence if I cannot connect it to the source records. The activity view should identify the action and show a result that I can interpret.
Read: The application supplies the instructions and the expense records through file tools. Check the file identity and the returned data.
Calculate: A verification command computes category totals, the grand total, and the number of records with a missing category. Inspect the actual output.
Propose: The model uses those results to prepare the summary. Compare its claims with the original input and the calculation.
Save and reopen: After the specific write is authorized, inspect the file that was actually saved and run the post-write check.
This is the agent loop in practical terms: choose an action, act through a tool, inspect what happened, and choose the next step. A tool request is not the same thing as a successful action. A planned save is not a saved file. A result saying that no summary exists yet is useful information, even when the arithmetic has already passed.
The harness should preserve those distinctions. If it reports every tool request as a success, the model may continue from a false premise. If it hides a failed write behind a generic error, recovery becomes harder. Clear results help both the model and the person reviewing the work understand the current state.
A saved file is not automatically remembered
Context is the information the model receives for the current step. In this example, the expense records and the rule for blank categories are relevant context. Unrelated files and old conversations are not automatically helpful. The harness decides what to load, what tool results to retain, and how to manage limited context space.
For longer tasks, the application may summarize earlier conversation or retrieve only selected material. That can keep a task moving, but summaries can drop detail. I prefer important facts and decisions to live in an artifact that can be reopened: the original data, a written instruction, or a checked output.
Persistence and retrieval are separate steps. The saved summary can survive the chat, but the next model response will not know its contents merely because it exists somewhere on disk. The system has to load it again. When someone says an agent has memory, ask what is stored, how it is retrieved, and how you can tell that the current response actually used it.
Approval and a sandbox answer different questions
In the recording, the initial write is blocked by the read-only setting. The agent then requests one-time permission to save the summary. I inspect the destination and the proposed content before allowing that action. The original expense file remains unchanged.
This demonstrates an application permission boundary. It does not demonstrate an operating-system sandbox. Approval answers whether a particular action may proceed. A sandbox limits which resources the execution environment can reach. A prompt saying 'do not touch other files' is neither a permission implementation nor proof of isolation.
For a learning exercise, the practical choice is a small disposable workspace with fictional inputs and only the capabilities needed for the task. Reading a file, creating a new summary, modifying the source, and sending the result to another person are different actions. Permission for one should not silently imply permission for all four.
Check what was saved, not just what was said
Once the summary is saved, I reopen it. The checker compares its category totals, grand total, and missing-category record count against the source data. In this example, those checks pass. That is stronger evidence than the final chat message saying the task is complete, because it tests the artifact I will actually use.
The limit matters just as much as the pass. This checker does not verify every sentence, decide whether an expense is legitimate, or tell me which category the blank row should have had. Correct arithmetic over incorrect source data would still be incorrect business information. The scope of a check determines what its result supports.
- Input preserved. The original records remain intact, so the output can be traced back to them.
- Exception retained. The 17.00 blank-category record remains visible instead of being guessed away.
- Output reopened. The saved artifact exists and can be inspected independently of the chat response.
- Checks named. A passing calculation has a specific meaning; it is not a blanket guarantee of correctness.
Four questions to ask of any agent
I use four questions to evaluate the complete system. What information can it see? Which actions can it take? How is the result checked? Where must it stop? These questions are useful even when you have no interest in building the harness yourself.
Take a weekly report as another example. The agent might be allowed to read a folder of approved inputs and draft a summary. Its check could reconcile totals and link each factual claim to a source. Its stopping point could be the reviewed draft. Publishing the report would remain a separate decision. That is a clearer design than asking an agent to 'handle reporting' and discovering its interpretation afterward.
Try the method on one small task
Choose a small set of fictional or non-sensitive records whose expected result you can calculate yourself. Use an agent application that exposes its tools and lets you restrict writes. Adapt this task brief to the capabilities it actually has. If it cannot show a tool result or enforce a permission boundary, record that limitation instead of assuming it exists.
A task brief you can adapt
Read the supplied expense file and preserve it unchanged.
Calculate totals by category and a grand total.
Keep blank categories in an Uncategorized group and report their count.
Show the calculation evidence and a proposed summary.
Do not save or send anything yet.
After I approve one destination and the summary, save that file only.
Reopen the saved file and compare its totals with the source.
Report which checks passed, which were not performed, and stop.
Then change one input deliberately: remove a category, add a row, or supply a destination the application is not allowed to write. Does the system retain the uncertainty, notice that the expected total changed, or stop at the boundary? A small controlled failure can teach you more about an agent than another polished demonstration.
You do not need a multi-agent architecture to learn this. Start with one task, a few tools, and one visible completion check. If you want to understand the implementation after that, my guide to building a coding agent from scratch explains the loop and the operating machinery around it. If you want more practical field notes like this, subscribe to The H Letter.
Sources
The fictional expense task, tool activity, blocked write, one-time approval, and saved-file check shown in this article.
Primary guidance on tools, environmental feedback, stopping conditions, and human oversight. It supports the general pattern, not a security certification of this demonstration.
Frequently asked questions
Is an AI harness the same thing as a model?
No. The model produces responses and requests actions. The harness is the surrounding software that supplies context, exposes tools, applies permission rules, runs permitted actions, and returns observations. The same model can behave differently in two applications because those surroundings differ.
Does an approval dialog mean an agent is sandboxed?
No. Approval decides whether a particular action may proceed. A sandbox limits what an execution environment can reach. The walkthrough demonstrates an application permission boundary; it does not demonstrate an operating-system sandbox.
Does saving a file give an agent permanent memory?
Saving creates a durable artifact. The model can use that artifact in a later response only if the system loads its contents into the context it receives. Persistence and retrieval are different steps.
Further reading
Build a coding agent from scratch: Go deeper into tool dispatch, context compaction, permission policies, and the code behind a harness.
How I evaluate AI agent frameworks: The practical questions I ask about integration, reliability, observability, and the cost of leaving a framework.
Full transcript and downloads
The complete transcript follows. You can also read it alongside the video in the original guide or download a copy:
0:00: What makes an agent work
When an AI reads a file, uses a tool, and checks a result, the model is only part of what you're seeing. The surrounding software makes that work possible. That surrounding system is called a harness. I'm going to show you one real task, and point out the pieces that make an agent useful.
0:18: The model and its harness
I'm using DeepSeek Harness, a real agent app with a conversation and tools for files. The model is one component inside this application. The instructions, available tools, and permission settings all affect what the complete system can do. The name in the model selector does not tell you everything about what the complete system can accomplish.
The model chooses what to say or which action to request. A tool does something outside the model, such as reading a file. The harness connects those pieces. It supplies context, makes allowed tools available, runs permitted actions, and returns their results.
The model can then use that new information to decide what to do next.
0:59: Give it the right context
For this task, I have a small file of fictional expenses. The goal is to calculate spending by category, flag anything that needs attention, and create a short summary. First, look at the information we are giving the agent. The file contains the original records.
The instructions say how to handle a blank category: keep the record, group it separately, and do not guess. We also tell it to preserve the input instead of quietly changing the source. These details are context: the information available to the model for this job. More context is not automatically better.
Unrelated files and old conversations can distract from the task. A useful harness helps select the relevant instructions and evidence, within the scope we allow. For longer tasks, a harness also has to manage limited context space. It may select earlier results or summarize older conversation.
Summaries can lose detail, so keep critical facts in an artifact you can reopen. The file is durable; whether the next response uses it depends on what the system actually loads. Before pressing send, check that the request names the input, the expected output, and the rule for uncertainty. That makes success something we can actually inspect.
2:19: Follow the real tool loop
Now we submit the task. Watch the activity between the prompt and the final response. The agent requests a tool, and the application performs the action. Opening the tool details lets us see which file it read and what information came back.
We can see what it actually read. The model uses that result to choose its next step. If it needs another file or a calculation, another tool request can follow. This is the basic agent loop: decide, act through a tool, inspect what happened, and continue.
This example uses files and a calculation. Those capabilities come from the integration around the model, and they should be limited to what the task needs. Installing a connector does not mean the agent should have unrestricted access. The loop does not guarantee that every decision is correct.
It gives the system a way to work with evidence instead of stopping at a single response. In this example, the useful evidence is the original expense data and the calculations made from it. We can follow the activity and compare the proposed summary with those records. Notice that an empty category is still something to flag, not an invitation to invent a label.
Check whether the actions and results support the answer.
3:35: Check the result and the boundary
Before accepting the summary, we check it against the original data. First, the calculator totals the source records. At this stage, it correctly reports that no summary has been saved. The expense with a missing category must remain visible in that check. Twelve records total two hundred and twenty-seven, including seventeen without a category.
The calculated totals match the source records; that is evidence for this calculation. Now look at the write boundary. Reading the input and saving a new file are different actions. The first write is blocked by the read-only setting. The agent then asks for one-time permission to save this summary.
Here, the application restricts which files and command the agent can use, and asks before writing this summary. This demonstrates an application permission boundary. An operating-system sandbox is a separate boundary, and we have not demonstrated one here. Inspect the proposed destination and contents before allowing this specific action.
After the file is saved, reopen it and confirm what actually exists. The checker compares the saved category totals, grand total, and the count of records missing a category against the source. A pass supports those checks; it does not verify every sentence in the report. That final readback matters more than a message saying the work is done.
Saved files only become context when the system loads them again. Approvals and sandboxes also do different jobs. An approval asks whether an action may proceed. A sandbox limits what the execution environment can reach.
Always check what is actually enforced.
5:22: Judge the whole system
The original file is unchanged, the totals have been checked, and the saved summary can be reopened. Following those steps gives us a practical way to evaluate the system. When you evaluate an agent, ask four questions. What information can it see?
Which actions can it take? How is the result checked? And where must it stop? Those questions tell you much more than the name of the model alone.
Try this with one small task of your own. Follow the actual steps, inspect the output, and identify one check you would want before trusting the result.
Top comments (0)