Making Local AI Tool Calls More Reliable
fix: improve tool call reliability and add automated tests for repairs
#20
I fixed an intermittent error where the assistant sometimes returned a text response instead of calling the required local-data tool. I added a safe one-time retry with a required tool choice, protected write operations from duplicate execution, and verified the solution with 67 automated tests and a live local Gemma model.
List the tests or checks you ran.
- [x] I updated my section in the matching weekly report.
- [x] I reviewed the final changes and ran
scripts/verify-contribution.ps1. - [x] I confirmed that this branch uses the correct base branch and that the pull request targets the intended branch.
While testing our local-first AI assistant, I found an intermittent problem: the model sometimes answered with plain text instead of calling the tool needed to read or update local data.
The original system used tool_choice="auto". This normally worked, but it allowed the model to skip a required tool call. A prompt can guide a model, but it cannot guarantee that the model will always follow the tool protocol.
I fixed this by adding a safe recovery step. The first request still uses automatic tool selection, so normal conversation works as before. If an explicit local-data request returns no tool call, the assistant retries once with tool_choice="required". The retry happens only before any tool has run, which prevents duplicate database writes.
The assistant also checks whether the recovered tool belongs to the correct read or write group. During streaming, it buffers the first response so an incorrect, ungrounded answer is not shown before recovery completes.
I tested the change with automated regression tests and the running local Gemma model. The live recovery flow was:
auto -> required -> auto
The main lesson was simple: prompts describe expected behavior, but reliable agent systems also need program-level checks around model decisions.
Top comments (6)
The retry boundary matters more than the retry itself here. If the first call skipped a required tool, forcing the second call is fine only while the system can still prove no side effect has happened. Once a write-capable tool has run, the safe move is usually a repair ticket or a human handoff, not another model attempt against the same request.
The boring check I would add is a tiny per-run ledger: requested tool, tool actually called, side-effect flag, retry allowed. It makes these failures much easier to debug later.
I have faced the same issue when I did my digital twin agent
Yeahsometimes AI is not stable when calling tools. Do you have any idea how we can make it more stable programmatically
I don’t think there’s a truly reliable way to get 100% accuracy in tool calling yet. Frameworks like LangChain and the OpenAI SDK make things smoother, but the biggest gains I’ve seen come from basics: tighten the prompt, make tool descriptions unambiguous, and add light guardrails (temp=0, validate args, short retry + timeout). Keep tools small and predictable, and keep the model’s return payloads concise.
I covered these in my Digital Twin write-up.
Nice minimal fix. The auto->required->auto recovery is a good pattern for the single-call case. One thing worth adding from the loop side: in our agent loop the same failure shows up at a different layer - the model returns a turn with no tool call even though the task required one. We handle it structurally rather than per-request: the loop validates each round's output against the task's declared tool requirements, and only then decides whether to retry, finish, or escalate. That catches the "model thinks it's done but the tool was mandatory" case, which tool_choice alone won't (auto can legitimately produce a conversational answer). The two patterns compose: your retry handles the transport-level skip; round-boundary validation handles the task-level skip. Also +1 to the side-effect point - the retry window closing once a write has run is the right boundary.
This is a really nice example of moving an important reliability property outside the model.
The part I especially like is that the retry happens before any tool has executed. That makes the auto → required recovery much safer, because you're not turning a tool-selection failure into a potential duplicate side effect.
The buffering during streaming is also a good detail. It prevents an ungrounded model response from becoming user-visible while the system is still determining whether the required tool path was followed.
One thing I'd be particularly interested in testing further is the boundary between tool selection correctness and tool execution authorization. The retry can establish that a tool call exists, but the resulting tool name, arguments, target data and operation still need to be validated independently before a write is allowed.
So I really like the broader lesson here: prompts can express the intended behavior, but the security and reliability properties that matter should be enforced and tested at the program boundary. The 67 regression tests + live-model verification make this much more convincing than relying on prompt adherence alone. 🔐