Making Local AI Tool Calls More Reliable
fix: improve tool call reliability and add automated tests for repairs
#20
I fixed an intermittent error where the assistant sometimes returned a text response instead of calling the required local-data tool. I added a safe one-time retry with a required tool choice, protected write operations from duplicate execution, and verified the solution with 67 automated tests and a live local Gemma model.
List the tests or checks you ran.
- [x] I updated my section in the matching weekly report.
- [x] I reviewed the final changes and ran
scripts/verify-contribution.ps1. - [x] I confirmed that this branch uses the correct base branch and that the pull request targets the intended branch.
While testing our local-first AI assistant, I found an intermittent problem: the model sometimes answered with plain text instead of calling the tool needed to read or update local data.
The original system used tool_choice="auto". This normally worked, but it allowed the model to skip a required tool call. A prompt can guide a model, but it cannot guarantee that the model will always follow the tool protocol.
I fixed this by adding a safe recovery step. The first request still uses automatic tool selection, so normal conversation works as before. If an explicit local-data request returns no tool call, the assistant retries once with tool_choice="required". The retry happens only before any tool has run, which prevents duplicate database writes.
The assistant also checks whether the recovered tool belongs to the correct read or write group. During streaming, it buffers the first response so an incorrect, ungrounded answer is not shown before recovery completes.
I tested the change with automated regression tests and the running local Gemma model. The live recovery flow was:
auto -> required -> auto
The main lesson was simple: prompts describe expected behavior, but reliable agent systems also need program-level checks around model decisions.
Top comments (4)
The retry boundary matters more than the retry itself here. If the first call skipped a required tool, forcing the second call is fine only while the system can still prove no side effect has happened. Once a write-capable tool has run, the safe move is usually a repair ticket or a human handoff, not another model attempt against the same request.
The boring check I would add is a tiny per-run ledger: requested tool, tool actually called, side-effect flag, retry allowed. It makes these failures much easier to debug later.
Nice minimal fix. The auto->required->auto recovery is a good pattern for the single-call case. One thing worth adding from the loop side: in our agent loop the same failure shows up at a different layer - the model returns a turn with no tool call even though the task required one. We handle it structurally rather than per-request: the loop validates each round's output against the task's declared tool requirements, and only then decides whether to retry, finish, or escalate. That catches the "model thinks it's done but the tool was mandatory" case, which tool_choice alone won't (auto can legitimately produce a conversational answer). The two patterns compose: your retry handles the transport-level skip; round-boundary validation handles the task-level skip. Also +1 to the side-effect point - the retry window closing once a write has run is the right boundary.
I have faced the same issue when I did my digital twin agent
Yeahsometimes AI is not stable when calling tools. Do you have any idea how we can make it more stable programmatically