Why Ollama Tool Calls Fail at a 4,096 Context Length

📝 Originally published (in Japanese) at forge.workstyle.tech.

TL;DR The app passed 29 tools and expected a context window of 16,384, but the model ran at 4,096. The OpenAI-compatible API path did not apply the expected window during model loading, and oversized input was truncated without a 400 response. Routing tool calls through /api/chat and setting options.num_ctx to 16,384 aligned the request path with the model context setting.

Conclusion First

In Ollama, the tool wasn't invoked, and the response was cut off mid-way. However, when using the OpenAI-compatible API (a connection method that sends requests in OpenAI's API format), the input was truncated from the beginning without returning a 400 error. The cause was that the window, which was expected to be 16,384, wasn't reflected when loading the model, and it was actually running with the default value of 4,096. I also sent the tool invocation to /api/chat and verified by specifying options.num_ctx (the context window setting value passed when loading the model) as 16,384.

Why Didn’t I Get a 400 Error and Couldn’t Pinpoint the Cause?

Initially, I thought the tool descriptions were too long and assumed reducing their number would solve the issue. Previously, I fixed a case where a 400 error was returned when the context window was exceeded. From the error message, I remembered the window size and reduced the number of tools in the next request. Additionally, if the window was exceeded, I truncated the old results and resent them.

With this mechanism, a 400 error could serve as a clue. However, in this case, no 400 error was returned, and even reducing the tools didn’t clarify what was happening. I couldn’t determine whether the model wasn’t calling the tools, the tool definitions weren’t reaching it, or the conversation history was being cut off midway—the response alone didn’t provide enough information.

So, I suspected that the estimated window size might be too small and investigated how the app determined this value.

Why the limit changed from 16,384 to 4,096

In our initial window estimation, we compared the value obtained from the model information with 16,384 and used the smaller of the two. Since the model information specified 262,144, the app was expected to use 16,384. Here, tokens refer to the units the model uses to process text.

However, when the model was actually loaded, the context_length was 4,096.

This wasn't just a simple estimation error. In the OpenAI-compatible API /v1/chat/completions, the expected window value was not being reflected in the model loading. The model was running at the default of 4,096.

Furthermore, when the window limit was exceeded, Ollama did not return a 400 error; instead, it silently truncated the input from the beginning. In other words, the signal the system relied on—"seeing a 400 to remember the window size"—never arrived, and the process to filter down the tools never began.

The mere fact that we passed 29 tools wasn't enough to realize the cause. The number of tools passed during transmission and what the model was actually able to read were two different things.

How I fixed the tool call destination

Simply changing the window size on the application side wasn't enough; I had to rethink the entire path through which tool calls are sent.

In this fix, tool calls are sent to /api/chat. Along this route, I specify options.num_ctx as 16,384 and pass the window value when the model is loaded.

If I had only toggled the regular conversation and left the tool usage logic on the old route, the same issue would persist. Therefore, I moved the tool calls through /api/chat as well, converting the arguments and history to match the expected format of the destination. I avoided specifying tool_choice: required, as Ollama does not support it.

With this, I synchronized the window estimation with the actual loading settings using the same path.

Did it reach the end after specifying 16,384?

In the post-fix verification, I was able to specify 16,384 for the context window, and the 400 errors were zero. I was also able to pass all 29 tools.

The logs show that the process progressed from tool calling to searching and retrieval, through to the final response. Unlike before, I did not need to wait for errors before reducing the number of tools.

What I confirmed here is that the window specification and tool transmission are working as intended. However, simply fixing the window size does not guarantee the model's response content or speed. Differences in responses will remain depending on the specific model.

Initially, I was looking for the presence of the 400 error. However, in this path, a 400 is not generated even if the input is truncated. The absence of an error was not evidence that the entire input had arrived.

When investigating issues with tool calls, in addition to estimating the window size, I verify the actual transmission path and the specified values. The value obtained from model information and the value used during loading are not necessarily the same. If the ability to specify the window varies by the destination, I align the path, including the number of tools.

Key takeaways

• Check the window value from model information.
• Check context_length when the model loads.
• Verify the tool call route and whether it sets the context window.
• Route tool calls through /api/chat and set options.num_ctx to 16,384.
• After the change, check the 400 count and the record from tool call through final response.