DEV Community

Cover image for How MCP and tool calling really work: I built an AI agent from scratch in Java and traced every step
Abiral Khanal
Abiral Khanal

Posted on

How MCP and tool calling really work: I built an AI agent from scratch in Java and traced every step

#java #ai #llm #mcp

I asked a local AI model to add "buy milk" to my notes file. It did. Then I traced every step to find out what actually happened.

To do that, I built the whole thing from scratch:

  • an MCP server
  • an MCP client and
  • an agent loop in plain Java and Spring Boot,

no AI frameworks and no MCP SDK, talking to Qwen 2.5 7B running locally through Ollama.

"From scratch" means no frameworks, not no help: Claude co-authored the code and this article, because writing every line by hand in 2026 would apparently make me a caveman.

Yes, I know: MCP has been around for about two years. I'm late to the party. But if even one person walks away with a new understanding, that's the best outcome I could ask for.

What we're building

One plain-text file, data/db, holds my notes. The MCP server is the only thing that touches it, through five tools: read, write (replace everything), append, update (find and replace) and clear. A simple File I/O method, ref

You ask the host in plain English ("Add buy milk"), it asks the model which tool fits, and runs it on the server.

What you need

  • Docker with Compose: runs the MCP server and the host.
  • Ollama, installed natively (ollama.com), not in Docker, so the model can use your GPU. On Linux, set OLLAMA_HOST=0.0.0.0 so the containers can reach it.
  • The model: ollama pull qwen2.5:7b (about 4.7 GB; 8 GB RAM minimum, 16 GB comfortable).
  • Git. No Java needed unless you run outside Docker. 
git clone https://github.com/abir4l/mcp_server
git clone https://github.com/abir4l/local_agent
ollama pull qwen2.5:7b          # with Ollama running
cd local_agent && docker compose up --build
Enter fullscreen mode Exit fullscreen mode

The map

Architecture: the host sits in the middle and talks to both Ollama and the MCP server, which never talk to each other

  • local_agent (port 8081) is the host. It runs the agent loop, keeps the conversation and makes every network call.
  • Ollama (port 11434) runs the model. Text in, text out.
  • mcp_server (port 8080) owns the tools. It's the only thing that touches data/db, the notes file.

Part 1: The layers

1. The model is just numbers

ollama pull qwen2.5:7b downloads about 4.7 GB of weights: billions of numbers. Together with a function that predicts the next token, that's the whole model. It doesn't listen on a port, doesn't speak JSON or HTTP, and does nothing on its own. A file of numbers can't write to your disk. Something else has to.

2. Ollama is a database server, but for models

I used to think Ollama, ChatGPT and Claude were abstractions over the model, so my own app would skip them and talk to the model directly. But the model is just numbers, and an app can't talk to numbers. Something has to load it and give you an interface to send text and get text back: a model server.

Locally that's Ollama; in the cloud it's the OpenAI or Anthropic API. That's the layer you can't bypass.

ChatGPT and the Claude app are one level higher: they're hosts like my local_agent, with a UI, history and tools on top of that API. Calling the OpenAI API isn't calling ChatGPT.

Layer Local OpenAI Anthropic
Model qwen2.5:7b on your disk GPT (private weights) Claude (private weights)
Model server (API) Ollama OpenAI API Anthropic API
Host / app local_agent, or Ollama's own chat app ChatGPT Claude app

ChatGPT is to the OpenAI API what my local_agent is to Ollama. Build on the API and you write the ChatGPT part: history, tools, the loop.

One twist: locally, Ollama sits in two rows at once. Its server (the API on port 11434) is the model-server layer, but its desktop chat window and ollama run in the terminal are hosts, thin clients of that same API. Imagine OpenAI shipping the API and ChatGPT in one download. My local_agent skips Ollama's chat window and talks to the API directly, the same way your own app would skip ChatGPT and call the OpenAI API.

3. There is no standard for talking to models (MCP vs function calling)

Every vendor has its own JSON dialect. OpenAI returns tool arguments as a JSON string, Ollama as a JSON object, and Anthropic uses tool_use content blocks instead of tool_calls. Same idea, three shapes, so every host needs an adapter.

That's also the answer to "MCP vs function calling". Function calling is the model side: you describe tools to the model and it replies with a request to use one. MCP standardized the tool side, not the model side: how a host finds and calls tools on a separate server. A real agent uses both. Since MCP's inputSchema and Ollama's parameters are both plain JSON Schema, my adapter between them is a single map().

Part 2: The illusions

4. Are LLMs stateless? Yes. Chat is an illusion

Maybe you already know this one: the model remembers nothing between calls. The host resends the entire conversation every turn, so "memory" is just a list in my Java process. That's all it ever is.

5. How tool calling actually works: it's just text

There is no special channel. Ollama's chat template renders your JSON tools into the prompt (see it with ollama show qwen2.5:7b --template). Trimmed, the model sees:

<|im_start|>system
# Tools
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "append", "description": "Append content to the end of the file", ...}}
...
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags
<|im_end|>
<|im_start|>user
Add buy milk to my notes<|im_end|>
<|im_start|>assistant
Enter fullscreen mode Exit fullscreen mode

The model predicts the most likely next tokens, which, because it was trained on this format, are:

<tool_call>
{"name": "append", "arguments": {"content": "buy milk"}}
</tool_call>
Enter fullscreen mode Exit fullscreen mode

Ollama parses that text back into a neat tool_calls JSON field. The "function call" is a string that looks like one. (Start Ollama with OLLAMA_DEBUG=1 to see the full rendered prompt.)

6. The model never touches the file

Ollama returns that tool_calls JSON to my host, and it's done. It has no idea an MCP server exists. My host reads the request, calls the MCP server over HTTP, and pastes the result back into the conversation as a role: "tool" message.

Ollama and the MCP server never talk to each other. The model requests. The host decides and acts. That's why the host is the real control point for safety, not the model.

How to add your own data to an LLM (no training required)

This is what I actually set out to learn. When the model calls read, my notes come back as a role: "tool" message and sit in the conversation like any other text. The model "knows" my file because the host pasted it into the prompt.

Every popular way to give an LLM your own data is this trick, except one:

  • Prompt stuffing: paste the data in yourself.
  • Tool calling / MCP: the model asks, the host fetches (this project).
  • RAG: the host searches your documents and pastes in the relevant chunks.
  • Fine-tuning: changes the weights. Good for behaviour and style, usually wrong for "make it know my documents".

So "train ChatGPT on my own data" usually means one of the first three. Choosing what goes into the context window is context engineering, and RAG exposed as an MCP tool is agentic RAG.

Part 3: MCP on the wire

7. MCP is JSON-RPC on one endpoint, not REST

An MCP server has exactly one endpoint, POST /mcp, and the operation lives in the body's method field (JSON-RPC 2.0). Since the 2026-07-28 spec, MCP is also stateless: no initialize handshake and no session ID. Every request carries its protocol version in params._meta, and the method and tool name are copied into headers, so a gateway can route without parsing the body.

What an MCP server's API looks like

If you build REST APIs for a living, this is the light-bulb view. Here is my server's entire API, OpenAPI-style:

POST /mcp                         the only endpoint (GET, DELETE -> 405)

method            params              returns
server/discover   (none)              supported versions, capabilities
tools/list        (none)              [{ name, description, inputSchema }]
tools/call        name, arguments     { content: [...], isError }
Enter fullscreen mode Exit fullscreen mode

And here is the API the model actually sees, as returned by tools/list:

tool      arguments          what it does
read      (none)             return the whole file
write     content            replace the whole file
append    content            add a line at the end
update    search, replace    find and replace text
clear     (none)             empty the file
Enter fullscreen mode Exit fullscreen mode

That's the whole thing: two lists. Think of tools/list as an OpenAPI doc written for a model instead of a developer. The full spec adds the same list-and-read pattern for resources (resources/list, resources/read) and prompts (prompts/list, prompts/get), but tools are where the action is. One call on the wire looks like this:

POST /mcp
MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: append

{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"append","arguments":{"content":"buy milk"},
 "_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}}}
Enter fullscreen mode Exit fullscreen mode

MCP vs API: isn't MCP just an API?

Technically, yes. The difference is who it's designed for. A normal API is designed for a developer, who reads the docs and writes a custom client. An MCP server describes itself to a model: tools/list returns every tool with a plain-English description and a JSON Schema, so any MCP host can use any MCP server with no custom code. MCP usually wraps your existing APIs rather than replacing them.

8. There are two kinds of errors

  • Protocol errors (bad JSON, wrong headers, unknown tool) come back as a JSON-RPC error. They're client bugs, for the developer.
  • Tool errors (missing argument, wrong type) come back as a normal result with isError: true. They're for the model, so it can fix its call and retry. 
// unknown tool is a protocol error, bad arguments are a tool error the model can fix
try {
    registry.validate(toolName, params);
} catch (ToolNotFoundException e) {
    throw RpcException.invalidParams("Unknown tool: " + toolName, HttpStatus.OK);
} catch (ToolArgumentException e) {
    return CallToolResult.error(e.getMessage(), resultMeta);
}
Enter fullscreen mode Exit fullscreen mode

Small models make small mistakes. Hand back Param (content) is required instead of throwing, and they usually fix it on the next step.

Part 4: The full trace

One /agent request, every hop.

1. You → host

curl -s localhost:8081/agent -H 'Content-Type: application/json' -d '{"message":"Add buy milk to my notes"}'
Enter fullscreen mode Exit fullscreen mode

2. Host → MCP server: tools/list (id 1) returns the five tools.

3. Host → Ollama

POST /api/chat
{
  "model": "qwen2.5:7b",
  "messages": [ { "role": "user", "content": "Add buy milk to my notes" } ],
  "tools": [
    { "type": "function", "function": { "name": "append",
        "description": "Append content to the end of the file",
        "parameters": { "type": "object",
          "properties": { "content": { "type": "string", "description": "Content that gets appended" } },
          "required": ["content"], "additionalProperties": false } } },
    ... read, write, update, clear ...
  ],
  "stream": false,
  "keep_alive": "30m"
}
Enter fullscreen mode Exit fullscreen mode

4. Ollama → host: the model wants a tool, and says nothing else (content is empty).

{ "message": { "role": "assistant", "content": "",
    "tool_calls": [ { "function": { "name": "append", "arguments": { "content": "buy milk" } } } ] },
  "done": true }
Enter fullscreen mode Exit fullscreen mode

5. Host → MCP server: the tools/call request shown in moment 7. This is the only moment anything touches the disk.

{"jsonrpc":"2.0","id":2,"result":{"resultType":"complete","content":[{"type":"text","text":"Success"}],"isError":false,
 "_meta":{"io.modelcontextprotocol/serverInfo":{"name":"mcp_server","version":"0.0.1-SNAPSHOT"}}}}
Enter fullscreen mode Exit fullscreen mode

6. Host → Ollama again, with the whole history

"messages": [
  { "role": "user", "content": "Add buy milk to my notes" },
  { "role": "assistant", "content": "",
    "tool_calls": [ { "function": { "name": "append", "arguments": { "content": "buy milk" } } } ] },
  { "role": "tool", "content": "Success", "tool_name": "append" }
]
Enter fullscreen mode Exit fullscreen mode

7. Ollama → host: "content": "I've added \"buy milk\" to your notes." with no tool_calls, so the loop stops.

8. Host → you: the reply, plus a list of every tool call made.

Seven HTTP calls between three processes for two words in a text file. The model was in two of them, receiving text and returning text.

Part 5: The harness

9. The agent harness matters more than people think

Everything around the model is the agent harness. Here, that's my host. The loop itself is tiny (logging trimmed):

for (int step = 1; step <= properties.maxSteps(); step++) {
    Message reply = ollamaClient.chat(messages, tools);
    messages.add(reply); // the tool_calls must stay in history

    if (!reply.hasToolCalls()) {
        return new Result(reply.content(), invocations);
    }
    for (ToolCall call : reply.toolCalls()) {
        ToolInvocation invocation = invoke(call.function().name(), call.function().arguments());
        invocations.add(invocation);
        messages.add(Message.toolResult(call.function().name(), invocation.output()));
    }
}
throw new IllegalStateException("Agent did not finish within " + properties.maxSteps() + " steps");
Enter fullscreen mode Exit fullscreen mode

The behaviour lives in the details:

  • An iteration guard (agent.max-steps=5), because a model that calls tools forever is a real failure mode.
  • Keeping the model's own tool call in history, or the tool result makes no sense to it.
  • Tool descriptions are written for the model. My write tool says "Write all contents to file". To a 7B model asked to "add" something, that can look fine, and then your notes are gone. "Replaces the entire file. To add a note, use append." works better.

A 7B model is slow on a laptop and not deterministic, but it's good enough to learn on, because every failure is visible.

What's next

10. Frameworks hide all of this

Spring AI can do everything above with an annotation like @McpTool. I'd use it in production. But now that Claude and I built it without one, I know what it hides, and when something breaks I know which of the seven hops to check.

  • v2: the same project with Spring AI, compared line by line.
  • v3: multiple notes files, then agentic RAG with a search tool.

FAQ

Do I need to train an LLM on my own data? Usually not. Put the data in the context window with a prompt, a tool/MCP server or RAG. Fine-tuning changes behaviour, not knowledge.

MCP vs RAG? RAG retrieves relevant chunks into the prompt. MCP lets the model call tools, and search can be one. Combined, that's agentic RAG.

Does the model execute the tool? No. It writes text asking for one. The host runs the tool and sends back the result.

Is MCP the same as function calling? No. Function calling is how a model asks for a tool. MCP is how a host finds and calls tools on another server.

Do I need MCP for tool calling? No. You can define tools in your app directly. MCP pays off when tools are shared across apps or teams.

Can a local model use MCP tools? Yes, if it was trained for tool calling (Qwen 2.5, Llama 3.1, Qwen 3). The model never talks to the MCP server anyway.

What changed in MCP 2026-07-28? It's stateless: no initialize, no sessions, and context travels in params._meta plus mirrored headers.

Try it yourself

Clone both repos, run docker compose up --build in local_agent, and send the curl from hop 1. Ollama runs natively on your machine, and Docker runs the rest.

The model is just numbers. Everything else is software, and software is something we already know how to read.

Co-written with Claude. If this was useful, a star on the repo helps.

Top comments (0)