Use host.docker.internal to reach local Ollama from Mule

#mule #ollama #inferenceconnector #docker

Problem

The flow failed with a connection refused error when the configuration used http://localhost:11434/v1 as the base URL. The MuleSoft Inference Connector enables text generation workflows docs.

Input

{
  "prompt": "Reply with exactly one word: pong"
}

Working configuration

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:http="http://www.mulesoft.org/schema/mule/http"
      xmlns:ms-inference="http://www.mulesoft.org/schema/mule/ms-inference"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="
        http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
        http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
        http://www.mulesoft.org/schema/mule/ms-inference http://www.mulesoft.org/schema/mule/ms-inference/current/mule-ms-inference.xsd">

  <http:listener-config name="http-listener-config">
    <http:listener-connection host="0.0.0.0" port="8081" />
  </http:listener-config>

  <ms-inference:text-generation-config name="ollama-openai-compatible">
    <ms-inference:openai-compatible-connection
        openAICompatibleModelName="qwen2.5:14b"
        openAICompatibleURL="http://host.docker.internal:11434/v1"
        apiKey="ollama"
        maxTokens="16"
        temperature="0"
        topP="1"
        timeout="120" />
  </ms-inference:text-generation-config>

  <flow name="ask">
    <http:listener config-ref="http-listener-config" path="/ask" />
    <ms-inference:chat-answer-prompt config-ref="ollama-openai-compatible">
      <ms-inference:prompt>#[payload.prompt]</ms-inference:prompt>
    </ms-inference:chat-answer-prompt>
  </flow>
</mule>

Output

{"response": {"response": "Pong"}, "status": 200}

The response field contains the model output, and status shows 200.

The trap

Using http://localhost:11434/v1 as the base URL causes a connection refused error.

What I do now

I configure openAICompatibleURL to http://host.docker.internal:11434/v1 in the connector configuration.

Runs shown: Mule 4 Community kernel 4.12.0

104 DataWeave patterns + 8 Exchange modules with 208 MUnit tests: github.com/shakarbisetty/mulesoft-cookbook | 60-second walkthroughs: youtube.com/@SanThaParv

DEV Community