Wiring Ollama's OpenAI-Compatible API to Android: Local LLM Inference Over the Network Without a Cloud Dependency

---
title: "Wiring Ollama to Android: Local LLM Inference Without the Cloud"
published: true
description: "Point Android's OkHttp at a local Ollama instance, stream tokens via SSE, manage network lifecycle, and build a ViewModel that degrades gracefully when the server is unreachable."
tags: android, kotlin, architecture, mobile
canonical_url: https://mvpfactory.co/blog/ollama-android-local-llm
---

What We Are Building

By the end of this workshop you will have an Android client that streams token responses from a local Ollama instance over Wi-Fi, handles mid-stream network loss without crashing, and surfaces meaningful error states to the user — all without a cloud dependency or on-device model weights.

Here is the pattern I use in every project that needs local-network inference: treat Ollama exactly like any other REST endpoint, but respect the one rule that makes or breaks streaming — never set a finite read timeout.

---

Prerequisites

• Ollama running on a machine reachable over your local network (default port 11434)
• Android project targeting API 28+
• OkHttp added to your dependencies
• Basic familiarity with Kotlin Flows and ViewModel

---

Step 1 — Configure OkHttp for Ollama

Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions. Here is the minimal setup to get this working:

val client = OkHttpClient.Builder()
    .connectTimeout(5, TimeUnit.SECONDS)
    .readTimeout(0, TimeUnit.MILLISECONDS) // critical for SSE — no read timeout
    .build()

readTimeout(0) is non-negotiable. A finite timeout will cut the connection mid-generation on any response longer than a few sentences.

---

Step 2 — Stream Tokens With a Kotlin Flow

Ollama streams tokens as Server-Sent Events when you pass "stream": true. Each data: line is a JSON delta. OkHttp has no native SSE parser, but a callbackFlow handles it cleanly:

fun streamCompletion(prompt: String): Flow<String> = callbackFlow {
    val body = buildJsonRequest(prompt, stream = true)
    val request = Request.Builder()
        .url("$baseUrl/v1/chat/completions")
        .post(body)
        .build()

    val call = client.newCall(request)
    call.enqueue(object : Callback {
        override fun onResponse(call: Call, response: Response) {
            response.body?.source()?.use { source ->
                while (!source.exhausted()) {
                    val line = source.readUtf8Line() ?: break
                    if (line.startsWith("data: ") && line != "data: [DONE]") {
                        val delta = parseDelta(line.removePrefix("data: "))
                        trySend(delta)
                    }
                }
            }
            close()
        }
        override fun onFailure(call: Call, e: IOException) = close(e)
    })
    awaitClose { call.cancel() }
}

The two helpers that make this runnable use org.json, which ships with Android — no extra dependencies:

private fun buildJsonRequest(prompt: String, stream: Boolean): RequestBody {
    val json = JSONObject().apply {
        put("model", "llama3")
        put("stream", stream)
        put("messages", JSONArray().put(
            JSONObject().apply {
                put("role", "user")
                put("content", prompt)
            }
        ))
    }.toString()
    return json.toRequestBody("application/json".toMediaType())
}

private fun parseDelta(json: String): String {
    return try {
        JSONObject(json)
            .getJSONArray("choices")
            .getJSONObject(0)
            .getJSONObject("delta")
            .optString("content", "")
    } catch (e: JSONException) { "" }
}

---

Step 3 — ViewModel With Graceful Degradation

The ViewModel covers three states: server reachable, server unreachable, and mid-stream loss. Catch IOException broadly, then discriminate:

class ChatViewModel(private val repo: OllamaRepository) : ViewModel() {
    private val _uiState = MutableStateFlow<ChatState>(ChatState.Idle)
    val uiState: StateFlow<ChatState> = _uiState.asStateFlow()

    fun send(prompt: String) {
        viewModelScope.launch {
            _uiState.value = ChatState.Streaming("")
            repo.streamCompletion(prompt)
                .catch { e ->
                    _uiState.value = when (e) {
                        is ConnectException -> ChatState.ServerUnreachable
                        is SocketTimeoutException -> ChatState.Error("Connection timed out mid-stream")
                        is IOException -> ChatState.Error("Network interrupted: ${e.message}")
                        else -> ChatState.Error(e.message ?: "Unknown error")
                    }
                }
                .collect { token ->
                    val current = (_uiState.value as? ChatState.Streaming)?.text ?: ""
                    _uiState.value = ChatState.Streaming(current + token)
                }
            if (_uiState.value is ChatState.Streaming) {
                _uiState.value = ChatState.Complete((_uiState.value as ChatState.Streaming).text)
            }
        }
    }
}

In ServerUnreachable, surface a nudge: "Local AI server not found — check that Ollama is running on your network."

---

Step 4 — Cancel on Network Loss

Register a NetworkCallback in your repository. The docs do not mention this, but lifecycle management is where most implementations leak:

init { connectivityManager.registerDefaultNetworkCallback(networkCallback) }

fun close() { connectivityManager.unregisterNetworkCallback(networkCallback) }

Wire close() to ViewModel.onCleared(). Without unregistration, you accumulate duplicate cancellations across configuration changes. Do not retry silently on loss — surface the interruption. Silent retry loops on a lossy home network produce garbled partial responses.

---

Gotchas

Cleartext traffic. Android blocks plain HTTP on API 28+ by default. Add android:usesCleartextTraffic="true" to your <application> tag, or define a network security config that permits your local IP range. Skip this and you get a cryptic CLEARTEXT communication not permitted exception with nothing pointing at the real cause.

Finite read timeout. Setting any value other than 0 on readTimeout will silently terminate long responses. This is the first thing to check when streaming appears to work but truncates output.

Catching too narrowly. ConnectException means the server is down. SocketTimeoutException means the stream died mid-flight. They need different messages and different recovery paths — do not collapse them into a single catch.

Latency assumptions. A congested 2.4 GHz network can push first-token latency past 500 ms. Benchmark on your actual hardware. Local-network Ollama over 5 GHz Wi-Fi hits 80–200 ms against a GPU server — faster than a typical cloud API round-trip. On 2.4 GHz that advantage can evaporate.

---

Tradeoffs at a Glance

Approach	First token	Peak memory (Android)	Offline
Cloud API	300–800 ms	~2 MB	No
Ollama local network	80–200 ms	~3 MB	No
On-device quantized 3B	1,500–4,000 ms	2,500–4,000 MB	Yes

Measured on a Pixel 7 over 5 GHz Wi-Fi against a local RTX 3090.

---

Conclusion

Two settings make or break this setup: readTimeout(0) on OkHttp and usesCleartextTraffic in your manifest. Get those right and the rest of the wiring is straightforward. The architecture scales cleanly — the same Flow-based repository pattern works whether you later swap Ollama for an on-device engine or a cloud endpoint.

The moment the device leaves the local network, you lose inference entirely. Design for that from day one, and this is a genuinely practical middle path between cloud costs and on-device memory constraints.