Build an On-Device LLM Chatbot with Kotlin and TensorFlow Lite

Large language models are usually accessed through cloud APIs, but modern Android devices can also run smaller AI models locally. This makes it possible to build applications that work offline and keep sensitive prompts on the device.

In this tutorial, we will design the architecture of an on-device LLM chatbot using Kotlin and TensorFlow Lite. The focus is on the mobile integration layer, model execution, prompt handling, and performance considerations.

What We Will Build

The application will have:

• A Kotlin Android UI
• A local TensorFlow Lite model
• A tokenizer layer
• Prompt construction
• Background inference
• Streaming-style response updates
• Basic memory and performance management

The exact model and tokenizer implementation depends on the model architecture you choose. Always use a model converted and packaged for the runtime supported by your Android application.

Architecture

A simple architecture looks like this:

Chat UI
   |
ViewModel
   |
LLM Repository
   |
Tokenizer
   |
TensorFlow Lite Interpreter
   |
Local Model

Keeping model execution behind a repository makes it easier to replace the model later.

Project Setup

Create an Android project with Kotlin and add TensorFlow Lite dependencies appropriate for the runtime and model you selected.

For example:

dependencies {
    implementation("org.tensorflow:tensorflow-lite:<version>")
}

Use the current compatible TensorFlow Lite version rather than copying an old version number from a tutorial.

Place your model in:

app/src/main/assets/

For example:

assets/
└── model.tflite

Loading the Model

Create a small model runner responsible for loading the TensorFlow Lite interpreter.

class LlmRunner(
    private val context: Context
) {
    private val interpreter: Interpreter by lazy {
        val model = loadModel("model.tflite")
        Interpreter(model)
    }

    private fun loadModel(name: String): MappedByteBuffer {
        val fileDescriptor = context.assets.openFd(name)

        FileInputStream(fileDescriptor.fileDescriptor).use { input ->
            return input.channel.map(
                FileChannel.MapMode.READ_ONLY,
                fileDescriptor.startOffset,
                fileDescriptor.declaredLength
            )
        }
    }
}

The model runner should not be called directly from the main thread.

Tokenization

LLMs operate on tokens rather than normal strings. Your tokenizer must convert the user's prompt into the integer representation expected by the model.

Conceptually:

val prompt = "Explain Kotlin coroutines"
val tokens = tokenizer.encode(prompt)

After inference, the generated token IDs need to be decoded back into text.

val text = tokenizer.decode(outputTokens)

The tokenizer must match the model. Using an incompatible tokenizer can produce invalid input or meaningless output.

Running Inference in the Background

Inference can be computationally expensive, so use a coroutine dispatcher designed for CPU work.

class ChatViewModel(
    private val runner: LlmRunner
) : ViewModel() {

    fun generate(prompt: String) {
        viewModelScope.launch {
            val result = withContext(Dispatchers.Default) {
                runner.generate(prompt)
            }

            // Update UI state here
        }
    }
}

This prevents long inference operations from blocking Android's UI thread.

Handling Generated Tokens

A production chatbot should avoid waiting unnecessarily before showing output. Depending on the model runtime, you can expose generated tokens or chunks as they become available.

A simplified API could look like:

interface LlmRunner {
    suspend fun generate(
        prompt: String,
        onToken: (String) -> Unit
    )
}

Then the ViewModel can update the chat state incrementally.

runner.generate(prompt) { token ->
    _response.update { it + token }
}

The actual implementation depends on whether the selected model/runtime supports incremental generation.

Managing Conversation History

Sending the complete conversation to the model on every request increases the token count.

Keep a bounded history:

data class ChatMessage(
    val role: String,
    val content: String
)

Before inference, construct a prompt from only the relevant recent messages.

For example:

val recentMessages = messages.takeLast(10)

For larger applications, consider summarizing older messages instead of keeping everything.

Quantization and Mobile Performance

On-device models can consume significant memory. Quantization can reduce model size and sometimes improve inference performance.

Common approaches include:

• FP16
• INT8
• Weight-only quantization

The best option depends on the model and target device.

You should benchmark:

• Model loading time
• First-token latency
• Tokens per second
• RAM usage
• Battery impact
• Thermal throttling

Avoiding UI Freezes

Never execute model inference inside a click listener directly:

button.setOnClickListener {
    runner.generate(prompt)
}

Instead, move the work into a coroutine or another background execution mechanism.

This is especially important for larger models.

Error Handling

Local inference can fail because of:

• Insufficient memory
• Unsupported operators
• Incorrect tensor shapes
• Invalid tokenizer configuration
• Corrupt model files
• Unsupported device acceleration

Wrap model execution with appropriate error handling and expose useful UI states:

sealed interface ChatState {
    data object Idle : ChatState
    data object Generating : ChatState
    data class Success(val text: String) : ChatState
    data class Error(val message: String) : ChatState
}

Security Considerations

One advantage of local inference is that prompts do not need to leave the device. However, the model itself is part of your application package and may be extracted.

Avoid embedding secrets in the model or APK.

Conclusion

An on-device LLM chatbot combines Kotlin application development with model inference, tokenization, concurrency, and mobile optimization.

The most important production lesson is that AI inference should be treated as a resource-intensive workload. Model size, memory consumption, latency, and thermal behavior all matter on mobile devices.

Once the basic architecture works, you can extend it with conversation memory, retrieval-augmented generation, voice input, function calling, or multimodal models.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx