Build a RAG-Based AI Assistant in Kotlin with a Vector Database

A normal LLM answers questions from information contained in its model. A Retrieval-Augmented Generation (RAG) system adds an external knowledge layer so an application can answer questions using private or frequently changing documents.

In this tutorial, we will build the architecture for a Kotlin client that communicates with a backend RAG service.

Why Use RAG?

Suppose an application contains company documentation.

Instead of sending the entire documentation to an LLM for every question, the system can:

1. Split documents into chunks.
2. Generate embeddings.
3. Store embeddings in a vector database.
4. Convert the user's question into an embedding.
5. Retrieve the most relevant chunks.
6. Send those chunks to the LLM.
7. Return a grounded answer.

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database

At query time:

User Question
   ↓
Embedding
   ↓
Vector Search
   ↓
Relevant Chunks
   ↓
LLM
   ↓
Answer

Kotlin Client Architecture

A clean Android architecture can look like:

Compose UI
   |
ViewModel
   |
RagRepository
   |
API Client
   |
RAG Backend

The vector database and LLM should normally remain on the backend rather than being exposed directly to the mobile application.

API Model

Define a request:

data class AskRequest(
    val question: String,
    val conversationId: String?
)

And a response:

data class AskResponse(
    val answer: String,
    val sources: List<Source>
)

data class Source(
    val title: String,
    val chunk: String
)

Retrofit Client

A Retrofit interface can expose the backend:

interface RagApi {

    @POST("api/ask")
    suspend fun ask(
        @Body request: AskRequest
    ): AskResponse
}

The backend endpoint is responsible for retrieval and generation.

Repository

Keep network details outside the ViewModel:

class RagRepository(
    private val api: RagApi
) {
    suspend fun ask(
        question: String,
        conversationId: String?
    ): AskResponse {
        return api.ask(
            AskRequest(question, conversationId)
        )
    }
}

ViewModel

Use a UI state model:

sealed interface ChatState {
    data object Idle : ChatState
    data object Loading : ChatState
    data class Success(val response: AskResponse) : ChatState
    data class Error(val message: String) : ChatState
}

Then:

fun ask(question: String) {
    viewModelScope.launch {
        state.value = ChatState.Loading

        try {
            val response = repository.ask(
                question,
                conversationId
            )

            state.value = ChatState.Success(response)
        } catch (e: Exception) {
            state.value = ChatState.Error(
                e.message ?: "Request failed"
            )
        }
    }
}

Document Ingestion

The backend ingestion pipeline might look like:

PDF / Markdown / HTML
       ↓
Text Extraction
       ↓
Chunking
       ↓
Embedding Model
       ↓
Vector Database

A chunk might contain:

{
  "text": "Kotlin coroutines provide structured concurrency...",
  "document": "kotlin-guide.md",
  "section": "Coroutines"
}

The metadata is useful when displaying citations to the user.

Embeddings

An embedding model converts text into a vector.

Conceptually:

"Kotlin coroutines"
        ↓
[0.021, -0.41, 0.73, ...]

The vector database stores this representation and supports similarity search.

Retrieval

When the user asks:

How does structured concurrency work?

The backend generates an embedding for the question and searches for similar chunks.

The top results are then added to the LLM prompt.

A simplified prompt could be:

Use the following context to answer the question.

Context:
{retrieved_chunks}

Question:
{user_question}

Grounding and Citations

A useful RAG assistant should return sources rather than only an answer.

For example:

LazyColumn {
    items(response.sources) { source ->
        Text(source.title)
        Text(source.chunk)
    }
}

This gives users a way to verify the generated answer.

Streaming Responses

For a better chat experience, the backend can stream generated tokens.

Depending on your backend protocol, Kotlin can consume Server-Sent Events or another streaming protocol.

The UI can then append chunks as they arrive:

_response.update { current ->
    current + token
}

Handling Retrieval Failures

A RAG system should not blindly answer every question.

If retrieval produces weak matches, the backend can return:

I could not find enough information in the provided documents.

This is often safer than allowing the model to invent an answer.

Security

Never put LLM API keys or vector database credentials directly into an Android application.

Use:

Android App
    ↓ HTTPS
Backend API
    ↓
Vector Database
    ↓
LLM Provider

Authenticate mobile users at the API layer and enforce authorization when retrieving private documents.

Production Improvements

Once the basic pipeline works, add:

• Hybrid keyword + vector search
• Reranking
• Conversation memory
• Document-level permissions
• Streaming generation
• Response caching
• Source citations
• Rate limiting
• Observability
• Evaluation datasets

Conclusion

A Kotlin RAG assistant is more than an AI chat screen. The Android application should focus on user experience and secure API communication, while the backend manages embeddings, retrieval, document permissions, and LLM orchestration.

This architecture scales much better than embedding AI credentials and vector database logic directly inside the mobile application.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx