Integrating LLM into Desktop Applications with Long Context Workloads

#engineering #oxlo #ai

We are going to build a lightweight desktop CLI tool that ingests an entire local project directory into a single long-context prompt and answers questions about the code. This is useful when you need to onboard a new repository, audit legacy logic, or trace dependencies without spinning up a vector database. Because we are sending the full text directly to the model, a request-based provider like Oxlo.ai keeps the cost predictable even when the context grows. See https://oxlo.ai/pricing for details.

What you'll need

Python 3.10 or newer.
An Oxlo.ai API key from https://portal.oxlo.ai.
The OpenAI SDK: pip install openai.
A local project folder you want to analyze.

Step 1: Set up the client

Create a new file named codebase_chat.py and initialize the Oxlo.ai client. I read the API key from the environment so it is not hardcoded.

import os
from pathlib import Path
from openai import OpenAI

OXLO_API_KEY = os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
TARGET_DIR = Path("./sample-project")

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=OXLO_API_KEY,
)

Step 2: Ingest files

Recursively collect text files from the target directory, skip noisy build folders, and concatenate everything into one context string. I cap the output at roughly 120,000 characters to stay safely inside the context window of the model we will use.

TEXT_EXTENSIONS = {
    ".py", ".md", ".txt", ".js", ".ts", ".json",
    ".yaml", ".yml", ".rs", ".go", ".java", ".c",
    ".cpp", ".h", ".cs", ".rb", ".swift", ".kt",
}

SKIP_DIRS = {
    ".git", "__pycache__", "node_modules", ".venv",
    "venv", "dist", "build", ".idea", ".vscode",
    "target", ".pytest_cache",
}

def ingest_directory(root: Path, max_chars: int = 120_000) -> str:
    parts = []
    total = 0

    for path in sorted(root.rglob("*")):
        if any(skip in path.parts for skip in SKIP_DIRS):
            continue
        if path.is_file() and path.suffix in TEXT_EXTENSIONS:
            try:
                text = path.read_text(encoding="utf-8", errors="ignore")
            except Exception:
                continue
            header = f"\n--- FILE: {path.relative_to(root)} ---\n"
            segment = header + text
            if total + len(segment) > max_chars:
                parts.append("\n[Additional files truncated due to context limit]\n")
                break
            parts.append(segment)
            total += len(segment)

    return "".join(parts)

context = ingest_directory(TARGET_DIR)
print(f"Ingested {len(context)} characters from {TARGET_DIR}")

Step 3: Define the system prompt

The system prompt grounds the model. It tells the agent to cite filenames and admit when it cannot find an answer.

SYSTEM_PROMPT = """You are a senior software engineer reviewing a local codebase.
Answer questions using only the provided file contents.
Cite filenames when referencing specific logic.
If the answer is not in the context, say you cannot find it."""

Step 4: Query with long context

Pass the full codebase as part of the user message. I use kimi-k2.6 because its 131K context window handles large codebases well, and Oxlo.ai charges per request rather than per token, so a massive prompt does not inflate the cost.

def ask_codebase(question: str) -> str:
    user_message = f"{question}\n\nCODEBASE CONTEXT:\n{context}"

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        stream=True,
    )

    reply = ""
    for chunk in response:
        if chunk.choices[0].delta.content:
            piece = chunk.choices[0].delta.content
            reply += piece
            print(piece, end="", flush=True)
    print()
    return reply

ask_codebase("What does the main entry point do, and what dependencies does it use?")

Step 5: Wrap in an interactive loop

Wrap the logic in a small REPL so the tool feels like a desktop assistant. You can ask follow-up questions without reloading the files.

if __name__ == "__main__":
    print("Codebase loaded. Ask a question or type 'exit'.")
    while True:
        try:
            question = input("\n> ").strip()
        except (KeyboardInterrupt, EOFError):
            break
        if question.lower() in {"exit", "quit"}:
            break
        if not question:
            continue
        ask_codebase(question)

Run it

Point the script at a project folder and start asking questions.

$ export OXLO_API_KEY="oxlo_..."
$ python codebase_chat.py
Ingested 84,231 characters from sample-project

Codebase loaded. Ask a question or type 'exit'.

> What does the main entry point do?

The main entry point is `src/main.py`. It initializes a FastAPI application, registers routers from `src/api/routes.py`, and starts an HTTP server on port 8000. It depends on `uvicorn`, `fastapi`, and `pydantic` as seen in `requirements.txt`.

> How is authentication handled?

Authentication is handled in `src/middleware/auth.py`. It validates Bearer tokens against a local SQLite database defined in `src/db/session.py`.

> exit

Next steps

Add a file watcher with watchdog so the context refreshes automatically when you save changes. If you need to support repositories larger than the context window, split the ingestion layer into a local SQLite cache and retrieve only relevant files via keyword matching before sending the request to Oxlo.ai.

DEV Community