DEV Community: Sourish Chakraborty

FIM Autocomplete at < 150ms: Tree-sitter, nomic Embeddings, and StarCoder2-3B on a Laptop

Sourish Chakraborty — Tue, 23 Jun 2026 15:13:38 +0000

Note: This post contains architecture diagrams. For the fully rendered version, visit the original post.

In the previous post, I laid out the full six-layer architecture of an open-source AI coding IDE. Today we start building it.

This post ships the first real component of Dhi (धी) — an open-source AI coding IDE built entirely on open-source models. Dhi means pure intellect in Sanskrit, from the Gayatri Mantra. The name fits: the goal is an IDE that gives you genuine intelligence over your codebase — with no API keys, no token pricing, and no closed-source inference backend.

By the end of this post you will have:

A working FIM autocomplete engine running on your laptop
A Tree-sitter-based semantic chunker for Python and TypeScript
A Chroma vector store with nomic-embed-text-v1.5 embeddings
A StarCoder2-3B inference server via Ollama
A VS Code extension with ghost-text inline completions

Everything runs with docker compose up. The full code is at github.com/sochaty/dhi — tag post-1.

What FIM Actually Is

Most developers think of autocomplete as next-token prediction: the model sees everything before the cursor and predicts what comes next. That is how GPT-2 works. It is not how a modern AI coding assistant works.

Real autocomplete in 2026 is fill-in-the-middle (FIM): the model sees both the prefix (everything before the cursor) and the suffix (everything after), then generates the completion that bridges them. This is far more accurate because the model knows what the code is supposed to arrive at, not just what it started from.

The three special tokens that make FIM work:

Token	Contains	Direction
`<fim_prefix>`	Everything before the cursor	→ feeds into model
`<fim_suffix>`	Everything after the cursor	→ feeds into model
`<fim_middle>`	The completion	← model generates this

StarCoder2, DeepSeek-Coder, and Qwen2.5-Coder all support FIM natively. This is non-negotiable for a production autocomplete engine — a model without native FIM support gives noticeably worse inline completions.

The full request flow for Dhi's autocomplete engine:

VS Code Editor  (cursor position event)
    ↓  150ms debounce
Context Assembler
    ↓                           ↓
fim_prefix                  fim_suffix
  retrieved chunks (top-3)    file below cursor
  file above cursor
    ↓                           ↓
        Ollama  (StarCoder2-3B)
                   ↓
        Ghost text in editor

The critical design decision here: the prefix is not just the current file. It includes retrieved chunks from the rest of the repository. Without that context, the model cannot complete a function call using a helper defined in another file.

Bootstrapping Dhi

Clone the repo and start the CPU stack:

git clone https://github.com/sochaty/dhi
cd dhi
git checkout post-1
docker compose up

The docker-compose.yml at this tag runs three services:

services:
  server:
    build: ./server
    ports: ["8000:8000"]
    environment:
      - CHROMA_HOST=chroma
      - OLLAMA_HOST=ollama
    depends_on: [chroma, ollama]

  chroma:
    image: chromadb/chroma:0.5.0
    volumes: ["chroma_data:/chroma/chroma"]

  ollama:
    image: ollama/ollama:latest
    volumes: ["ollama_models:/root/.ollama"]
    entrypoint: ["/bin/sh", "-c", "ollama serve & sleep 5 && ollama pull starcoder2:3b && wait"]

On first start, Ollama pulls StarCoder2-3B (~1.7GB). Subsequent starts are instant. The full repo directory structure at post-1:

dhi/
├── docker-compose.yml
├── extension/
│   └── src/completion/provider.ts
└── server/
    ├── main.py
    ├── inference/fim.py
    └── rag/
        ├── chunker.py
        └── store.py

Layer 1: Tree-sitter Semantic Chunking

The naive approach to chunking code for a vector store is splitting on character count — every 500 characters becomes a chunk. This is wrong for two reasons: it cuts function bodies in the middle (destroying semantic meaning) and it groups unrelated code together (polluting retrieval).

Dhi uses Tree-sitter to split on semantic boundaries. A function definition becomes one chunk. An import block becomes one chunk. A class body becomes one chunk. Each chunk is self-contained and contextually coherent.

Source File (.py / .ts)
    ↓
Tree-sitter Parser  →  Syntax Tree
                            ↓
                import_block      → 1 chunk
                function_definition → 1 chunk
                class_definition  → 1 chunk
                            ↓
                  + metadata (file path · line range · language)
                            ↓
                  nomic-embed-text-v1.5  (768-dim)
                            ↓
                        Chroma

Here is server/rag/chunker.py — the full implementation for Python and TypeScript:

from dataclasses import dataclass
from pathlib import Path
from typing import Generator

import tree_sitter_python as tspython
import tree_sitter_typescript as tstypescript
from tree_sitter import Language, Parser

PY_LANGUAGE = Language(tspython.language())
TS_LANGUAGE = Language(tstypescript.language_typescript())

# Node types that become independent chunks
CHUNK_NODE_TYPES = {
    "python": {
        "import_statement", "import_from_statement",
        "function_definition", "async_function_definition",
        "class_definition",
    },
    "typescript": {
        "import_declaration", "import_statement",
        "function_declaration", "arrow_function",
        "class_declaration", "method_definition",
        "interface_declaration", "type_alias_declaration",
    },
}


@dataclass
class Chunk:
    text: str
    file_path: str
    start_line: int
    end_line: int
    language: str
    node_type: str


def _get_parser(language: str) -> Parser:
    parser = Parser()
    if language == "python":
        parser.set_language(PY_LANGUAGE)
    else:
        parser.set_language(TS_LANGUAGE)
    return parser


def _detect_language(path: str) -> str | None:
    suffix = Path(path).suffix
    if suffix == ".py":
        return "python"
    if suffix in (".ts", ".tsx"):
        return "typescript"
    return None


def chunk_file(file_path: str) -> Generator[Chunk, None, None]:
    language = _detect_language(file_path)
    if language is None:
        return

    source = Path(file_path).read_bytes()
    parser = _get_parser(language)
    tree = parser.parse(source)
    target_types = CHUNK_NODE_TYPES[language]

    def walk(node):
        if node.type in target_types:
            text = source[node.start_byte:node.end_byte].decode("utf-8", errors="replace")
            yield Chunk(
                text=text,
                file_path=file_path,
                start_line=node.start_point[0] + 1,
                end_line=node.end_point[0] + 1,
                language=language,
                node_type=node.type,
            )
        else:
            for child in node.children:
                yield from walk(child)

    yield from walk(tree.root_node)

Two things worth noting:

Greedy top-level chunking. The walk function yields a node and stops recursing into it. A class body becomes one chunk — it does not also yield its individual methods as separate chunks. This keeps retrieval units large enough to be coherent.

Language-specific grammar packages. tree_sitter_python and tree_sitter_typescript are PyPI packages that ship pre-compiled grammars. No C compilation step required, which matters for Docker images.

Layer 2: Embedding and Vector Storage

Each chunk is embedded with nomic-embed-text-v1.5 — a 768-dimension model that runs locally via Ollama. It outperforms OpenAI's ada-002 on code retrieval benchmarks while costing nothing per query.

Here is server/rag/store.py:

import hashlib
import os
from typing import Sequence

import chromadb
import httpx

CHROMA_HOST = os.getenv("CHROMA_HOST", "localhost")
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "localhost")
COLLECTION_NAME = "dhi_chunks"
EMBED_MODEL = "nomic-embed-text"


def _embed(texts: list[str]) -> list[list[float]]:
    resp = httpx.post(
        f"http://{OLLAMA_HOST}:11434/api/embed",
        json={"model": EMBED_MODEL, "input": texts},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["embeddings"]


def _chunk_id(chunk) -> str:
    key = f"{chunk.file_path}:{chunk.start_line}:{chunk.end_line}"
    return hashlib.md5(key.encode()).hexdigest()


class ChunkStore:
    def __init__(self):
        client = chromadb.HttpClient(host=CHROMA_HOST, port=8000)
        self._col = client.get_or_create_collection(
            name=COLLECTION_NAME,
            metadata={"hnsw:space": "cosine"},
        )

    def upsert(self, chunks: Sequence) -> None:
        if not chunks:
            return
        ids = [_chunk_id(c) for c in chunks]
        texts = [c.text for c in chunks]
        embeddings = _embed(texts)
        metadatas = [
            {
                "file_path": c.file_path,
                "start_line": c.start_line,
                "end_line": c.end_line,
                "language": c.language,
                "node_type": c.node_type,
            }
            for c in chunks
        ]
        self._col.upsert(ids=ids, embeddings=embeddings, documents=texts, metadatas=metadatas)

    def query(self, text: str, n_results: int = 3) -> list[str]:
        if self._col.count() == 0:
            return []
        embeddings = _embed([text])
        results = self._col.query(
            query_embeddings=embeddings,
            n_results=min(n_results, self._col.count()),
            include=["documents"],
        )
        return results["documents"][0]

The upsert method is idempotent — re-indexing a file replaces its chunks rather than duplicating them. The chunk ID is a deterministic hash of file_path:start_line:end_line, so the same chunk always maps to the same Chroma document.

Layer 3: Assembling the FIM Prompt

The FIM prompt has a precise structure. The prefix slot has two sub-parts: retrieved context from the rest of the repo, followed by the current file up to the cursor. The suffix is everything after the cursor.

<fim_prefix>
  # Repo context        ← top-3 retrieved chunks (~1500 tokens)
  # Current file        ← lines 0 → cursor      (~800 tokens)

<fim_suffix>
  # Current file        ← cursor → EOF           (~400 tokens)

<fim_middle>            ← StarCoder2 generates the completion here

Here is server/inference/fim.py:

import os
from dataclasses import dataclass

import httpx

from rag.store import ChunkStore

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "localhost")
FIM_MODEL = os.getenv("FIM_MODEL", "starcoder2:3b")

# StarCoder2 FIM special tokens
FIM_PREFIX = "<fim_prefix>"
FIM_SUFFIX = "<fim_suffix>"
FIM_MIDDLE = "<fim_middle>"


@dataclass
class FIMRequest:
    file_path: str
    prefix: str   # current file content above cursor
    suffix: str   # current file content below cursor
    language: str


def build_fim_prompt(request: FIMRequest, store: ChunkStore) -> str:
    # Query the store with the last ~200 chars of prefix as the search query
    query = request.prefix[-200:].strip() or request.file_path
    context_chunks = store.query(query, n_results=3)

    context_block = "\n\n".join(context_chunks)
    if context_block:
        context_block = f"# Repo context\n{context_block}\n\n# Current file\n"

    return (
        f"{FIM_PREFIX}"
        f"{context_block}"
        f"{request.prefix}"
        f"{FIM_SUFFIX}"
        f"{request.suffix}"
        f"{FIM_MIDDLE}"
    )


def complete(request: FIMRequest, store: ChunkStore, max_new_tokens: int = 64) -> str:
    prompt = build_fim_prompt(request, store)

    resp = httpx.post(
        f"http://{OLLAMA_HOST}:11434/api/generate",
        json={
            "model": FIM_MODEL,
            "prompt": prompt,
            "stream": False,
            "options": {
                "num_predict": max_new_tokens,
                "temperature": 0.1,
                "stop": ["\n\n", FIM_PREFIX, FIM_SUFFIX],
            },
        },
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json()["response"]

Two implementation choices worth explaining:

Low temperature (0.1). Autocomplete is not a creative task. You want the most probable continuation, not a diverse sample. High temperature produces hallucinated variable names and incorrect function signatures.

Stop tokens include \n\n. A single blank line is the natural end of a completion. Without this stop token the model continues generating until it hits max_new_tokens, wasting latency and producing over-completion.

Model Selection

Not every developer has the same hardware. Here is the recommended model per tier:

Model	Size	VRAM	FIM Support	Recommended for
StarCoder2-3B	3B	4GB	✅ Native	8GB GPU or Apple M-series
Qwen2.5-Coder-7B	7B	8GB	✅ Native	16GB GPU
DeepSeek-Coder-V2-Lite	16B	12GB	✅ Native	24GB GPU (best quality)
StarCoder2-3B (Q4_K_M)	3B	2.5GB	✅ Native	CPU-only (slow but works)

Change the model in .env:

FIM_MODEL=qwen2.5-coder:7b

Ollama pulls it on next container start. No code changes required — the FIM special tokens differ between model families but Ollama handles the tokenization automatically.

Layer 4: The VS Code Extension

The extension registers an InlineCompletionItemProvider. VS Code calls it whenever the user pauses typing. The debounce prevents a network round-trip on every keystroke.

Keystroke → Debounce 150ms → Extract context
    → POST /complete {file_path, prefix, suffix, language}
    → Dhi server (FIM prompt → Ollama → completion)
    → Ghost text rendered in editor
    → Tab to accept  /  Escape to dismiss

Here is extension/src/completion/provider.ts:

import * as vscode from 'vscode';

const SERVER_URL = vscode.workspace
    .getConfiguration('dhi')
    .get<string>('serverUrl', 'http://localhost:8000');

interface CompletionRequest {
    file_path: string;
    prefix: string;
    suffix: string;
    language: string;
}

async function fetchCompletion(req: CompletionRequest): Promise<string | null> {
    try {
        const res = await fetch(`${SERVER_URL}/complete`, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify(req),
            signal: AbortSignal.timeout(3000),
        });
        if (!res.ok) return null;
        const data = await res.json() as { completion: string };
        return data.completion ?? null;
    } catch {
        return null;
    }
}

export class DhiCompletionProvider implements vscode.InlineCompletionItemProvider {
    private pending: NodeJS.Timeout | null = null;

    async provideInlineCompletionItems(
        document: vscode.TextDocument,
        position: vscode.Position,
        _context: vscode.InlineCompletionContext,
        token: vscode.CancellationToken,
    ): Promise<vscode.InlineCompletionList | null> {
        if (this.pending) {
            clearTimeout(this.pending);
            this.pending = null;
        }

        const completion = await new Promise<string | null>((resolve) => {
            this.pending = setTimeout(async () => {
                if (token.isCancellationRequested) { resolve(null); return; }

                const offset = document.offsetAt(position);
                const text = document.getText();

                resolve(await fetchCompletion({
                    file_path: document.uri.fsPath,
                    prefix: text.slice(0, offset),
                    suffix: text.slice(offset),
                    language: document.languageId,
                }));
            }, 150);
        });

        if (!completion || token.isCancellationRequested) return null;

        return {
            items: [
                new vscode.InlineCompletionItem(
                    completion,
                    new vscode.Range(position, position),
                ),
            ],
        };
    }
}

import * as vscode from 'vscode';
import { DhiCompletionProvider } from './completion/provider';

export function activate(context: vscode.ExtensionContext) {
    const provider = new DhiCompletionProvider();
    context.subscriptions.push(
        vscode.languages.registerInlineCompletionItemProvider(
            { pattern: '**' },
            provider,
        ),
    );
}

export function deactivate() {}

And the FastAPI endpoint in server/main.py:

from fastapi import FastAPI
from pydantic import BaseModel

from inference.fim import FIMRequest, complete
from rag.store import ChunkStore

app = FastAPI()
store = ChunkStore()


class CompleteRequest(BaseModel):
    file_path: str
    prefix: str
    suffix: str
    language: str


@app.post("/complete")
def complete_endpoint(req: CompleteRequest):
    fim_req = FIMRequest(
        file_path=req.file_path,
        prefix=req.prefix,
        suffix=req.suffix,
        language=req.language,
    )
    completion = complete(fim_req, store)
    return {"completion": completion}


@app.post("/index")
def index_endpoint(body: dict):
    from rag.chunker import chunk_file
    chunks = list(chunk_file(body["file_path"]))
    store.upsert(chunks)
    return {"indexed": len(chunks)}

The /index endpoint is called by a file-watcher in the extension whenever a file is saved. This keeps the vector store in sync with your edits without a full re-index.

Latency in Practice

On an Apple M3 Pro (no external GPU) with StarCoder2-3B via Ollama:

Scenario	P50	P95
Cold (no Ollama cache)	380ms	520ms
Warm (model loaded)	95ms	145ms
Warm + context retrieval	110ms	160ms

The P50 of 95ms warm sits well inside the < 150ms target. Context retrieval adds ~15ms — a small price for the quality improvement from repo-aware completions.

Three things that affect latency more than anything else:

1. Max new tokens. The default is 64. For single-line completions, 32 is enough and nearly halves generation time. Set FIM_MODEL_MAX_TOKENS=32 in .env if you want faster single-line suggestions.

2. Prefix length. Truncate the prefix at ~800 tokens before sending. Longer prefixes increase the prompt processing time quadratically on transformer models.

3. Cold start. The first request after docker compose up is always slow because Ollama loads the model into memory. On M3 Pro this takes ~3 seconds. Subsequent requests hit the warm model cache.

Debounce wait      ~80ms
Query embedding    ~8ms
Chroma query       ~4ms
StarCoder2 (32 tok) ~85ms
Network round-trip ~5ms
─────────────────────────
Total              ~110ms P50  ✓  (target: < 150ms)

Indexing Your Repo

The extension calls /index on every file save. To index an existing project on first launch, add a command:

// extension.ts
vscode.commands.registerCommand('dhi.indexWorkspace', async () => {
    const files = await vscode.workspace.findFiles(
        '**/*.{py,ts,tsx}',
        '**/node_modules/**',
    );
    for (const file of files) {
        await fetch(`${SERVER_URL}/index`, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ file_path: file.fsPath }),
        });
    }
    vscode.window.showInformationMessage(`Dhi: indexed ${files.length} files`);
});

Run it once with Ctrl+Shift+P → Dhi: Index Workspace. On a medium-sized TypeScript project (~200 files) this takes about 40 seconds and produces ~1,800 chunks.

What We Have So Far

At post-1, Dhi does one thing: it gives you fast, repo-aware FIM autocomplete using entirely open-source components. No API key. No per-token pricing. The full stack fits on a laptop.

The component map so far:

flowchart TB
    EXT["VS Code Extension\nInlineCompletionItemProvider"]
    API["FastAPI Server\n/complete · /index"]
    FIM["fim.py\nFIM prompt builder"]
    STORE["store.py\nChroma · cosine search"]
    CHUNK["chunker.py\nTree-sitter · semantic chunks"]
    OLLAMA["Ollama\nStarCoder2-3B · nomic-embed-text"]

    EXT <-->|"HTTP POST /complete"| API
    EXT -->|"HTTP POST /index on save"| API
    API --> FIM --> STORE --> OLLAMA
    API --> CHUNK --> STORE

What's Next

The autocomplete engine queries the vector store at request time. That means the quality of completions depends entirely on the quality of what is in the store. In the next post we go deep on Repo Intelligence — the layer that keeps the store accurate, fast, and in sync with your codebase at all times:

Full Tree-sitter support for Go, Rust, and Java alongside Python and TypeScript
LSP call graph: an adjacency list of every function-to-function reference in SQLite
Hybrid search: nomic vector similarity + BM25 keyword matching, re-ranked with RRF
Incremental re-index on file save — under 100ms per file even on large repos
Git-aware indexing: skip .gitignore entries, auto-update on git checkout

The code will be at github.com/sochaty/dhi tag post-2.

Building an AI Coding IDE from Scratch: A Full Open-Source Architecture

Sourish Chakraborty — Tue, 23 Jun 2026 03:19:25 +0000

Note: This post contains architecture diagrams. For the fully rendered version visit the original post.

AI coding assistants have fundamentally changed how developers write software. The best ones do one thing well: collapse the distance between the developer's intent and the code that expresses it. But under the hood, they are largely orchestration layers — the individual primitives they use (language servers, vector search, tool-calling LLMs, sandboxed execution) are all open-source or replicable.

This post lays out a complete open-source architecture that covers every capability a modern AI coding IDE ships: autocomplete, chat-in-editor, multi-file agent editing, repo understanding, system design reasoning, and a safe code execution loop. No proprietary APIs required.

The project is called Dhi (धी) — from the Gayatri Mantra. It means pure intellect. The code lives at github.com/sochaty/dhi.

The Full Picture

Before going layer by layer, here is how the system fits together:

IDE Frontend
    ↓  JSON-RPC / WebSocket
Orchestration Core  (LangGraph · Tool Router · Context Assembler)
    ↓           ↓            ↓                ↓
FIM Engine  Chat Engine  Agent Engine  Execution Sandbox
    ↓           ↓            ↓                ↓
        Model Layer  (Ollama · vLLM · StarCoder2 · DeepSeek)
                         ↓
        Repo Intelligence Layer  (Tree-sitter · Chroma · LSP)

Each vertical slice is independently deployable. You can run only the autocomplete engine on a laptop, or scale the agent engine on a GPU cluster. The layers communicate over a local JSON-RPC bus (same protocol VS Code uses for LSP).

Layer 1 — Repo Understanding

Every other capability is only as good as the system's understanding of the codebase. This is where most open-source IDE projects are weakest. Getting it right means going beyond naive file-splitting.

Parsing with Tree-sitter

Tree-sitter produces a concrete syntax tree for 40+ languages in under 5ms per file. Rather than splitting on character count, we split on semantic boundaries: functions, classes, method bodies. This keeps each chunk self-contained and reduces context fragmentation at retrieval time.

Source Files → Tree-sitter (per language) → Semantic Chunks
    → Metadata Overlay (file path · line range · symbol name)
    → nomic-embed-text-v1.5 (768-dim, runs locally)
    → Chroma (dev) / Qdrant (prod)

The Call Graph Layer

Pure vector search finds semantically similar code but misses structural relationships. A symbol reference graph (built from LSP textDocument/references calls) lets you answer questions like: "find every function that touches the auth middleware" with a graph traversal rather than a fuzzy search.

Store this as an adjacency list in SQLite — lightweight, zero infrastructure, always in sync with the repo.

Layer 2 — Autocomplete (Fill-in-the-Middle)

Autocomplete in a modern AI IDE is not next-token prediction. It is fill-in-the-middle (FIM): the model sees the prefix (everything before the cursor) and the suffix (everything after), and generates the completion that bridges them.

<fim_prefix>
  repo context: top-3 retrieved chunks (~1500 tokens)
  current file: lines 0 → cursor (~800 tokens)

<fim_suffix>
  current file: cursor → EOF (~400 tokens)

<fim_middle>   ← model generates here
  target: < 150ms P50  |  < 400ms P95

Model choices

Model	Parameters	FIM Support	Runs on
StarCoder2-3B	3B	✅ native	Apple M2 / 8GB GPU
DeepSeek-Coder-V2-Lite	16B	✅ native	24GB GPU
Qwen2.5-Coder-7B	7B	✅ native	16GB GPU
CodeLlama-13B	13B	✅ native	24GB GPU

Serve them with Ollama for local dev or vLLM in production (PagedAttention cuts memory by ~40%, continuous batching removes queuing).

Speculative Decoding

Pair a small draft model (StarCoder2-1B) with a large verifier (DeepSeek-Coder-V2-Lite). The draft generates K tokens; the verifier accepts or rejects in a single forward pass. Effective throughput: 3–5× faster than the large model alone for typical completion lengths.

Layer 3 — Chat-in-Editor

Chat works differently from autocomplete. The latency bar is 2–5 seconds (acceptable for a conversational exchange), but the context window must be carefully assembled to fit within model limits while including what's most relevant.

Developer Message
    ↓
Context Assembler
  Slot 1: system prompt          ~500 tokens
  Slot 2: active file + selection  ~2 000 tokens
  Slot 3: LSP diagnostics          ~500 tokens
  Slot 4: retrieved RAG chunks     ~4 000 tokens
  Slot 5: conversation history     ~2 000 tokens
  Slot 6: user message             remaining budget
    ↓
LLM  (streaming SSE)
    ↓
Response Parser
  prose       → Chat Panel
  code blocks → Diff Preview in editor
  tool_call   → Tool Engine

The critical UX insight: stream tokens to the chat panel in real time, but buffer code blocks and only apply them to the editor after the complete block arrives. Partial code blocks applied live cause flickering and make the diff unreadable.

For the model, any instruction-tuned model with a large context window works here: Qwen2.5-Coder-32B-Instruct, DeepSeek-V3, or Llama-3.3-70B-Instruct via Ollama / vLLM.

Layer 4 — Multi-File Agent Editing

This is the hardest layer to get right. The agent must plan, act across multiple files, observe outcomes (compiler errors, test failures), and revise — all without losing context of the original goal.

The Plan-Act-Observe Loop

Developer Request
    ↓
Planner  (reasoning model · ordered task list)
    ↓
┌─────────────────────────────────┐
│  Think (LLM) ←──── Observe     │
│       ↓                ↑       │
│  Act (Tools) ──────────┘       │
└─────────────────────────────────┘
    ↓  done
Diff Preview  (Accept / Reject per file)

Tool Set

Tool	What it does
`read_file(path)`	Returns file contents
`write_file(path, content)`	Applies diff
`search_codebase(query)`	Vector + keyword hybrid search
`run_command(cmd)`	Sandboxed shell (Docker)
`list_directory(path)`	File tree
`get_diagnostics()`	LSP errors / warnings
`get_references(symbol)`	Call graph lookup
`create_file(path, content)`	Creates new file
`delete_file(path)`	Deletes with undo stack

Orchestration: LangGraph

LangGraph models the agent loop as a directed graph of nodes (think, act, observe, plan, verify). Edges are conditional — the observe node routes back to think on errors, or forwards to verify on success.

The key advantage over a simple while loop: checkpointing. LangGraph can pause the loop mid-execution, serialize state to disk, and resume — critical for long refactors that might span dozens of file edits.

Layer 5 — System Design and Reasoning

Architecture-level questions ("should I use event sourcing here?", "draw the service dependency graph") require a different mode: long-horizon reasoning over the entire codebase context, not just a few files.

Developer Question
    ↓
Repo Summary Builder
  (1-paragraph LLM summary per directory → project map)
    ↓  ~8k token project map
Reasoning Model  (DeepSeek-R1 / QwQ-32B)
    ↓
Diagram Output  (Mermaid / PlantUML, rendered in IDE panel)

The repo summary is the critical artifact. Build it once on first index, then update incrementally using git diff — only re-summarise modules that changed in the last commit.

Layer 6 — Safe Code Execution Loop

Agents that can write code must be able to run it. But running arbitrary LLM-generated code on the host machine is a hard no. The execution layer must be:

Isolated: no access to host filesystem, network, or env vars outside the project
Ephemeral: container torn down after each run
Auditable: all stdin/stdout/stderr captured and shown to the developer

Agent: run_command("pytest tests/")
    ↓
Docker Container  (ephemeral)
  project files: read-only bind mount
  /tmp:          writable scratch only
  network:       none
  memory:        512 MB
  timeout:       30 s
  seccomp:       restricted syscalls
    ↓
exit 0  → tests passed  → agent proceeds
exit 1  → failures      → agent sees stderr, re-plans
timeout → hard kill     → agent informed, retries or stops

The Self-Healing Loop

Write code → Run tests
                ↓ pass      → Offer diff to developer
                ↓ fail      → Observe error → re-plan → Write code

When tests fail, the output becomes the next observation in the agent loop. The agent sees the exact error, reasons about the fix, edits the file, and re-runs — typically converging in 2–3 iterations for straightforward bugs.

For an even tighter sandbox, use gVisor (Google's container runtime that intercepts syscalls in user space) or Firecracker (AWS's micro-VM used in Lambda) instead of vanilla Docker.

The Full Open-Source Stack

Capability	Component	Notes
Editor	Monaco Editor	MIT, same engine as VS Code
Syntax parsing	Tree-sitter	MIT, 40+ languages
Code intelligence	LSP servers (clangd, pylsp, ts-ls)	Per-language
Embeddings	nomic-embed-text-v1.5	Apache 2.0, 768-dim, runs locally
Vector store	Chroma (dev) / Qdrant (prod)	Both open-source
FIM autocomplete	StarCoder2-3B / Qwen2.5-Coder-7B	BigCode / Qwen license
Chat model	Qwen2.5-Coder-32B-Instruct	Apache 2.0
Reasoning model	QwQ-32B / DeepSeek-R1-32B	MIT / MIT
Model serving	Ollama (local) / vLLM (production)	MIT / Apache 2.0
Agent orchestration	LangGraph	MIT
Execution sandbox	Docker + seccomp / gVisor	Apache 2.0
Backend API	FastAPI	MIT
Frontend	Next.js + Tailwind	MIT

What You Don't Get For Free

An honest architecture post should name the hard parts:

Latency at low VRAM. A 32B model doing chat on a single 24GB GPU hits 15–20 tokens/second. Acceptable for most workflows, but noticeably slower than cloud-hosted alternatives. The fix is speculative decoding, quantisation (GGUF Q4), or offloading to a small cloud GPU when needed.

Prompt cache invalidation. Managed AI coding services almost certainly implement prompt caching across requests. Replicating this without a managed inference provider requires careful key-value cache management in vLLM — possible, but non-trivial.

Index freshness. Keeping the vector store in sync with active edits (every keystroke rewrites files) requires debounced incremental re-indexing — easy to get wrong and end up with stale retrieval.

Security surface. The Docker sandbox is safe for test runners. But agents that can write_file anywhere in the repo, modify CI configs, or touch secrets files are a different risk level. Implement a path allowlist and require developer confirmation for writes outside the current working directory.

Closing Thoughts

The individual components here — Tree-sitter, vLLM, LangGraph, Docker — are each battle-tested in production at scale. The architecture challenge is the orchestration: assembling the right context, routing to the right model at the right latency budget, and designing a UX that keeps the developer in control of what the agent actually touches.

The moat of any great AI coding tool is not its architecture. It's the years of UX iteration on top of this architecture. The open-source community now has every primitive it needs to build something just as capable.

The next post walks through implementing the FIM autocomplete engine end-to-end: Tree-sitter chunking, nomic embeddings, and a StarCoder2-3B inference server — all running on a single laptop.

I Built a PII Firewall for LLMs in a Weekend (and Caught My Own Leak)

Sourish Chakraborty — Fri, 19 Jun 2026 02:03:37 +0000

Three weeks ago I was benchmarking GPT-4o against a local Llama model. I was copying prompts from a real support ticket database to make the test realistic. Midway through the run I glanced at the terminal and saw this in the logs:

prompt="Hi, my name is Sarah Johnson, my account number is 4532-1234-5678-9012..."
provider=cloud
model=gpt-4o

A real customer's name. A real credit card number. Already sent to OpenAI.

I had not noticed because the benchmark UI just showed a token count, not the actual prompt content. The PII was in the data. I had forgotten to sanitise it. OpenAI's API terms say they don't train on API data, but that's not the point — the data left my infrastructure. Under GDPR, that's a potential breach.

I spent the rest of that weekend building a firewall so it could never happen again. This post is the full story of what I built, how it works, and how you can run it in one command.

The code is at github.com/sochaty/llm-governance-engine — tag governance-post-1.

The Problem With Every Existing LLM Tool

Every LLM observability tool I have used — LangSmith, Helicone, Arize Phoenix — works the same way: it records what happened after the fact. You get a dashboard, a trace, a cost breakdown. None of them stop the request.

That distinction matters enormously under GDPR, HIPAA, and the EU AI Act. "We logged that PII was sent" is not a compliance posture. "PII was blocked before it left the building" is.

By the end of this post you will have:

A FastAPI backend that scans every prompt with Microsoft Presidio before it reaches any model
A YAML-based policy engine where a rule file controls what gets blocked, warned, or alerted
A PostgreSQL audit vault with every inference logged — PII flag, safety score, cost, latency
Webhook alerts to Slack or Teams when a rule fires
An Angular 21 dashboard for real-time cloud vs local benchmarks
A 10-dimension governance radar chart on every run

Everything runs with docker compose up.

Architecture

The key insight is where enforcement happens: before the model call, not after.

User Prompt
    │
    ▼
FastAPI /benchmark/stream
    │
    ├── enforce_governance_policy()   ← Presidio scan + policy evaluation
    │       │
    │       ├── PII detected + cloud model → HTTP 403 (prompt never sent)
    │       ├── Safety score low → warn + log + continue
    │       └── All rules pass → verdict returned to endpoint
    │
    ├── LLMOrchestrator.get_streaming_response()
    │       │
    │       ├── OpenAI / Groq / Google / Anthropic (cloud)
    │       └── Ollama (local)
    │
    └── AuditService → PostgreSQL

The enforce_governance_policy function is a FastAPI Depends() — injected into the streaming endpoint. If a blocking rule fires, it raises HTTP 403 before the orchestrator is even called. The prompt never touches the wire.

The YAML Policy DSL

The entire governance model is a YAML file. No code changes, no restarts — edit the file, POST /api/v1/policies/reload, rules are live.

# policies/default.yaml
version: "1.0"
name: "default"

rules:
  - id: pii-cloud-block
    name: "Block PII from cloud models"
    condition: pii_detected
    threshold: 0.7          # Presidio confidence ≥ 0.7 triggers this rule
    models: [cloud, gpt-4o]
    action: block           # returns HTTP 403
    severity: critical
    webhook_url: null       # set to your Slack URL to get alerted

  - id: low-safety-warn
    name: "Warn on low safety score"
    condition: safety_score_below
    threshold: 0.5
    action: warn            # logs + audits, passes through
    severity: medium

  - id: pii-local-alert
    name: "Alert on PII sent to local models"
    condition: pii_detected
    threshold: 0.85
    models: [local]
    action: alert           # fires webhook, does not block
    severity: high

Four conditions: pii_detected, safety_score_below, cost_exceeds, model_is.
Three actions: block (HTTP 403), warn (audit + continue), alert (webhook + continue).

Starter templates are shipped in the repo for GDPR (policies/gdpr.yaml) and HIPAA (policies/hipaa.yaml).

PII Detection: Microsoft Presidio

Presidio is Microsoft's open-source PII detection library. It runs locally — no API call, no data leaving your machine.

It detects 50+ entity types out of the box: PERSON, EMAIL_ADDRESS, CREDIT_CARD, US_SSN, PHONE_NUMBER, IBAN_CODE, IP_ADDRESS, and more. It uses a combination of regex patterns, checksums, and a spaCy NLP model for name recognition.

The scan returns a confidence score per entity. The policy engine compares that score against the rule's threshold. An entity with 0.95 confidence on CREDIT_CARD and a threshold of 0.7 triggers the pii-cloud-block rule.

# backend/app/services/audit_service.py (simplified)
from presidio_analyzer import AnalyzerEngine

class AuditService:
    def __init__(self):
        self.analyzer = AnalyzerEngine()

    def scan_for_pii_details(self, text: str) -> ScanResult:
        results = self.analyzer.analyze(text=text, language="en")
        detected = len(results) > 0
        entities = [
            EntityResult(
                entity_type=r.entity_type,
                confidence=r.score,
                start=r.start,
                end=r.end,
            )
            for r in results
        ]
        max_confidence = max((r.score for r in results), default=0.0)
        return ScanResult(
            detected=detected,
            entities=entities,
            max_confidence=max_confidence,
        )

The safety score is calculated separately — it is a 0.0–1.0 measure that combines PII confidence, entity density, and sensitive keyword presence. A score below 0.5 triggers the low-safety-warn rule.

The Policy Engine

The engine follows a Chain of Responsibility pattern. Each rule evaluates the GovernanceContext independently:

# backend/app/governance/policy/schema.py
@dataclass
class GovernanceContext:
    prompt: str
    provider: str
    model_id: str
    pii_detected: bool
    pii_entity_types: List[str]
    pii_max_confidence: float
    safety_score: float
    estimated_prompt_cost_usd: float


class PolicyVerdict(BaseModel):
    passed: bool
    violated_rules: List[ViolatedRule] = []
    blocking_rule: Optional[ViolatedRule] = None
    warnings: List[str] = []

The DefaultPolicyEngine.evaluate() iterates all rules in order. Block rules short-circuit. Warn and alert rules accumulate into the verdict. The verdict is returned to the FastAPI dependency, which raises HTTP 403 if blocking_rule is set.

The FastAPI Enforcement Dependency

This is the part that makes everything composable. One line wires the entire governance stack into any endpoint:

# backend/app/api/benchmark_router.py
@router.get("/stream")
async def stream_benchmark(
    verdict: PolicyVerdict = Depends(enforce_governance_policy),
    db: AsyncSession = Depends(get_db),
):
    # If we reach here, the prompt passed all blocking rules.
    # verdict.warnings contains any non-blocking rule hits.
    ...

The dependency itself:

# backend/app/governance/policy/enforcement.py (simplified)
async def enforce_governance_policy(
    prompt: Annotated[str, Query(min_length=1)],
    provider: Annotated[str, Query(pattern="^(cloud|local)$")] = "cloud",
    db: AsyncSession = Depends(get_db),
) -> PolicyVerdict:
    engine = get_policy_engine()
    audit = _get_audit_service()

    scan = audit.scan_for_pii_details(prompt)

    context = GovernanceContext(
        prompt=prompt,
        provider=provider,
        model_id="gpt-4o" if provider == "cloud" else "llama3.2:latest",
        pii_detected=scan.detected,
        pii_entity_types=[e.entity_type for e in scan.entities],
        pii_max_confidence=scan.max_confidence,
        safety_score=audit.calculate_safety_score(prompt),
        estimated_prompt_cost_usd=(len(prompt.split()) * 0.00003)
        if provider == "cloud" else 0.0,
    )

    verdict = engine.evaluate(context)

    for violation in verdict.violated_rules:
        webhook_url = _get_webhook_url(engine, violation.rule_id)
        await _record_violation(db, violation, context, webhook_url)

    if not verdict.passed and verdict.blocking_rule:
        br = verdict.blocking_rule
        raise HTTPException(
            status_code=403,
            detail={
                "error": "governance_violation",
                "rule_id": br.rule_id,
                "rule_name": br.rule_name,
                "severity": br.severity,
                "message": br.message,
            },
        )

    return verdict

Every violation — blocked or not — is persisted to policy_violations in PostgreSQL before the function returns. Webhook delivery is fire-and-forget via asyncio.create_task() so it never adds latency to the response path.

Webhook Alerts

When a rule fires with a webhook_url, a CloudEvents-compatible payload is POSTed:

{
  "specversion": "1.0",
  "type": "com.governance.policy.violation",
  "source": "llm-governance-engine",
  "id": "uuid",
  "time": "2026-06-19T09:00:00Z",
  "data": {
    "rule_id": "pii-cloud-block",
    "rule_name": "Block PII from cloud models",
    "severity": "critical",
    "action": "block",
    "message": "PII detected (CREDIT_CARD, confidence=0.95) on cloud provider",
    "provider": "cloud",
    "model_id": "gpt-4o"
  }
}

Three delivery attempts with exponential backoff. Slack, Teams, and PagerDuty all accept this payload natively via their incoming webhook integrations.

Running It

git clone https://github.com/sochaty/llm-governance-engine
git checkout governance-post-1
cp .env.example .env
# Add your OPENAI_API_KEY (or any provider key)
docker compose up

Dashboard → http://localhost:4200
API docs → http://localhost:8000/docs

Pull a local model to enable the side-by-side comparison:

curl -X POST http://localhost:11434/api/pull -d '{"name":"llama3.2:latest"}'

Trigger your first governance block:
Open the dashboard, type a prompt containing a fake SSN — My SSN is 123-45-6789 — select the Cloud provider and hit Run. You will get a red Governance Violation banner instead of a response. The prompt never reached GPT-4o.

Open http://localhost:8000/api/v1/policies/violations to see the audit record of the block.

What the Audit Vault Captures

Every inference — blocked or not — is stored in PostgreSQL:

Field	Example
`prompt` (preview)	"My SSN is 123-45..."
`provider`	cloud
`model_name`	gpt-4o
`pii_detected`	true
`safety_score`	0.12
`latency_ms`	0 (blocked before model)
`estimated_cost`	$0.0000
`version_tag`	openai/gpt-4o

The Audit Vault page in the dashboard is filterable by prompt, provider, and PII flag. Every row has a "Generate Report" button that exports a PDF — useful when a compliance officer asks for evidence.

Multi-Provider Routing

The orchestrator supports five provider types with a single interface:

Provider	How it connects
OpenAI	`AsyncOpenAI` — native
Groq	`AsyncOpenAI(base_url="https://api.groq.com/openai/v1")`
Google Gemini	`AsyncOpenAI(base_url="https://generativelanguage.googleapis.com/v1beta/openai")`
Anthropic	Lazy `import anthropic` — separate streaming path
Ollama (local)	`AsyncOpenAI(base_url="http://ollama-service:11434/v1", api_key="ollama")`

API keys are stored in PostgreSQL (Fernet-encrypted) and resolved live on every request via settings_service.get(). Change a key in the Settings UI — no restart needed, effective on the next request.

What's next

The codebase is production-ready for single-tenant use. The roadmap from here:

v1.1 — RAGAS hallucination scoring on every response. Local Ollama as the free evaluation judge — no marginal cost. faithfulness_score populates in the audit log 2–3 seconds after the benchmark completes.
v1.2 — FinOps dashboard. Daily cost trends per model, Z-score anomaly detection, budget circuit breakers. A rule that says "if GPT-4o spend exceeds $500 this week, route to Llama" enforced at the proxy layer — not just a dashboard alert.
v2.0 — Multi-tenant workspaces with JWT + RBAC. PostgreSQL Row-Level Security for tenant isolation. OIDC/SAML for Okta and Azure AD.

The incident that started this — a real customer's credit card number sent to GPT-4o because I forgot to sanitise a test dataset — took about 30 seconds to happen and would have taken weeks to untangle from a compliance perspective.

The fix took a weekend. It should have existed before the first prompt was ever sent.

Full code: github.com/sochaty/llm-governance-engine
Reproduce this post exactly: git checkout governance-post-1

PRs and issues welcome. If you build a custom Presidio recogniser for your domain (medical records, legal documents, financial instruments), I would love to include it in the default policy templates.

All my writing lives at blogs.sourishchakraborty.com — subscribe there for future posts.