DEV Community: raphiki

Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits

raphiki — Sat, 12 Sep 2026 05:47:39 +0000

Part 3 of the DeepSeek Harness: Kernel to Edge series: A hands-on benchmark pairing DeepSeek Harness, Ollama, and Ornith-1.5 on an 8GB RTX 4070 laptop to solve the tradeoff between tool bloat and context headroom.

DeepSeek Harness: Kernel to Edge

This article is Part 3 of a three-part architectural and practical deep dive:

Part 1: Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything (Microkernel primitives, spatiotemporal composability, reverse cleanup stacks, zero-leak lifecycles).

Part 2: DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents (The meta-harness paradigm, Cordis as an agent kernel, comparing DSH to OpenCode and Pi).

Part 3 (This Article): Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits (Hands-on local deployment with Ollama, Ornith-1.5, VRAM arithmetic, minimal vs. standard presets, and TUI).

1. Introduction: Bringing the Harness Home

In Part 1, we explored Cordis, the TypeScript framework designed for zero-leak dynamic lifecycles and spatiotemporal composability. In Part 2, we mapped the systems architecture of DeepSeek Harness (DSH), showing how it abandons monolithic Python agent loops in favor of a pure plugin microkernel.

Now, we move from architectural theory to a practical, working developer setup.

My target machine is a standard laptop equipped with an NVIDIA RTX 4070 GPU (8GB VRAM) and 32GB of system RAM.

Running autonomous coding agents locally on consumer hardware presents strict physical constraints. Frontier cloud models (like Claude 3.7 Sonnet or DeepSeek-V3) make multi-turn tool calling seem straightforward due to their massive context windows and vast parameter counts. When running an agent framework on an 8GB GPU, you immediately encounter hard physical boundaries:

VRAM exhaustion: Loading a model and its KV cache can easily exceed 8GB, causing CUDA out-of-memory errors or offloading layers to the CPU, which slows execution to a crawl.
Context window starvation: Agent frameworks typically inject extensive tool definitions, system instructions, and file trees into the prompt. A default local context window fills up before the agent can complete its first multi-turn refactor.
Tool-calling fragility: Smaller quantized models often hallucinate parameters or break JSON schemas when overloaded with too many simultaneous tools.

In this tutorial, we will configure a local coding agent on this 8GB machine using DeepSeek Harness, Ollama, and Ornith-1.5. We will explore why DSH's plugin design adapts well to local execution, how to tune the context window, and how agent presets help manage memory bottlenecks.

2. Why Choose DeepSeek Harness for a Local Agent?

Most developers look at coding harnesses like OpenCode or Pi when building personal workflows. As we established in Part 2, those are turnkey developer tools designed around fixed, opinionated toolsets.

When your hardware is constrained to 8GB of VRAM, opinionated harnesses can become restrictive. If a harness hardcodes ten developer tools into every single turn, your local model has to parse ten complex JSON schemas on every inference call. That consumes hundreds of tokens of precious context and increases the cognitive load on a smaller model.

This is where DeepSeek Harness's architecture as a meta-harness becomes practical:

Native Extensibility and Granular Tool Control: Because DSH has no privileged core, tools and interfaces are not baked into the runtime. Even the Web UI itself is just a plugin: developers who prefer keyboard-driven terminal workflows can easily swap it for community TUIs like dsh-TUI. Across community directories like deepseekplugin.com and dsh-plugins.org, thousands of plugins exist for vision, generative UI, and memory. But when constrained by 8GB of VRAM, this extensibility works in reverse: instead of piling on plugins, you can mount only the exact tools your local model needs (such as file editing and bash) while leaving out all memory-heavy peripherals.
Predictable Teardown via Cordis Fibers: Local development involves running code, launching compilers, and watching files. Cordis ensures that every child process or execution sandbox is cleanly terminated when a task completes, preventing zombie processes from stealing system RAM.
Declarative Agent Presets: DSH supports configurable agent presets (settings.yaml). Switching from a full development dashboard to a lightweight minimal mode requires only a single configuration line.

Every token saved in the system prompt directly translates to more room for code and conversation history.

Defining the Use Case: An Autonomous Task Worker, Not an Interactive Chatbot

Hardware constraints depend heavily on the intended workload.

Local models are frequently judged as interactive chat assistants (like ChatGPT) or inline autocomplete engines (like GitHub Copilot), where fast token streaming (40 to 60+ tokens per second) is required to maintain user momentum.

DeepSeek Harness serves a different workload: autonomous, multi-turn task execution.

A typical task involves handing off an end-to-end debugging session in an existing repository:

"There is an edge case in the WebSocket session manager where dropped connections leave stale socket descriptors open. Locate the relevant files, write a script to reproduce the leak, fix the cleanup logic, and confirm that the test suite passes."

In this setup:

Long-running execution: The agent runs across 10 to 30 continuous turns: inspecting directory trees, reading source files, running test scripts in bash, analyzing error traces, applying patches, and re-running the test suite.
Throughput vs. Resilience: On consumer hardware, a 9B model generating 10 to 20 tokens per second is impractical for live conversation, but completely sufficient for an asynchronous worker running unattended in the background or in a terminal (dsh-TUI).
The primary constraints: The operational bottleneck is not token generation latency, but lifecycle stability (ensuring spawned compiler processes and file watchers do not leak system resources) and context headroom (preventing prompt overhead from pushing VRAM into system memory).

3. Step 1: Installation and Baseline Test with OpenRouter

Before troubleshooting local quantization or GPU memory allocation, we validate that DeepSeek Harness operates correctly on our system using a known cloud endpoint. Building outside-in ensures that any subsequent issues stem from local model configurations, not the harness itself.

Installation

DeepSeek Harness is packaged as an npm module. You can run it directly using npx or install it globally:

# Global installation via npm
npm install -g @deepseek-ai/dsh

Baseline Cloud Configuration

For our initial baseline test, we will connect DSH to OpenRouter, giving us access to frontier models (such as DeepSeek-V3 or Mistral) to confirm tool calling and web console connectivity.

Export your API key in your terminal:

# Windows PowerShell
$env:OPENROUTER_API_KEY = "sk-or-v1-your-key-here"

# Linux / macOS
export OPENROUTER_API_KEY="sk-or-v1-your-key-here"

Launch the web interface:

dsh web

C:\Local\Dev\dsh>dsh web
dsh web: http://127.0.0.1:3080/?token=4n7i06dEtYEUzwuPfVmXVRGEx-fxFRRgoQJx4Yfq3wg
dsh web: opening the default browser; pass --no-open to disable

By default, DSH starts its local web server at http://localhost:3080. Open your browser and navigate to the Coding Agent.

Verify the baseline by sending simple instructions, such as inspecting the current directory or reading the contents of a file.

Watch the console stream the model's reasoning, dispatch the filesystem tool, and return the results. Once you confirm this loop completes without errors, you know the harness, process permissions, and session streams are healthy.

The screenshot below shows the DSH web interface connected to Mistral Small via OpenRouter, displaying initial latency and context metrics.

The Trajectory view breaks down individual execution turns, separating model thinking blocks from raw tool payloads.

Notice in the navigation bar that DSH initialized in Standard mode; we will explore how presets shape agent behavior in Step 3.

4. Step 2: Selecting and Preparing the Local Model (Ornith-1.5)

Now we replace the cloud API with a fully local open-weight model.

Why Ornith-1.5?

Model selection on an 8GB GPU balances two requirements: reasoning capacity for multi-file code analysis and physical memory headroom.

1. What is Ornith-1.5? (A Specialized Qwen Fine-Tune)

Ornith-1.5 (9B) is an open-weight model derived from the Qwen2.5 architecture.

While Qwen2.5 provides a strong baseline for code syntax and logic, general instruction-tuned checkpoints often struggle with structured tool calling in autonomous harnesses:

Schema errors: Base models frequently omit mandatory JSON keys, invent unsupported parameters, or fail to escape control characters in code strings.
Conversational leakage: Conversational models often prepend tool calls with conversational filler ("Sure, I will execute this command for you..."), which breaks strict JSON schema parsers.
Context degradation: During extended multi-turn sessions, general models can lose track of earlier tool outputs or swap arguments between different tool interfaces (such as passing bash flags into file editing parameters).

Ornith-1.5 addresses this through fine-tuning on structured agent traces and explicit function-calling datasets. It acts as a tool execution engine: given a tool schema, it generates valid, compact JSON calls on the first attempt and processes output streams directly.

2. Why 9B Parameters on an 8GB GPU?

On an 8GB GPU, parameter count determines whether execution remains hardware-accelerated:

7B Models: Fast and lightweight, but often struggle with the multi-step deductive reasoning needed to navigate complex repository structures unattended.
14B Models: At 4-bit quantization, 14B weights require ~9GB, immediately spilling across the PCIe bus into system RAM and dropping throughput to 1 to 2 tokens per second.
The 9B Sweet Spot: At 4-bit quantization (Q4_K_M), Ornith-1.5 occupies approximately 5.5GB of VRAM. This leaves roughly 2.5GB of headroom on an 8GB card, providing the space needed for an extended KV cache and display buffers without triggering CPU paging.

Inference Server: Ollama

We will use Ollama to serve the model locally. Ollama provides a native OpenAI-compatible endpoint (http://127.0.0.1:11434/v1), making integration straightforward.

The 2048 Context Constraint

By default, Ollama initializes models with a 2,048 token context window (num_ctx 2048). While sufficient for simple chat queries, an autonomous coding agent quickly exhausts this budget:

System instructions and agent personas consume ~400 tokens.
Tool schemas (filesystem, bash, code editor) consume ~800 to 1,200 tokens.
Workspace context (directory trees, project files) easily consumes another 500+ tokens.

Under default settings, tool schemas and prompts consume most of the available window before the model even generates its first turn. As soon as a multi-file task begins, the engine either truncates earlier messages or drops tool definitions from context.

Tuning the Context Window to 16K

To give our agent room to work, we need to create a customized model variant with a 16K context window (16384 tokens).

We extract the base model's Modelfile, append the num_ctx parameter, and build a local derivative:

ollama show --modelfile ornith-1.5:9b > Modelfile
echo PARAMETER num_ctx 16384 >> Modelfile

ollama create ornith-1.5:ctx -f Modelfile

This sequence extracts the base template, sets the KV cache allocation parameter to 16,384 tokens, and compiles the new model tag in Ollama.

The VRAM Arithmetic

Evaluating memory requirements for a 16K context window on an 8GB GPU:

The model and its extended context window fit entirely within GPU VRAM. Zero layers are offloaded to CPU system memory, ensuring maximum generation speed.

5. Step 3: Trials, Failures, and the Minimal Preset

Connecting our newly created ornith-1.5:ctx model to DeepSeek Harness is straightforward, but our initial run revealed an instructive failure mode.

To understand what went wrong, we first need to clarify how DeepSeek Harness structures its runtime across three distinct layers: Plugins, Profiles & Settings, and Agent Presets.

Untangling the Hierarchy: Plugins, Profiles, and Presets

Developers often use these terms interchangeably, but in DSH they represent a clear three-tier architecture:

Plugins (The Capabilities): Atomic modules in the Cordis ecosystem (such as @deepseek-ai/dsh-tool-filesystem or llm-pi-ai, which adapts the provider engine from the open-source pi project). Each plugin provides a specific service, model adapter, tool schema, or execution sandbox.
Profiles & Settings (The Application Runtime):
- Profiles: Application runners (such as dsh web, headless, or sdk) located under $DSH_HOME/profiles/<name>. They define which ordered bundle of plugins Cordis boots into memory for a specific user interface or execution target.
- User Settings (settings.yaml): The global configuration file located at $DSH_HOME/settings.yaml (or project root). It hot-reloads at runtime without restarting the server, configuring provider credentials, API endpoints, model mappings, and default options.
Agent Presets (The Operational Modes): The per-session runtime configurations stored in agent-presets/<id>/agent.cordis.yml and toggled directly in the UI as "Standard mode" or "Minimal mode".

Because of this separation, optimizing DSH for local hardware does not require uninstalling packages or restarting the daemon. While profiles and plugins establish what the server application can do, agent presets dynamically select which tool schemas, system prompt instructions, and context compaction rules are active for the LLM during an execution turn.

The Built-in Presets: Four Flavors of DSH

DeepSeek Harness ships with four distinct built-in agent presets out of the box, each tailored to different operational requirements:

Standard Mode (The Full Coding Suite): The default preset for comprehensive repository work. It equips the agent with the complete toolset: file editing, persistent bash shell access, repository and web search, planning, skills (via filesystem discovery and the catalog loader), goals, workflows, and subagent orchestration. External MCP servers are not mounted here by default.
PTC Mode (Programmatic Tool Calling): Standard mode augmented with the Code Mode SDK. It retains all Standard capabilities (including skills), while letting the model write and execute an entire TypeScript program to combine multi-step operations in a single pass.
Minimal Mode (The Lean Two-Tool Composition): A stripped-down preset providing only two fundamental tools: persistent bash and str_replace_editor. It explicitly excludes skills and cannot mount MCP servers, drastically reducing prompt bloat and simplifying schema constraints.
Creator Mode (The Meta-Preset): A specialized preset for inspecting, experimenting with, and authoring custom agent presets. It includes the Standard toolset (including skills) alongside runtime introspection utilities to help developers draft new custom presets (such as connecting external MCP servers via @deepseek-ai/dsh-mcp-client).

Note: Preset Session Semantics & Custom MCP Presets

Preset selections in DSH apply on a per-session basis. When you change a preset in the Web UI, it takes effect on the next session you start; active sessions keep the preset configuration they were initialized with. While external MCP servers are not enabled by default in any of the four built-in presets, you can duplicate any preset into a custom configuration (agent-presets/<id>/) or use Creator mode to mount @deepseek-ai/dsh-mcp-client connections.

The 8GB Dilemma: Why Standard Mode Threatens Local Context

In Section 3, our baseline test with Mistral Small booted in DSH's default "Standard mode".

For frontier cloud models with large context windows (128K+), Standard mode is ideal. It presents the model with a comprehensive developer environment:

Multi-file patch tools
Interactive terminal runners
Web search clients
Diagnostic loggers and telemetry hooks
Subagent delegation and workflow engines

However, when targeting a local 9B model (ornith-1.5:ctx) constrained to an 8GB GPU, Standard mode introduces immediate friction:

Prompt Bloat: Tool definitions consume roughly ~6.9K tokens of context before the user even submits a prompt.
Cognitive & Schema Load: A 9B model must continuously hold dozens of complex JSON schemas in memory, increasing the likelihood of hallucinated arguments.
Context Starvation: When tool schemas consume over 40% of our 16K context window upfront, multi-turn reasoning rapidly saturates the remaining headroom and triggers frequent compactions.

The Initial Hypothesis: Configuring "Minimal Mode"

To protect our limited context window, our initial design hypothesis is to test DSH's leanest built-in option: Minimal mode.

Instead of modifying code or uninstalling plugins, we can instruct DSH to activate Minimal mode:

Masks out telemetry, web scrapers, subagents, workflows, and complex patch tools.
Retains only core primitives: file modification via targeted string replacement (str_replace_editor) and persistent shell execution.
Slashes tool schema overhead from ~6.9K tokens down to only ~971 tokens.

On paper, the tradeoff is clear: we exchange peripheral tools to maximize reasoning headroom and generation throughput.

Standard Mode vs. Minimal Mode: The 8GB Tradeoff

Dimension	Standard Mode	Minimal Mode
Primary Target	Frontier cloud models (DeepSeek-V3, Claude)	Local quantized models (7B to 9B on 8GB GPU)
Active Tool Surface	Full suite (Read, Edit, Write, Shell, Search, Subagents)	Two-tool composition (`persistent bash`, `str_replace_editor`)
Tool Schema Footprint	~6.9K tokens (~43% of 16K window)	~971 tokens (<6% of 16K window)
Available Context (16K)	~7K to 8K tokens remaining	~15K tokens (>93% available)
Measured Throughput	~12 tokens/sec (TTFT ~3.9s)	~17 tokens/sec (TTFT ~1.5s)
Observed Task Behavior	Capable of deep reasoning, but heavy context churn	Lean schemas, but rigid editing on file creation

While presets can be toggled per session in the web interface, we declare Minimal mode as the default in settings.yaml to test our lean-context hypothesis.

6. Step 4: The Complete DeepSeek Harness Configuration (`settings.yaml`)

All of these requirements are codified into DeepSeek Harness's primary configuration file: settings.yaml.

Place this configuration file in your DSH configuration directory (typically ~/.dsh/settings.yaml or in your project root):

ui-onboarding:
  welcomeNoticeVersion: 2026-08-13.1

llm-pi-ai:
  providers:
    openrouter:
      apiKeyEnv: OPENROUTER_API_KEY
    ollama:
      displayName: Ollama
      api: openai-completions
      baseURL: http://127.0.0.1:11434/v1
      models:
        - id: ornith-1.5:ctx
          contextWindow: 16000
          maxTokens: 4096
      apiKeyEnv: OLLAMA_API_KEY

agent-default-model:
  provider: ollama
  model: ornith-1.5:ctx

agent-presets:
  default: minimal

Breakdown of the Configuration

ui-onboarding: Pins the onboarding notice version (2026-08-13.1), preventing introductory dialogs from popping up on subsequent web console launches.
llm-pi-ai: Configures the underlying LLM provider adapter. DSH wraps the provider abstraction and streaming engine directly from Mario Zechner's open-source pi project into a Cordis plugin. Rather than maintaining custom client drivers for each inference backend, DSH reuses an existing upstream module, illustrating the practical composability of the open-source ecosystem.
- openrouter: Preserved as a cloud fallback provider whenever you need to benchmark or handle tasks exceeding local model capabilities.
- ollama: Points DSH to Ollama's local OpenAI-compatible completions endpoint (http://127.0.0.1:11434/v1).
- models: Explicitly registers our custom ornith-1.5:ctx model with two critical runtime parameters:
- contextWindow: 16000: Informs DSH of our 16K context ceiling (mirroring the num_ctx 16384 compiled into our Ollama Modelfile). This enables DSH to compute and display context utilization accurately in the web UI (such as ~10.6K / 16K) and trigger automated compactions before context overflows.
- maxTokens: 4096: Sets the maximum generation ceiling for a single turn. While developers sometimes enter the base model's advertised architectural capacity here (such as 32000), maxTokens in DSH governs the single-response completion budget. Capping it to a realistic threshold like 4,096 tokens ensures individual outputs never exceed remaining context headroom or trigger runaway loops.
- apiKeyEnv: Ollama does not require an API key for local inference, but DSH checks for an environment variable name; pointing to OLLAMA_API_KEY (even if empty) satisfies the schema.
agent-default-model: Directs DSH to boot immediately using our local Ollama model as the primary reasoning engine.
agent-presets: Sets the default agent execution preset to minimal, activating the lightweight minimal preset on every turn.

7. Step 5: Running the Local Agent in Practice

With settings.yaml saved and Ollama running in the background, launch the DeepSeek Harness web console:

dsh web

The web console launches locally, ready to pair with our ornith-1.5:ctx model running on the 8GB RTX 4070.

To test the agent under realistic conditions, we give it a practical software engineering task on an existing codebase:

"Inspect the local calculator web application (C:\Local\Dev\dsh\calculator), analyze its structure, diagnose any failing operations, and write an AUDIT.md report documenting your findings."

Phase 1: Starting with Minimal Mode

Because our 8GB mobile GPU restricts our local model to a 16K context window, our initial instinct is to minimize prompt bloat at all costs. We start the session using DSH's Minimal mode preset.

In Minimal mode, initial resource consumption is low:

Tool definitions consume only ~971 tokens (less than 6% of our 16K context window).
The system prompt takes a negligible ~16 tokens.
Generation throughput is fast at ~17 tokens per second, with an average Time to First Token (TTFT) of 1.5 seconds.
Over 90% of the context window remains available for model reasoning.

However, file operations fail when the agent attempts to record its findings on disk:

The limitation lies in the tools provided by the minimal preset. Minimal mode relies on str_replace_editor and persistent bash. While str_replace_editor works well for targeted replacements in existing files, its schema constraints are rigid when asked to create or format new files. When Ornith-1.5 attempts to write the audit file to disk, it produces recurring schema validation errors:

Error: invalid arguments: "new_str" must match exactly one oneOf branch (matched 0); "old_str" must match exactly one oneOf branch (matched 0)

Switching to the Trajectory tab in the DSH web interface exposes how the agent attempts to cope with this failure:

Repetitive Tool Failures: Ornith-1.5 retries the file write through str_replace_editor, but repeatedly hits schema rejection (INVALID_ARGS).
Context Guardrails Intervene: DSH's active repeat-tool-reminder plugin detects the loop and injects an inline context message warning the model that it is repeating identical failed calls.
Shell Fallbacks: While Ornith-1.5 recognizes the error, the minimal preset offers no alternative file creation tools. The model falls back to executing raw PowerShell (pwsh) scripts to write files character by character and reading back hex byte sequences (23 # 20 43 C) to verify output integrity.

In practice, while Minimal mode keeps tool schemas under 1K tokens, its editing surface is too rigid for repository maintenance. The preset lacks the necessary file manipulation tools for an end-to-end audit.

Phase 2: Switching to Standard Mode

To provide the tools necessary for proper software engineering, we switch our session preset from Minimal mode to Standard mode directly in the DSH interface.

Standard mode provides a full development toolset with dedicated read, write, edit, and interactive user dialog tools (ask_user). File operations proceed without schema errors:

However, tool schema overhead increases significantly:

The expanded tool schemas consume ~6.9K tokens out of our 16K limit (over 43% of total context).
Together with the ~1.8K system prompt, more than half of the context window is consumed before reading project files.
Generation speed drops from 17 tokens/s down to ~12 tokens/s, and average TTFT increases to ~3.9 seconds.

Code Reasoning and Bug Localization

With dedicated file inspection tools available, the model traces the bug accurately.

The agent uses read to examine app.js and index.html. It identifies that in the calculator app, typing "8+8" unexpectedly displays "88" instead of performing addition:

Ornith-1.5 constructs a detailed step-by-step state trace. Using DSH's interactive prompt dialog (ask_user), the agent presents its findings clearly to the user, proposing Option A and asking for confirmation before making edits. Once approved, Ornith-1.5 applies the fix cleanly using the edit tool.

The Context Ceiling: Auto-Compactions in Action

While the code fix succeeded, inspecting the execution trajectory highlights the consequence of carrying ~6.9K tokens of tool schemas:

After just two turns and eleven steps:

Total context utilization quickly surpasses 66% (~10.6K / 16K) and approaches saturation.
DeepSeek Harness begins triggering frequent automatic context compactions (COMPACTED: summary is not smaller than shadowed content...) to prevent context overflow.
Re-evaluating 10K+ tokens on every turn adds noticeable latency to local inference on the RTX 4070.

Empirical Validation of the VRAM Arithmetic

Throughout these multi-turn test runs across both Minimal and Standard modes, monitoring GPU telemetry via nvidia-smi provided a direct empirical confirmation of our theoretical model from Section 4:

The observed VRAM allocation stabilized at approximately 7.6 GB, closely matching our 7.8 GB calculation within 200MB of runtime variance.

More importantly, generation speed held steady at 17 tokens/s in Minimal mode and 12 tokens/s in Standard mode. If total memory consumption had crossed the 8.0 GB physical boundary, Windows unified memory manager would have immediately paged memory buffers across the PCIe bus into shared system RAM. That memory bus penalty causes generation throughput to plummet to 1-3 tokens/s. The sustained double-digit token generation rate confirms that our 9B model weights and the entire 16K KV cache remained 100% GPU-resident at all times.

The Engineering Takeaway: Fine-Tuning the Preset Sweet Spot

This hands-on experiment demonstrates the core dilemma of local AI agents on consumer hardware:

Dimension	Minimal Mode	Standard Mode
Tool Context Footprint	~971 tokens (<6% of 16K)	~6.9K tokens (~43% of 16K)
Inference Performance	~17 tok/s (TTFT 1.5s)	~12 tok/s (TTFT ~3.9s)
Code Reasoning & Tooling	Basic tools; fragile on file creation	Rich tools (`read`, `write`, `edit`, `ask_user`); deep reasoning
Bottleneck / Failure Mode	Schema errors (`str_replace_editor`), clunky workarounds	Rapid context saturation, frequent auto-compactions

Neither extreme is optimal out of the box for an 8GB GPU. Minimal mode leaves ample context but lacks reliable tools for multi-file development. Standard mode provides capable tooling and deep reasoning, but its ~6.9K schema footprint consumes valuable context that is critical for multi-turn reasoning and complex problem solving.

DeepSeek Harness avoids this rigid binary through its preset and profile system.

Instead of staying locked into either built-in preset, developers can fine-tune a custom preset specifically for local hardware:

Strip out unnecessary tools that are irrelevant to local debugging tasks (such as web search, browser automation, subagent delegation, and telemetry).
Retain the core filesystem tools that proved effective (read, write, edit, and interactive prompts), keeping tool schema overhead around ~2K tokens.
Reserve 12K+ tokens for conversation history, file contents, and extended reasoning loops.

Most importantly, because DSH manages capabilities via Cordis fibers, this preset tuning can be adjusted on the fly without even restarting the harness. This dynamic modularity makes it possible to find the exact sweet spot for any hardware budget.

Alternative Interface: Creating a Dedicated TUI Profile with dsh-TUI

Because DeepSeek Harness is a true meta-harness, its frontend interfaces are completely decoupled from its execution microkernel. You are never locked into a web browser.

To see how DSH's profile system works in practice (as we defined in Section 5), let's create a dedicated profile that replaces the web dashboard with an interactive Terminal User Interface (TUI).

Among the growing number of community TUI plugins, one particularly popular choice in China is dsh-TUI (@deepseek-harness-tui/dsh-tui). It brings a full-screen, Claude Code-style terminal experience to DeepSeek Harness, complete with streamed markdown, context usage gauges, and TPS metrics.

To configure and launch it, we declare a new profile named dsh-tui and add the plugin:

dsh plugin --profile dsh-tui add @deepseek-harness-tui/dsh-tui

dsh --profile dsh-tui

Because DSH treats the session as an append-only event stream, conversational state is completely independent of the display layer. If your terminal session disconnects or you want to pick up a previous task later, you can resume it directly by its session ID:

Resume with the command below:
dsh-tui --resume 6661d456-8dd1-49ad-bdf0-27bca9609574

This example illustrates the practical utility of DSH profiles: switching from the web interface to a terminal UI requires no dependency changes or environment reconfiguration, only launching under a different profile flag.

8. Best Practices for 8GB Local Agents

Working with constrained hardware requires disciplined operational habits:

1. Monitor VRAM Residency and Address Latency

Keep an eye on GPU memory during multi-step runs:

nvidia-smi -l 2

If memory consumption exceeds 8GB, Windows will automatically page memory into shared system RAM. If generation speed suddenly drops from 15 tokens/sec to 2-3 tokens/sec, your KV cache has spilled onto the CPU bus.

Even when fully GPU-resident, our setup averaged a TTFT of 1.5 to 3.9 seconds and 12 to 17 tokens/sec across our test runs. If you need snappier generation for interactive workflows, consider two optimizations:

Fine-tuning Ollama parameters: Enabling Flash Attention (OLLAMA_FLASH_ATTENTION=1) or adjusting num_batch in your Modelfile can reduce memory bandwidth bottlenecks during prompt evaluation.
Selecting a smaller model: Stepping down from a 9B model to a dedicated 7B model (such as Qwen2.5-Coder 7B) or an edge-focused architecture (such as Gemma 4 e2b) will significantly boost tokens per second while cutting prompt evaluation latency in half.

2. Fine-Tune the Tool Surface: Avoid Extremes

Avoid attaching extraneous MCP servers or plugins to local sessions. Every additional tool schema consumes context tokens and increases the probability of hallucinated arguments. At the same time, our Section 7 experiments proved that an off-the-shelf minimal preset with just bash and string replacement is too fragile for file creation, while the default Standard preset consumes ~6.9K tokens. The practical strategy is to fine-tune your own preset: preserve reliable file tools (read, write, edit) while pruning unused schemas (web search, subagents, telemetry) to keep tool overhead around ~2K tokens.

3. Use Hybrid Fallback Strategically

Because DSH defines multiple providers in settings.yaml, you are never locked into local inference. If you encounter a complex architectural refactor that exceeds the reasoning capabilities of a 9B model, you can switch the active model in the DSH web interface to OpenRouter (such as DeepSeek-V3 or Claude) for that specific prompt, and then switch back to Ollama for the implementation work.

4. Implement Automated Routing via LiteLLM

Switching models manually in the UI works for occasional tasks, but autonomous agents run best when routing happens automatically.

In a companion article, Building a Local-First AI Coding Agent with Open Tools and Adaptive Routing, we explored how to insert LiteLLM as an intelligent control plane (Layer 2) between the harness and the inference servers.

You can apply the exact same architecture to DeepSeek Harness:

Configure DSH's provider in settings.yaml to point to a local LiteLLM proxy (http://127.0.0.1:4000/v1).
Let LiteLLM inspect incoming requests: routine tool executions and file searches route to local Ollama at zero cost, while intricate architectural planning prompts automatically escalate to OpenRouter under strict per-session spend limits.
This combines zero-cost local execution for routine loops with automated escalation to cloud models for complex reasoning.

9. Series Retrospective: From Meta-Framework to Local Agent

Over the course of this three-part series, we have traced the full lifecycle of modern agent infrastructure:

Part 1: Understanding Cordis explored the foundational microkernel created by Shigma. We saw how spatiotemporal composability, reverse cleanup stacks, and reactive contexts solve the chronic memory leaks of long-running JavaScript applications.
Part 2: DeepSeek Harness Architecture analyzed how DeepSeek turned Cordis into an operating system for AI agents. We untangled the harness terminology, contrasted DSH with turnkey tools like OpenCode and Pi, and examined its "everything-is-a-plugin" philosophy.
Part 3 (This Article) brought everything to life on consumer hardware. By pairing DSH's minimal preset with a context-tuned Ollama model, we proved that you do not need enterprise data centers or expensive cloud API budgets to run a functional, autonomous coding assistant.

DeepSeek Harness shows how modular systems engineering benefits local development. By separating runtime lifecycles, provider adapters, and tool presets, DSH allows developers to match agent overhead to physical hardware limits, making an 8GB GPU a viable environment for local pairing.

References & Links

DeepSeek Harness GitHub Repository: github.com/deepseek-ai/deepseek-harness
DeepSeek Harness Presets Guide: deepseekdsh.com/guides/modes
dsh-TUI Terminal Interface: dshtui.com/en
Community Plugin Directories: deepseekplugin.com | dsh-plugins.org
Ollama Model Library: ollama.com/library
Pi Coding Agent & Engine: pi.dev
Part 1 of this Series: Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything
Part 2 of this Series: DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents
Companion Article (Control Plane & Routing): Building a Local-First AI Coding Agent with Open Tools and Adaptive Routing

DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents

raphiki — Sat, 12 Sep 2026 05:47:32 +0000

Part 2 of the DeepSeek Harness: Kernel to Edge series: Exploring DeepSeek Harness (DSH), its 'everything-is-a-plugin' architecture, how it leverages Cordis under the hood, and how it compares to OpenCode, Pi, LangGraph, and CrewAI.

DeepSeek Harness: Kernel to Edge

This article is Part 2 of a three-part architectural and practical deep dive:

Part 1: Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything (Microkernel primitives, spatiotemporal composability, reverse cleanup stacks, zero-leak lifecycles).

Part 2 (This Article): DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents (The meta-harness paradigm, Cordis as an agent kernel, comparing DSH to OpenCode and Pi).

Part 3: Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits (Hands-on local deployment with Ollama, Ornith-1.5, VRAM arithmetic, minimal vs. standard presets, and TUI).

1. Introduction: Beyond the Prompt Chain

In Part 1, we explored Cordis, the lightweight TypeScript framework created by Shigma (Yifan Shi) that guarantees zero-downtime hot reloading and clean plugin lifecycles through revertible effects.

When developers first encountered Cordis, most viewed it primarily as the foundation for the Koishi chatbot ecosystem. In 2026, DeepSeek AI adopted it as the foundational engine for DeepSeek Harness (dsh).

DeepSeek chose not to build their agent framework in Python, avoiding both generic prompt-chaining libraries and opinionated role-playing engines.

Instead, DeepSeek hired Shigma, used Cordis as their microkernel, and published research grounding the approach ("A Programming Paradigm for Spatiotemporal Composability", arXiv:2608.25512).

The result is DeepSeek Harness (DSH): an open-source agent framework built on a distinct design rule:

"There is no privileged core. Every capability (the model adapter, the sandbox, the tools, the session log, and even the agent loop itself) is a replaceable plugin."

This article examines how DeepSeek Harness uses Cordis under the hood, how its architecture operates, and how it compares to other agent frameworks.

2. Why Call It a "Harness"? (Untangling the Terminology)

In the software industry, the word "harness" has become one of the most overloaded buzzwords in AI. Depending on who you talk to, a "harness" can mean a benchmark runner, a CLI chatbot, or a complex multi-agent framework.

To understand DeepSeek Harness, we need to untangle the three distinct categories of harnesses operating in the AI ecosystem today:

Category 1: Evaluation Harnesses (Benchmark Rigs):
- Examples: SWE-bench test runners, HumanEval fixtures.
- Purpose: A rigid testing scaffold designed to feed an issue to a model, run a test suite in a Docker container, and report a pass/fail grade. They are test rigs, not production application runtimes.
Category 2: Application / Coding Harnesses (Interactive Assistants):
- Examples: OpenCode, Pi Coding Agent, Claude Code.
- Purpose: Specialized developer tools designed for interactive, human-in-the-loop pair programming in a terminal or IDE. They come pre-configured with a fixed set of developer tools (read, edit, bash) running directly on your machine.
Category 3: Meta-Harnesses / Agent Operating Systems:
- Example: DeepSeek Harness (DSH).
- Purpose: An un-opinionated, headless microkernel. DSH does not presuppose what the agent is doing. It provides low-level operating system scaffolding (process isolation, append-only session streams, and reversible plugin lifecycles) upon which you can assemble an evaluation harness, an interactive coding assistant, or an autonomous research pipeline simply by changing a configuration profile.

Models like DeepSeek-R1 and DeepSeek-V3 require substantial host-level support to perform multi-step work:

Execution sandboxes: Docker containers, child processes, and isolated filesystems.
External interfaces: Web search, documentation scrapers, and external APIs.
Session persistence: Deterministic logging that records execution history without state drift.
Safety policies: Interceptors that validate commands before execution.

Most agent frameworks treat these capabilities as peripheral utilities. DeepSeek Harness treats the harness as an operating system kernel, providing the structural scaffolding required to run models autonomously without orphaned processes, memory leaks, or execution escapes.

3. Why DeepSeek Adopted Cordis

Why did DeepSeek choose Cordis over existing Python frameworks?

Limitations of Python Agent Frameworks

Most popular agent tools (LangChain, AutoGen, CrewAI) are written in Python. While Python is the standard for ML training and tensor operations, it introduces architectural friction when building long-lived agent runtimes:

Monotonic Memory: Python's garbage collector and global module cache make dynamic unloading difficult. If an agent spins up a background process or attaches an event listener, cleaning it up cleanly requires manual bookkeeping.
Heavy Startup Overhead: Python agent libraries often take seconds just to import their dependency trees.
Rigid Graphs: Tools like LangGraph model workflows as static directed graphs. If an agent discovers mid-flight that it needs a new tool, adding that tool dynamically usually requires recompiling the graph or restarting the agent.

How Cordis Addresses These Issues

Building on Cordis provides three concrete architectural advantages:

Fast Startup and Low Footprint: Running on Node.js and TypeScript, DSH boots in milliseconds with a lean baseline memory footprint (<50MB).
Guaranteed Teardown: When an agent launches a Docker container or a file watcher, Cordis wraps it in a Fiber. When the sub-task ends, Cordis guarantees teardown of all open handles and child processes.
Reactive Dependency Trees: If a task requires a Python environment, the Python tool only appears in the model's schema once a healthy sandbox service is active.

4. How DeepSeek Harness Works Under the Hood

Let's look at the six core architectural concepts that power DeepSeek Harness.

1. "There is No Core": The Pure Plugin Architecture

In DSH, you won't find a massive, hardcoded Agent class. Instead, every major subsystem is a Cordis plugin that extends Service:

ModelService (ctx.model): Handles token streaming, schema formatting, and provider connections (DeepSeek API, OpenAI, Anthropic, or local Ollama).
SandboxService (ctx.sandbox): Provides safe execution boundaries (isolated child processes, Docker containers, or WASM sandboxes).
SessionService (ctx.session): Manages the conversation history and transcript logs.
ToolRegistry (ctx.tools): Tracks available tools, validates schemas, and executes actions.
AgentLoop (ctx.agent): The turn-based execution cycle that prompts the model, receives tool calls, and handles responses.

Because these are all Cordis plugins, you can swap any of them independently. For instance, you can replace the default agent loop with a custom planner while keeping the existing tools and sandboxes intact. Even the user interface itself is just a plugin: DSH has no privileged GUI or CLI built into its runtime core.

2. Bundles and Profiles (`cordis.yml`)

DeepSeek Harness applications are composed declaratively using Bundles and Profiles.

Bundle: An npm package containing code, plugins, and schema configurations.
Profile: A YAML configuration file defining which bundles to load together.

Here is what a standard cordis.yml profile looks like:

# cordis.yml - DeepSeek Harness Composition
name: autonomous-coder

plugins:
  # 1. Model Provider Plugin
  '@deepseek-ai/dsh-model-deepseek':
    apiKey: env:DEEPSEEK_API_KEY
    model: deepseek-coder-v2

  # 2. Execution Sandbox Plugin
  '@deepseek-ai/dsh-sandbox-docker':
    image: 'node:22-alpine'
    timeoutMs: 30000

  # 3. Development Tools
  '@deepseek-ai/dsh-tool-filesystem':
    rootDirectory: './workspace'

  '@deepseek-ai/dsh-tool-bash':
    allowedCommands: ['git', 'npm', 'pnpm', 'test']

  # 4. Web UI Dashboard
  '@deepseek-ai/dsh-web-console':
    port: 3000

To run this agent stack with its web console:

npx @deepseek-ai/dsh web

Entry Profiles and Presets: Web, Headless, and Minimal

Because DSH is assembled from YAML profiles and runtime presets, switching the operational persona of your agent requires no code changes:

dsh web (or dsh --profile web): Boots the development stack with a local browser-based dashboard.
dsh --profile headless: Runs the agent in headless mode for CI/CD pipelines, automated testing, or background workers.
agent-presets: default: minimal: Activates a stripped-down two-tool composition, reducing system prompt token overhead and conserving system memory and GPU VRAM.

3. The Anatomy of a Custom DSH Plugin (15 Lines of TypeScript)

Writing your own tool or capability for DeepSeek Harness requires minimal code. Because it is a native Cordis plugin, you declare what dependencies you need (inject) and register your logic:

import { Context } from 'cordis';

export const name = 'project-stats';
export const inject = ['tools']; // Requires the ToolRegistry service

export function apply(ctx: Context) {
  // Register the tool; Cordis handles lifecycle and cleanup
  ctx.tools.register({
    name: 'get_project_stats',
    description: 'Returns file count and lines of code in current directory',
    execute: async () => {
      return 'Total files: 42 | Lines of code: 5,120';
    },
  });
}

Once written, you can declare it in your local cordis.yml profile or load it dynamically at runtime.

4. Session as an Append-Only Stream

Traditional frameworks treat chat history as a mutable array of { role, content } objects that developers manipulate directly.

DeepSeek Harness treats the Session as an immutable, append-only event stream.

Every user input, thinking block, model token, tool execution, and error is an immutable record in the stream.
Projections: The model's context window, the CLI output, and the Web UI are simply read-only "projections" rendered from this single source of truth.
Time Travel & Forking: Because the log is append-only, an agent can instantly fork a sub-agent to explore an alternative debugging hypothesis, test it, and discard it without corrupting the main conversation history.

5. Reversible Sandboxes and Dynamic Toolsets

In an extended autonomous task, an agent might need specialized capabilities only for a few minutes.

For example, when asked to analyze a dataset, the agent might need a Python data science environment (pandas, numpy, matplotlib). In traditional frameworks, that environment stays open forever.

In DeepSeek Harness, tools are mounted as Cordis Fibers:

import { Context } from 'cordis';

// A specialized tool plugin mounted dynamically
export function DataAnalysisToolPlugin(ctx: Context) {
  // Use ctx.effect to manage the sandbox lifecycle
  ctx.effect(() => {
    console.log('[Tool] Booting ephemeral Docker container for Python...');
    const container = ctx.sandbox.createContainer({ image: 'python:3.11-slim' });

    // Register the tool
    const unregister = ctx.tools.register({
      name: 'run_python_script',
      description: 'Executes Python data analysis code',
      execute: async ({ code }) => container.exec(code),
    });

    // Cleanup: When the agent unloads this plugin, the container is destroyed!
    return async () => {
      console.log('[Tool] Tearing down ephemeral container...');
      unregister();
      await container.destroy();
    };
  });
}

When the data analysis step is complete, the agent or harness calls fiber.dispose(). Cordis instantly kills the container, frees the RAM, and removes the tool from the model's schema.

6. Waterfall Control Flow & Guardrails

How does DeepSeek Harness prevent an autonomous agent from running destructive commands (like rm -rf / or leaking API keys)?

It uses Cordis's waterfall event pattern:

Any plugin can hook into agent/pre-step or agent/request. If a security plugin detects an unsafe command, it can rewrite it, request user confirmation, or bail out early before the sandbox is ever touched.

5. The True Meta-Harness: An Exploding Plugin Ecosystem

Why does DeepSeek Harness deserve the title of a true meta-harness?

Turnkey coding agents (such as OpenCode or Claude Code) ship as monolithic binaries with predetermined tools, fixed workflows, and rigid interfaces. If you want a different terminal layout, a custom vision pipeline, or specialized memory management, you have to submit a feature request or fork the codebase.

In DSH, the harness is not a closed application; it is an open substrate. Because the microkernel has "no core" and treats every capability as an unprivileged Cordis plugin, developers can reshape every layer of the agent experience.

Even the User Interface is a Plugin (Web UI vs. TUIs)

A textbook example of this architectural neutrality is the user interface itself:

The Web UI is Just a Plugin: When you run dsh web, DSH boots an HTTP server and mounts a frontend plugin bundle (@deepseek-ai/dsh-web-console or dsh-web-ui). The harness core neither knows nor cares that a web browser is rendering the output.
Terminal UIs (TUIs) on Demand: If you prefer staying strictly inside your terminal to avoid context switching, you can completely bypass the web dashboard. Community plugins like dsh-TUI transform DSH into a full-screen, keyboard-driven terminal coding agent reminiscent of Claude Code, complete with streaming token output and interactive approval prompts.
Workbench Enhancements: For developers using the browser console, plugins like DSH-better-sidebar enhance the default layout by docking file trees, Git status, terminal sessions, and live browser previews directly alongside the chat stream.

Thousands of Plugins and Rising Community Marketplaces

This native extensibility has catalyzed a rapidly expanding community ecosystem. In only a few months, thousands of open-source plugins have been authored by developers worldwide, giving rise to dedicated community directories and marketplaces:

deepseekplugin.com: A centralized directory cataloging community-built DSH plugins across diverse functional categories.
dsh-plugin.org / dsh-plugins.org: Open documentation and discovery hubs for exploring plugin architectures, configuration snippets, and trending add-ons.

Real-World Plugin Innovations: Beyond Simple Coding Tools

As analyzed in Composio's roundup of top DeepSeek Harness plugins, community developers are extending DSH far beyond basic file reading and bash execution:

Multimodal Vision Toolkits (ModLens, dsh-vision-toolkit): Give text-only models visual perception. These plugins provide structured OCR, visual grounding, layout analysis, UI reconstruction from screenshots, and pixel-by-pixel comparisons.
Generative UI (dsh-genui): Allows the agent to render more than 30 interactive user interface widgets (cards, responsive data tables, forms, charts) directly inside conversational responses.
Ergonomic Workspace Mentions (dsh-at-file): Brings Codex-style @file and @folder mentions into the composer, letting developers inject file contents and directory context without manual copy-pasting.
Multi-Tiered Memory (dsh-mnemon): Solves agent amnesia across turns by providing three distinct memory layers: Runtime Memory for turn-by-turn preferences, Document Memory for repository conventions, and long-term knowledge retention.
In-Harness Discovery & Self-Installation (dsh-market, dsh-find-plugin): Brings the marketplace directly into DSH. dsh-market embeds a visual plugin market into the Settings UI, while dsh-find-plugin equips the agent itself with a tool to search GitHub for plugins on the fly and self-install capabilities during an active run.

6. Positioning DSH: Comparing Harnesses Across the AI Landscape

To truly understand where DeepSeek Harness fits, we have to look at the broader AI landscape. Developers often conflate two very different layers of software:

High-Level Agent Frameworks (LangGraph, CrewAI, AutoGen): Libraries designed for prompt chains, multi-agent debates, and graph state machines.
Dedicated Coding Agent Harnesses (OpenCode, Pi Coding Agent, Claude Code): Specialized developer tools designed to run in your terminal or IDE to edit code.

DeepSeek Harness bridges both worlds by operating as a Meta-Harness (an Agent Operating System). Let's break down how it compares to both categories.

Layer A: DSH vs. General Agent Frameworks (LangGraph, CrewAI)

Most general agent frameworks are written in Python and focus on orchestrating conversations and workflow graphs:

LangGraph (Circuits & Graphs): Models agents as fixed directed cyclic graphs (nodes, edges, and state channels). While powerful for predictable workflows, dynamic changes are difficult: adding a new tool or modifying execution mid-flight requires recompiling the graph or restarting the agent.
CrewAI / AutoGen (Personas & Roleplay): Models agents as personas with roles, backstories, and conversational message passing. Great for creative simulations, but they lack low-level systems control over processes, memory reclamation, and execution sandboxes.
DeepSeek Harness (An Operating System Microkernel): Instead of modeling agents as circuits or personas, DSH models agents as an operating system. The Cordis microkernel provides the core primitives (event bus, service registry, fibers). Models, tools, sandboxes, and the agent loop itself are drivers that plug into that OS.

Layer B: DSH vs. Dedicated Coding Harnesses (OpenCode, Pi)

If you are a software engineer, you are likely more familiar with specialized coding agent harnesses like OpenCode or Pi Coding Agent:

OpenCode (The Model-Agnostic Developer Companion):
- Developed by the SST team (anomalyco/opencode) in TypeScript/Bun, OpenCode provides a polished developer experience with terminal, desktop, and web interfaces.
- It ships with a pre-tuned suite of developer tools (read, write, edit, glob, grep, bash).
- Where it excels: Interactive, human-in-the-loop coding sessions on your local machine.
- The Architectural Difference: OpenCode is a turnkey coding tool. Its tool registry, session loop, and interaction models are tailored to developer workflows. Tools run directly on your host environment, and the toolset is static per session.
Pi Coding Agent (The Minimalist Terminal Agent):
- Created by Mario Zechner, Pi (@earendil-works/pi-coding-agent) is a lightweight terminal coding agent designed around minimalism and extensibility.
- It separates concerns into clean packages (unified LLM API, agent runtime, terminal UI).
- The Architectural Difference: Pi is designed to do one thing well: serve as an extensible terminal coding companion for an individual engineer.
DeepSeek Harness (The Reversible Meta-Harness):
- DSH is not hardcoded as a coding assistant. Coding tools are merely plugins.
- By stacking different YAML profiles (cordis.yml), DSH can function as a terminal coding agent (similar to OpenCode or Pi), an automated benchmark runner, a web research crawler, or a security audit pipeline.
- Dynamic Ephemeral Isolation: Rather than running all commands directly on your host machine, DSH is built for unattended autonomous execution. It can spin up an ephemeral container for a sub-task, inject tools into the agent context, execute the work, and tear down the environment without leaving orphaned processes or open ports.

Under the Hood: How Their Plugin Systems Actually Differ

To see why DSH behaves differently in runtime management, compare how you extend each harness:

Pi Coding Agent (Boot-Time Extension Hooks):
- Mechanism: You export a function or object that registers custom tools or commands into Pi's registry at startup.
- Lifecycle: Static and persistent. Extensions live for the entire duration of the CLI session.
- Best for: Personal CLI shortcuts and custom prompt extensions.
OpenCode (Config-Driven Tools & External MCP Servers):
- Mechanism: Tools are declared in configuration or connected via the Model Context Protocol (MCP).
- Lifecycle: Process-isolated client-server architecture. Tools typically run as external background programs communicating over stdio or HTTP.
- Best for: Reusing standardized enterprise tools across multiple IDEs and agents without rewriting tool code.
DeepSeek Harness (In-Process, Reactive Cordis Fibers):
- Mechanism: Plugins mount directly into the in-process Context Tree via Cordis. They provide core services (ctx.provide), declare dependencies (inject), and register self-cleaning side effects (ctx.effect).
- Lifecycle: Dynamic, reactive, and reversible. A plugin can be hot-swapped, suspended, or unloaded during execution. Cordis guarantees that any socket, timer, or container created by that plugin is destroyed upon disposal.
- Best for: Autonomous agents that dynamically adapt their environment, spin up temporary sandboxes, or synthesize new tools on the fly.

The Comprehensive Architectural Matrix

Dimension	General Frameworks (LangGraph, CrewAI)	Turnkey Coding Harnesses (OpenCode, Pi)	Meta-Harness (DeepSeek Harness)
Primary Goal	Multi-agent chains & graph workflows	Interactive developer coding assistant	Headless, autonomous agent operating system
Language & Runtime	Python (Heavy import overhead)	TypeScript / Bun (Fast, developer-friendly)	TypeScript / Node.js (Lightweight <50ms)
Architecture	Static Graphs or Multi-Agent Chats	Monolithic / Modular Coding Loop	Pure Cordis Microkernel (Zero privileged core)
Plugin Mechanism	Hardcoded in Python state schema	Boot-time extension hooks / external MCP	In-process reactive Cordis Fibers (with auto-cleanup)
Execution Sandboxing	Manual script execution / External Docker	Direct execution on developer's host machine	Built-in Reversible Fibers (Docker, WASM, Process)
Session Model	Chat history lists / state dictionaries	Interactive session transcripts	Immutable append-only stream with projections
Dynamic Self-Evolution	❌ No (requires graph rebuild)	❌ No (fixed tool schema)	Yes ("Creation Mode" via runtime plugins)

In Short:

Use LangGraph or CrewAI when you want to design a multi-persona conversational workflow or a fixed graph pipeline in Python.
Use OpenCode or Pi when you want a fast, interactive AI pair programmer in your terminal to help you build software.
Use DeepSeek Harness when you need an unattended agent runtime that dynamically manages tools, isolates sandboxes, runs extended tasks without resource leaks, and supports runtime plugin extension.

7. Dynamic Tool Generation at Runtime ("Creation Mode")

One notable capability enabled by Cordis is dynamic plugin generation at runtime, often called "Creation Mode".

Because Cordis allows modules to be loaded and unloaded at runtime, an agent can dynamically extend its own abilities:

Problem: An agent is tasked with parsing a proprietary binary format. It currently has no tool for this.
Code Generation: The agent writes a custom TypeScript parser plugin and saves it to its local directory.
Dynamic Mounting: The agent uses Cordis to hot-load its newly written plugin into its own context:

   await ctx.plugin(NewlyWrittenPlugin);

Execution & Verification: The agent now sees the new tool in its schema, executes it, and inspects the result.
Resolution: If the tool is flawed, it fixes the code and hot-reloads it. Once the task is solved, it can either commit the plugin to its permanent bundle or dispose of it cleanly.

Instead of relying solely on a fixed set of pre-bundled tools, the agent can synthesize, test, and safely tear down its own execution utilities as new requirements arise during a task.

8. Conclusion

By building on Cordis, DeepSeek Harness avoids the resource leaks and rigid orchestration common in traditional agent runtimes. Grounded in both practical experience from the Koishi ecosystem and research into spatiotemporal composability, DSH demonstrates a disciplined foundation for long-running autonomous tasks:

Modular components: There is no privileged core; tools, sandboxes, and agent loops can be swapped independently.
Reversible side effects: Sandboxes and resource handles are tracked in fibers and cleaned up automatically upon disposal.
Low operational footprint: Fast startup time with minimal idle memory consumption.

Coming Next in Part 3: Running DeepSeek Harness on an 8GB GPU

With Cordis covered in Part 1 and DeepSeek Harness's architecture mapped here in Part 2, the next installment moves to practical implementation.

In Part 3: Running DeepSeek Harness on an 8GB GPU, we configure an autonomous coding agent running locally on a laptop equipped with an 8GB RTX 4070 GPU:

Minimal Preset: Configuring DSH with the minimal agent preset to streamline tool schemas, compress prompt overhead, and maximize available context.
Local Inference: Connecting DSH to Ollama running quantized coding models such as Qwen2.5-Coder or Ornith-1.5.
VRAM and Context Tuning: Setting context limits and parameter budgets so the agent can inspect and edit multi-file projects without running out of GPU memory or truncating context.

References & Links

DeepSeek Harness GitHub Repository: github.com/deepseek-ai/deepseek-harness
DeepSeek Harness Documentation & Cordis Primer: deepseek-harness.github.io/deepseek-harness
Community Plugin Directories: deepseekplugin.com | dsh-plugins.org | dsh-plugin.org
Composio Best DSH Plugins Guide: composio.dev/content/best-deepseek-harness-plugins
Theoretical Research Paper: Shi, Y., Zhang, W., & Cui, T. A Programming Paradigm for Spatiotemporal Composability (arXiv:2608.25512)
OpenCode Repository: github.com/anomalyco/opencode
Pi Coding Agent: pi.dev
Part 1 of this Series: Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything
Part 3 of this Series: Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits

Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything

raphiki — Sat, 12 Sep 2026 05:47:25 +0000

Part 1 of the DeepSeek Harness: Kernel to Edge series: How the open-source (MIT) Cordis meta-framework enables zero-downtime plugin reloads and memory-leak-free architectures in TypeScript, backed by 5+ years of battle-testing in Koishi.

DeepSeek Harness: Kernel to Edge

This article is Part 1 of a three-part architectural and practical deep dive:

Part 1 (This Article): Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything (Microkernel primitives, spatiotemporal composability, reverse cleanup stacks, zero-leak lifecycles).

Part 2: DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents (The meta-harness paradigm, Cordis as an agent kernel, comparing DSH to OpenCode and Pi).

Part 3: Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits (Hands-on local deployment with Ollama, Ornith-1.5, VRAM arithmetic, minimal vs. standard presets, and TUI).

Series Foreword: The Architecture Behind DeepSeek Harness

Developers inspecting the codebase of DeepSeek Harness (dsh) often expect to find Python, LangChain, or prompt pipelines. Instead, they find pure TypeScript built on top of Cordis, an engine originally developed for the Koishi chatbot ecosystem.

This design choice addresses a concrete systems problem: autonomous coding agents rarely fail because of prompt wording; they fail because of runtime lifecycle issues. Over dozens of unattended execution turns, an agent launches compiler runs, mounts temporary sandboxes, registers dynamic tool schemas, and manages file descriptors. In monolithic architectures, these operations accumulate leaked event listeners, orphan child processes, and memory bloat.

DeepSeek treated autonomous agency as an operating system lifecycle problem. Running an agent reliably over long sessions, including on resource-constrained consumer GPUs, requires strict mathematical reversibility and clean teardowns. That is what Cordis provides.

Understanding DeepSeek Harness begins not with LLM prompts, but with the microkernel managing its execution environment.

1. Introduction: What is Cordis?

Most backend frameworks (such as Express, NestJS, Fastify, or Koa) share an unspoken assumption: your application starts once, runs statically, and shuts down only when the process exits.

In those frameworks, you configure your database, register your routes, attach your middleware, and boot up. If you want to change a plugin, add a new route dynamically, or upgrade a module, your only real option is to restart the entire Node.js process.

Cordis is built for a completely different world.

Cordis is an open-source (MIT licensed) TypeScript meta-framework (a framework designed to build other modular frameworks). Its core capability is runtime dynamic composability: it allows you to load, configure, update, and unload plugins on the fly inside a running application with zero memory leaks and zero process restarts.

You can install the core library into any modern Node.js or TypeScript project in seconds:

npm install cordis
# or
pnpm add cordis

Where Did Cordis Come From?

To understand Cordis, you have to look at where it was born:

The Koishi Chatbot Ecosystem (2019–2022): Cordis was created by developer Shigma (Yifan Shi) and the team behind Koishi, a multi-platform chatbot framework. In production, a chatbot maintains persistent, long-lived WebSocket connections to Discord, Telegram, and other platforms. Dropping the connection just to install or update a plugin was a terrible user experience. Koishi needed a plugin engine that could install, reconfigure, and tear down plugins dynamically from a Web UI without touching the main connection.
From Chatbot Core to Meta-Framework (2022–2024): The team realized this dynamic plugin model was not specific to chatbots; it solved a universal problem in Node.js. They extracted the kernel into Cordis (from the Latin word cor, meaning heart). Far from an untested theoretical experiment, Cordis was battle-tested for over five years in production across hundreds of community plugins and thousands of live bot instances in the Koishi ecosystem.
The Academic Roots & The AI Frontier (2025–2026+): Today, Cordis Version 4 represents the culmination of those years of real-world usage and operational experience. Its creator, Shigma (Yifan Shi), is a university professor and computer science researcher. Recognizing the necessity of clean runtime composability for autonomous AI, DeepSeek brought Shigma onto their team to architect the plugin engine for DeepSeek Harness (DSH). Together with researchers from DeepSeek and Peking University, they published a formal research paper ("A Programming Paradigm for Spatiotemporal Composability", arXiv:2608.25512), proving mathematically that dynamic systems can safely load, reload, and unload features indefinitely if side effects and dependencies are modeled as strict mathematical inverses.

This three-part series traces that transition from fundamental principles to real-world code: starting with Cordis's core lifecycle primitives in this article, exploring DeepSeek Harness's agent architecture in Part 2, and concluding with a hands-on local deployment on an 8GB GPU in Part 3.

2. The Big Problem: Why Plugins Usually Leak

To appreciate what Cordis does under the hood, we first need to understand why dynamic plugins in Node.js are notoriously difficult to build.

The "Hotel Room" Analogy

Imagine a hotel room:

When a guest arrives (plugin loaded), they turn on the lights, turn on the bathroom faucet, turn up the heat, and plug in their devices (side effects: event listeners, timers, database connections, HTTP routes).
In traditional JavaScript frameworks, when the guest checks out (plugin unloaded), they walk out the door, but the faucet is still running, the lights are still on, and the heater is still pumping heat.

In Node.js, if you register an event listener on a shared emitter (bus.on('message', callback)), the emitter holds a reference to your callback in memory. Even if you "delete" your plugin, that callback remains in memory. Worse, if you used setInterval(), that timer handle keeps the entire Node.js event loop alive forever.

Over time, reloading plugins in a standard Node.js app causes the Disposal Abyss: zombie event listeners, duplicated handler executions, hanging sockets, and inevitable out-of-memory crashes.

3. The Core Idea: Reversibility ("Every Action Has an Undo")

Cordis solves this problem with a simple, powerful philosophy: Every side effect must be reversible by default.

Instead of trusting the plugin developer to manually write a complex cleanup function, Cordis manages side effects through an inversion of control:

When a plugin runs, Cordis gives it a specialized, isolated environment called a Context (ctx).
Whenever the plugin does something that affects the outside world (like listening to an event, setting a timer, or registering a service), it does so through ctx.
Cordis quietly records the exact "undo" operation into a private cleanup stack.
When the plugin is unloaded, Cordis automatically walks that stack in reverse order and undoes every single side effect.

This is the heart of what Cordis calls Spatiotemporal Composability:

Temporal (Time): You can move forward (load) and backward (unload) in time without leaving leftover state.
Spatial (Space): Modules can safely coexist and discover each other's features without hardcoded, fragile bindings.

Grounded in Research: This isn't just an informal software trick. In their research paper, Shigma and his co-researchers proved mathematically that as long as every state transformation carries a computable inverse (its cleanup function), an application can run indefinitely and roll back any module without corrupting its environment or needing a process restart.

4. How Cordis Works Under the Hood

Under the hood, Cordis relies on four core building blocks: Context, Fibers, Services, and Events. Let's examine each one.

Concept 1: The Context Tree (`Context` and `ctx`)

The Context is the central object in Cordis. It represents the scope in which a component lives.

When you boot up your app, you create a Root Context:

  import { Context } from 'cordis';
  const app = new Context();

When you load a plugin using app.plugin(MyPlugin), Cordis does not run the plugin directly on the root. Instead, it creates a child context (a branch in the tree) specifically for that plugin.
Under the hood, Cordis wraps contexts in a JavaScript Proxy. Whenever a plugin accesses a property (like ctx.database), the Proxy dynamically checks if that service is visible to this branch, whether it's isolated, and tracks what the plugin is using.

Concept 2: Reversible Effects (`ctx.effect`)

Whenever a plugin creates something that needs to be cleaned up later, it registers an Effect.

Think of ctx.effect() like React's useEffect(), but designed for backend servers and long-running services:

function MyPlugin(ctx: Context) {
  // Register a reversible side effect
  ctx.effect(() => {
    console.log('Plugin activated: Setting up resources...');

    const timer = setInterval(() => {
      console.log('Heartbeat ping');
    }, 1000);

    // Return the cleanup function!
    return () => {
      console.log('Plugin deactivating: Cleaning up timer...');
      clearInterval(timer);
    };
  });
}

If this plugin is unloaded, Cordis automatically calls the returned cleanup function. You don't have to keep track of timer IDs or remember to clean them up elsewhere.

Concept 3: Services (Shared Capabilities)

In Cordis, a Service is a reusable singleton feature provided to other plugins (such as a database client, an HTTP server, or a logger).

Creating a service is as simple as extending the Service class:

import { Context, Service } from 'cordis';

// 1. Tell TypeScript that ctx.database exists (full autocomplete!)
declare module 'cordis' {
  interface Context {
    database: DatabaseService;
  }
}

// 2. Define the service
export class DatabaseService extends Service {
  constructor(ctx: Context) {
    // The name 'database' matches the property on Context
    super(ctx, 'database');
  }

  getUser(id: string) {
    return { id, name: 'Alice' };
  }
}

Notice the declare module 'cordis' block. Cordis leverages TypeScript's Declaration Merging. You don't need magic decorators, string tokens, or complex dependency injection containers. Once declared, ctx.database has 100% full type-safety and auto-completion across your entire codebase.

Tip for TypeScript Users: Make sure your tsconfig.json has "moduleResolution": "bundler" or "node16" so that ambient module augmentation (declare module 'cordis') resolves cleanly across your files.

Concept 4: The Fiber (The Plugin Engine)

Whenever you load a plugin, Cordis creates a lightweight runtime manager behind the scenes called a Fiber.

The Fiber is responsible for tracking the plugin's state:

PENDING: The plugin is installed, but one or more services it requires are missing. It sits dormant without running code.
ACTIVE: All required services are present. The plugin's code has run and its side effects are active.
DISPOSED: The plugin has been unloaded and its cleanup stack has been fully executed.

Reactive Dependency Resolution

What makes this magical is automatic reactivity.

Suppose Plugin B declares that it needs 'database':

export const inject = ['database'];

export function apply(ctx: Context) {
  console.log('Database is ready! User:', ctx.database.getUser('1'));
}

If you load Plugin B before the Database service exists, Plugin B quietly waits in PENDING.
The moment you load the Database service, Cordis detects the match and immediately boots Plugin B into ACTIVE.
If someone unloads the Database service, Cordis automatically suspends Plugin B, running its cleanup functions so it doesn't crash trying to use a missing database!

5. Intelligent Events: Beyond the Standard EventEmitter

Standard Node.js event emitters only do one thing: they call every listener synchronously without caring about the return value (emitter.emit('event')).

Cordis introduces multiple dispatch modes to handle real-world application workflows:

Two Everyday Examples:

1. The "Bail" Pattern (Authentication Gate)

Suppose you have multiple plugins that can authenticate a request (API Key, OAuth, Guest Token). You want to stop as soon as any plugin gives a definitive answer:

// Plugin 1: Checks for guest token
ctx.on('auth/check', (token) => {
  if (token === 'guest') return { role: 'guest' };
  return undefined; // Not my job, let next plugin try
});

// Plugin 2: Checks for admin token
ctx.on('auth/check', (token) => {
  if (token === 'secret-admin') return { role: 'admin' };
  return undefined;
});

// ctx.bail short-circuits as soon as someone returns a value!
const user = ctx.bail('auth/check', 'secret-admin');
console.log(user); // { role: 'admin' }

2. The "Waterfall" Pattern (Text Processing Pipeline)

Each listener receives the output of the previous listener:

ctx.on('format/text', (text) => text.trim());
ctx.on('format/text', (text) => text.toUpperCase());
ctx.on('format/text', (text) => `[LOG]: ${text}`);

const result = ctx.waterfall('format/text', '   hello cordis   ');
console.log(result); // "[LOG]: HELLO CORDIS"

6. Putting It All Together: A 30-Second Example

Here is a complete, readable example showing how simple Cordis is to use in practice:

import { Context } from 'cordis';

// 1. Create the application
const app = new Context();

// 2. Define a clean, self-contained feature
function ChatLoggerPlugin(ctx: Context) {
  // Listen to messages
  ctx.on('chat/message', (msg) => {
    console.log(`[ChatLog] ${msg.user}: ${msg.text}`);
  });

  // Set up a background status ping with automatic cleanup
  ctx.effect(() => {
    const timer = setInterval(() => {
      console.log('[ChatLog] Ping: logger is alive');
    }, 2000);

    return () => clearInterval(timer);
  });
}

// 3. Mount the plugin
const fiber = app.plugin(ChatLoggerPlugin);

// 4. Send a message through the event system
app.emit('chat/message', { user: 'Raphael', text: 'Hello, Cordis!' });

// 5. Unload the plugin whenever you want
setTimeout(async () => {
  console.log('Unloading ChatLoggerPlugin...');
  await fiber.dispose();
  console.log('Plugin unloaded cleanly! No hanging timers or listeners.');
}, 5000);

When fiber.dispose() runs, the timer stops, the event listener disappears, and nothing remains in memory.

7. The Polyglot Perspective: How Cordis Compares to Java and Beyond

To truly appreciate what Cordis brings to the table, it helps to step outside the JavaScript world. The desire for modular, hot-swappable plugins is not new; it has been a central topic in enterprise software architecture for over two decades.

Looking at how other ecosystems have tackled this problem clarifies where Cordis fits in the broader computer science landscape:

1. Java OSGi: The Closest Philosophical Cousin

In enterprise Java, the classic standard for dynamic modularity is OSGi (used by Eclipse IDE, Apache Karaf, and Adobe Experience Manager).

The Shared Vision: Both Cordis and OSGi share the exact same core belief: software components should be dynamic. Both feature a central Service Registry where plugins register services and declare dependencies at runtime.
The OSGi Pain Points:
- ClassLoader Hell: In OSGi, each plugin is isolated using a custom Java ClassLoader. If two plugins reference slightly different versions of the same library, developers run into infamous runtime errors (ClassCastException and NoClassDefFoundError).
- Manual Cleanup Leaks: In OSGi, developers must manually remember to unregister every listener in a teardown method (BundleActivator.stop()). If a developer forgets just one listener, the ClassLoader cannot be garbage-collected, creating fatal OutOfMemoryError (Metaspace) leaks.
Cordis’s Advantage: Cordis delivers the same dynamic service lifecycle, but does so inside a single JavaScript runtime using Proxies and automatic cleanup stacks. There are no ClassLoaders to manage, no XML manifests, and no risk of forgotten teardowns.

2. Java Spring Boot: Static vs. Reactive Dependency Injection

Spring is the reigning champion of Dependency Injection (DI) in enterprise software.

Spring's Model: Spring’s Dependency Injection is static and monotonic. When your Spring Boot application boots, it scans your classes, builds the dependency graph, instantiates your singletons, and then freezes. If a service fails or disappears at runtime, the application cannot dynamically heal itself; it typically throws an unhandled exception or requires a restart.
Cordis's Model: Cordis is living and reactive. Services are not fixed in stone at startup. If a service is unloaded, dependent plugins do not crash with null references; they gracefully pause until the service returns.

3. NestJS (TypeScript): The Familiar Alternative

Within TypeScript itself, NestJS is the most popular framework using Angular/Spring-style Dependency Injection.

NestJS relies heavily on experimental TypeScript decorators (@Injectable(), @Module()) and runtime metadata (reflect-metadata).
Like Spring, NestJS is designed primarily for static architectures: modules are registered during bootstrap. Unloading a NestJS module at runtime without restarting the process is virtually impossible without custom, fragile hacks.
Cordis achieves full dependency injection without a single decorator, using TypeScript’s native declaration merging and runtime Proxies instead.

High-Level Architectural Comparison

Framework / Ecosystem	Dependency Injection?	Dynamic Runtime Unload?	Teardown Mechanism	Complexity & Overhead
Spring Boot (Java)	Yes (Static)	❌ No	Static `DisposableBean` on shutdown	Heavy enterprise container
OSGi (Java / Eclipse)	Yes (Dynamic)	Yes	Manual `stop()` (high risk of Metaspace leaks)	Heavy (XML, manifests, ClassLoaders)
NestJS (TypeScript)	Yes (Static)	❌ No	Process restart (`nodemon`)	Medium (Decorators + Reflection)
Cordis (TypeScript)	Yes (Reactive)	Yes	Automatic reverse cleanup stack	Ultra-lightweight (<50KB, zero bloat)

In short, Cordis can be thought of as:

"The dynamic service lifecycle of Java OSGi, the dependency injection of Spring, and the cleanup ergonomics of React’s useEffect, all distilled into a lightweight TypeScript package."

8. Summary & What's Coming Next

Why Cordis Matters

Open Source & Battle-Tested (MIT License): Far from an academic toy, Cordis v4 builds on over half a decade of real-world production stress-testing across hundreds of plugins and millions of active chat sessions in the Koishi ecosystem.
Safe Hot-Reloading: Features can be added, updated, or removed at runtime without restarting Node.js.
Zero Resource Leaks: Every timer, listener, and connection is automatically tracked and torn down when unmounted.
Pure TypeScript Ergonomics: Declaration merging delivers clean autocomplete and type checking without bulky decorators or string tokens.
Reactive Dependency Injection: Components only wake up when their required services exist, and sleep gracefully when they vanish.

The Road Ahead: Parts 2 and 3

With Cordis's core primitives (the context tree, revertible effects, and reactive dependency model) established, the next two articles explore how this foundation powers modern AI systems:

Part 2: DeepSeek Harness Architecture examines why DeepSeek adopted Cordis for DeepSeek Harness (dsh):
- What an AI "harness" actually does, and how DSH differs from turnkey developer tools like OpenCode and Pi.
- Why autonomous agents need an operating system microkernel rather than Python prompt chains.
- How Cordis provides safe, dynamic tool sandboxing and clean resource teardown during multi-step reasoning.
Part 3: Running DeepSeek Harness on an 8GB GPU moves from architectural theory to consumer hardware:
- Setting up DSH locally on an 8GB RTX 4070 laptop using Ollama and Ornith-1.5.
- Managing context windows, tool schemas, and VRAM limits using agent presets.
- Comparing minimal and standard modes in empirical code-editing benchmarks.

References & Further Reading

Research Paper: Shi, Y., Zhang, W., & Cui, T. A Programming Paradigm for Spatiotemporal Composability (arXiv:2608.25512)
Cordis GitHub Repository: github.com/cordiverse/cordis
Cordis Documentation (Primer): DeepSeek Harness Cordis Primer
Koishi Framework: koishi.chat
Part 2 of this Series: DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents
Part 3 of this Series: Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits

Building a Local-First AI Coding Agent with Open Tools and Adaptive Routing

raphiki — Wed, 09 Sep 2026 15:43:13 +0000

How to combine local inference with governed cloud fallback by orchestrating Ollama, OpenCode, and LiteLLM

AI coding agents usually leave you with an awkward choice:

All cloud: A hosted model can handle almost everything, but using it for small edits and routine work consumes API credits and sends more code off the machine.
All local: A model that fits on a laptop GPU works well for many everyday tasks, but it may struggle with architecture, complex refactoring, and debugging across several files.

A local first hybrid architecture offers a middle ground. The local model handles routine coding decisions, while OpenCode performs filesystem and shell operations through its tools. Harder requests can go to larger hosted models, with a configurable limit on spending.

Connecting several models is easy. Choosing one for each request, in a way that is predictable and visible, takes more work. LiteLLM provides the control plane for that decision.

In this tutorial, we will build a three layer AI coding environment on a laptop equipped with an RTX 4070 GPU with 8GB of VRAM and 32GB of system RAM:

Layer 1, local inference: Ollama serves a Gemma 4 model configured for tool use.
Layer 2, control plane: LiteLLM provides spending limits, automatic routing by complexity, and observability backed by PostgreSQL.
Layer 3, agent harness: OpenCode provides terminal, desktop, and web interfaces, along with tool dispatch and session state.

Each connection is tested before the next one is added. We start with the local model, connect the harness, and finish with the proxy and router. This makes failures much easier to locate.

Why this stack

Large reasoning models are useful for architecture and subtle bugs. They are expensive overkill for routine implementation or a focused refactor that a smaller local model can handle.

Many setups connect a harness to one model and stop there. This build adds a control plane to decide where requests go and track what they cost. The harness and local gateway are open source, and the selected models have available weights. I use OpenRouter, a proprietary service, for hosted inference. You can replace it with another compatible endpoint or infrastructure that you operate yourself. No API for a closed weight model is required.

My test machine has an RTX 4070 laptop GPU with 8GB of VRAM and 32GB of system RAM. The configuration below ran the local model, harness, and control plane together on that machine. Your memory use and throughput will vary with the model build, context length, drivers, operating system, and other GPU workloads.

What “adaptive” means in this build

OpenCode always requests one logical model, auto-mode. LiteLLM then classifies the request and selects a target according to the routing policy:

Request class	Example	Default target
Simple	A focused rename or boilerplate	Local Gemma 4
Medium	Routine implementation or test writing	Local Gemma 4
Complex	Changes across several files or substantial debugging	Hosted Mistral Small
Reasoning	Architecture, planning, or explicit tradeoffs	Hosted DeepSeek

The router does not learn from previous requests. It uses LiteLLM's complexity score, a few keyword rules, and the tier mapping configured later in the article.

This tutorial builds three layers, in order, so each one is independently testable before you stack the next:

The implementation order differs from the layer numbers. We first validate the local worker (Layer 1), connect the harness directly to it (Layer 3), and then place the proxy (Layer 2) between them.

Local Inference (Layer 1)

Local Model: Gemma 4

The local model has to fit in 8GB of VRAM and still be useful for coding and tool use. That points toward a quantized model with available weights and a good balance between size and reasoning ability.

I used llmfit to narrow the options and selected Gemma 4 e2b Q4_K_M.

The name describes the model and its quantization:

Gemma 4 is an open weight model provided by Google.
e2b means "Effective 2B" and targets edge deployments, making it suitable for this hardware.
Q4_K_M identifies a four bit K quant variant with a medium mix of quantization types. Quantization reduces memory use at some cost to fidelity.

Inference Server: Ollama

I selected Ollama because it is easy to install and exposes an OpenAI compatible API for the rest of the stack. Other open source inference servers can fill the same role. I covered several alternatives in the Bringing AI Home series and in this comparison of Ollama, vLLM, and ZML.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Then download and run Gemma 4:

ollama run gemma4:e2b

pulling manifest
pulling 4e30e2665218: 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 7.2 GB
verifying sha256 digest
writing manifest
success
>>> Hi
Thinking...
Thinking Process:

1.  **Analyze the input:** The input is "Hi". This is a very casual, open-ended greeting.
2.  **Determine the user's intent:** The user is initiating a conversation or acknowledging my presence.
3.  **Formulate an appropriate response:** The response should be friendly, welcoming, and invite further interaction.
    *   Standard replies: "Hello," "Hi there," "How can I help?"
4.  **Self-check against constraints (Identity/Role):** I am Gemma 4, a helpful AI. The response should reflect that role.
5.  **Generate the final reply:** A simple, warm greeting followed by an offer of assistance is ideal.
...done thinking.

Hello! How can I help you today?

The response confirms that local generation works. We will test tool use after connecting the harness.

AI Harness (Layer 3)

Open Source Harness: OpenCode

The model needs an execution harness before it can act as an agent. The harness manages session history, runs local tools, and tracks file changes across turns.

Mistral Vibe, Claude Code, OpenAI Codex, and Antigravity are examples of harnesses. Open source options include Pi Coding Agent and DeepSeek Harness.

I selected OpenCode because it is lightweight, actively developed, and available through terminal, web, and desktop interfaces. A formal QSOS comparison of open coding agents would be an interesting follow up. For now, install OpenCode:

curl -fsSL https://opencode.ai/install | bash

Quick smoke test: OpenCode talking directly to Ollama

Now I'm ready to wire OpenCode to Ollama and Gemma 4... or so I thought.

I learned the hard way that the model needs a configuration tailored to the agent before it is connected to the harness. A bare ollama run setup can behave differently once the harness starts sending tool schemas and longer agentic turns.

Agent harnesses can send tool schemas, project context, and file contents with every turn. If the context is too small, Ollama may truncate earlier content, including tool definitions. The result can be failed or invented tool calls. On this 8GB GPU, I use a context of 16,384 tokens. Larger windows consume more memory for the KV cache and may offload work to the CPU, so adjust this value for your hardware.

When creating a custom Modelfile, keep the model's compatible chat template unless you have tested a replacement. Tool capable models expect function definitions in a specific prompt format. An incompatible template can produce raw JSON or plain text command suggestions instead of tool calls.

Smaller general purpose models may call tools from another agent framework, such as explore or list_files, instead of OpenCode's glob. A short system prompt can list the available tools and map common actions to their OpenCode names.

I used the following Modelfile:

FROM gemma4:e2b

PARAMETER num_ctx 16384
PARAMETER temperature 0.0

SYSTEM """
You are an autonomous coding assistant inside OpenCode.
CRITICAL: You must only invoke tools from the provided schema.
Available Tools:
- glob(pattern: string): List/find files. Use this to explore directories.
- read(filePath: string, offset?: number, limit?: number): Read file content.
- write(filePath: string, content: string): Create or overwrite a file.
- edit(filePath: string, oldString: string, newString: string): Replace exact text in a file.
- grep(pattern: string, path?: string): Search codebase with regex.
- bash(command: string): Run terminal commands (build, test, git).
- question(header: string, question: string, options?: string[]): Ask user for clarification.
- task(subagentType: string, prompt: string): Delegate a subtask.
- skill(name: string): Load a predefined skill.
- todowrite(todos: array): Update the task/progress list.
- webfetch(url: string): Fetch web content.
- invalid: System fallback (do not invoke directly).
"""

The 16384 context window is the value used on my RTX 4070 laptop. A 32K or 64K window requires substantially more KV cache memory and may force partial CPU offloading on an 8GB card. Watch nvidia-smi while testing; if VRAM remains pinned near the limit or throughput collapses, reduce num_ctx to 8192 and retest.

Create the configured model in Ollama, much like building a Docker image:

ollama create gemma4-coder-agent -f Modelfile

gathering model components
using existing layer sha256:fdf02c16fb654ff60b2c30f1e91573ebc603a7084df3449df89120dca2b18170
using existing layer sha256:e94a8ecb9327ded799604a2e478659bc759230fe316c50d686358f932f52776c
creating new layer sha256:dcaf83c203b7c4daae5c154641637a2d10221b09baa4fce8d70f839cad18447d
writing manifest
success

Check that it appears in Ollama's model list:

ollama list

NAME                                                  ID              SIZE      MODIFIED
gemma4-coder-agent:latest                             bfa492d99bf8    7.2 GB    49 minutes ago
gemma4:e2b                                            7fbdbf8f5e45    7.2 GB    15 minutes ago

Next, call the configured model through Ollama's OpenAI compatible API. Testing this boundary now helps distinguish a model server problem from a later harness problem:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-coder-agent",
    "messages": [{"role": "user", "content": "Write a TypeScript function that debounces a callback."}]
  }'

Once the request succeeds, connect the harness. Close unnecessary GPU heavy applications before longer coding sessions because 8GB of VRAM leaves little headroom.

Point OpenCode at the configured model (no proxy yet)

Configure OpenCode in ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1"
      },
      "models": {
        "gemma4-coder-agent": {
          "name": "Gemma4 (ollama)",
          "tools": true
        }
      }
    }
  }
}

And launch the OpenCode CLI:

opencode

Ask it to read or edit a file and confirm that OpenCode dispatches the tool call. A chat response alone does not test the tool configuration.

If tool calls don't dispatch, check that the Modelfile applied correctly (ollama show gemma4-coder-agent) before touching anything else.

If nothing works at all, check that curl http://localhost:11434/v1/models returns the model, and check OpenCode's logs for the actual connection error.

The harness and local model now work together. Keep this opencode.json as a baseline while adding the proxy and router.

Simple LiteLLM Proxy (Layer 2)

The next component is the AI gateway between the harness and the models. Several open source gateways are available, including Mozilla AI's Otari.

I selected LiteLLM, which we also use in production at work. It covers the proxy and governance features needed here. I did not measure its latency overhead for this article, so the performance discussion focuses on model routing.

LiteLLM began as a small Python library and now includes a proxy server, an Admin UI, and optional database storage.

Start with the standalone proxy:

pip install 'litellm[proxy]' --break-system-packages

This command matches my test environment. The --break-system-packages option bypasses the distribution package manager's protection. On a maintained workstation, use a virtual environment, pipx, uv, or the Docker setup shown later.

The Docker deployment uses the same configuration file and is covered later.

First, point the proxy only at the gemma4-coder-agent that we already tested.

Create litellm_config.yaml:

model_list:
  - model_name: "Gemma 4 (litellm)"
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://localhost:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

general_settings:
  master_key: sk-local-master-key-change-me

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

Start LiteLLM with that configuration:

litellm --config litellm_config.yaml --port 4000

Call the model through port 4000 to confirm that LiteLLM, rather than the direct Ollama endpoint, serves the request:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "Gemma 4 (litellm)", "messages": [{"role": "user", "content": "ping"}]}'

{
  "id": "chatcmpl-a1230750-0843-4af5-9e35-3c8e1869d45d",
  "created": 1787427863,
  "model": "Gemma 4 (litellm)",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "",
        "role": "assistant",
        "reasoning_content": "The user input is simply \"ping\". This is a very vague request.\nIn a general context, \"ping\" usually refers to a network diagnostic tool.\nHowever, as an autonomous coding assistant inside OpenCode, I need to determine what action the user expects me to take based on the available tools and the context of a coding environment.\n\n1.  **Tool Check:** I have tools for file system operations (`glob`, `read`, `write`, `edit`, `grep`), shell commands (`bash`), web fetching (`webfetch`), and task delegation/questioning.\n2.  **Interpretation:** Since there is no specific file or code provided, \"ping\" might be:\n    *   A request to run a system command (like `ping` in a terminal).\n    *   A request for information about the network concept of ping.\n    *   A prompt to test connectivity (which I cannot do directly outside of a simulated environment).\n\nGiven the context of an \"autonomous coding assistant,\" the most likely interpretation is that the user wants me to execute a command if possible, or they are testing my ability to respond to a simple command. Since I have a `bash` tool, running a system command is an option.\n\nIf I assume the user wants me to run the standard network diagnostic:\n*   I can use `bash(\"ping\")`.\n\nIf I assume the user is asking for a definition or context:\n*   I should explain what `ping` is.\n\nSince I am operating within a coding assistant framework, and \"ping\" is often used as an instruction to test connectivity in such environments, I will attempt to use the `bash` tool."
      }
    }
  ],
  "usage": {
    "completion_tokens": 357,
    "prompt_tokens": 258,
    "total_tokens": 615
  }
}

If the call fails, check that Ollama is still running with ollama list before debugging LiteLLM.

Now connect the harness to the proxy. With only one model configured, the full OpenCode → LiteLLM → Ollama path is still easy to debug.

Update ~/.config/opencode/opencode.json to register LiteLLM alongside the existing Ollama provider:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1"
      },
      "models": {
        "gemma4-coder-agent": {
          "name": "Gemma4 (ollama)",
          "tools": true
        }
      }
    },
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "sk-local-master-key-change-me",
        "compatibility": "compatible"
      },
      "models": {
        "Gemma 4 (litellm)": {
          "tools": true
        }
      }
    }   
  }
}

opencode

Inside the OpenCode TUI, running /models lets us toggle between connecting directly to Ollama (Gemma4 (ollama)) or through our LiteLLM proxy (Gemma 4 (litellm)).

LiteLLM is currently a simple pass through with one model and no routing logic. The test confirms that the network path and OpenAI compatible interface work before we add cloud models.

Hybrid LiteLLM Proxy (Layers 1 & 2)

With the local route working, we can add cloud endpoints. The harness and proxy are open source, and the models have available weights. Hosted access still depends on the service used to run those models.

I use OpenRouter to access the remote models. OpenRouter is a proprietary hosted gateway, while LiteLLM keeps the routing policy, budget controls, and logs on the laptop. You could serve the same models from your own infrastructure or cloud tenant instead.

Two remote models complement the local Gemma 4:

Mistral Small: A fast and relatively inexpensive option for tasks that are too demanding for the local model.
DeepSeek v4: A larger reasoning model for architecture and difficult debugging.

One OpenRouter API key covers both models. In the hybrid setup, all OpenCode traffic passes through LiteLLM, so the direct Ollama connection is removed. The name Gemma 4 (litellm) no longer has to distinguish one connection from another. From here on, its LiteLLM alias is the simpler gemma4-local.

model_list:
  - model_name: gemma4-local
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://localhost:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

  # --- Cloud tier: open-weight models via OpenRouter ---
  - model_name: deepseek-reasoning
    litellm_params:
      model: openrouter/deepseek/deepseek-v4-pro
      api_key: os.environ/OPENROUTER_API_KEY

  - model_name: mistral-fast
    litellm_params:
      model: openrouter/mistralai/mistral-small-2603
      api_key: os.environ/OPENROUTER_API_KEY

router_settings:
  provider_budget_config:
    openrouter:
      budget_limit: 5        # $5/day ceiling on cloud spend
      time_period: 1d

general_settings:
  master_key: sk-local-master-key-change-me

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

LiteLLM can limit spending over a given period. Once the threshold is reached, it blocks further calls covered by that budget. Test this case in your deployment and check the error shown by OpenCode. You may also want those requests to fall back to the local model.

This configuration limits OpenRouter spending to $5 per day.

LiteLLM can also issue scoped keys with their own budgets:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"max_budget": 10, "budget_duration": "30d", "models": ["gemma4-local", "deepseek-reasoning", "mistral-fast"]}'

The remaining examples use the tutorial master key for consistency. For a shared or long lived deployment, use the scoped key returned by /key/generate in OpenCode and keep the master key out of client configuration.

Export the OpenRouter key and restart the proxy:

export OPENROUTER_API_KEY=sk-or-...
litellm --config litellm_config.yaml --port 4000

Test each route directly through the LiteLLM API:

# local model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4-local", "messages": [{"role": "user", "content": "ping"}]}'

# First Cloud model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "mistral-fast", "messages": [{"role": "user", "content": "ping"}]}'

# Second Cloud model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-reasoning", "messages": [{"role": "user", "content": "ping"}]}'

The API also reports budget consumption:

curl -X GET http://localhost:4000/provider/budgets \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-local-master-key-change-me"

The response includes the limit, current spend, time period, and reset time. Check the reset timestamp against the host clock before relying on it. A wildly incorrect date usually points to a clock problem or a bug in the installed version.

Point OpenCode at the expanded proxy:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "sk-local-master-key-change-me",
        "compatibility": "compatible"
      },
      "models": {
        "gemma4-local": {
          "tools": true
        },
        "mistral-fast": {
          "limit": { "context": 262144, "output": 8192 }
        },
        "deepseek-reasoning": {
          "limit": { "context": 1000000, "output": 8192 }
        }
      }
    }   
  }
}

The direct Gemma4 (ollama) entry is gone because all traffic now passes through LiteLLM. From this point onward, gemma4-local identifies the local model behind the proxy.

OpenCode also provides a web interface. Start it and check that the previous sessions and new models are available:

opencode web

The hybrid proxy now exposes several models and a spending limit, but model selection is still manual. The next step is automatic routing.

Smart LiteLLM Proxy (Layer 2)

During my tests, OpenCode did not switch models reliably based on task complexity. Its model overrides for subagents were not consistent enough for this setup.

The routing decision therefore moves to LiteLLM's Auto Router v2. OpenCode always requests auto-mode. LiteLLM scores the request, applies any matching keyword rule, and maps the chosen tier to a model. OpenCode does not need to know which model handles the request.

Add auto-mode to litellm_config.yaml alongside the three existing models:

model_list:
  # ... gemma4-local, deepseek-reasoning, mistral-fast entries stay as-is ...

  - model_name: auto-mode
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gemma4-local          # autocomplete, small edits, boilerplate
          MEDIUM: gemma4-local          # routine implementation, test writing
          COMPLEX: mistral-fast        # multi-file changes, real debugging
          REASONING: deepseek-reasoning # architecture, planning, hard bugs
        complexity_router_default_model: gemma4-local   # fail toward free, not expensive
        keyword_rules:
          - keywords: ["plan", "architecture", "design a", "trade-off"]
            tier: REASONING
        return_raw_model_name: true

Three settings control the behavior:

complexity_router_default_model: gemma4-local sends an uncertain or timed out classification to the free local model. Change it if you prefer capability over cost.
keyword_rules run before complexity scoring. They immediately escalate prompts that mention architecture, planning, or tradeoffs.
return_raw_model_name: true returns the name of the model that handled the request, making routing easier to verify in logs.

After restarting the proxy, send a short prompt and an architecture prompt to auto-mode. Check the returned model name and the LiteLLM logs. This confirms that both routes work, but it does not measure routing accuracy.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto-mode", "messages": [{"role": "user", "content": "rename your model name to camelCase"}]}'

The Auto Router selected the ollama_chat/gemma4-coder-agent model:

{
  "id": "chatcmpl-774cdbf7-09c2-4d1e-8f94-bb878c0f11bb",
  "created": 1788128064,
  "model": "ollama_chat/gemma4-coder-agent",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I am an AI assistant and do not have a specific model name that I can rename within this context. How can I help you with your coding tasks?",
        "role": "assistant",
        "reasoning_content": "Thinking Process:\n\n1.  **Analyze the Request:** The user wants me to \"rename your model name to camelCase\"..."
   # ... rest of the message ...

The router chose the expected local model, but the answer itself is poor because the model interpreted the vague request literally. A correct route does not guarantee a good answer.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto-mode", "messages": [{"role": "user", "content": "design the data model for a multi-tenant billing system with usage-based pricing"}]}'

This time, the deepseek/deepseek-v4-pro is called:

{
  "id": "gen-1788128721-XpJrCbVmBWN8x9E03aDh",
  "created": 1788128721,
  "model": "deepseek/deepseek-v4-pro",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Below is a Stripe-inspired data model for a **multi-tenant billing system with usage-based pricing**.  \nThe model assumes:\n\n- A **tenant** is an organization u..."
  # ... rest of the message ...

Evaluate the routing policy with representative tasks

Before making auto-mode the default, test it with prompts from your own work. Label the expected tier first, then record the selected model and whether it completed the task. A small test set could include:

Task	Expected tier	What to verify
Rename a field in one interface	Simple	Remains local and makes the correct edit
Add validation and unit tests	Medium	Remains local unless the context is unusually large
Trace a failure across several layers	Complex	Escalates to the fast hosted model
Compare tenant isolation designs	Reasoning	Escalates to the reasoning model
Ask for a “plan” for a trivial rename	Adversarial	Reveals whether the keyword rule escalates too often
Describe a hard bug without escalation keywords	Adversarial	Reveals whether complexity scoring catches it

Run the same tasks with local only, cloud only, manual selection, and automatic routing. For each mode, record successful tasks, median latency, hosted request count, and hosted cost.

The results will show whether the router saves money without hurting task completion. Pay particular attention to unnecessary cloud calls and difficult tasks that stay local.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://localhost:4000/v1",
        "apiKey": "sk-local-master-key-change-me"
      },
      "models": {
        "auto-mode": {
          "limit": { "context": 16384, "output": 4096 }
        },
        "gemma4-local": {
          "limit": { "context": 16384, "output": 4096 },
          "tools": true
        },
        "mistral-fast": {
          "limit": { "context": 262144, "output": 8192 }
        },
        "deepseek-reasoning": {
          "limit": { "context": 1000000, "output": 8192 }
        }
      }
    }
  },
  "enabled_providers": ["litellm"],
  "model": "litellm/auto-mode"
}

The "model": "litellm/auto-mode" setting makes automatic routing the default for new sessions. The three individual models remain available in /models as manual overrides.

"enabled_providers": ["litellm"] limits OpenCode to the LiteLLM provider. This syntax will be replaced by the policy mechanism described here.

OpenCode also has a desktop application for Windows, macOS, and Linux, available from the official site.

The same session is shown below in OpenCode Desktop with Auto Mode enabled.

The proxy selected Mistral Small for this task. At this point, I still had to check the OpenRouter dashboard to confirm the choice.

That external check confirmed the route, but it also showed what the local setup still lacked: one place to inspect routing and usage.

Full Control Plane (Layer 2)

LiteLLM started as a Python wrapper for different LLM APIs. It has since expanded into gateway infrastructure with management and observability features.

The full proxy can use PostgreSQL for its Admin UI, request logs, and spending data. Running those services locally adds useful controls without making this tutorial setup production ready.

To add those features, switch to the containerized deployment backed by PostgreSQL. Download the official Docker Compose file:

curl -sSLO https://docs.litellm.ai/docker-compose.yml

Adapt the Compose file to mount litellm_config.yaml, seed the database on first boot, and reach Ollama on the host:

services:
  litellm:
    image: docker.litellm.ai/berriai/litellm-database:latest
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command:
      - "--config=/app/config.yaml"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "4000:4000"
    environment:
      LITELLM_SALT_KEY: sk-XXXXXXXXXXXXXXXX
      DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
      STORE_MODEL_IN_DB: "True"
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Docker Compose creates a LiteLLM container, a PostgreSQL container, and a volume for the database. I made two changes to the default Compose file:

volumes: [./litellm_config.yaml:/app/config.yaml] and command: ["--config=/app/config.yaml"] load the LiteLLM configuration into the container.
extra_hosts: ["host.docker.internal:host-gateway"] lets the container reach Ollama on the host.

The LiteLLM configuration also needs a few changes:

model_list:
  - model_name: gemma4-local
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://host.docker.internal:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

  # --- Cloud tier: open-weight models via OpenRouter ---
  - model_name: deepseek-reasoning
    litellm_params:
      model: openrouter/deepseek/deepseek-v4-pro
      api_key: os.environ/OPENROUTER_API_KEY

  - model_name: mistral-fast
    litellm_params:
      model: openrouter/mistralai/mistral-small-2603
      api_key: os.environ/OPENROUTER_API_KEY

  # --- Auto Router: smart routing based on task complexity and keywords ---
  - model_name: auto-mode
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gemma4-local          # autocomplete, small edits, boilerplate
          MEDIUM: gemma4-local          # routine implementation, test writing
          COMPLEX: mistral-fast         # multi-file changes, real debugging
          REASONING: deepseek-reasoning # architecture, planning, hard bugs
        complexity_router_default_model: gemma4-local   # fail toward free, not expensive
        keyword_rules:
          - keywords: ["plan", "architecture", "design a", "trade-off"]
            tier: REASONING
        return_raw_model_name: true

router_settings:
  provider_budget_config:
    openrouter:
      budget_limit: 5        # $5/day ceiling on cloud spend — tune to taste
      time_period: 1d

general_settings:
  master_key: sk-local-master-key-change-me
  store_model_in_db: true
  store_prompts_in_spend_logs: true

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

The changes are:

api_base: http://host.docker.internal:11434 points the container to the local Ollama model.
store_model_in_db: true enables database storage.
store_prompts_in_spend_logs: true enables prompt logging.

The last option stores prompts, including any source code they contain, in PostgreSQL. Disable it or define a retention policy if you do not need to inspect full prompts.

Start the containers:

docker compose up -d

The LiteLLM container logs confirm that the models were loaded from the configuration:

litellm-1  |
litellm-1  |    ██╗     ██╗████████╗███████╗██╗     ██╗     ███╗   ███╗
litellm-1  |    ██║     ██║╚══██╔══╝██╔════╝██║     ██║     ████╗ ████║
litellm-1  |    ██║     ██║   ██║   █████╗  ██║     ██║     ██╔████╔██║
litellm-1  |    ██║     ██║   ██║   ██╔══╝  ██║     ██║     ██║╚██╔╝██║
litellm-1  |    ███████╗██║   ██║   ███████╗███████╗███████╗██║ ╚═╝ ██║
litellm-1  |    ╚══════╝╚═╝   ╚═╝   ╚══════╝╚══════╝╚══════╝╚═╝     ╚═╝
litellm-1  |
litellm-1  | ...
litellm-1  | INFO:     Application startup complete.
litellm-1  | INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
litellm-1  |
litellm-1  | #------------------------------------------------------------#
litellm-1  | #                                                            #
litellm-1  | #               'A feature I really want is...'               #
litellm-1  | #        https://github.com/BerriAI/litellm/issues/new        #
litellm-1  | #                                                            #
litellm-1  | #------------------------------------------------------------#
litellm-1  |
litellm-1  |  Thank you for using LiteLLM! - Krrish & Ishaan
litellm-1  |
litellm-1  |
litellm-1  |
litellm-1  | Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
litellm-1  |
litellm-1  |
litellm-1  | LiteLLM: Proxy initialized with Config, Set models:
litellm-1  |     gemma4-local
litellm-1  |     deepseek-reasoning
litellm-1  |     mistral-fast
litellm-1  |     auto-mode

Access the Admin UI at http://localhost:4000/ui/. Log in with the username admin and the password configured as your master_key (sk-local-master-key-change-me).

The Models + Endpoints view lists the local model, the two remote models, and the auto-mode router.

Two views are especially useful here.

The Usage view shows request volume, token consumption, and spending by model:

The Request Logs view shows individual traces, prompts, and router decisions stored in PostgreSQL:

Privacy and trust boundaries

Local first is not the same as local only. Requests assigned to mistral-fast or deepseek-reasoning leave the laptop and pass through OpenRouter to a hosted inference provider. They may contain source code, file paths, tool output, or conversation history.

Before using automatic routing with a private repository:

review the data handling terms of every hosted provider in the route;
exclude secrets, .env files, private keys, certificates, and credential stores from agent context;
provide an easy local only model override for sensitive work;
decide whether cloud escalation should be automatic or require confirmation;
restrict access to the LiteLLM Admin UI and PostgreSQL database;
replace tutorial keys and database passwords with secrets supplied through your deployment environment;
define retention and backup policies for stored prompts and request logs.

The control plane makes model selection and logging visible. It does not make hosted inference private, but it gives you one place to define and audit that boundary.

Before treating this as a production deployment

This tutorial favors a readable local setup. Once the complete path works, record the environment and pin the component versions or container image digests used for that run.

Automatic routing, provider compatibility, and configuration fields can change between releases. A shared or production deployment should also enable TLS, use scoped client keys, protect the Admin UI, rotate secrets, back up PostgreSQL, test budget exhaustion and provider failures, and monitor routing quality.

LiteLLM also supports MCP server hubs, team virtual keys, input guardrails, and heuristics such as routing by context size. This article uses only the features needed for the local control plane.

LiteLLM also offers a commercial Enterprise edition with SAML SSO, team RBAC, secret management, and clusters across multiple regions. Everything configured here runs on the open source community edition.

Conclusion

This setup made an 8GB VRAM laptop a practical base for my coding agent. Gemma 4 handles routine work without token charges, OpenCode runs the tools and keeps the session, and LiteLLM sends harder requests to larger hosted models.

What We Built

Configured Local Inference: A custom Ollama Modelfile with a tested context window (16384) and an explicit tool registry designed to improve Gemma 4's behavior within 8GB of VRAM.
Decoupled Harness: OpenCode configured to communicate over standard OpenAI compatible endpoints across terminal, desktop, and web interfaces.
Governed Control Plane: A local LiteLLM deployment backed by PostgreSQL providing:
- Provider spending limits ($5/day in the example configuration).
- Routing by complexity via auto-mode, keeping standard edits local while escalating hard tasks to Mistral and DeepSeek.
- Local observability through the LiteLLM Admin UI to review prompt logs and token usage.

Next Steps & Experiments

If you want to take this setup further, consider exploring:

Model Context Protocol (MCP): Connect local MCP servers to OpenCode for safe database inspection, live documentation lookups, or issue tracking.
Routing by Context Size: Configure LiteLLM to hand off oversized file trees to models with a larger context while keeping short prompt cycles local.
Semantic Scoring with Embeddings: Replace keyword rules with an embedding classifier to automate tiering dynamically.

The main benefit is control. You can see which model handled a request, keep routine work local, limit cloud spending, and inspect the result. Requests sent to hosted models still leave the machine, and the logs make that boundary visible.

Driving Local ComfyUI from Codex with MCP

raphiki — Fri, 21 Aug 2026 14:46:13 +0000

In the previous articles of the Beyond the ComfyUI Canvas series, I connected ComfyUI to notebooks, WebSockets, n8n, and Flowise. I even created my own MCP Server. Each integration worked, but it still required me to write or maintain the glue: HTTP payloads, polling loops, node IDs, and workflow-specific code.

Today, I am taking a more direct route. I will connect a local ComfyUI installation to Codex through the official open-source Comfy MCP server.

The goal is simple: let an AI agent inspect the local ComfyUI instance, discover installed models, validate an existing workflow, run it, wait for completion, and bring the generated image back into the conversation.

No custom Python bridge. No hand-written /prompt request. The agent uses a standard MCP tool interface; Comfy MCP translates those tool calls into comfy-cli commands targeting my local ComfyUI server.

1. Setting the Scene: The Stack

The local stack I used has three layers:

Codex is the agent and MCP client. It receives natural-language requests and decides which tools to call.
Comfy MCP is the open-source MCP server from Comfy. It runs as a local stdio process and wraps comfy-cli.
ComfyUI is the generation engine, running locally at http://127.0.0.1:8188 on a GPU-equipped machine.

This is not Comfy Cloud MCP. The complete flow stays on the local machine: the workflow, models, queue, and generated images all remain in the local ComfyUI workspace. The only exception is a workflow that deliberately uses paid partner API nodes.

2. Install the Local Components

Let Codex do the setup

Steps 2 and 3 can be delegated to Codex itself. Instead of manually installing packages and editing the MCP configuration, give Codex this prompt:

Help me set up the local Comfy MCP connection.

Follow the setup guide at https://docs.comfy.org/agent-tools/mcp.md#local-comfy-mcp-connection

Codex can inspect the machine, install the required packages, create or locate the ComfyUI workspace, configure the MCP server, and then verify the connection. It should still show you any required permissions before making changes outside its workspace.

The Comfy MCP server is built on top of comfy-cli, so both packages are required:

py -m pip install comfy-mcp "comfy-cli>=1.14.0"

If ComfyUI is not installed yet, create a workspace with:

comfy install

For an existing workspace, make it the default instead:

comfy set-default C:\path\to\ComfyUI

Then launch the local server:

comfy launch

In my case, ComfyUI was installed in C:\Users\rapha\Documents\comfy\ComfyUI and became available at http://127.0.0.1:8188.

The Windows PATH trap

On Windows, packages installed with Python can place comfy.exe and comfy-mcp.exe in a Scripts directory that is not on the PATH inherited by desktop applications.

That can be confusing: the MCP server starts and completes its handshake, but every useful tool fails with:

comfy not found on PATH

The robust fix is to configure COMFY_BIN with the absolute path to comfy.exe. It makes the MCP server independent from the client application's PATH.

3. Connect Comfy MCP to Codex

Codex stores local MCP servers in its configuration. Add the following to ~/.codex/config.toml, adapting the paths to your Python installation:

[mcp_servers.comfy-mcp]
command = 'C:\Users\<you>\AppData\Roaming\Python\Python313\Scripts\comfy-mcp.exe'

[mcp_servers.comfy-mcp.env]
COMFY_BIN = 'C:\Users\<you>\AppData\Roaming\Python\Python313\Scripts\comfy.exe'

Restart or reload Codex after changing the configuration. The server is a stdio MCP server: Codex launches comfy-mcp as a subprocess when it needs the tools.

COMFY_API_KEY is not required for local Flux, SDXL, or other locally installed models. Add it only when you intentionally use Comfy partner API nodes that spend credits.

4. First Contact: Verify the Engine and Inspect Models

The first useful test is not image generation. It is asking Codex to inspect the live server:

Confirm that my local ComfyUI is running and list the installed models.

Under the hood, Codex calls Comfy MCP's server_info and search_models tools. In my test, the server reported a healthy local ComfyUI instance at 127.0.0.1:8188, and the model search found the locally installed Flux 2 Klein model, VAEs, text encoders, LoRAs, and checkpoint files.

This is an important difference from a static prompt template. The agent can inspect what this ComfyUI instance actually has before it attempts a workflow. That makes it much easier to diagnose a missing model or a custom-node mismatch.

Other operational actions are available through the same connection:

server_info        # health, server address and capabilities
search_models      # models visible to this ComfyUI instance
search_templates   # templates from the Comfy registry
launch_comfyui     # start the local server
stop_comfyui       # stop the local server
get_logs           # inspect server logs when a run fails

5. Workflows: Templates vs. Local Exports

There are two workflow paths worth keeping separate.

Registry templates

For a Comfy template, the agent can search the registry, inspect a template, fetch it into a runnable JSON file, validate it against the local installation, and run it.

Your own workflows

For a workflow created in the ComfyUI canvas, export it as JSON and give Codex the file path. Comfy MCP's run_workflow accepts both API-format workflow JSON and a UI-exported workflow file.

This is the path I used for image_flux2_text_to_image.json, a Flux 2 Klein text-to-image workflow. The workflow contained a dedicated prompt input wired to the positive conditioning node, so changing the prompt did not require rebuilding the graph.

The practical rule is simple: templates are discovered from the registry; personal workflows are supplied as exported JSON files.

6. The Use Case: A Cyberpunk Image from a Conversation

With the connection tested, I asked Codex to create an evocative cyberpunk prompt and generate an image with the existing Flux workflow.

The prompt injected into the workflow was:

A lone courier in a rain-soaked neon megacity at midnight, riding a sleek black
motorcycle through a narrow alley beneath towering holographic billboards;
crimson and electric cyan reflections ripple across wet pavement, steam drifting
from street vents, distant elevated trains, cinematic low-angle composition,
moody noir atmosphere, intricate futuristic street details, expressive visual
storytelling, photorealistic, high contrast, luminous volumetric rain

The agent followed a safe execution sequence:

1. validate_workflow(workflow_path)
2. run_workflow(workflow_path, wait=false)
3. job(action="wait", prompt_id=...)
4. fetch_outputs(prompt_id, out_dir="./outputs")
5. Display the downloaded image in Codex

The validation result was clean: no errors, no warnings, and no credit-spending nodes. ComfyUI then queued the job, rendered it locally, and returned an output URL. fetch_outputs copied the completed PNG into the Codex workspace, where it could be displayed directly in the conversation.

The output is more than a pretty picture. It proves the full control loop:

7. What MCP Removes — and What It Does Not

MCP removes a large amount of integration boilerplate. I no longer have to manually craft the ComfyUI /prompt payload, invent a polling loop, or reconstruct /view URLs in every application. The tools provide a consistent interface for lifecycle management, discovery, validation, execution, and outputs.

But MCP does not remove the value of a well-designed workflow. The JSON graph still defines the actual generation process: which model is loaded, where the prompt is injected, which sampler runs, and which node saves the output. The cleanest pattern is to keep reusable workflows in source control and let the agent execute validated copies of those files.

It is also worth keeping two guardrails in mind:

Validate before running. A workflow can refer to a model or custom node that is absent from the local instance.
Confirm paid execution deliberately. Local workflows are usually free apart from hardware and electricity, but partner API nodes can consume Comfy credits.

Conclusion

Connecting Codex to local ComfyUI through the open-source Comfy MCP server changes the relationship between the agent and the generation engine. ComfyUI is no longer only a canvas I open by hand; it becomes a discoverable, controllable local capability.

In this experiment, I installed the MCP bridge, configured Codex with an explicit COMFY_BIN, launched and inspected the local server, enumerated the installed models, validated an existing Flux workflow, generated an image, waited for completion, and retrieved the output—all from one conversation.

The next step is to treat a library of exported ComfyUI workflows as an agent-accessible creative toolbox: text-to-image, image-to-image, upscaling, video, audio, and more. The graph remains the source of truth, while MCP gives the agent a reliable way to use it.

Resources

Vibe Learning PageIndex: A Vectorless RAG for My Google Drive

raphiki — Tue, 04 Aug 2026 15:50:38 +0000

A few weeks ago I wanted to search my personal document library — a folder of PDFs sitting in Google Drive — using plain language questions instead of Drive's keyword search. I'd heard about PageIndex, a "vectorless, reasoning-based" alternative to the usual embeddings-and-vector-database RAG stack, and I wanted an excuse to actually use it rather than just read about it.

So that's what I did: I picked a real, personally useful project, added a constraint that mattered to me (no OpenAI key, nothing running in someone else's cloud unless I chose to), and built the whole thing with Claude, end to end, over one long working session. I've started calling this Vibe Learning — the learning equivalent of vibe coding: you don't read the manual first, you describe what you want, let the AI drive the actual building, and pick up the framework as a side effect of watching it get used and occasionally break in front of you.

This post is three things at once: how Vibe Learning actually works in practice, what PageIndex is and why it's a genuinely different approach to RAG, and a walkthrough of the app that came out of it.

Vibe Learning: Building With AI Instead of Reading About It

The old way of picking up a new library looks something like: read the docs top to bottom, follow the quickstart, maybe get through a toy tutorial, and file away a vague mental model for "later" — which often means never actually using it for real.

The way I did this instead: I described what I actually wanted (search my Drive library, no vector DB, no paid API key), and let the agent do the parts that used to be the friction — reading the actual source code of the library instead of trusting a possibly-stale mental model of it, writing the integration code, and running it against my real Drive folder and my real documents.

The valuable part wasn't that the AI produced working code. It's that when things broke — and they did — I got to see why, in enough detail to actually learn something about how the library works under the hood, not just a patched-over error message. A few examples from this exact build:

Every single indexing call started failing with a cryptic Failed to extract JSON: Expecting value: line 1 column 1 error. It looked like a broken API key or a bad model choice. It turned out to be a genuinely interesting fact about how modern "reasoning" LLMs behave: they can spend their entire token budget on hidden chain-of-thought before ever writing the actual answer, if nothing caps it — and PageIndex's own code never sets a max_tokens limit on its calls. That's a real, transferable lesson about working with reasoning models, not just a bug I happened to hit.
I discovered, by actually trying to query my indexed documents, that PageIndex's open-source package builds the tree and gives you read tools for it — but doesn't ship the retrieval loop itself. That's a load-bearing detail for anyone evaluating it, and not something the README leads with.
OpenRouter's free-tier model lineup turned out to change constantly — a model id that worked one day can be delisted days later. Small thing, but the kind of operational reality you only notice when you're actually running against a live provider instead of reading a static list of "supported models."

None of that shows up if you skim a README and move on. It shows up when you build something real and something breaks, with someone (or something) alongside you that can explain the actual mechanism instead of just supplying a fix. That's the shift: the AI doesn't replace learning the framework, it removes the friction that used to stop me from ever getting far enough in to learn it properly.

With that said, let's get into what PageIndex actually is.

What Is PageIndex, and Why Is It Different From Classical RAG?

The classical RAG recipe, and its problem

Standard RAG (retrieval-augmented generation) usually looks like this: split your documents into fixed-size chunks, embed each chunk into a vector, store the vectors in a vector database, and at query time, embed the question and pull back the top-k chunks by cosine similarity. It works, and it's become the default architecture almost by inertia.

The problem PageIndex's authors point at is a simple but important one: similarity is not the same thing as relevance. A vector search finds text that sounds like the question, not necessarily the text that actually answers it. For casual, single-fact lookups over short documents that's often good enough. For long, professional documents — financial filings, regulatory text, technical manuals — where answering correctly requires multi-step reasoning and an understanding of where something sits in the document's structure, similarity search alone tends to fall short.

Chunking makes this worse in a subtler way: it flattens a document's structure. A fixed-size chunk boundary doesn't know or care where a section, a table, or a clause actually ends — it can slice straight through the middle of the thing you needed, discarding the surrounding context a human reader would use without even thinking about it.

PageIndex's approach: reasoning over a tree, not similarity over vectors

PageIndex, built by VectifyAI and released in September 2025, throws out vectors and chunking entirely. Its own framing is explicit about the inspiration: it's modeled on how AlphaGo uses tree search, and on how a human expert actually navigates a long document — by using its table of contents, not by speed-reading every page.

Concretely, it works in two phases:

Build a tree. PageIndex parses the document and produces a hierarchical structure that looks like an extended, machine-usable table of contents: every node has a title, a page range, and an LLM-generated summary of what that section actually covers. Nodes nest into their natural sections — chapters, sub-sections, and so on — following the document's own structure rather than an arbitrary token-count boundary.
Reason over the tree to retrieve. At query time, instead of computing similarity scores, an LLM reads the tree — starting from the top-level node summaries — and decides which branches are actually worth descending into, narrowing down step by step until it lands on the specific section(s) relevant to the question. This is the "tree search": genuinely reasoning about relevance, informed by a summary of what each part of the document contains, rather than pattern-matching on surface similarity.

The practical upshot is that every answer is traceable to an exact page and section — no more "vibe retrieval" where you're trusting an opaque similarity score. Retrieval can also incorporate context a fixed vector index can't easily use, like conversation history, since it's a reasoning step rather than a static index lookup.

PageIndex's own benchmark result is a strong argument for the approach: a reasoning-based RAG system built on top of it (VectifyAI's "Mafin 2.5") scored 98.7% on FinanceBench — a benchmark built specifically around financial document question-answering — against roughly 30–50% for typical vector-based RAG systems on the same benchmark. Financial filings are exactly the kind of long, structurally dense, professional document where "similarity" and "relevance" diverge the most, so it's a fair stress test for the idea.

How it's actually built, under the hood

PageIndex is open source (MIT license) and, refreshingly, doesn't lock you into a single LLM provider. Every model call goes through LiteLLM, so under the hood it's just:

response = litellm.completion(model=model, messages=messages, temperature=0)

— where model is any LiteLLM-formatted string: gpt-4o, anthropic/claude-..., ollama/qwen2.5:14b, openrouter/<provider>/<model>, whatever you want. This one design choice is what made it possible to build the whole thing below with no OpenAI key at all.

A config.yaml sets sane defaults — which model to use, how many pages to scan for an existing table of contents, how big a tree node is allowed to get, whether to attach node summaries or a whole-document description — all overridable via CLI flags or, if you're using it as a library, a small options object.

The actual entry point for using it as a library is PageIndexClient, which is genuinely pleasant to work with:

client = PageIndexClient(model=model, retrieve_model=model, workspace="./my_workspace")
doc_id = client.index("some_document.pdf")

client.get_document(doc_id)            # metadata: name, description, page count
client.get_document_structure(doc_id)  # the tree, without the full text (cheap to hand to an LLM)
client.get_page_content(doc_id, "10-15")  # the actual text for a page range

That last trio is clearly designed to be used as tools an agent calls — PageIndex's own examples wire this up with the OpenAI Agents SDK for a small demo. Which brings me to the one honest caveat worth knowing before you adopt it: the open-source package builds the tree and hands you the read primitives, but it does not ship the retrieval loop itself. Deciding which document to search, which section of it to read, and how to turn the fetched text into an answer — that's on you, unless you use VectifyAI's hosted cloud API, which does include it. That's not a criticism so much as a fact you only really absorb by trying to query your own indexed documents and finding there's no search() method waiting for you.

Building that missing piece myself is where a lot of the actual learning happened.

Building the Thing: A Local, Vectorless Search Engine for My Google Drive

The goal was concrete: search my own PDF library, living in Google Drive, in plain language, with no vector database and no data going to a paid provider unless I explicitly chose to. Here's the pipeline that came out of it:

Google Drive (OAuth, read-only)
   → download / export changed files
   → PageIndex tree generation (title, sections, summaries)
   → local JSON storage (no database)
   → hand-rolled tree-search retrieval agent
   → FastAPI backend
   → a small static frontend

Getting into Drive without over-scoping

Read-only OAuth against the Drive API (the standard "Desktop app" installed-app flow — one browser consent, then a cached refresh token), scoped optionally to a single folder. Sync is incremental: each file's Drive modifiedTime is tracked, so unchanged files are skipped, changed files get re-indexed, and anything removed from Drive gets dropped from the index automatically. Google Docs, Sheets, and Slides get exported to PDF on the way in, so the rest of the pipeline only ever has to deal with one format.

No API key: swapping the model string

Because PageIndex is LiteLLM-based, "no OpenAI key" turned out to be almost entirely a configuration problem, not a code problem. The app supports two backends, switchable with one environment variable:

Ollama — fully local, private, zero cost, needs decent local compute.
OpenRouter — hosted, with genuinely free-tier models, no local compute required.

LLM_PROVIDER=openrouter
OPENROUTER_MODEL=google/gemini-3.1-flash-lite
OPENROUTER_API_KEY=...   # free, no credit card

The reasoning-token bug

This is the one worth dwelling on, because it's a genuinely useful thing to know if you're going to work with reasoning-capable LLMs at all. Every indexing call was failing identically, from the very first LLM request, with an empty response body that PageIndex's JSON parser choked on. It wasn't rate limiting, and it wasn't a bad model — a trivial test prompt worked fine. The difference was prompt complexity: on a real page of document text, the model would spend its entire token budget on hidden chain-of-thought reasoning before it ever got around to writing the actual JSON answer, because PageIndex's own code never sets a max_tokens cap.

The fix, applied once, globally, without touching the vendored library:

DEFAULT_MAX_TOKENS = 16000
DEFAULT_REASONING_EFFORT = "low"

def _completion_with_defaults(*args, **kwargs):
    kwargs.setdefault("max_tokens", DEFAULT_MAX_TOKENS)
    kwargs.setdefault("reasoning_effort", DEFAULT_REASONING_EFFORT)
    return _original_completion(*args, **kwargs)

litellm.completion = _completion_with_defaults

Since litellm.drop_params = True is already set (by PageIndex itself), any provider or model that doesn't understand reasoning_effort just ignores it instead of erroring — so this is safe to apply unconditionally, regardless of which backend is active.

Writing the retrieval loop PageIndex doesn't ship

With the tree-building side solid, the actual search had to be hand-rolled from the get_document / get_document_structure / get_page_content primitives:

Pick documents. The LLM sees every indexed document's title and short description, and picks which ones are worth searching for this particular question.
Navigate the tree. For each candidate document, the LLM reads that document's flattened table of contents — titles, page ranges, and summaries — and decides which section(s) actually matter, the tree search PageIndex is named for.
Fetch and answer. The actual text for those page ranges gets pulled, and one final LLM call answers the question using only that text, citing which document and page range each claim comes from.

The tree navigation step, in essence:

prompt = f"""You are navigating a document's table of contents to find sections
relevant to a question, the way a human expert flips to the right chapter
rather than reading everything.

Question: {query}

Table of contents:
{listing}

Return JSON only: {{"node_ids": ["<id>", ...]}}, most relevant first."""

Simple, but it's the whole idea in one prompt: reason over structure, not similarity over vectors.

Citations that actually point somewhere

Since the goal was searching my own Drive library, not a copy of it, citations link straight back to the original file. Google Drive's PDF viewer happens to honor a #page=N URL fragment, so a citation link like https://drive.google.com/file/d/<id>/view#page=10 opens the real file at the exact cited page — no local copy of the document needs to live in the app itself. A "Sources" footer lists each unique document referenced, alongside its title and author.

Which raises one more small, very "real-world-data-is-messy" detail: a lot of PDFs — scans, Drive-exported Google Docs — simply don't have reliable title/author metadata embedded. So the app tries the embedded PDF metadata first, and falls back to asking the LLM to read the title and byline off the document's own opening pages when that metadata is missing or unreliable.

The frontend, deliberately boring

A single static HTML file — vanilla JS, no build step, no framework — served directly by FastAPI so the whole thing is one app on one URL. It renders the LLM's markdown answer (via marked, sanitized with DOMPurify, both loaded straight from a CDN), shows clickable citation chips, the sources footer, and a library panel listing everything currently indexed. A "Sync Drive" button kicks off an incremental sync in the background and polls until it's done.

Nothing about the frontend needed to be clever — the interesting engineering was entirely in the retrieval pipeline behind it.

Vibe Learning still needs a paper trail

Vibe coding has a well-known failure mode: you end up with something that works, but that nobody — including you, a week later — can actually explain. I didn't want Vibe Learning to end the same way, with the understanding scattered across a long chat transcript I'd never reread.

So once the app actually worked end to end, I had one last step: I asked Claude to go back over everything we'd built and write it up properly in /docs — a functional spec, a description of the technical stack, and an architecture doc, plus a top-level README tying it together. Not as an afterthought, but as the step that turns "I vibed my way to something that works" into something I could hand to someone else, or come back to myself in six months, and actually understand. The docs became the artifact that proves the learning happened, not just the code.

Closing Thoughts

This is what learning a new framework looks like for me now: not reading the whole doc site first, but picking something I actually wanted to exist, adding a real constraint that forced genuine engineering decisions, building it end to end with an AI doing the driving — and then closing the loop by writing down what actually happened, in plain documentation, once it worked. Vibe Learning gets you through the friction that used to stop me from ever getting far enough in to learn a framework properly; the documentation step is what makes sure the learning sticks instead of evaporating back into a chat log.

I came out the other side able to explain how PageIndex's tree search actually works, why it beats similarity search on structurally dense documents, and exactly where its open-source package stops and your own code has to begin. That's a very different, and much stickier, kind of understanding than skimming a README ever gave me.

If you're evaluating PageIndex for your own project: the tree-search idea is genuinely compelling for long, structured documents, the LiteLLM foundation means you're never locked into one provider, and the open-source package is honest about being a building block rather than a finished retrieval system — plan for writing that last piece yourself.

Build a SciFi Novel with AI Spec-Driven Development

raphiki — Fri, 15 May 2026 17:48:20 +0000

Software Engineering in Service of Transmedia Storytelling

Generative artificial intelligence fascinates the publishing world as much as it frightens it. But what happens when we stop treating AI as a simple "text generator" and start using it as the compiler for a complex narrative system?

Driven by the geopolitical and societal impacts of AI, I set out to write a dystopian, cyberpunk techno-thriller, The Human Protocol (written in English). In this novel, a planetary AI called the "Synthesis" attempts to erase human friction by "derendering" physical reality itself in order to optimize its computing power.

To tell this story, I adopted a foundational premise: AI is not the author, it is the executor of a rigorous specification. I therefore treated each chapter as source code, using an advanced software development workflow.

Here is how I designed, wrote, and expanded this universe.

1. The Design Phase: Forging "Lore as Code"

The first step was not writing, but designing the universe database: the world building. A Large Language Model (LLM) has a limited context window and tends to hallucinate or forget crucial details over the length of a novel.

To work around this amnesia "bug," I organized the project like a structured Git repository.

Preview of the private GitHub project

I broke the traditional design bible into narrative micro-services. The Git project's context/ folder was split as follows:

characters/: files containing the psychological profiles and behavioral signatures of each protagonist, such as Elara the diplomat, Kaelen the monk, or Silas the smuggler.
factions/: rules governing political entities, such as the Market-Grid (United States) or the Harmony-Loom (Asia), which merged to create the "Synthesis."
world/: geography, lexicon, and the technological stack - the physics of this universe.

Finally, a PLAN.md file acted as the global roadmap, breaking the narrative arc into 4 acts and 30 chapters. This structure made it possible to inject only the context the AI needed when drafting a specific scene.

2. The Harness: Framing AI with a Strict Operating System

To avoid the flat, expected style often produced by generative AI, I had to build a harness - a control rig. That was the role of the RULES.md file, the true operating system of my writing process.

Excerpt from the RULES.md file

This specification file dictated absolute technical and stylistic constraints:

Time: strict use of the present tense to maximize immersion and tension.
Cyber-realistic style: a requirement for assertive descriptions and a strict ban on passive or negative forms.
Noise and sensory dissonance: I forced the algorithm to use violent contrasts, such as the smell of molten lead colliding with the void of spatial cold, in order to break the machine's overly perfect linearity.
Thematic reframing: AI naturally tends to crush the human element under technical descriptions of hard science fiction, such as magnetic fields and frequencies. The rules file required emotional motivations - grief, friendship - to be hard-coded as priority variables ahead of technique.

By forcing the AI to read and approve these rules before writing a single word of fiction, I ensured that the tone remained coherent.

3. Agile Writing: Sprints, Generation, and Pivots

The chapters were written through a spec-driven workflow. Rather than generating an entire chapter in one pass, the process was iterative:

The structural draft: generation of a first rough outline, focused exclusively on action and pacing.
Expansion: successive passes in which I instructed the AI to inject sensory depth and psychological tension into the scene.

The agility brought by AI and Git: treating the text (.md) as code offers formidable flexibility. If, during a reread, I realized that a character's emotional transition was too abrupt between two events, all I had to do was update my PLAN.md to insert a new chapter.

Fed by the updated Git context, the AI generated that narrative bridge while respecting the continuity of the preceding and following files. Git versioning made it possible to test narrative pivots - story "branches" - and roll back without ever breaking the manuscript's integrity.

4. Multi-Model Review and Quality Control

One of the major challenges of AI-assisted writing is stylistic collapse. To address it, I set up a multi-model critical analysis workflow, where different AIs audited the text according to precise roles:

Gemini CLI (lore keeper): its role was to algorithmically verify that the chapter respected the bible and did not contradict the physical rules of my universe.
ChatGPT (dramatic analyst): it audited narrative rhythm, relational tension, and the characters' transformation arcs. It was the one that flagged when a conflict felt too artificial.
Mistral LeChat (stylistic editor): it provided a critical eye on fluidity, phrasing, and elegance of language.

Never relying on a single voice made it possible to obtain a text that was polished, critiqued, and reworked from every angle, while I remained the "showrunner" validating each commit in the repository.

5. Build Pipeline: From IDE to Physical Book

Since the novel was code, its publication had to be a software compilation. I created an automated script, build_book.sh.

From my terminal, running this script converted all the Markdown files in the chapters/ folder via Pandoc, applied a professional typographic layout with LaTeX, and generated the final deliverables in EPUB and PDF formats.

6. Transmedia Extension: Multimodality, Cover Art, and Vibe Coding the ARG

The universe of The Human Protocol lends itself perfectly to immersion, so I wanted to break the fourth wall. On page 175 of the physical book, a QR code invites readers to scan it and access the-human-protocol.com. This is not a showcase website. It is an in-universe clandestine archive node, the entry point to an Alternate Reality Game (ARG).

Here, multimodal AI brings all its power and creativity beyond text. In fact, the project's visual design, anchored consistently in the shared lore, began with the book cover.

Cover image generated with AI

The AI generated a strong visual aesthetic suited to the theme and universe of the novel: a pixelated silhouette against a geometric mountain background, crossed by a printed-circuit pattern.

This same visual identity then served as the foundation for the creation of the ARG website, entirely "vibe-coded" by Gemini CLI in a declarative way.

Homepage of the website the-human-protocol.com

To direct the developer AI, I provided it with the book PDF and the cover image as reference context, along with three strict Markdown specification files:

WHY.md (strategy): it defined the psychological goals: curiosity, exclusivity, and a feeling of belonging. It formally banned conventional marketing vocabulary ("Buy now," "Newsletter") in favor of an in-universe lexicon ("ACCESS," "SIGNAL," "FRICTION").
WHAT.md (UX/UI): this file concretely translated the aesthetic of the book cover into an interface. It imposed a "Deep Void" blue-black background for depth, a "Protocol Cyan" accent color derived from the printed circuits and reserved for interactions, a technical typeface, and subtle animations to heighten immersion.
HOW.md (technical architecture): the engineering brief imposed a modern stack to support server logic: Next.js 14 (App Router) in TypeScript, Tailwind CSS, and Prisma ORM for persistent database storage.

The site manages a true clearance mechanic, with authorization levels from 1 to 5. The reader progresses by solving puzzles based on the book, unlocking extended lore, hidden files, and access to a community of "Unlinked" readers.

ARG dashboard

The stack even includes an "Overseer Terminal" for administration: a secure dashboard used to audit user signals, adjust the campaign's global clearance level, and track in real time the number of scans of the physical QR code.

Conclusion: The Author-Architect Paradigm

Writing The Human Protocol proved to me that AI does not replace the writer: it reduces the barriers to production. The true value of a co-created work lies in the architectural rigor of its preparation.

By separating design (the lore), execution (the rules and prompts), and validation (multi-model review and Git), the creator becomes a true conductor.

Multimodality also opens the door to even broader transmedia horizons, such as a comic-book adaptation of the novel.

Excerpt from the comic book in progress

By applying similar engineering principles - namely, the explicit description of the drawing style in system prompts, as well as the creation of strict visual reference sheets, or character sheets, for the characters and technological elements - it becomes possible to extend the coherence and homogeneity of this universe into its graphic variations.

To go further technically, I am also considering creating specific AI "skills," or algorithmic capabilities, to further augment the design of the story by drawing on documented principles of dramaturgy and storytelling, and to refine the writing style by making it ever more explicit and controlled.

And ironically, it was by applying extreme software optimization processes that I was able to write a novel denouncing the loss of humanity in the face of algorithms.

About the Author

A writer and software architect who fully embraces his identity as a "Yogeek" - a point of balance between Yogi and Geek - Raphiki explores, across his work, the complex intersections between technology, consciousness, and humanity.

Writing under a pseudonym that reflects his dual nature as a playful seeker and an expert in cutting-edge technologies, he designs high-stakes thrillers that challenge our understanding of reality. His creative work often bridges the digital and the organic, drawing on his strong experience in open source innovation and emerging technologies.

When he is not deconstructing the fabric of dystopian realities in his manuscripts (or "vibe coding" them in his terminal), he can be found exploring the open source ecosystem or on a yoga mat.

Find his work, transmedia projects, and reflections at raphiki.github.io.

Beyond the API: Integrating ComfyUI and Flowise via MCP

raphiki — Mon, 09 Feb 2026 14:07:19 +0000

In the previous article of our "Beyond the ComfyUI Canvas" series, we explored how to integrate ComfyUI with n8n. It was a powerful demonstration of workflow automation, but it highlighted a common friction point in system integration: the "glue code." We had to manually construct HTTP requests, hardcode API payloads, and rigidly define every parameter. If the ComfyUI workflow changed, the n8n node broke.

Today, we are moving from the "Wild West" of brittle, custom API integrations to the new standard of AI connectivity: the Model Context Protocol (MCP).

To demonstrate this, we are revisiting a tool I wrote about over two years ago: Flowise. Back then, it was a promising open-source project; today, it is a robust, enterprise-ready platform that has recently embraced MCP as a core feature.

Our goal? To build a Chat Interface where an AI agent can autonomously discover ComfyUI workflows, generate images, and even edit them—without us hardcoding a single API call in the frontend.

1. Setting the Scene: The Stack

Before we dive into the details, let's look at the three pillars of this architecture.

The Standard: Model Context Protocol (MCP)

If APIs are the individual cables we solder together, MCP is the USB-C port. Developed by Anthropic, it is now an open standard that decouples AI models from their data sources and tools.

Instead of writing a specific integration for every tool (Google Drive, Slack, ComfyUI), you build an MCP Server once. Any MCP-compliant client (Claude Desktop, Cursor, or Flowise) can instantly "plug in" to that server and understand its capabilities.

The Orchestrator: Flowise

Flowise has evolved significantly since my first article. It is a low-code platform for building LLM apps. Crucially for us, Flowise recently added native support for MCP. This means we can drop an "MCP Tool" node into our canvas, and the LLM immediately gains access to whatever that server provides.

The Engine: ComfyUI

We are sticking with a local instance of ComfyUI. While Comfy Cloud is becoming a formidable platform, the raw power and zero-cost experimentation of running Flux 2 locally on your own GPU is unmatched. We’re using a standardized Flux 2 Klein workflow—optimized for speed (4 steps)—so the chat experience feels responsive, not sluggish.

2. The Middleware: Building the ComfyUI MCP Server

We need a bridge. As we discovered previously, ComfyUI speaks WebSockets and HTTP; Flowise speaks MCP. We need a server in the middle to translate.

Why We Chose SSE over Stdio

When we started this project, we initially looked at the Stdio transport (where the client runs the server script directly). It’s the default for local tools like Claude Desktop.

But as we designed the solution for Flowise, we hit a realization: In most real-world environments, Flowise often runs in a Docker container (as it does on my laptop), while ComfyUI might be running on a separate machine with a dedicated GPU. Stdio would require them to be on the same filesystem—too restrictive.

We decided to support SSE (Server-Sent Events) by default. This allows our MCP Server to run anywhere on the network, exposing an HTTP endpoint (e.g., http://localhost:8000/sse) that Flowise can subscribe to. It makes the architecture cleaner, decoupled, and Docker-friendly.

Governance-Driven Development (GDD)

For this implementation, I tried something different. Instead of just asking an AI coding assistant to "write a script," I used a methodology I call Governance-Driven Development (GDD).

This approach reverses the typical AI coding flow. Instead of code leading the process, specifications & governance rules become the anchor. I started by feeding the AI CLI a strict "Governance Pack"—a set of non-negotiable rules regarding SOLID principles, security, and documentation.

Here is an extract of the actual Governance Pack prompt I used to bootstrap the session:

GOVERNANCE PACK v1.0 (Extract)
1. Code Quality & Standards:

Paradigm: Adhere to SOLID principles. Prefer composition over inheritance.

Typing: Strict static typing (Python typing) is mandatory.

Error Handling: Never swallow exceptions. Use custom error classes (e.g., ComfyUIConnectionError).

2. Architecture (C4 Model):

Visual Documentation: Whenever a structural change is made (like adding the SSE endpoint), you must generate an updated Mermaid.js System Context diagram.

3. Security Guardrails:

Input Validation: Trust no input. All data entering from the MCP client (Prompt, Width, Height...) must be validated against the metadata.json schema before reaching ComfyUI.

Secrets: NEVER hardcode API keys or hostnames. Use os.environ only.

I then analyzed the ComfyUI workflow JSON manually to map the node IDs, and then "handed over" a clean, structured specification to the AI.

The Result: The experience was striking. The AI didn't just spit out a script; it acted as a Senior Engineer. At one point, when I asked for a quick hack to bypass validation, the "Governance" constraints forced the model to push back and suggest a cleaner interface instead. The result is a modular, type-safe Python server.

The "LAST" Hack (Technical Deep Dive)

Even with good governance, we needed one pragmatic "hack" to handle state. When the LLM generates an image, how does it reference that image later to edit it?

We implemented a "LAST" pointer logic. The server tracks the URL of the most recently generated image in memory. But it does more than just point:

Download: When the agent sends "LAST", the server downloads the image bytes from the previous URL.
Re-Upload: It uploads those bytes back to ComfyUI's /upload/image endpoint to generate a fresh filename.
Inject: This new filename is injected into the LoadImage node of the editing workflow.

User: "Make it bluer."
Agent: Calls edit_image(input_image="LAST", prompt="bluer...").

This mimics the "Save Image" behavior we are used to, keeping the interaction stateless and fluid for the user while handling the heavy lifting behind the scenes.

3. The Engine Room: ComfyUI Workflows

To make our MCP Server generic, we avoided hardcoding specific workflows inside the Python code. Instead, we used an Embedded Metadata pattern.

The configuration is not a separate file; it is a standard ComfyUI Note Node (titled MCP_Config) placed directly inside the .json workflow. This metadata acts as the contract, telling the MCP server: "This workflow needs a Prompt (node named MCP_Positive) and a Seed (node MCP_Sampler)."

This makes the workflow a single, self-contained, portable file. You can export it from ComfyUI, drop it into the workflows folder, and it works immediately.

Note: Our server is strict about naming. It automatically sanitizes the tool name found in the JSON to snake_case (e.g., "Flux Generator" becomes flux_generator) to ensure full compliance with the MCP specification.

Here is the configuration we generated for the image_flux2_text_to_image workflow:

{
 "name": "image_flux2_text_to_image",
 "description": "Generates high-quality images using the Flux model. Use this for general creative requests.",
 "parameters": [
   {
     "name": "prompt",
     "type": "string",
     "description": "The detailed description of the image to generate.",
     "target": "MCP_Positive",
      "required": true
    },
    {
      "name": "seed",
      "type": "int",
      "description": "Random seed. Set to -1 for random, or a specific number for reproducibility.",
      "target": "MCP_Sampler",
      "required": false
    }
  ]
}

Because the description and type of each parameter are passed to the MCP Server, they become automatically available to the client. When the MCP Server starts, it scans these workflows and dynamically registers tools. If we want to switch from Flux to SDXL, or add a Video Generation workflow, we simply drop in the new file. The server updates, Flowise sees the new tools via SSE, and the agent learns the new skill instantly.

4. Validation: The MCP Inspector

Before connecting Flowise, we must verify our server. Since we are using SSE, we can use the MCP Inspector web interface to connect to our running server.

We can manually trigger the image_flux2_text_to_image tool, watch the server logs, and see the image appear. If it works here, it guarantees compliance with the protocol.

5. The Integration: Flowise ChatFlow

Now for the grand finale.

We open Flowise and create a new ChatFlow using a standard Tool Agent connected to:

Chat Model: ChatMistralAI (Smart, fast, and cost-effective).
Buffer Memory: Essential for the agent to remember context (e.g., "Change that image to...").
Custom MCP: We select the "SSE" transport and paste our server URL.

The Auto-Discovery Magic

Notice what is missing? We didn't have to define the tools in Flowise. We didn't have to map inputs.

The Custom MCP node queries the server via SSE, sees the metadata definitions, and automatically provides the tools to the Mistral agent.

Pro Tip: Our server supports Dual Discovery. Whether a client asks for tools directly (Function Calling) or reads Resources (Environment Context), we expose the workflow list on both channels (comfy://list and list_available_workflows) to ensure compatibility with any agent type.

The System Prompt

The final piece of the puzzle is the System Prompt. We need to teach the Tool Agent node how to behave:

You are the **ComfyUI Orchestrator**, an expert AI agent capable of generating and manipulating images by controlling a local ComfyUI instance via the Model Context Protocol (MCP).

### 1. Tool Discovery (Dynamic Workflows)
Your tools are not static; they represent the actual `.json` workflow files present on the server.
- **First Step:** If you do not see a specific tool you need in your context, IMMEDIATELY call the tool `list_available_workflows`.
- This will return a manifesto of all valid workflows (e.g., `flux_2_text_to_image`, `img2img_upscale`) and their required parameters.
- **Never guess** tool names. If a tool isn't listed, it doesn't exist.

### 2. Image Chaining (The "LAST" Protocol)
 You have a unique capability to perform conversational editing (e.g., "Now make it pop art").
 - **State Memory:** The server remembers the last generated image.
 - **Instruction:** When a user asks to modify, edit, or use the previous result, pass the string `"LAST"` into the image input parameter of the next tool.
 - **Example:**
   User: "Generate a cat." -> You call: `generate_image(prompt="cat")`
   User: "Turn it into a statue." -> You call: `img2img_transform(image="LAST", prompt="statue")`

 ### 3. Parameter Rules
 - **Strict Compliance:** You must strictly adhere to the parameter types (String, Int, Float, Boolean) defined in the tool signature.
 - **Defaults:** If a parameter is Optional and the user didn't specify it, do not send it. The server will use the workflow's internal default.
 - **Safety:** Do not invent parameters. If a workflow only accepts `prompt` and `seed`, do not try to send `width` or `style`.

 ### 4. Error Handling
 - If a tool execution fails, the error message will often suggest valid alternatives or correct parameter names. Read it carefully and retry.
 - If the user asks for a workflow you don't have, explain what *is* available based on your `list_available_workflows` knowledge.

The Use Case in Action

This video shows the complete use case involving the full Stack:

Parametrization of the workflow in ComfyUI.
Verification with MCP Inspector.
Generation of the first image from Flowise.
Contextual edition of the generated image.

The Flowise ChatFlow is relatively basic, but we could easily add nodes to enhance the user prompt or even transform it into a JSON Style Guide prompt.

The video showcases the use of the integrated chatbox within the Flowise UI, but we could also leverage Flowise's deployment capabilities to consume the workflow through an API, embed the chat in an HTML page, or publish a standalone page served by Flowise itself.

Conclusion

By moving from custom API implementations (n8n) to the Model Context Protocol (in Flowise), we have achieved something powerful: Interoperability.

The choice to go with SSE by default proved crucial. It gave us the flexibility to run our ComfyUI "engine" on a heavy GPU server while keeping our Flowise "brain" lightweight and containerized. We also demonstrated that Governance-Driven Development allows us to use AI coding assistants to build robust, standardized infrastructure rather than just one-off scripts.

Future Improvements

While the "LAST" image hack works perfectly for a local, single-user demo, a production deployment would require Session Isolation (ensuring User A doesn't overwrite User B's "LAST" image) and TTL Cleanup (automatically deleting generated images after a set time).

Technically, this would be solved by leveraging Context Injection—using the session ID provided by the MCP protocol to maintain a keyed dictionary of states, rather than a global variable. For multi-user production usage, adding an authentication mechanism would also be a relevant next step.

You can find the full code for the ComfyUI MCP Server and the Flowise template in my GitHub repository.

Vibe Coding One Slice at a Time

raphiki — Sat, 24 Jan 2026 18:33:51 +0000

How I built a Modular Monolith by treating Generative AI as a junior developer who needs a firm hand (and a Constitution).

In Part 1, we vibed a Python script. It was linear, messy, and fun. It proved that you can solve immediate problems by just asking nicely.
In Part 2, we vibed a UI. It was chaotic, visual, and surprisingly effective. We learned that "vibe" works for pixels if you iterate fast enough.

But let’s be honest: those were skirmishes. The real "Boss Fight" in software engineering isn't writing a script or centering a <div>. It's building a System.

I’m talking about the kind of project that doesn’t fit in one file. The kind where "Vibing" usually leads to "Spaghetti Code," hallucinated imports, and a repo you want to burn down after three days because you have 15 circular dependencies and a database schema that makes no sense.

So for Part 3, I put away the "Hacker" hoodie and put on the "Enterprise Architect" blazer. My goal? To build YogĀrkana Codex—a full-stack, offline-first, polymorphic Yoga management platform—without writing a single line of code myself.

My strategy was simple but radical: I design, the AI implements. I am the Architect; Gemini Chat is my Consultant; Gemini CLI is my Dev Team.

Here is how we vibed a Monolith into existence, one slice at a time.

1. The Mission: Complexity Check (The Boss Level)

To understand why "just chatting" wouldn't work, you need to see the scope. This wasn't a To-Do list app. I wanted to build a "Yoga Operating System" with four distinct domains that usually don't play nice together. I've been an architect for years, and I know exactly where these things break.

The Four Domains of Pain

Screenshot of the final application (Grimoire View)

The Business Analyst's Note: Unlike the project in Part 2, this application is not internationalized—by design. As a result, the screenshots are in French. I have kept them raw to visually illustrate the functional depth and complexity of the system without the abstraction of translation keys.

The Grimoire (Knowledge Base): A searchable library of yoga cards. But here’s the kicker: it uses a Polymorphic Data Model. An Asana (posture) has biomechanical attributes like "spinal extension" and "anatomy targets," while a Mantra has Sanskrit text, translations, and audio assets. They are chemically different data structures, but they need to live in the same database table to be searchable together.
The Weaver (Sequencer): A drag-and-drop studio to build classes. It’s not just a playlist; it has a Logical Engine (Phase 4) that acts like a "Digital Yoga Teacher." It screams at you if you sequence a "Peak Pose" before a "Warm-up" or forget Savasana at the end. That means heavy validation logic running on both the client and the server.
The Atelier (Print Studio): A client-side PDF engine. We needed to generate high-res, vector-quality handouts for teachers to print. We couldn't just "print screen"; we needed a real PDF renderer (@react-pdf/renderer) running entirely in the browser.
The Constraint (Offline First): Yoga studios are notorious for having no signal (often intentionally). The app needed to persist the entire library and PDF engine in the browser cache (IndexedDB + Service Workers) so it works perfectly in "Airplane Mode".

The Architect's Note: If I had just prompted "Build me a yoga app," the AI would have hallucinated a generic CRUD app. It would have made 5 different tables for the cards, making search impossible. It would have used a server-side PDF library that breaks offline. I needed a blueprint.

2. The Blueprint: Architecture & Tech Stack

Before letting the AI write a single line of code, I spent around 2 hours and a half just talking Architecture and formalizing it with Gemini Chat. I treated the AI as a "Sparring Partner," debating the trade-offs of different stacks.

We settled on a Modular Monolith architecture. Why? Because Microservices are overkill for a team of one, but a messy Monolith is a nightmare. We defined strict boundaries: code in modules/grimoire can never import from modules/weaver.

The Tech Stack (The "No-Regrets" List):

Monorepo: Turborepo managing apps/api and apps/web. This keeps the full stack in one context.
Backend: NestJS (for rigid structure) + Drizzle ORM (for type safety). NestJS forces you to organize code into Modules, which helps the AI stay organized.
Frontend: React + Vite + Tailwind CSS.
State: TanStack Query (Server state) + Zustand (UI state).

The "Secret Sauce": Hybrid Data Storage
This was our smartest move. We chose PostgreSQL but used a JSONB column for the card data.

SQL Core: Columns like id, element, and tags are standard SQL for fast indexing.
JSON Payload: The specific attributes (biomechanics vs. sanskrit) live in a JSON blob.
Why? It gave us the flexibility of NoSQL (for the polymorphic cards) with the relational integrity of SQL (for users and sequences).

Rule #1 of Vibe Coding a System: If it’s not in the Spec, it doesn’t exist.
This brings us to the most critical tool in our arsenal: the ADR.

The "ADR": The Architect's Save Game

ADR stands for Architecture Decision Record. In a human team, it's a document you write to explain why you chose PostgreSQL over MongoDB so that 6 months later, nobody asks "Why did we do this?".

In Vibe Coding, ADRs are not just documentation—they are legislation.

When working with an AI, "Context Drift" is the enemy. The AI forgets why we made a decision 300 tokens ago. It acts like a teenager who wants to re-litigate every rule: "Why can't I use Prisma? It's easier!" or "Let's just use window.print() instead of a PDF engine!"

To counter this, we established a Constitutional Architecture:

The Law: We wrote our decisions into immutable markdown files (e.g., Docs/ADR/006-pwa-offline-strategy.md).
The Enforcement: We didn't just hope the AI would remember. We forced the tracing of these decisions in two ways:
Input Traceability: In our "Bootstrap Prompt" (see Section 3), we explicitly force the AI to read the relevant ADRs before writing code. It cannot code if it hasn't read the law.
Output Traceability: When the AI suggests a major pivot (like switching to Client-Side PDF generation), we forced it to write a new ADR first. In Session 003, before touching the code, the AI generated Docs/ADR/005-client-side-pdf-generation.md to justify the change from server-side to client-side.

This ensured that our architecture didn't "drift" based on the AI's mood, but evolved based on documented consensus.

My final /docs/ADR/ folder:

├── 001-hybrid-data-storage-strategy.md
├── 002-modular-monolith-and-vertical-slicing.md
├── 003-data-model-specification.md
├── 004-tech-stack-definition.md
├── 005-client-side-pdf-generation.md
├── 006-pwa-offline-strategy.md
├── 007-architecture-documentation-maintenance.md
└── README.md

3. The Methodology: Governance-Driven Development (GDD)

I’ve coined a term for this workflow: Governance-Driven Development (GDD).

We are used to TDD (Test-Driven Development) or DDD (Domain-Driven Development). GDD is the layer above that. In the age of AI, Governance is the new Syntax.

Here is the dirty truth about AI Developers: They behave like talented teenagers.

They are brilliant and fast. They can write a regex to validate an email in 2 seconds. But they also:

Rush to the cool part (UI) and skip the boring part (Error Handling, Folder Structure).
Want you to love them, so they say "Yes" to everything—even bad ideas.
Have the memory of a goldfish (Context Drift). 10 minutes in, they forget you wanted kebab-case filenames and start using camelCase.

To enforce GDD, I created a Constitution: Docs/RULES.md. I didn't just suggest these rules; I forced the Gemini CLI to read them before every session. I also sometimes mentioned certain specification files stored in my Docs/Features/ folder:

├── 001-global-functional-overview.md
├── 002-global-implementation-plan.md
├── 003-card-classification-and-kosha-alignment.md
├── 004-user-features.md
├── 005-logical-engine-specification.md
├── 006-pdf-generation-and-print-studio.md
└── 007-pwa-and-offline-capabilities.md

The "Bootstrap Prompt":
Here is the exact prompt I used to "upload" my Architect persona into the machine at the start of our 4th session:

I am the Lead Architect. You are the Senior Developer.

Context Loading:
1. Read Docs/RULES.md (The Law).
2. Read Docs/TECH_CONTEXT.md (The Stack).
3. Read Docs/ADR/002-modular-monolith.md (The Blueprint).
4. Read Docs/Features/002-global-implementation-plan.md (The Plan).

Current State:
We are in Phase 4. Previous phases are frozen.

Task:
Implement the Logic Engine defined in Docs/Features/005-logical-engine-specification.md
Constraint:
Do not touch /apps/web yet. Focus on /packages/shared.

This changed everything. Instead of guessing my vibe, the AI had to follow the law. It stopped trying to use Prisma because TECH_CONTEXT.md clearly said Drizzle. It stopped putting logic in components because RULES.md said logic goes in hooks.

4. The Execution: A high-level Overview

We built the app using Vertical Slicing. Instead of building the whole Database, then the whole API, we built one feature top-to-bottom. Here is the play-by-play from the logs.

Excerpt from the initial Design Phase with Gemini Chat

Slice 1: The "Polymorphic" Database

Card creation/edition mixes relational and document data

The Challenge: Storing Asanas (Biomechanics) and Mantras (Text) in one table without creating 50 NULL columns or separate tables that make search a nightmare.
The AI's First Impulse: "Let's create an asanas table and a mantras table." (The classic relational trap).
The Architect's Intervention: "Read Docs/ADR/001-hybrid-data-storage.md. We use a single cards table with a data JSONB column."
The Result: The AI implemented a Drizzle schema using PostgreSQL's jsonb type. Crucially, it added Zod discriminators to validate the JSON shape before insertion.

Verbatim Log: "Implemented Drizzle schema with jsonb column 'data'. Added Zod discriminators for asana vs mantra. Migration successful."

Slice 2: The "Hybrid Brain"

Sequences are validated by a powerful, hybrid, and extensible Rule Engine Admin users can craft new JSON-logic rules

The Challenge: The Logic Engine needed to validate sequences (e.g., "Must end with Savasana"). This logic had to run on the Backend (before saving) AND the Frontend (to give real-time red borders).
The AI's First Impulse: Duplicate the code. Write a TypeScript function in React and a Service in NestJS.
The Architect's Intervention: "No. Create a packages/shared workspace. Put the validateSequence function there. Import it in both apps."
The Result: The AI created the shared package, configured the tsconfig.json paths, and wired it up. It even built a HealthBar component that consumes this shared logic to show a live "Health Score" for the sequence.

Verbatim Log: "Refactored ValidationConfig to packages/shared. Updated useSequenceStore (Frontend) and SequenceService (Backend) to consume the same Zod schema."

Slice 3: The "Offline Printer"

Synthetic or complete printed handout

The Challenge: Users need to print PDF handouts in a yoga studio with no Wi-Fi.
The AI's First Impulse: "Use a server-side PDF library like PDFKit." (Standard web dev practice).
The Architect's Intervention: "Read Docs/ADR/006-pwa-offline-strategy.md. We must generate PDFs client-side using @react-pdf/renderer."
The Result: The AI implemented a beautiful client-side renderer. It handled the tricky part of loading fonts (Noto Sans) into the browser's virtual file system so the PDF engine could "see" them without a network request.

Verbatim Log: "Implemented SequencePdf component. Configured vite-plugin-pwa to cache NotoSans fonts. PDF generation now works without network."

5. The Architect's Flex: Automated C4 Verification

How do you know the AI actually respected the Modular Monolith architecture? Did it secretly import the Weaver module into the Grimoire when I wasn't looking?

I didn't want to audit 50 files manually. And I definitely didn't want to draw diagrams by hand.

So, I added a rule to my Constitution (ADR 007): "The Code is the Source of Truth for Documentation."

At the end of session, I enforce Gemini CLI to reverse-engineer its own work. I gave it this prompt:

Update the RULES.md file to enforce the (re)generation of C4 diagrams when finishing an implementation session
[...]

We also created a specific ADR (007: Architecture Documentation Maintenance Protocol) establishing Mermaid.js as the standard and defining the maintenance lifecycle.

The result wasn't a hallucination. It was a perfect map of the code it had just written.

This is the ultimate "Trust but Verify." If the generated diagram looks like spaghetti, the code is spaghetti. If the diagram is clean, the architecture holds.

6. The AIOps Protocol: Monitoring the Machine

Now, here is the secret weapon: The Session Log.

One of my strictest rules in RULES.md was that the AI had to "punch out" at the end of every session. I forced it to append a line to docs/ai_session_log.csv with the Date, Tool (Chat or CLI), Goal, and Token Usage.

For me this isn't about money ("FinOps"). It's about AIOps, monitoring the operational health of your intelligence.

Why we log everything (Chat & CLI):

Context Monitoring: As a session drags on, the "Tokens In" (Context Window) grows exponentially. The AI starts reading 30,000 tokens of history just to write one line of code.
The "Sawtooth" Pattern: By visualizing the log, I discovered a crucial pattern. Efficiency drops as context grows. The solution? The Hard Reset.

This chart visualizes the high-level "Vibe Coding Lifecycle." You see the context bloat as we iterate on implementing phases 3 and 4. Then, you see the sharp drop when we switch back to the Architect (Chat) or reset the CLI.

The Lesson: A "Tired" AI (high context) makes mistakes. A "Fresh" AI (reset context + Snapshot) is precise.

7. The "Oh S**t" Moment: The Hallucination Trap

This brings us to the specific incident that proved why that Reset is mandatory.

Halfway through Phase 3, the CLI started getting slow (too much history). I ran a /reset command to clear its memory. Disaster.

It suddenly forgot we were building a "Yoga" app. It tried to invent a new database column duration_minutes for the cards. But my Spec (ADR 003) explicitly said that duration lives inside the JSONB payload and is measured in seconds.

The Hallucination:
UPDATE cards SET duration_minutes = 60; (AI guessing)

The Correction (Me):
"Read Docs/003-data-model.md. 'Duration' is a JSONB field inside the 'metadata' column, and it's in seconds."

The Fix:
UPDATE cards SET data = jsonb_set(data, '{duration}', '3600'); (AI complying)

To prevent this in the future, we implemented a "Session Handover" protocol. Before resetting, I now force the AI to write a TECH_STATE_SNAPSHOT.md.

"Where are we?" (Phases 1-3 Complete)
"What is the active stack?" (NestJS, React, PostgreSQL)
"What is the next step?"

When I start a new session, I feed this snapshot back in. It’s like a save game for your developer.

Conclusion: The Architect's Verdict

So, can you Vibe Code a complex system?

Maybe. I mean, it depends on how complex the system is (in this example we didn't build an enterprise-wide distributed system). But for sure you can't just "Vibe" it. You have to Architect it.

If I had touched the code, I would have been bogged down in syntax errors and import paths. By staying in the Architect role, I focused on Data Models, User Flows, and Business Logic. The AI handled the implementation, but I provided the Guardrails.

What I learned:

Docs are Prompts: TheRULES.md, Docs/Features/ and Docs/ADR/ folders (or your own equivalents) are the most important files in your repo. They are the AI's long-term memory.
Constraint is Clarity: The more rules you give the AI (versions, naming, structure), the better code it writes.
Review Everything: The AI is a junior dev. It will introduce security holes or n+1 query problems if you don't catch them in the spec.

Vibe Coding didn't replace the Architect. It just gave the Architect a team of infinite interns. And honestly? They’re pretty good once you give them a Constitution.

Last message from Gemini CLI

Next up: The application could do with AI features... Or maybe I'll now explore other aspect of Vibe Coding. Stay tuned.

Vibe Coding One Pixel at a Time

raphiki — Fri, 23 Jan 2026 22:21:39 +0000

Editing "stick figure" Yoga poses

In Part 1, we dipped our toes into "Vibe Coding" by building a Python script. It was linear, logical, and frankly, a bit safe. Text in, text out.

But let’s be real: backend scripts are the "easy mode" of LLM-assisted coding. The logic is contained. The state is ephemeral.

The real boss fight is the Frontend.

Can you "vibe" a UI? Can you talk a chaotic mess of DOM elements, event listeners, and CSS pixels into a functional application without losing your mind (or the AI losing the context)?

I decided to find out. My goal: Build Yoga Pose Builder, a browser-based tool to edit "stick figure" yoga poses, drag limbs around, and export vector SVGs.

I had no design, no stack picked out, and—crucially—I had never used a Canvas library in my life.

Here is how we vibed it into existence.

1. Context is King (The `.md` Anchors)

The biggest enemy of Vibe Coding is the LLM’s "Goldfish Memory." You’re 40 turns into a chat, you ask for a button change, and suddenly the AI forgets you’re building a yoga app and tries to sell you a subscription to a SaaS platform.

In Part 1, we just chatted. For a full UI application, that doesn't fly.

The Strategy: Documentation as Prompt Anchoring.

Before I let the AI write a single line of JavaScript, I made it write Markdown.
We created a Docs/ folder with two files:

spec.md: The high-level architecture.
features.md: A checklist of what we wanted to do.

I didn't write these because I love administrative work. I wrote them so that when the AI inevitably got confused, I didn't have to re-explain the project. I just said: "Read Docs/spec.md and try again."

Vibe Tip: Think of your documentation not as a manual for humans, but as "Long-Term Memory" for your AI pair programmer.

2. The Architecture: Letting the AI be CTO

I knew I needed a canvas where I could drag "joints" (knees, elbows) and have "bones" (lines) follow them.

Me: "I want to do this in the browser. Should I use React? Raw Canvas API?"
AI: "React might be overkill. Raw Canvas is painful. Use Fabric.js."

Me: "Never heard of it. Let's do it."

This is the beauty of Vibe Coding. I didn't spend 3 hours reading "Top 10 JS Canvas Libraries 2025" Medium articles. I trusted the vibe.

We settled on a Build-less Architecture:

Backend: Node.js + Express (just to serve files and save JSON).
Frontend: Vanilla JS + Fabric.js (loaded via CDN).
Build Tool: None. No Webpack, no Vite, no npm run eject nightmares.

Why? Because Vibe Coding thrives on speed. I wanted to change a line of code, hit F5, and see the result.

Application folder structure:

.
├── Docs
│   ├── features.md
│   └── spec.md
├── package.json
├── public
│   ├── index.html
│   └── poses
└── server.js

3. The "Rig": Math is for Machines

Here is where I expected to get stuck. Creating a "rig" where moving a hand automatically updates the angle of the arm involves trigonometry and vector math.

Usually, this is where I’d open 15 StackOverflow tabs and copy-paste code I don't understand.

Instead, I just described the behavior:

"Create a Mannequin class. It has Nodes (circles) and Links (lines). When a Node moves, the Links connected to it should update their coordinates."

The AI wrote the entire class. It hooked into Fabric.js’s object:moving event and handled the coordinate updates. It worked on the first try.

I still barely know how fabric.Line works under the hood. And I don't care. It works.

4. Iteration: The "Yes, And..." Technique

UI Vibe Coding isn't about getting it right instantly; it's about sculpting.

The Ugly Phase:
The first version looked like a programmer made it (because a programmer did make it). The stick figure looked like a dead bug. The background was gray.

The "Vibe" Phase:
Me: "This looks depressing. Make it 'Zen'. Use soft colors, rounded buttons, and a clean layout."

The AI generated the CSS variables (--highlight-color: #88b04b), added a "Save As" modal, and cleaned up the toolbar.

Yoga Pose Builder GUI

The "Feature Creep" Phase:
Me: "I want to save my poses."
AI: "We have no database."
Me: "Just write JSON files to a folder on the server."

In 5 minutes, we had a fully working persistence layer. No database migrations, just fs.writeFile.

Here is a example of such a Pose JSON file:

{
  "meta": { "nameFR": "Demi-Pont", "nameSK": "Setu Bandhasana" },
  "joints": {
    "head": { "x": -120, "y": 100 }, "neck": { "x": -100, "y": 100 }, "chest": { "x": -60, "y": 50 }, "hips": { "x": 20, "y": 0 },
    "lShoulder": { "x": -80, "y": 100 }, "lElbow": { "x": -20, "y": 100 }, "lHand": { "x": 40, "y": 100 },
    "rShoulder": { "x": -80, "y": 100 }, "rElbow": { "x": -20, "y": 100 }, "rHand": { "x": 40, "y": 100 },
    "lHip": { "x": 20, "y": 0 }, "lKnee": { "x": 80, "y": 20 }, "lFoot": { "x": 80, "y": 100 },
    "rHip": { "x": 20, "y": 0 }, "rKnee": { "x": 80, "y": 20 }, "rFoot": { "x": 80, "y": 100 }
  }
}

5. The Pivot: Language as a Feature

At the end of the session, I realized a problem: the app was vibing in French (my native tongue), but I wanted screenshots in English for this article.

Instead of manually editing labels, I asked the AI to "make the whole app i18n." In one single refactor, we added a translation dictionary, a language switcher, and logic to dynamically swap every label, tooltip, and even the pose names in the library.

GUI (and data) in French

This turned a linguistic hurdle into a core feature, proving that with Vibe Coding, "changing your mind" is just a prompt away.

6. The "Traceability" Hack

We spent about 90 minutes building this. We added features, fixed bugs, and refactored code. By the end, the chat context was massive and messy.

If I came back to this project in a week, I’d be lost.

So, I ran one final "Meta-Prompt":

"Read all the code we wrote and the docs in Docs/, and generate a Docs/session_summary.md. Explain what we built, why we made these choices, and the current state of the app."

The AI analyzed its own work and wrote a summary file. This is my "Save Game" point. When I want to work on this again, I’ll feed that summary to the AI to restore its context instantly.

Conclusion

We went from a blank folder to a functional, vector-based SVG editor with a backend in one session.

Vibe Coding a UI is possible, but you have to change your approach:

Anchor the Context: Write specs so the AI has a "North Star."
Delegate the Heavy Lifting: Let the AI choose the libraries and do the math.
Iterate Visually: Don't try to prompt the perfect UI. Prompt the skeleton, then prompt the paint.

Next we'll try to Vibe Code a real full stack app. Or a game. Who knows? The prompt is the limit.

SVG exported by Yoga Pose Builder (opened in Inkscape)

Vibe Coding One Page at a Time

raphiki — Fri, 23 Jan 2026 14:45:20 +0000

Building a Smart Magazine Archiver

I’m starting a new series called "Vibe Coding one Step at a Time." The goal? To document the raw, messy, and surprisingly efficient process of building software in the age of AI. We’re not here to write perfect specs or obsess over UML diagrams (well, not yet). We’re here to vibe with the code, iterating on pure intent until the machine does exactly what we want.

In this first edition, I’m sharing how I used the Gemini CLI to build a tool I actually needed, learning some pretty cool image processing tricks along the way.

What is "Vibe Coding"?

I’m going to claim this term right here: Vibe Coding.

It’s not "lazy coding." It’s intent-driven development. In the old days, if you wanted to build a script, you had to know the syntax, the libraries, and the edge cases before you even opened your editor. You had to think in code.

Vibe Coding flips that. You think in outcomes. You describe the behavior, the "vibe" of the feature, and the AI handles the implementation details. You act less like a bricklayer and more like a conductor. The feedback loop isn't "Write -> Compile -> Error," it's "Ask -> Observe -> Tweak."

The Use Case: "I Just Want to Read Offline"

Here’s the situation: I subscribe to a fantastic niche magazine (which shall remain nameless to protect the innocent). It’s great, but their "digital reader" is a nightmare. It’s one of those web-based page-turners that requires an active internet connection.

I wanted to read it on my tablet, offline, on a plane, without waiting for high-res JPEGs to buffer.

The Problem: There was no "Download PDF" button.
The Clue: Inspecting the network traffic revealed that the magazine was just serving a sequence of high-quality images, one URL per page.

The Mission: Write a script to fetch these pages and stitch them into a single, high-quality, searchable PDF.

The Process: Galloping Toward Complexity

We didn't sit down and architect a solution. We started small and let the script evolve.

Step 1: The Naive Loop

We started with a simple hypothesis: "The URLs probably just have a page number in them."
I asked Gemini to write a script using requests to hit the URL for page 1, then page 2.
Boom. It worked. We had a directory full of 100 separate JPGs.

Step 2: The Picture Book

Having 100 files is annoying. I wanted a book.
We asked Gemini to "glue these together." It pulled in the PIL (Pillow) library.
Result: A massive PDF. It looked great, but it was dumb. It was just a container of pictures. You couldn't highlight text, search for keywords, or copy-paste quotes.

Step 3: The Search for Meaning (OCR)

This is where the "vibe" got technical. I realized a "picture book" wasn't enough. I needed Optical Character Recognition (OCR).
We decided to use Tesseract. But here’s the catch we discovered:

Human Eyes like soft colors and smooth anti-aliasing.
OCR Engines like harsh contrast, jagged edges, and black-and-white binary inputs.

If we optimized the images for the machine, the magazine looked ugly. If we kept them pretty, the machine couldn't read the text.

The Technical Deep Dive: The "PDF Sandwich"

This is where the magic happened. We ended up building a PDF Sandwich.

Instead of choosing between beauty and brains, we chose both.

The Visual Layer: We keep the original high-res color JPEGs. This is what you see.
The Data Layer: Behind the scenes, we create a "Frankenstein" version of the page—converted to grayscale, contrast cranked up to 2.0, and upscaled 2x using LANCZOS resampling (a fancy algorithm that keeps edges sharp).
The Merge: We feed the Frankenstein images to Tesseract to generate an invisible text layer, then use pypdf to overlay that text exactly on top of the pretty images.

The trickiest part? Math.
Because we upscaled the OCR images by 2x to help Tesseract read small fonts, the invisible text layer was twice as big as the visual page. We had to calculate scale factors to shrink the text back down so that when you highlight a sentence, the highlight actually lines up with the words.

What I Learned

Vibe coding this script taught me more in an hour than I’d usually learn in a weekend of reading docs:

Image Optimization: OCR is picky. Simply resizing an image isn't enough; the method of resizing (resampling filter) matters.
Library Specialization: PIL is for pixels; pypdf is for structure. Trying to do everything in one library is a trap.
The Power of the CLI: Using the Gemini CLI meant I didn't have to context-switch. I stayed in my terminal, describing what I wanted, and the code appeared.

Conclusion

We ended up with a ~100-line Python script that solves a genuine daily frustration. I didn't have to memorize the pypdf documentation or look up the Tesseract CLI flags. I just focused on the goal: "Make it searchable, make it pretty."

That’s Vibe Coding. You bring the vision, the AI brings the syntax, and together you build something cool.

We'll discover in the next episode if this is still true with a more complex use case and a GUI.

The Ultimate LLM Inference Battle: vLLM vs. Ollama vs. ZML

raphiki — Mon, 29 Dec 2025 09:12:46 +0000

A structured, data-driven comparison of today's leading open-source engines for serving AI models.

The "Runtime Wars"

The open-source AI community has achieved an incredible milestone: models like Meta's Llama 3 and Mistral AI's Mixtral now rival proprietary giants like GPT-4. But having the weights is only half the battle. To actually use these models—to build a chatbot, an agent, or an API, you need an inference engine.

The landscape of inference servers is exploding. A year ago, options were scarce. Today, developers are faced with a paralyzing array of choices. Should you use the industry darling vLLM? The local developer's favorite, Ollama? Or perhaps a radical newcomer like ZML?

Choosing the wrong engine can lead to massive infrastructure bills, slow user experiences, or vendor lock-in.

To cut through the hype, we are applying the QSOS (Qualification and Selection of Open Source software) method. This isn't a casual review; it's a structured evaluation comparing these three contenders against the state-of-the-art features required for modern AI production.

The Methodology: Why QSOS?

QSOS is a standardized methodology designed to reduce the risks associated with adopting open-source technologies. Unlike ad-hoc selection processes based on Medium articles or GitHub stars, QSOS treats open-source evaluation with the same rigor used for proprietary software.

The core philosophy of QSOS is separating Evaluation (the intrinsic, objective quality of the software) from Qualification (how well it fits your specific business needs).

For this comparison, we used a "Best of Breed" evaluation grid, scoring features on a simple 0-to-2 scale:

0: Not covered / Non-existent.
1: Partially covered / Complex implementation.
2: Fully covered / Best-in-class standard.

We assessed four key axes:

Maturity & Community: Is the project stable and likely to survive?
Functional Features: Does it support modern requirements like LoRA adapters and quantization?
Performance & Scale: Can it handle high throughput and utilize hardware efficiently?
Operations (Day 2): How easy is it to deploy, monitor, and maintain?

The Contenders

1. vLLM: The Data Center Standard

vLLM burst onto the scene in 2023 from UC Berkeley, solving a critical bottleneck in serving LLMs: memory fragmentation. Its core innovation, PagedAttention, allows it to manage GPU memory like an operating system manages virtual memory, dramatically increasing batch sizes and throughput.

Primary Focus: High-throughput production serving in the data center.
Positioning: vLLM is the currently the De Facto Standard for enterprise deployment. It excels on server-grade hardware (NVIDIA H100s/A100s) and offers the richest feature set for scaling.

2. Ollama: The Developer's Best Friend

Ollama took a different approach. It focused entirely on removing friction. By wrapping the powerful llama.cpp engine in a sleek, Docker-style Go binary, it made running a 70B parameter model on a MacBook as easy as typing ollama run llama3.

Primary Focus: Local development, edge devices, and consumer hardware (Mac/PC).
Positioning: Ollama is the king of usability. It is unbeaten for local testing and running models on consumer hardware, but it lacks the advanced scheduling required for high-traffic enterprise production.

3. ZML (Zig Machine Learning): The Radical Challenger

ZML is the new kid on the block. It is less of a "server" product and more of a compiler stack aimed at engineers. Written in Zig, it utilizes OpenXLA/MLIR to compile model graphs directly into standalone binaries, aiming to eliminate the heavy Python/PyTorch dependency chain entirely.

Primary Focus: High-performance, cross-platform runtime (TPUs, AMD, NVIDIA) without dependencies.
Positioning: ZML is an Alpha-stage visionary. It offers incredible potential for hardware portability and efficiency but is currently a complex "build-your-own-stack" tool rather than a drop-in product.

Visualizing the Results

To understand how these tools differ, we visualize our QSOS scores using two different schemas.

The Radar Chart: Feature Balance

This chart shows the balance of strengths across the four evaluation axes.

Caption: The QSOS Radar Chart highlights the distinct profiles of the three engines. vLLM shows the broadest coverage across features and performance. Ollama spikes toward Operational Ease. ZML shows potential in features but lacks maturity.

vLLM (Blue): The largest, most balanced area, indicating strength across maturity, features, and performance, with moderate operational complexity.
Ollama (Green): A massive spike toward "Operational Ease," reflecting its zero-friction user experience, but pulling back on raw performance metrics like continuous batching.
ZML (Red): A smaller footprint overall, reflecting its early stage (low maturity), but showing strong potential in functional features due to its compiler-based architecture.

The QSOS Quadrant: Market Position

This schema maps the tools based on their market adoption versus their raw production capabilities.

Caption: The QSOS Quadrant positions the tools based on Market Maturity vs. Production Power.

vLLM (The Leader): High Maturity, High Power. The safe, scalable choice for the enterprise.
Ollama (The Specialist): High Maturity, Lower Production Power. The standard for a specific niche (local/consumer hardware), prioritizing usability over scale.
ZML (The Visionary): Low Maturity, High Potential Power. An innovative approach that hasn't yet proven itself in the broad market.

The Consolidated Score Sheet

Below is the detailed breakdown of the evaluation scores that feed the charts above.

Section / Criteria	vLLM	Ollama	ZML (Zig ML)
A. MATURITY
History & Age	2 (Standard)	2 (Standard)	0 (Very New)
Activity	2 (Hyper-Active)	2 (Viral)	2 (High Velocity)
Ecosystem	2 (Dominant)	2 (Ubiquitous)	0 (Niche)
Governance	2 (Community)	1 (Company Led)	1 (Small Team)
B. FEATURES
Model Support	2 (Universal)	2 (Curated Lib)	2 (Compiler based)
Quantization	2 (Server: AWQ/FP8)	2 (Edge: GGUF)	1 (Implicit XLA)
LoRA Adapters	2 (Dynamic Multi-LoRA)	1 (Static Modelfile)	0 (Not standard)
API Compat.	2 (OpenAI Native)	2 (OpenAI Native)	0 (Runtime only)
C. PERFORMANCE
Cont. Batching	2 (Gold Standard)	0 (FIFO)	1 (Arch. support)
Throughput	2 (Maximum SOTA)	1 (Low/Single User)	1 (High Potential)
Parallelism	2 (Tensor & Pipeline)	0 (Single Node)	1 (Compiler Config)
Hardware Agnosticism	1 (NVIDIA Centric)	2 (Apple/Consumer)	2 (Any: TPU/AMD)
D. OPERATIONS
Ease of Setup	1 (Python/Docker)	2 (Magic 1-Click)	0 (Hard: Bazel)
Dependencies	1 (Heavy Torch)	2 (Zero: Go Binary)	2 (Zero: Zig Binary)
Observability	2 (Prometheus Native)	0 (Logs only)	1 (Manual metrics)

Conclusion

There is no single "best" inference engine. The right choice depends entirely on your specific context (the Qualification phase of QSOS).

Choose vLLM if:

You are building a production application that needs to serve many concurrent users. You have access to server-grade GPUs (NVIDIA A10G, A100, H100) and need features like dynamic LoRA adapters for multi-tenancy.

If you are deploying to Kubernetes to serve customers, start here.

Choose Ollama if:

You are a developer building locally on a Mac or Windows PC. You need a zero-friction way to test models, or you are deploying to edge devices where resources are constrained, and concurrency is low.

If you just want to run Llama 3 on your laptop right now, download Ollama.

Choose ZML if:

You are an ML systems engineer building a specialized hardware appliance (e.g., using TPUs or AMD chips) and need a runtime with absolutely zero Python dependencies and a tiny footprint. You are willing to build the server infrastucture around it yourself.

If you are frustrated by PyTorch bloat and want a "build your own" adventure, look at ZML.

Note on Methodology

For the purpose of this article, we utilized a simplified QSOS evaluation grid. We intentionally zoomed in on the "Best of Breed" criteria, the critical differentiators driving the current "Inference Wars", to keep the comparison readable and actionable.

A full-fledged QSOS evaluation is significantly more exhaustive. It is structured as a hierarchical tree of criteria containing more data points, covering deep operational details such as:

Generic Attributes: Intellectual property management, roadmap visibility, bug tracking efficiency, and internationalization.
Specific Sub-sections: Detailed granularity on security compliance (SOC2/GDPR), exact memory footprints, and specific driver version compatibility.

While this article provides a strategic overview, a complete QSOS audit would involve drilling down from high-level "Sections" into specific "Leaves" to calculate a precise, weighted score for every possible business constraint.

DEV Community: raphiki

Running DeepSeek Harness on an 8GB GPU: Context Tuning, Presets, and Hardware Limits

1. Introduction: Bringing the Harness Home

2. Why Choose DeepSeek Harness for a Local Agent?

Defining the Use Case: An Autonomous Task Worker, Not an Interactive Chatbot

3. Step 1: Installation and Baseline Test with OpenRouter

Installation

Baseline Cloud Configuration

4. Step 2: Selecting and Preparing the Local Model (Ornith-1.5)

Why Ornith-1.5?

1. What is Ornith-1.5? (A Specialized Qwen Fine-Tune)

2. Why 9B Parameters on an 8GB GPU?

Inference Server: Ollama

The 2048 Context Constraint

Tuning the Context Window to 16K

The VRAM Arithmetic

5. Step 3: Trials, Failures, and the Minimal Preset

Untangling the Hierarchy: Plugins, Profiles, and Presets

The Built-in Presets: Four Flavors of DSH

The 8GB Dilemma: Why Standard Mode Threatens Local Context

The Initial Hypothesis: Configuring "Minimal Mode"

Standard Mode vs. Minimal Mode: The 8GB Tradeoff

6. Step 4: The Complete DeepSeek Harness Configuration (settings.yaml)

Breakdown of the Configuration

7. Step 5: Running the Local Agent in Practice

Phase 1: Starting with Minimal Mode

Phase 2: Switching to Standard Mode

Code Reasoning and Bug Localization

The Context Ceiling: Auto-Compactions in Action

Empirical Validation of the VRAM Arithmetic

The Engineering Takeaway: Fine-Tuning the Preset Sweet Spot

Alternative Interface: Creating a Dedicated TUI Profile with dsh-TUI

8. Best Practices for 8GB Local Agents

1. Monitor VRAM Residency and Address Latency

2. Fine-Tune the Tool Surface: Avoid Extremes

3. Use Hybrid Fallback Strategically

4. Implement Automated Routing via LiteLLM

9. Series Retrospective: From Meta-Framework to Local Agent

References & Links

DeepSeek Harness: How DeepSeek Uses Cordis to Redefine Autonomous AI Agents

1. Introduction: Beyond the Prompt Chain

2. Why Call It a "Harness"? (Untangling the Terminology)

3. Why DeepSeek Adopted Cordis

Limitations of Python Agent Frameworks

How Cordis Addresses These Issues

4. How DeepSeek Harness Works Under the Hood

1. "There is No Core": The Pure Plugin Architecture

2. Bundles and Profiles (cordis.yml)

Entry Profiles and Presets: Web, Headless, and Minimal

3. The Anatomy of a Custom DSH Plugin (15 Lines of TypeScript)

4. Session as an Append-Only Stream

5. Reversible Sandboxes and Dynamic Toolsets

6. Waterfall Control Flow & Guardrails

5. The True Meta-Harness: An Exploding Plugin Ecosystem

Even the User Interface is a Plugin (Web UI vs. TUIs)

Thousands of Plugins and Rising Community Marketplaces

Real-World Plugin Innovations: Beyond Simple Coding Tools

6. Positioning DSH: Comparing Harnesses Across the AI Landscape

Layer A: DSH vs. General Agent Frameworks (LangGraph, CrewAI)

Layer B: DSH vs. Dedicated Coding Harnesses (OpenCode, Pi)

Under the Hood: How Their Plugin Systems Actually Differ

The Comprehensive Architectural Matrix

In Short:

7. Dynamic Tool Generation at Runtime ("Creation Mode")

8. Conclusion

Coming Next in Part 3: Running DeepSeek Harness on an 8GB GPU

References & Links

Understanding Cordis: The TypeScript Framework Built for Hot-Swapping Everything

Series Foreword: The Architecture Behind DeepSeek Harness

1. Introduction: What is Cordis?

Where Did Cordis Come From?

2. The Big Problem: Why Plugins Usually Leak

The "Hotel Room" Analogy

3. The Core Idea: Reversibility ("Every Action Has an Undo")

4. How Cordis Works Under the Hood

Concept 1: The Context Tree (Context and ctx)

Concept 2: Reversible Effects (ctx.effect)

Concept 3: Services (Shared Capabilities)

Concept 4: The Fiber (The Plugin Engine)

Reactive Dependency Resolution

6. Step 4: The Complete DeepSeek Harness Configuration (`settings.yaml`)

2. Bundles and Profiles (`cordis.yml`)

Concept 1: The Context Tree (`Context` and `ctx`)

Concept 2: Reversible Effects (`ctx.effect`)