Building a Local-First AI Coding Agent with Open Tools and Adaptive Routing

How to combine local inference with governed cloud fallback by orchestrating Ollama, OpenCode, and LiteLLM

---

AI coding agents usually leave you with an awkward choice:

1. All cloud: A hosted model can handle almost everything, but using it for small edits and routine work consumes API credits and sends more code off the machine.
2. All local: A model that fits on a laptop GPU works well for many everyday tasks, but it may struggle with architecture, complex refactoring, and debugging across several files.

A local first hybrid architecture offers a middle ground. The local model handles routine coding decisions, while OpenCode performs filesystem and shell operations through its tools. Harder requests can go to larger hosted models, with a configurable limit on spending.

Connecting several models is easy. Choosing one for each request, in a way that is predictable and visible, takes more work. LiteLLM provides the control plane for that decision.

In this tutorial, we will build a three layer AI coding environment on a laptop equipped with an RTX 4070 GPU with 8GB of VRAM and 32GB of system RAM:

• Layer 1, local inference: Ollama serves a Gemma 4 model configured for tool use.
• Layer 2, control plane: LiteLLM provides spending limits, automatic routing by complexity, and observability backed by PostgreSQL.
• Layer 3, agent harness: OpenCode provides terminal, desktop, and web interfaces, along with tool dispatch and session state.

Each connection is tested before the next one is added. We start with the local model, connect the harness, and finish with the proxy and router. This makes failures much easier to locate.

---

Why this stack

Large reasoning models are useful for architecture and subtle bugs. They are expensive overkill for routine implementation or a focused refactor that a smaller local model can handle.

Many setups connect a harness to one model and stop there. This build adds a control plane to decide where requests go and track what they cost. The harness and local gateway are open source, and the selected models have available weights. I use OpenRouter, a proprietary service, for hosted inference. You can replace it with another compatible endpoint or infrastructure that you operate yourself. No API for a closed weight model is required.

My test machine has an RTX 4070 laptop GPU with 8GB of VRAM and 32GB of system RAM. The configuration below ran the local model, harness, and control plane together on that machine. Your memory use and throughput will vary with the model build, context length, drivers, operating system, and other GPU workloads.

What “adaptive” means in this build

OpenCode always requests one logical model, auto-mode. LiteLLM then classifies the request and selects a target according to the routing policy:

Request class	Example	Default target
Simple	A focused rename or boilerplate	Local Gemma 4
Medium	Routine implementation or test writing	Local Gemma 4
Complex	Changes across several files or substantial debugging	Hosted Mistral Small
Reasoning	Architecture, planning, or explicit tradeoffs	Hosted DeepSeek

The router does not learn from previous requests. It uses LiteLLM's complexity score, a few keyword rules, and the tier mapping configured later in the article.

This tutorial builds three layers, in order, so each one is independently testable before you stack the next:

---

The implementation order differs from the layer numbers. We first validate the local worker (Layer 1), connect the harness directly to it (Layer 3), and then place the proxy (Layer 2) between them.

Local Inference (Layer 1)

Local Model: Gemma 4

The local model has to fit in 8GB of VRAM and still be useful for coding and tool use. That points toward a quantized model with available weights and a good balance between size and reasoning ability.

I used llmfit to narrow the options and selected Gemma 4 e2b Q4_K_M.

The name describes the model and its quantization:

• Gemma 4 is an open weight model provided by Google.
• e2b means "Effective 2B" and targets edge deployments, making it suitable for this hardware.
• Q4_K_M identifies a four bit K quant variant with a medium mix of quantization types. Quantization reduces memory use at some cost to fidelity.

Inference Server: Ollama

I selected Ollama because it is easy to install and exposes an OpenAI compatible API for the rest of the stack. Other open source inference servers can fill the same role. I covered several alternatives in the Bringing AI Home series and in this comparison of Ollama, vLLM, and ZML.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Then download and run Gemma 4:

ollama run gemma4:e2b

pulling manifest
pulling 4e30e2665218: 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 7.2 GB
verifying sha256 digest
writing manifest
success
>>> Hi
Thinking...
Thinking Process:

1.  **Analyze the input:** The input is "Hi". This is a very casual, open-ended greeting.
2.  **Determine the user's intent:** The user is initiating a conversation or acknowledging my presence.
3.  **Formulate an appropriate response:** The response should be friendly, welcoming, and invite further interaction.
    *   Standard replies: "Hello," "Hi there," "How can I help?"
4.  **Self-check against constraints (Identity/Role):** I am Gemma 4, a helpful AI. The response should reflect that role.
5.  **Generate the final reply:** A simple, warm greeting followed by an offer of assistance is ideal.
...done thinking.

Hello! How can I help you today?

The response confirms that local generation works. We will test tool use after connecting the harness.

AI Harness (Layer 3)

Open Source Harness: OpenCode

The model needs an execution harness before it can act as an agent. The harness manages session history, runs local tools, and tracks file changes across turns.

Mistral Vibe, Claude Code, OpenAI Codex, and Antigravity are examples of harnesses. Open source options include Pi Coding Agent and DeepSeek Harness.

I selected OpenCode because it is lightweight, actively developed, and available through terminal, web, and desktop interfaces. A formal QSOS comparison of open coding agents would be an interesting follow up. For now, install OpenCode:

curl -fsSL https://opencode.ai/install | bash

Quick smoke test: OpenCode talking directly to Ollama

Now I'm ready to wire OpenCode to Ollama and Gemma 4... or so I thought.

I learned the hard way that the model needs a configuration tailored to the agent before it is connected to the harness. A bare ollama run setup can behave differently once the harness starts sending tool schemas and longer agentic turns.

Agent harnesses can send tool schemas, project context, and file contents with every turn. If the context is too small, Ollama may truncate earlier content, including tool definitions. The result can be failed or invented tool calls. On this 8GB GPU, I use a context of 16,384 tokens. Larger windows consume more memory for the KV cache and may offload work to the CPU, so adjust this value for your hardware.

When creating a custom Modelfile, keep the model's compatible chat template unless you have tested a replacement. Tool capable models expect function definitions in a specific prompt format. An incompatible template can produce raw JSON or plain text command suggestions instead of tool calls.

Smaller general purpose models may call tools from another agent framework, such as explore or list_files, instead of OpenCode's glob. A short system prompt can list the available tools and map common actions to their OpenCode names.

I used the following Modelfile:

FROM gemma4:e2b

PARAMETER num_ctx 16384
PARAMETER temperature 0.0

SYSTEM """
You are an autonomous coding assistant inside OpenCode.
CRITICAL: You must only invoke tools from the provided schema.
Available Tools:
- glob(pattern: string): List/find files. Use this to explore directories.
- read(filePath: string, offset?: number, limit?: number): Read file content.
- write(filePath: string, content: string): Create or overwrite a file.
- edit(filePath: string, oldString: string, newString: string): Replace exact text in a file.
- grep(pattern: string, path?: string): Search codebase with regex.
- bash(command: string): Run terminal commands (build, test, git).
- question(header: string, question: string, options?: string[]): Ask user for clarification.
- task(subagentType: string, prompt: string): Delegate a subtask.
- skill(name: string): Load a predefined skill.
- todowrite(todos: array): Update the task/progress list.
- webfetch(url: string): Fetch web content.
- invalid: System fallback (do not invoke directly).
"""

The 16384 context window is the value used on my RTX 4070 laptop. A 32K or 64K window requires substantially more KV cache memory and may force partial CPU offloading on an 8GB card. Watch nvidia-smi while testing; if VRAM remains pinned near the limit or throughput collapses, reduce num_ctx to 8192 and retest.

Create the configured model in Ollama, much like building a Docker image:

ollama create gemma4-coder-agent -f Modelfile

gathering model components
using existing layer sha256:fdf02c16fb654ff60b2c30f1e91573ebc603a7084df3449df89120dca2b18170
using existing layer sha256:e94a8ecb9327ded799604a2e478659bc759230fe316c50d686358f932f52776c
creating new layer sha256:dcaf83c203b7c4daae5c154641637a2d10221b09baa4fce8d70f839cad18447d
writing manifest
success

Check that it appears in Ollama's model list:

ollama list

NAME                                                  ID              SIZE      MODIFIED
gemma4-coder-agent:latest                             bfa492d99bf8    7.2 GB    49 minutes ago
gemma4:e2b                                            7fbdbf8f5e45    7.2 GB    15 minutes ago

Next, call the configured model through Ollama's OpenAI compatible API. Testing this boundary now helps distinguish a model server problem from a later harness problem:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4-coder-agent",
    "messages": [{"role": "user", "content": "Write a TypeScript function that debounces a callback."}]
  }'

Once the request succeeds, connect the harness. Close unnecessary GPU heavy applications before longer coding sessions because 8GB of VRAM leaves little headroom.

Point OpenCode at the configured model (no proxy yet)

Configure OpenCode in ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1"
      },
      "models": {
        "gemma4-coder-agent": {
          "name": "Gemma4 (ollama)",
          "tools": true
        }
      }
    }
  }
}

And launch the OpenCode CLI:

opencode

Ask it to read or edit a file and confirm that OpenCode dispatches the tool call. A chat response alone does not test the tool configuration.

If tool calls don't dispatch, check that the Modelfile applied correctly (ollama show gemma4-coder-agent) before touching anything else.

If nothing works at all, check that curl http://localhost:11434/v1/models returns the model, and check OpenCode's logs for the actual connection error.

The harness and local model now work together. Keep this opencode.json as a baseline while adding the proxy and router.

Simple LiteLLM Proxy (Layer 2)

The next component is the AI gateway between the harness and the models. Several open source gateways are available, including Mozilla AI's Otari.

I selected LiteLLM, which we also use in production at work. It covers the proxy and governance features needed here. I did not measure its latency overhead for this article, so the performance discussion focuses on model routing.

LiteLLM began as a small Python library and now includes a proxy server, an Admin UI, and optional database storage.

Start with the standalone proxy:

pip install 'litellm[proxy]' --break-system-packages

This command matches my test environment. The --break-system-packages option bypasses the distribution package manager's protection. On a maintained workstation, use a virtual environment, pipx, uv, or the Docker setup shown later.

The Docker deployment uses the same configuration file and is covered later.

First, point the proxy only at the gemma4-coder-agent that we already tested.

Create litellm_config.yaml:

model_list:
  - model_name: "Gemma 4 (litellm)"
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://localhost:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

general_settings:
  master_key: sk-local-master-key-change-me

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

Start LiteLLM with that configuration:

litellm --config litellm_config.yaml --port 4000

Call the model through port 4000 to confirm that LiteLLM, rather than the direct Ollama endpoint, serves the request:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "Gemma 4 (litellm)", "messages": [{"role": "user", "content": "ping"}]}'

{
  "id": "chatcmpl-a1230750-0843-4af5-9e35-3c8e1869d45d",
  "created": 1787427863,
  "model": "Gemma 4 (litellm)",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "",
        "role": "assistant",
        "reasoning_content": "The user input is simply \"ping\". This is a very vague request.\nIn a general context, \"ping\" usually refers to a network diagnostic tool.\nHowever, as an autonomous coding assistant inside OpenCode, I need to determine what action the user expects me to take based on the available tools and the context of a coding environment.\n\n1.  **Tool Check:** I have tools for file system operations (`glob`, `read`, `write`, `edit`, `grep`), shell commands (`bash`), web fetching (`webfetch`), and task delegation/questioning.\n2.  **Interpretation:** Since there is no specific file or code provided, \"ping\" might be:\n    *   A request to run a system command (like `ping` in a terminal).\n    *   A request for information about the network concept of ping.\n    *   A prompt to test connectivity (which I cannot do directly outside of a simulated environment).\n\nGiven the context of an \"autonomous coding assistant,\" the most likely interpretation is that the user wants me to execute a command if possible, or they are testing my ability to respond to a simple command. Since I have a `bash` tool, running a system command is an option.\n\nIf I assume the user wants me to run the standard network diagnostic:\n*   I can use `bash(\"ping\")`.\n\nIf I assume the user is asking for a definition or context:\n*   I should explain what `ping` is.\n\nSince I am operating within a coding assistant framework, and \"ping\" is often used as an instruction to test connectivity in such environments, I will attempt to use the `bash` tool."
      }
    }
  ],
  "usage": {
    "completion_tokens": 357,
    "prompt_tokens": 258,
    "total_tokens": 615
  }
}

If the call fails, check that Ollama is still running with ollama list before debugging LiteLLM.

Now connect the harness to the proxy. With only one model configured, the full OpenCode → LiteLLM → Ollama path is still easy to debug.

Update ~/.config/opencode/opencode.json to register LiteLLM alongside the existing Ollama provider:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://127.0.0.1:11434/v1"
      },
      "models": {
        "gemma4-coder-agent": {
          "name": "Gemma4 (ollama)",
          "tools": true
        }
      }
    },
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "sk-local-master-key-change-me",
        "compatibility": "compatible"
      },
      "models": {
        "Gemma 4 (litellm)": {
          "tools": true
        }
      }
    }   
  }
}

opencode

Inside the OpenCode TUI, running /models lets us toggle between connecting directly to Ollama (Gemma4 (ollama)) or through our LiteLLM proxy (Gemma 4 (litellm)).

LiteLLM is currently a simple pass through with one model and no routing logic. The test confirms that the network path and OpenAI compatible interface work before we add cloud models.

Hybrid LiteLLM Proxy (Layers 1 & 2)

With the local route working, we can add cloud endpoints. The harness and proxy are open source, and the models have available weights. Hosted access still depends on the service used to run those models.

I use OpenRouter to access the remote models. OpenRouter is a proprietary hosted gateway, while LiteLLM keeps the routing policy, budget controls, and logs on the laptop. You could serve the same models from your own infrastructure or cloud tenant instead.

Two remote models complement the local Gemma 4:

• Mistral Small: A fast and relatively inexpensive option for tasks that are too demanding for the local model.
• DeepSeek v4: A larger reasoning model for architecture and difficult debugging.

One OpenRouter API key covers both models. In the hybrid setup, all OpenCode traffic passes through LiteLLM, so the direct Ollama connection is removed. The name Gemma 4 (litellm) no longer has to distinguish one connection from another. From here on, its LiteLLM alias is the simpler gemma4-local.

model_list:
  - model_name: gemma4-local
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://localhost:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

  # --- Cloud tier: open-weight models via OpenRouter ---
  - model_name: deepseek-reasoning
    litellm_params:
      model: openrouter/deepseek/deepseek-v4-pro
      api_key: os.environ/OPENROUTER_API_KEY

  - model_name: mistral-fast
    litellm_params:
      model: openrouter/mistralai/mistral-small-2603
      api_key: os.environ/OPENROUTER_API_KEY

router_settings:
  provider_budget_config:
    openrouter:
      budget_limit: 5        # $5/day ceiling on cloud spend
      time_period: 1d

general_settings:
  master_key: sk-local-master-key-change-me

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

LiteLLM can limit spending over a given period. Once the threshold is reached, it blocks further calls covered by that budget. Test this case in your deployment and check the error shown by OpenCode. You may also want those requests to fall back to the local model.

This configuration limits OpenRouter spending to $5 per day.

LiteLLM can also issue scoped keys with their own budgets:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"max_budget": 10, "budget_duration": "30d", "models": ["gemma4-local", "deepseek-reasoning", "mistral-fast"]}'

The remaining examples use the tutorial master key for consistency. For a shared or long lived deployment, use the scoped key returned by /key/generate in OpenCode and keep the master key out of client configuration.

Export the OpenRouter key and restart the proxy:

export OPENROUTER_API_KEY=sk-or-...
litellm --config litellm_config.yaml --port 4000

Test each route directly through the LiteLLM API:

# local model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma4-local", "messages": [{"role": "user", "content": "ping"}]}'

# First Cloud model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "mistral-fast", "messages": [{"role": "user", "content": "ping"}]}'

# Second Cloud model through the proxy
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-reasoning", "messages": [{"role": "user", "content": "ping"}]}'

The API also reports budget consumption:

curl -X GET http://localhost:4000/provider/budgets \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-local-master-key-change-me"

The response includes the limit, current spend, time period, and reset time. Check the reset timestamp against the host clock before relying on it. A wildly incorrect date usually points to a clock problem or a bug in the installed version.

Point OpenCode at the expanded proxy:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "sk-local-master-key-change-me",
        "compatibility": "compatible"
      },
      "models": {
        "gemma4-local": {
          "tools": true
        },
        "mistral-fast": {
          "limit": { "context": 262144, "output": 8192 }
        },
        "deepseek-reasoning": {
          "limit": { "context": 1000000, "output": 8192 }
        }
      }
    }   
  }
}

The direct Gemma4 (ollama) entry is gone because all traffic now passes through LiteLLM. From this point onward, gemma4-local identifies the local model behind the proxy.

OpenCode also provides a web interface. Start it and check that the previous sessions and new models are available:

opencode web

The hybrid proxy now exposes several models and a spending limit, but model selection is still manual. The next step is automatic routing.

Smart LiteLLM Proxy (Layer 2)

During my tests, OpenCode did not switch models reliably based on task complexity. Its model overrides for subagents were not consistent enough for this setup.

The routing decision therefore moves to LiteLLM's Auto Router v2. OpenCode always requests auto-mode. LiteLLM scores the request, applies any matching keyword rule, and maps the chosen tier to a model. OpenCode does not need to know which model handles the request.

Add auto-mode to litellm_config.yaml alongside the three existing models:

model_list:
  # ... gemma4-local, deepseek-reasoning, mistral-fast entries stay as-is ...

  - model_name: auto-mode
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gemma4-local          # autocomplete, small edits, boilerplate
          MEDIUM: gemma4-local          # routine implementation, test writing
          COMPLEX: mistral-fast        # multi-file changes, real debugging
          REASONING: deepseek-reasoning # architecture, planning, hard bugs
        complexity_router_default_model: gemma4-local   # fail toward free, not expensive
        keyword_rules:
          - keywords: ["plan", "architecture", "design a", "trade-off"]
            tier: REASONING
        return_raw_model_name: true

Three settings control the behavior:

• complexity_router_default_model: gemma4-local sends an uncertain or timed out classification to the free local model. Change it if you prefer capability over cost.
• keyword_rules run before complexity scoring. They immediately escalate prompts that mention architecture, planning, or tradeoffs.
• return_raw_model_name: true returns the name of the model that handled the request, making routing easier to verify in logs.

After restarting the proxy, send a short prompt and an architecture prompt to auto-mode. Check the returned model name and the LiteLLM logs. This confirms that both routes work, but it does not measure routing accuracy.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto-mode", "messages": [{"role": "user", "content": "rename your model name to camelCase"}]}'

The Auto Router selected the ollama_chat/gemma4-coder-agent model:

{
  "id": "chatcmpl-774cdbf7-09c2-4d1e-8f94-bb878c0f11bb",
  "created": 1788128064,
  "model": "ollama_chat/gemma4-coder-agent",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I am an AI assistant and do not have a specific model name that I can rename within this context. How can I help you with your coding tasks?",
        "role": "assistant",
        "reasoning_content": "Thinking Process:\n\n1.  **Analyze the Request:** The user wants me to \"rename your model name to camelCase\"..."
   # ... rest of the message ...

The router chose the expected local model, but the answer itself is poor because the model interpreted the vague request literally. A correct route does not guarantee a good answer.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto-mode", "messages": [{"role": "user", "content": "design the data model for a multi-tenant billing system with usage-based pricing"}]}'

This time, the deepseek/deepseek-v4-pro is called:

{
  "id": "gen-1788128721-XpJrCbVmBWN8x9E03aDh",
  "created": 1788128721,
  "model": "deepseek/deepseek-v4-pro",
  "object": "chat.completion",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Below is a Stripe-inspired data model for a **multi-tenant billing system with usage-based pricing**.  \nThe model assumes:\n\n- A **tenant** is an organization u..."
  # ... rest of the message ...

Evaluate the routing policy with representative tasks

Before making auto-mode the default, test it with prompts from your own work. Label the expected tier first, then record the selected model and whether it completed the task. A small test set could include:

Task	Expected tier	What to verify
Rename a field in one interface	Simple	Remains local and makes the correct edit
Add validation and unit tests	Medium	Remains local unless the context is unusually large
Trace a failure across several layers	Complex	Escalates to the fast hosted model
Compare tenant isolation designs	Reasoning	Escalates to the reasoning model
Ask for a “plan” for a trivial rename	Adversarial	Reveals whether the keyword rule escalates too often
Describe a hard bug without escalation keywords	Adversarial	Reveals whether complexity scoring catches it

Run the same tasks with local only, cloud only, manual selection, and automatic routing. For each mode, record successful tasks, median latency, hosted request count, and hosted cost.

The results will show whether the router saves money without hurting task completion. Pay particular attention to unnecessary cloud calls and difficult tasks that stay local.

Register auto-mode in ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM Proxy",
      "options": {
        "baseURL": "http://localhost:4000/v1",
        "apiKey": "sk-local-master-key-change-me"
      },
      "models": {
        "auto-mode": {
          "limit": { "context": 16384, "output": 4096 }
        },
        "gemma4-local": {
          "limit": { "context": 16384, "output": 4096 },
          "tools": true
        },
        "mistral-fast": {
          "limit": { "context": 262144, "output": 8192 }
        },
        "deepseek-reasoning": {
          "limit": { "context": 1000000, "output": 8192 }
        }
      }
    }
  },
  "enabled_providers": ["litellm"],
  "model": "litellm/auto-mode"
}

The "model": "litellm/auto-mode" setting makes automatic routing the default for new sessions. The three individual models remain available in /models as manual overrides.

"enabled_providers": ["litellm"] limits OpenCode to the LiteLLM provider. This syntax will be replaced by the policy mechanism described here.

OpenCode also has a desktop application for Windows, macOS, and Linux, available from the official site.

The same session is shown below in OpenCode Desktop with Auto Mode enabled.

The proxy selected Mistral Small for this task. At this point, I still had to check the OpenRouter dashboard to confirm the choice.

That external check confirmed the route, but it also showed what the local setup still lacked: one place to inspect routing and usage.

Full Control Plane (Layer 2)

LiteLLM started as a Python wrapper for different LLM APIs. It has since expanded into gateway infrastructure with management and observability features.

The full proxy can use PostgreSQL for its Admin UI, request logs, and spending data. Running those services locally adds useful controls without making this tutorial setup production ready.

To add those features, switch to the containerized deployment backed by PostgreSQL. Download the official Docker Compose file:

curl -sSLO https://docs.litellm.ai/docker-compose.yml 

Adapt the Compose file to mount litellm_config.yaml, seed the database on first boot, and reach Ollama on the host:

services:
  litellm:
    image: docker.litellm.ai/berriai/litellm-database:latest
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command:
      - "--config=/app/config.yaml"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "4000:4000"
    environment:
      LITELLM_SALT_KEY: sk-XXXXXXXXXXXXXXXX
      DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
      STORE_MODEL_IN_DB: "True"
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Docker Compose creates a LiteLLM container, a PostgreSQL container, and a volume for the database. I made two changes to the default Compose file:

• volumes: [./litellm_config.yaml:/app/config.yaml] and command: ["--config=/app/config.yaml"] load the LiteLLM configuration into the container.
• extra_hosts: ["host.docker.internal:host-gateway"] lets the container reach Ollama on the host.

The LiteLLM configuration also needs a few changes:

model_list:
  - model_name: gemma4-local
    litellm_params:
      model: ollama_chat/gemma4-coder-agent
      api_base: http://host.docker.internal:11434
      max_tokens: 16384
      num_ctx: 16384
    model_info:
      supports_function_calling: true

  # --- Cloud tier: open-weight models via OpenRouter ---
  - model_name: deepseek-reasoning
    litellm_params:
      model: openrouter/deepseek/deepseek-v4-pro
      api_key: os.environ/OPENROUTER_API_KEY

  - model_name: mistral-fast
    litellm_params:
      model: openrouter/mistralai/mistral-small-2603
      api_key: os.environ/OPENROUTER_API_KEY

  # --- Auto Router: smart routing based on task complexity and keywords ---
  - model_name: auto-mode
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gemma4-local          # autocomplete, small edits, boilerplate
          MEDIUM: gemma4-local          # routine implementation, test writing
          COMPLEX: mistral-fast         # multi-file changes, real debugging
          REASONING: deepseek-reasoning # architecture, planning, hard bugs
        complexity_router_default_model: gemma4-local   # fail toward free, not expensive
        keyword_rules:
          - keywords: ["plan", "architecture", "design a", "trade-off"]
            tier: REASONING
        return_raw_model_name: true

router_settings:
  provider_budget_config:
    openrouter:
      budget_limit: 5        # $5/day ceiling on cloud spend — tune to taste
      time_period: 1d

general_settings:
  master_key: sk-local-master-key-change-me
  store_model_in_db: true
  store_prompts_in_spend_logs: true

litellm_settings:
  drop_params: true
  telemetry: false
  modify_params: false

The changes are:

• api_base: http://host.docker.internal:11434 points the container to the local Ollama model.
• store_model_in_db: true enables database storage.
• store_prompts_in_spend_logs: true enables prompt logging.

The last option stores prompts, including any source code they contain, in PostgreSQL. Disable it or define a retention policy if you do not need to inspect full prompts.

Start the containers:

docker compose up -d

The LiteLLM container logs confirm that the models were loaded from the configuration:

litellm-1  |
litellm-1  |    ██╗     ██╗████████╗███████╗██╗     ██╗     ███╗   ███╗
litellm-1  |    ██║     ██║╚══██╔══╝██╔════╝██║     ██║     ████╗ ████║
litellm-1  |    ██║     ██║   ██║   █████╗  ██║     ██║     ██╔████╔██║
litellm-1  |    ██║     ██║   ██║   ██╔══╝  ██║     ██║     ██║╚██╔╝██║
litellm-1  |    ███████╗██║   ██║   ███████╗███████╗███████╗██║ ╚═╝ ██║
litellm-1  |    ╚══════╝╚═╝   ╚═╝   ╚══════╝╚══════╝╚══════╝╚═╝     ╚═╝
litellm-1  |
litellm-1  | ...
litellm-1  | INFO:     Application startup complete.
litellm-1  | INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
litellm-1  |
litellm-1  | #------------------------------------------------------------#
litellm-1  | #                                                            #
litellm-1  | #               'A feature I really want is...'               #
litellm-1  | #        https://github.com/BerriAI/litellm/issues/new        #
litellm-1  | #                                                            #
litellm-1  | #------------------------------------------------------------#
litellm-1  |
litellm-1  |  Thank you for using LiteLLM! - Krrish & Ishaan
litellm-1  |
litellm-1  |
litellm-1  |
litellm-1  | Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
litellm-1  |
litellm-1  |
litellm-1  | LiteLLM: Proxy initialized with Config, Set models:
litellm-1  |     gemma4-local
litellm-1  |     deepseek-reasoning
litellm-1  |     mistral-fast
litellm-1  |     auto-mode

Access the Admin UI at http://localhost:4000/ui/. Log in with the username admin and the password configured as your master_key (sk-local-master-key-change-me).

The Models + Endpoints view lists the local model, the two remote models, and the auto-mode router.

Two views are especially useful here.

The Usage view shows request volume, token consumption, and spending by model:

The Request Logs view shows individual traces, prompts, and router decisions stored in PostgreSQL:

Privacy and trust boundaries

Local first is not the same as local only. Requests assigned to mistral-fast or deepseek-reasoning leave the laptop and pass through OpenRouter to a hosted inference provider. They may contain source code, file paths, tool output, or conversation history.

Before using automatic routing with a private repository:

• review the data handling terms of every hosted provider in the route;
• exclude secrets, .env files, private keys, certificates, and credential stores from agent context;
• provide an easy local only model override for sensitive work;
• decide whether cloud escalation should be automatic or require confirmation;
• restrict access to the LiteLLM Admin UI and PostgreSQL database;
• replace tutorial keys and database passwords with secrets supplied through your deployment environment;
• define retention and backup policies for stored prompts and request logs.

The control plane makes model selection and logging visible. It does not make hosted inference private, but it gives you one place to define and audit that boundary.

Before treating this as a production deployment

This tutorial favors a readable local setup. Once the complete path works, record the environment and pin the component versions or container image digests used for that run.

Automatic routing, provider compatibility, and configuration fields can change between releases. A shared or production deployment should also enable TLS, use scoped client keys, protect the Admin UI, rotate secrets, back up PostgreSQL, test budget exhaustion and provider failures, and monitor routing quality.

LiteLLM also supports MCP server hubs, team virtual keys, input guardrails, and heuristics such as routing by context size. This article uses only the features needed for the local control plane.

LiteLLM also offers a commercial Enterprise edition with SAML SSO, team RBAC, secret management, and clusters across multiple regions. Everything configured here runs on the open source community edition.

Conclusion

This setup made an 8GB VRAM laptop a practical base for my coding agent. Gemma 4 handles routine work without token charges, OpenCode runs the tools and keeps the session, and LiteLLM sends harder requests to larger hosted models.

What We Built

1. Configured Local Inference: A custom Ollama Modelfile with a tested context window (16384) and an explicit tool registry designed to improve Gemma 4's behavior within 8GB of VRAM.
2. Decoupled Harness: OpenCode configured to communicate over standard OpenAI compatible endpoints across terminal, desktop, and web interfaces.
3. Governed Control Plane: A local LiteLLM deployment backed by PostgreSQL providing:
   • Provider spending limits ($5/day in the example configuration).
   • Routing by complexity via auto-mode, keeping standard edits local while escalating hard tasks to Mistral and DeepSeek.
   • Local observability through the LiteLLM Admin UI to review prompt logs and token usage.

Next Steps & Experiments

If you want to take this setup further, consider exploring:

• Model Context Protocol (MCP): Connect local MCP servers to OpenCode for safe database inspection, live documentation lookups, or issue tracking.
• Routing by Context Size: Configure LiteLLM to hand off oversized file trees to models with a larger context while keeping short prompt cycles local.
• Semantic Scoring with Embeddings: Replace keyword rules with an embedding classifier to automate tiering dynamically.

The main benefit is control. You can see which model handled a request, keep routine work local, limit cloud spending, and inspect the result. Requests sent to hosted models still leave the machine, and the logs make that boundary visible.