DEV Community: René Zander

Claude Code with Local LLMs and ANTHROPIC_BASE_URL: Ollama, LM Studio, llama.cpp, vLLM

René Zander — Wed, 29 Apr 2026 05:53:48 +0000

Native Anthropic endpoints, tool-call compatibility, and context-window sizing for local Claude Code.

Last tested: April 2026. See Changelog at the bottom.

TL;DR cheat sheet

Goal	Use
MacBook Air	Gemma 4 26B-A4B Q4, 32K context, LM Studio or Ollama
MacBook Pro	Gemma 4 26B-A4B Q4 / UD-Q4, 64K context, llama.cpp or LM Studio
Claude Code minimum	32K context (anything below is a chat demo)
Best local backend	LM Studio or Ollama first; llama.cpp for advanced; vLLM for servers
Avoid	8K / 16K context, dense 31B Gemma 4 on 32 GB machines, old llama.cpp builds

The local-Claude-Code rule of thumb

Three things decide whether a local Claude Code session works:

Model quality decides whether the answer is smart.
Tool-call formatting decides whether Claude Code can act on the answer.
Context length decides whether the session survives past the first few edits.

For local coding agents: 32K is the floor. 64K is the sweet spot. Anything below 32K is a chat demo, not Claude Code.

Recommended setup

Use this first. Don't shop the buffet of alternatives until you've tried this one.

Backend: LM Studio (≥ 0.4.1) or Ollama (≥ v0.14.0) — both expose a native Anthropic compatible local endpoint, no proxy needed.
Model: gemma4:26b-a4b (Gemma 4 26B-A4B-it, Q4 quant). MoE active-param ≈ 3.88 B → laptop-friendly latency, tool-use trained directly into the model.
Context: 32K context on a MacBook Air, 64K context on a MacBook Pro M5 Pro/Max with 48 GB+ RAM.
Machine: 32 GB+ RAM strongly preferred. 24 GB works at 24K–32K with care.

If you don't have Anthropic-compatible mode and only have an OpenAI compatible local endpoint running, run LiteLLM in front (see section on LiteLLM).

1. Environment variables Claude Code reads

# Where Claude Code POSTs requests. Default: https://api.anthropic.com
ANTHROPIC_BASE_URL=http://localhost:11434

# Sent as auth. Local servers usually accept any non-empty value.
ANTHROPIC_AUTH_TOKEN=ollama

# Map Claude Code's "claude-opus-X-Y" / "claude-sonnet-X-Y" / "claude-haiku-X-Y"
# to model names your local backend serves.
ANTHROPIC_DEFAULT_OPUS_MODEL=gemma4:26b-a4b
ANTHROPIC_DEFAULT_SONNET_MODEL=gemma4:26b-a4b
ANTHROPIC_DEFAULT_HAIKU_MODEL=gpt-oss:20b

claude

Or override per-invocation:

claude --model gemma4:26b-a4b

If ANTHROPIC_BASE_URL is set but the URL doesn't respond with the right shape, Claude Code does not fall back to the cloud. It errors out.

2. Context length: the hidden failure mode

Claude Code is not a chat prompt. Before your actual request, the backend sees:

Claude Code's system prompt (~6–10K tokens by itself)
tool definitions for Read / Edit / Bash / Grep / Glob / TodoWrite
conversation history
file excerpts and full reads
diffs
command output
retry/error messages from failed tool calls

That means 8K and 16K contexts are misleading tests. They may answer a chat question, but they are not enough for reliable agentic coding. The session survives a handful of turns, then silently degrades — file edits truncate, tool calls drop arguments, the loop gets confused.

Practical context tiers

Context	Verdict	What happens
8K	Broken for Claude Code	System prompt + tools eat the window before your code arrives. Chat-only.
16K	Demo only	Tiny edits, short sessions. Not a real test of any model.
25K	LM Studio's stated minimum	Good enough for small tasks if tool calls are reliable.
32K	Real minimum (32K context).	Ollama recommends this floor. Use as your default.
64K	Sweet spot (64K context).	Best balance on 32GB+ machines. Handles medium repos and multi-file edits.
128K+	Diminishing returns	Prefill latency and KV-cache memory rise hard. Worth it only on high-memory servers, and only for repo-wide reads.

Apple Silicon context presets

Machine	Recommended context	Notes
MacBook Air M5, 16 GB	16K–24K	Use smaller models (≤8B). 26B-A4B is tight.
MacBook Air M5, 24 GB	24K–32K	32K is the target; keep other apps light.
MacBook Air M5, 32 GB	32K	Best Air setup. Higher rarely beats thermal throttling.
MacBook Pro M5 Pro, 24 GB	32K	Better sustained perf than Air at the same context.
MacBook Pro M5 Pro, 48/64 GB	64K	Sweet spot for serious local coding.
MacBook Pro M5 Max, 64/128 GB	64K default, 128K experimental	Use 128K for repo-wide analysis, not every edit loop.

Note: backend docs differ — LM Studio says "start at 25K, increase for better results," Ollama recommends 32K. Use 32K as the cross-backend baseline. Reading "25K" as "25K is enough" is the most common mistake.

3. Claude Code Ollama setup (native, v0.14.0+)

Ollama announced Anthropic Messages API compatibility on 2026-01-16. No proxy, no LiteLLM, no nothing.

# Set context length first — this is the most important knob
export OLLAMA_CONTEXT_LENGTH=32768   # 65536 on a Pro

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_BASE_URL=http://localhost:11434

claude --model gemma4:26b-a4b

Cloud-hosted Ollama models work too:

claude --model glm-4.7:cloud
claude --model minimax-m2.1:cloud

Two known limits of Ollama's Anthropic-compat layer (April 2026):

No prompt caching. Anthropic's cache_control doesn't apply — every Claude Code request re-processes the system prompt and conversation history from scratch.
No tool_choice. Claude Code occasionally uses tool_choice to force a specific tool call. Ollama's compat layer ignores it. When it matters, Claude Code may pick the wrong tool and get stuck in a loop.

4. Claude Code LM Studio setup (native, 0.4.1+)

LM Studio added the Anthropic-compatible /v1/messages endpoint on 2026-01-30. Streaming, tool calls, and message-shape are all supported natively.

# Set context to at least 32K in the LM Studio UI (or higher; see section 2)
lms server start --port 1234

export ANTHROPIC_BASE_URL=http://localhost:1234
export ANTHROPIC_AUTH_TOKEN=lmstudio

claude --model openai/gpt-oss-20b

For VS Code with the Claude Code extension (env vars from your shell are NOT inherited by VS Code):

// .vscode/settings.json
"claudeCode.environmentVariables": [
  { "name": "ANTHROPIC_BASE_URL", "value": "http://localhost:1234" },
  { "name": "ANTHROPIC_AUTH_TOKEN", "value": "lmstudio" }
]

LM Studio's docs say "at least 25K." Set 32K. See section 2.

5. Claude Code llama.cpp setup (Apple Silicon fast path for Gemma 4 26B-A4B)

If you're on Apple Silicon and want the absolute lowest overhead with Gemma 4 26B-A4B, llama.cpp's server is faster per-token than Ollama or LM Studio. You need a recent build (one that supports -hf for HuggingFace pulls and --jinja for chat templates).

./build/bin/llama-server \
  -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M \
  --host 127.0.0.1 \
  --port 8080 \
  -ngl 99 \
  -c 65536 \
  --jinja

export ANTHROPIC_BASE_URL=http://localhost:8080
export ANTHROPIC_AUTH_TOKEN=llama-cpp
claude --model gemma-4-26B-A4B

Flags that matter:

-c 65536 sets 64K context (drop to -c 32768 on tighter machines).
-ngl 99 offloads all layers to Metal/GPU.
--jinja is required for Gemma 4's chat template to render correctly. Without it, tool calls won't format and you'll see <unused24> / <unused49> tokens leaking into output.
-hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M pulls the GGUF straight from HuggingFace.

Caveat: llama.cpp's Anthropic-compat is partial. Works for chat and basic tool calling. Streaming-shape and some Anthropic-specific request fields are rougher than Ollama or LM Studio. If something breaks weirdly, fall back to Ollama. llama.cpp is the speed play, not the compatibility play.

6. Claude Code vLLM setup (native + tool parser)

vLLM ships an official Claude Code integration. Three things at server start: a tool-calling-capable model, --enable-auto-tool-choice, and the right --tool-call-parser.

vllm serve openai/gpt-oss-120b \
  --served-model-name my-model \
  --enable-auto-tool-choice \
  --tool-call-parser openai \
  --port 8000

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=dummy
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_DEFAULT_OPUS_MODEL=my-model
export ANTHROPIC_DEFAULT_SONNET_MODEL=my-model
export ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model

claude

The --tool-call-parser value depends on the model family — openai for the gpt-oss family, llama3_json for Llama 3.x, hermes for Hermes. Wrong parser → tool calls return as plain text and Claude Code's edit/grep/bash tools silently no-op.

7. LiteLLM — for fallbacks, not for translation

With Ollama, LM Studio, llama.cpp, and vLLM all speaking native Anthropic now, LiteLLM's role changes. It's no longer "the translator" — it's the router for fallbacks, request logging, per-tenant keys, and rate limits. Also the right answer if your only local option is an OpenAI compatible local endpoint.

# litellm-config.yaml
model_list:
  - model_name: claude-opus-4-7
    litellm_params:
      model: openai/my-vllm-model
      api_base: http://vllm:8000/v1

  - model_name: claude-sonnet-4-6
    litellm_params:
      model: ollama/gemma4:26b-a4b
      api_base: http://ollama:11434

  - model_name: claude-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks:
    - claude-opus-4-7: ["claude-haiku-4-5"]   # local fail → cloud Haiku

The single biggest win: when a local tool call silently fails, LiteLLM falls back to cloud Haiku transparently. Claude Code keeps working.

8. Common failures (the error strings developers google)

`tool_use parse error` / `invalid tool call` / `tool_use is not supported`

Three different symptoms, one root cause: the model is not emitting Anthropic-format tool_use content blocks.

The most deceptive symptom is the silent one — Claude Code starts, prints the model's plain-prose answer ("I would change the file like this..."), and nothing happens. No file edit, no error.

Common causes (April 2026):

vLLM: missing --enable-auto-tool-choice or wrong --tool-call-parser.
Ollama: model that wasn't trained for tool calling (avoid stock llama3.x instruct).
llama.cpp: missing --jinja. The chat template renders incorrectly and you see literal <unused24> / <unused49> tokens.
LM Studio: model file is fine but the loaded preset uses the wrong template.

`context length exceeded` / model stopped mid-edit

Claude Code's prompts overflow the configured window. The session may finish a single turn, then truncate the next file edit silently. Fix: raise context to at least 32K. If you're already at 32K and still hitting this, the model is reading too aggressively — drop to fewer tools or shorter file reads.

`empty assistant response`

Backend returned 200 OK with an empty content array. Causes:

Streaming SSE format mismatch (mostly llama.cpp).
Tool-call parser swallowed the message because it couldn't parse it.
Model emitted only a <unused24> / <unused49> token and the parser dropped the rest.

Fix: switch backend (Ollama or LM Studio if you were on llama.cpp), or upgrade llama.cpp to a build with the patched Gemma 4 chat template.

`model not found` / `404 the model X does not exist`

Claude Code asked for claude-opus-4-7 but the backend serves gpt-oss:20b or gemma4:26b-a4b. Fixes:

Set ANTHROPIC_DEFAULT_OPUS_MODEL (plus _SONNET_ and _HAIKU_) to the backend's actual model name.
Use claude --model <backend-name> per call.
Map the names in LiteLLM (the model_name: field is what Claude Code asks for; model: is what gets served).

`messages: Extra inputs are not permitted` (HTTP 422)

Some backends are stricter than Anthropic's own. They reject Anthropic-specific fields (cache_control, thinking, tools[].input_schema, metadata.user_id). Fix: upgrade the backend, or run a small middleware proxy that strips the unsupported fields before forwarding.

`ANTHROPIC_BASE_URL` ignored / Claude Code still calls the real API

Env var was set in .zshrc after the shell session started — restart the terminal.
~/.config/claude/config.json or a --api-key flag is overriding the env var.
VS Code: env vars from your shell are NOT inherited. Use claudeCode.environmentVariables in workspace settings (section 4).

echo $ANTHROPIC_BASE_URL inside the same shell that runs claude. If empty, you have a sourcing problem.

9. Debug flow

When something breaks, walk this tree before swapping backends:

Did the model load?
- No → check quant size vs RAM. 26B-A4B Q4 needs ~16 GB free; bigger quants need more.
Is the context at least 32K?
- No → raise to 32K (Air) or 64K (Pro). See section 2.
Are tool calls malformed? (Look for <unused24>, <unused49>, plain prose where you expected an edit.)
- Yes → switch to native Anthropic mode (Ollama/LM Studio), or for vLLM verify --tool-call-parser, or for llama.cpp add --jinja.
Does Claude Code stop mid-edit?
- Yes → context exhaustion. Lower context targets in your tools, or use a faster quant so the model finishes turns before the window reuse cycle.
Is the model hallucinating files that don't exist?
- Yes → the model isn't calling Read before Edit. Add a CLAUDE.md rule that requires reading before editing, or use a tool-finer model (Gemma 4 26B-A4B is solid here).

10. Smoke test

Verify your setup with one prompt. Ask Claude Code:

Create a small FastAPI app with one /health endpoint, add a pytest test for it, run pytest, and fix any failures.

Passes if:

It reads/writes files correctly (no hallucinated paths).
It runs the test command (you see real pytest output).
It patches a failure (e.g. missing dependency) without losing context.
It does not lose tool-call format (no <unused24> / <unused49> leakage).
It does not truncate after the first edit.

Expected terminal feel:

✓ model loaded     (gemma4:26b-a4b, Q4_K_M)
✓ context: 32768
✓ tool call parsed (Edit)
✓ edited file      (app.py)
✓ tool call parsed (Bash)
✓ tests passed

If you don't see all five, walk the debug flow above.

11. Compatibility matrix (April 2026)

Backend	Native Anthropic API	Tool calls	Context floor	Notes
Ollama (≥ v0.14.0)	Yes	Depends on model	32K context (cross-backend baseline)	Easiest setup. No prompt caching, no `tool_choice` (see section 3).
LM Studio (≥ 0.4.1)	Yes	Yes (out of the box)	Stated 25K, use 32K	Streaming + `tool_use` blocks supported natively. VS Code extension takes workspace env vars.
llama.cpp server	Partial	Yes with `--jinja`	32K, 64K context on Pro	Lowest overhead on Apple Silicon. Rougher Anthropic-compat. Best path for Gemma 4 26B-A4B.
vLLM	Yes	Yes with `--enable-auto-tool-choice` + correct parser	Model-dependent	Best throughput. Requires correct parser per model family.
LiteLLM	Routes to any backend	Whatever the backend supports	n/a	Use for fallbacks and logging, or to wrap an OpenAI compatible local endpoint as Anthropic.
Direct Ollama < v0.14.0	No	No	n/a	Upgrade.

12. Hardware × model × context × backend (the cheat-sheet table)

A developer should not have to infer what to use:

Machine	Model	Context	Backend	Verdict
MacBook Air M5, 16 GB	Gemma 4 E4B	16K–24K	LM Studio	usable for small tasks
MacBook Air M5, 24 GB	Gemma 4 26B-A4B Q4	24K–32K	Ollama / LM Studio	good
MacBook Air M5, 32 GB	Gemma 4 26B-A4B Q4	32K	Ollama / LM Studio	best Air setup
MacBook Pro M5 Pro, 48 GB	Gemma 4 26B-A4B Q4/UD-Q4	64K	llama.cpp / LM Studio	sweet spot
MacBook Pro M5 Max, 64 GB+	Gemma 4 26B-A4B or 31B	64K–128K	llama.cpp / vLLM	best local

This is the single most copied table in this gist. Bookmark it.

13. Gemma 4 26B-A4B: the Apple Silicon sweet spot

For Mac local Claude Code, the standout Gemma 4 variant is 26B-A4B-it, not the dense 31B. Reasons:

Google trained tool-use directly into Gemma 4 (not bolted on as a fine-tune). It works on the first try, not after three retries.
The 26B MoE activates only ~3.88 B params per inference, so latency is in the 4 B-model range — around 300 tok/sec on M2 Ultra.
Strong tool-use behavior, good enough coding quality for private/local workflows.
Fits at useful context sizes on high-memory MacBooks.

Why 26B-A4B instead of 31B?

Faster tool calls — every Claude Code turn is bottlenecked by tool-call latency, not single-shot quality.
Lower active-parameter count keeps prefill cheap.
Better fit for laptops — 31B dense needs more RAM and more thermal headroom.
Enough quality for iterative coding; the agent loop matters more than peak IQ.
31B may be better for single-shot answers — but Claude Code is many small turns, not one big answer.

For Gemma 4 local coding specifically: pick 26B-A4B unless you're on a 64 GB+ Pro and you've measured that 31B Q4 actually finishes turns faster on your hardware.

14. Other model picks for Claude Code (April 2026)

If Gemma 4 isn't available or you want to compare:

gpt-oss:20b — easy starting point. Tool calling reliable, runs on a single decent GPU. Recommended in Ollama's and LM Studio's official Claude Code blog posts.
gpt-oss:120b — much smarter on real codebases. The vLLM Claude Code integration page uses this as the example. Needs serious VRAM.
qwen3-coder — purpose-built for coding. Strong tool-call performance on Ollama. Frequently called the strongest local pick for Claude Code in March/April 2026 community threads.
qwen3.5 family — the 35B MoE variants are reported as the strongest agentic-coding open models in this size class. Verify tool-call support per quant.
glm-4.7-flash / glm-4.7:cloud — strong agentic coder. Available as an Ollama cloud model (no local GPU needed).
minimax-m2.1:cloud — newer Ollama cloud option, agentic-tuned.

What to avoid: stock llama3.x instruct models without tool fine-tuning. They will look like they work, then silently fail on file edits.

15. Setups I would avoid

8K context. Too small for Claude Code. The system prompt eats it before your code arrives.
16K context. Demos only. Don't judge a model by 16K behavior.
Old llama.cpp builds with Gemma 4. No --jinja or no patched chat template → <unused24> / <unused49> token leakage.
128K context on a 32 GB laptop. KV cache + prefill latency tax > the benefit.
Judging model quality before tool calls are stable. Fix the parser/template first, then evaluate the model.
Routing through LiteLLM when the backend is already native Anthropic. Adds a hop for nothing — only use LiteLLM for fallbacks or when wrapping an OpenAI compatible local endpoint.

16. Reusable startup script

Drop this in start-claude-code-local.sh and chmod +x. Default 32K context, override via env.

#!/usr/bin/env bash
set -euo pipefail

export OLLAMA_CONTEXT_LENGTH="${OLLAMA_CONTEXT_LENGTH:-32768}"
export ANTHROPIC_BASE_URL="${ANTHROPIC_BASE_URL:-http://localhost:11434}"
export ANTHROPIC_AUTH_TOKEN="${ANTHROPIC_AUTH_TOKEN:-ollama}"
export ANTHROPIC_DEFAULT_OPUS_MODEL="${ANTHROPIC_DEFAULT_OPUS_MODEL:-gemma4:26b-a4b}"
export ANTHROPIC_DEFAULT_SONNET_MODEL="${ANTHROPIC_DEFAULT_SONNET_MODEL:-gemma4:26b-a4b}"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="${ANTHROPIC_DEFAULT_HAIKU_MODEL:-gpt-oss:20b}"

echo "Starting Ollama with context=$OLLAMA_CONTEXT_LENGTH"
ollama serve &
OLLAMA_PID=$!

# Wait for Ollama to be ready
until curl -sf "$ANTHROPIC_BASE_URL/api/version" > /dev/null; do
  sleep 0.5
done

echo "Launching Claude Code → $ANTHROPIC_BASE_URL"
echo "Model: $ANTHROPIC_DEFAULT_OPUS_MODEL"

claude

kill $OLLAMA_PID 2>/dev/null || true

For LM Studio, swap ollama serve for lms server start --port 1234 and update the env vars accordingly.

This script (and additions for other backends as they ship) lives in the companion repo:

github.com/renezander030/local-ai-coding-stack — git clone, chmod +x scripts/start-claude-code-local.sh, run.

17. Production recommendation

For real work, do not let Claude Code talk directly to a single local endpoint without a fallback path:

Claude Code
   │  ANTHROPIC_BASE_URL
   ▼
LiteLLM (router + logger)
   │  primary
   ▼
Ollama / LM Studio / llama.cpp / vLLM (local)
   │  on tool-call failure or 5xx
   ▼
Cloud Claude Haiku (fallback)
   │
   ▼
Audit log

Model swaps without restarting Claude Code; transparent fallback when local tool calling silently fails; request logs you can grep when something goes wrong. Same five-contract pattern from agent-approval-gate.

18. When local models are the wrong choice

Repo-wide refactors. Multi-step tool flows compound silent tool-call failures. Local fine-tunes drop accuracy fast.
Security-sensitive edits without an approval gate. Use agent-approval-gate and the local-vs-cloud question becomes secondary.
Tool-heavy sessions (50+ tool calls). Every silent failure compounds.
Anything billed by your time. A failed local tool call costs your time; a successful Haiku call is roughly $0.001.

Local Claude Code is a fit for: chat-only assist on private code, classification/summarization sub-steps, air-gapped environments.

Series

This gist is part of Production AI Automation Notes — a running set of repos and gists on shipping AI agents outside demos. Other entries:

agent-approval-gate — production-safe approval pattern. Drop in front of any local-model agent that touches real systems.
Production AI Automation Notes #1: Agent Approval Gates
CLAUDE.md — 10 rules for Claude Code, edit-time and runtime
Context7 v2 — enterprise GraphQL MCP pattern

Sources

Reader contributions

If you get this working on a different Mac/RAM/model combo, comment with:

machine
RAM
backend
model + quant
context length
what worked / what failed

The compatibility matrix and hardware table are updated weekly from these reports.

Changelog

2026-04-28

Added TL;DR cheat sheet, Recommended setup section, smoke test, debug flow, reusable startup script, hardware × model × context × backend table.
Expanded error-string section to include <unused24> / <unused49> template-leak symptoms.
Added 26B-A4B vs 31B comparison bullets.
Added "Setups I would avoid."
Renamed Update log → Changelog.
Added Gemma 4 26B-A4B context recommendations.
Added MacBook Air vs Pro presets.
Added 32K / 64K Claude Code guidance.
Backend coverage rewritten: Ollama, LM Studio, vLLM all native Anthropic; llama.cpp added as Apple Silicon fast path.
LiteLLM repositioned as fallback router (and OpenAI-compat wrapper), not translator.

2026-04-22

Initial publish.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Voice AI in Production: From RunPod to Hosted Kubernetes

René Zander — Thu, 23 Apr 2026 13:10:51 +0000

Your voice model works in a demo. The same model in production stalls under concurrent load. The model file is identical. So is the GPU card. Only the deployment changed.

If your TTS service runs on a single RunPod pod, you've already met this wall. You handle one request per GPU at a time. A crash costs ninety seconds to reload the model. Failover isn't in the setup. Your marketing page says "generate narration instantly." Your infrastructure says "please form an orderly queue."

The gap between prototype and product sits in the infrastructure layer. The voice AI companies asking me for help want hosted Kubernetes because their engineering hours are going into pod management when they should be going into the model.

Single Pod Stops Working Around Four Concurrent Users

A voice model like Qwen3-TTS loads into GPU memory once. Each inference holds that memory plus a working buffer. On an H100 you fit the model plus maybe four to eight concurrent generations before latency goes off a cliff. On a 4090, less.

That number is the ceiling of your business on a single pod. You can buy a bigger GPU. You can't buy a second one attached to the same pod. The moment you need more than one machine, you're in distributed-systems territory whether you planned for it or not.

What Actually Breaks First

Cold starts are the obvious one. A pod that dies takes ninety seconds to reload the model into VRAM, and during those ninety seconds your users hit 502s. Kubernetes with a warm pool absorbs it.

Voice profile storage gets worse the moment you scale. On one pod a user's cloned voice sits on local disk. Spread that across ten pods and you need shared storage plus replication on every node that might serve that user. Miss one and the next request uses the wrong voice or errors out.

Then there's the cost trap. You rent preemptible GPUs at a third the price, and one afternoon the cloud provider takes them back with two minutes' warning. A single pod goes dark. A K8s cluster with a warm replica serves the next request from a different node and nobody sees the eviction.

Fine-tuning is the one that forces the decision. The moment you offer custom voice creation, you need training runs that don't block inference. That means another queue, another GPU pool, and priority rules that don't collide with live inference. A single pod can't multiplex that, and bolting it on later costs more than designing for it up front.

What the K8s Layer Actually Buys You

Keep model weights on the node, where they outlive any single pod. New pods scheduled to that node get a warm cache and start in under ten seconds instead of ninety.

Not every request needs an H100. Real-time low-latency responses can run on a 4090 nodepool, premium batch generations go to H100. Nodepool labels and taints handle the routing without the application code caring.

Pick queue depth as your autoscale signal. CPU metrics are useless here. GPU utilization also lies when the model is streaming. The number that maps to user-visible latency is requests waiting in the queue.

Show the queue depth back to the caller. "You're number four, about forty seconds" keeps users on the line. A thirty-second timeout with no feedback teaches them your service is broken.

None of this is visible in a Voicebox README.

Hosted K8s Is the Service

Voice AI companies keep asking for this because it's the gap between a model that works and a product that holds up under paying users. You can learn Kubernetes while trying to ship, but most founders can't afford both learning curves at once. Hiring a team is slow. Handing the layer off gets your engineering hours back on the model.

If your voice AI product is past the demo and breaking under real traffic, I run the K8s layer so your team stays on the model. Contact on the blog.

Your Model Is the Value. Your Pod Isn't.

Are your engineering hours going into the model or into the pod that serves it? If the answer is the pod, you're paying to solve the wrong problem twice. Handle the infrastructure properly or hand it off. A half-built version while your competitor ships isn't a strategy.

Originally published at renezander.com.

Where is your engineering time actually going right now: into the model or into the pod that serves it?

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Ten CLAUDE.md rules for Claude Code - four edit-time, six runtime

René Zander — Thu, 23 Apr 2026 04:06:42 +0000

Forrestchang's andrej-karpathy-skills CLAUDE.md is four rules aimed at the moment Claude is writing code. They work. What they don't cover is the moment Claude is running. Once a Claude-driven pipeline goes to production, a different failure mode shows up: confident outputs, silent budget overruns, destructive side-effects, prompt injection via user input.

These six extension rules are what I shipped into fixclaw — a Go pipeline engine where Claude drafts, classifies, and summarizes, but never executes. Deterministic code does. The rules below are what made that claim stick.

Merge with your own project rules. Tradeoff: these bias toward caution over autonomy.

Forrestchang's four (edit-time) — unchanged

Think Before Coding — state assumptions, surface tradeoffs, ask when unclear.
Simplicity First — minimum code, no speculative abstractions.
Surgical Changes — touch only what the task requires.
Goal-Driven Execution — define success criteria, loop until verified.

(Full text: forrestchang/andrej-karpathy-skills/CLAUDE.md.)

Six runtime rules — lessons from fixclaw

5. Deterministic First

Claude is for judgment calls. Plain code does everything else.

Fetching, filtering, routing, persisting, dispatching — none of it is a language task. Don't ask the model to "decide if we should retry" when a status code already answers. Use the model for: classification, drafting, summarization, extraction from unstructured text. That's the whole list.

The failure mode without this rule: the model makes a routing decision one week, a different routing decision the next, and you've reinvented flaky if-else at $0.003/token.

6. Declare Budgets, Halt On Breach

No silent overruns. Ever.

Every AI step runs under a token budget: per-step, per-pipeline, per-day. Exceeding any of the three halts the pipeline immediately, logs the breach, and surfaces it to the operator. Budgets live in config, not in prompts.

budgets:
  per_step_tokens: 2048
  per_pipeline_tokens: 10000
  per_day_tokens: 100000

The failure mode without this rule: a runaway loop burns $40 overnight and you find out from the invoice.

7. Human-In-The-Loop Is A First-Class Step Type

Label destructive actions. Require approval. No exceptions via flags.

Anything touching the outside world — sending an email, updating a CRM, posting a message — is an approval step, not an ai step. The approval is routed to an operator channel (Slack, Telegram, whatever) with approve/edit/reject controls. The pipeline blocks until a decision is recorded.

- name: approve-send
  type: approval
  mode: hitl
  channel: telegram

The failure mode without this rule: a hallucinated follow-up email goes to a real customer.

8. Validate AI Output Against A Schema

Unstructured strings don't belong in deterministic downstream code.

Every AI step declares an output schema. The runtime rejects anything that doesn't match — missing fields, wrong types, out-of-range numbers. Rejected outputs trigger a retry (under budget) or halt.

output_schema:
  type: object
  required: [match, reason, score]
  properties:
    match:  { type: boolean }
    reason: { type: string, maxLength: 280 }
    score:  { type: integer, minimum: 0, maximum: 100 }

The failure mode without this rule: a boolean comes back as the string "maybe" and a downstream if branches the wrong way.

9. Sanitize Operator Input Before It Reaches A Prompt

User-supplied text is not trusted.

Before any operator or external input enters a prompt, strip role markers (system:, assistant:, <|im_start|> variants), enforce length limits, and normalize markdown so formatting can't break prompt boundaries. This is prompt-injection defense, not input validation — the goal is to stop an attacker from pivoting the model mid-run.

10. Log Rejections Silently

Don't narrate to the attacker.

When input is rejected for sanitization or schema violations, log internally — never echo the rejection reason back to the source. A detailed error message is a free signal that tells the attacker which pattern to try next.

The "working if" test

The full ten rules are working if:

Diffs are smaller and more targeted (rules 1–4).
Pipeline runs have predictable token costs (rule 6).
No AI output ever reaches a production side-effect without a human approval record (rule 7).
Downstream code never branches on a malformed AI response (rule 8).
Operator-channel logs show silent rejections rather than echoed errors (rules 9–10).

If even one of those is failing, the rule isn't enforced — it's aspirational.

Originally published as a gist: https://gist.github.com/renezander030/2898eb5f0100688f4197b5e493e156a2 — weekly gists on Claude Code, MCP, and automation at @renezander030.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

95% of PII Redaction Doesn't Need an LLM. The Other 5% Is Where Your Masker Leaks.

René Zander — Tue, 21 Apr 2026 11:43:05 +0000

A VP at an SAP shop told me recently: "Every time we copy production to our lower environments, PII leaks. And no, we're not throwing an LLM at it. That's a thousand times the compute of what we already run."

He's right.

Most of the PII redaction problem in enterprise data isn't a neural network problem. It's a lookup table problem. And the incumbents already solve it. SAP TDMS, Delphix, Informatica, IBM InfoSphere Optim. All schema-aware. All row-level. All deterministic.

The 95% Where Deterministic Wins

In a SAP production database, the schema tells you almost everything. KNA1-NAME1 is a customer name. BSEG-IBAN is a bank account. USR02-BNAME is a user ID. A YAML rule says: "for this column type, replace with this pattern." Done.

The math is brutal. A regex plus a lookup table costs microseconds per row. A 1.5B-parameter model costs 10 to 50 milliseconds per row, even on a GPU. That's three to five orders of magnitude. A nightly batch copy that finishes by morning with TDMS would take weeks with an LLM in the loop.

Compute isn't even the main argument.

Referential integrity is. "Anna Müller" has to become "Person_47" consistently across 200 tables. KNA1, VBAK, VBKD, BSEG, wherever the customer ID travels. Deterministic pseudonymization with an HMAC and a scoped salt gives you that for free. Neural outputs drift.

Auditability is. A regulator asks: "show me the rule that masked this column." A YAML rule is defensible. A model output is not.

So for any SAP field with a known schema type, deterministic masking wins. Full stop. Don't let anyone sell you a neural-network-powered "modernization" of that layer.

Where a Fine-Tuned Model Earns Its Compute

Here's what TDMS, Delphix, and their peers silently miss.

Free-text columns. BSEG-SGTXT, the long-text field where someone typed "Ansprechpartner Anna Müller, Tel +49-170-...". Ticket descriptions from ServiceNow mirrored into dev. Email bodies stored as CLOBs. ADRC annotations. The column type is "text." The content is gold-mine PII.

Unstructured attachments. PDFs, scanned invoices, OCR'd contracts pulled into dev via ArchiveLink. Names and IBANs mid-prose, not in a column.

Schema drift. Consultants add Z-tables. The data steward hasn't classified them yet. Deterministic tools don't know the column holds PII. They pass the data through untouched.

On these, rule-based tools do one of two things. They wipe the whole column, destroying test fidelity, so the dev team can't debug against realistic data. Or they miss the PII entirely, and you get a compliance incident.

A German-specialized redactor earns its keep here because the alternative isn't "faster regex." It's "no coverage at all."

The Hybrid Architecture

This is the part that actually ships.

A classifier pass on the SAP copy. Cheap heuristics (column-name keywords, column type, sample-value regex) flag each column as structured_pii, free_text, or safe.
Deterministic masker handles structured_pii. TDMS or whatever you already run.
Fine-tuned LLM redactor runs only on free_text, attachments, and unclassified Z-columns.
A consistency bridge. Both paths share a pseudonym table keyed by HMAC(value, tenant_salt). "Anna Müller" becomes "Person_47" whether she was caught by regex or by the model.

Compute budget: the LLM runs on maybe 1 to 5 percent of the cells. Total cost is still dominated by the deterministic layer. You're not replacing TDMS. You're covering its blind spots.

What I Won't Claim

Three things I won't sell you:

The LLM is cheaper than a regex. It isn't. Ever.
It replaces your incumbent masking vendor. It doesn't.
A benchmark against TDMS on structured columns is meaningful. You lose that benchmark. Benchmark on free-text and attachments, where deterministic tools score near zero.

The honest pitch to the VP was this. "You're right. For the 90% structured case, keep TDMS. The model is the long-tail layer. It runs only over the free-text fields and attachments your current tools silently leak. Small job. Different problem."

That's the conversation that lands. Not "replace your stack." Not "AI-powered everything."

Regex for the schema. LLM for the shadows.

I reserve my audits for teams ready to take action on the results.

Book a 30-min call →

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

What llama.cpp's Pace Tells You About On-Prem LLM Readiness

René Zander — Tue, 14 Apr 2026 08:42:06 +0000

Your team asked for GPU budget for self-hosted inference. You said "not yet" because last time you checked, the tooling wasn't production-grade. That was true 18 months ago. It's not true now, and the delay is costing you leverage you don't know you're losing.

I'm writing this because most decision-makers I talk to are still running on an outdated mental model of what self-hosted LLM infrastructure looks like. The software moved. The org didn't.

The Team That Celebrated Too Early

I watched a team spin up on-prem inference, celebrate for a week, then watch it rot because nobody owned it. Six months later they were back on the API, having spent the budget anyway.

This is the failure mode nobody talks about. The software works. It's been working for a while now. The problem is everything around it.

Nobody owns the stack. Running self-hosted inference in production means someone on your team owns model updates, hardware failures, quantization tradeoffs, and latency tuning. That's a different job than calling an API. If you don't staff it, the deployment decays.

Procurement kills momentum. GPU capacity is a capital expenditure conversation, not a software download. If you don't already have data center access or cloud-GPU contracts, the blocker isn't the code. It's a procurement cycle that takes months. By the time the hardware arrives, the team that asked for it has moved on.

Model selection is real work. The quantized model that runs great for summarization falls apart on code generation. There is no default. Every use case needs evaluation, and evaluation takes time nobody budgets for.

These are solvable problems. But teams that skip them end up with on-prem deployments that nobody trusts, and leadership that says "see, I told you it wasn't ready" when the real issue was organizational, not technical.

What Changed While You Were Waiting

A year ago, I would have told you to hold off. Not anymore.

You can now split inference across multiple GPUs without patching anything yourself. The server mode handles concurrent requests behind a load balancer. 1-bit quantization means models that needed high-end hardware run on modest configs without catastrophic quality loss.

Multi-modal support landed. Speculative decoding shipped, cutting latency on long outputs. The API compatibility layer means your existing code that talks to cloud providers works against a self-hosted endpoint with a URL change.

I deployed a quantized model on a client's on-prem GPU last month. Set up the server, pointed the app at it, ran inference. It worked. First try. That sentence would have been fiction two years ago.

The gap between "experimental" and "production-ready" closed while most orgs were waiting for someone else to go first.

The Decision You're Actually Making

This isn't a permanent binary. It's a portfolio allocation.

Move workloads on-prem when:

Your inference volume is high enough that API costs became a material line item.
You need predictable latency without network variability.
Compliance or data residency requirements mandate it. But verify this. Many teams assume they need on-prem when they don't.
You have an engineer who wants to own the stack.

Stay on the API when:

You're prototyping or usage is unpredictable.
You need frontier models not available as open weights.
Nobody on your team can own the ops burden.

The mistake I see most often: treating this as all-or-nothing. Start with API. Move specific workloads to self-hosted when economics or data constraints force the conversation. The infrastructure to do it properly exists now. It didn't two years ago.

The Question for Your Next Planning Cycle

The software is ready. The open-weight models are good enough for most production use cases. The tooling matured past the point where "not ready yet" is a defensible position.

The real question isn't whether the technology works. It's whether your org is set up to operate it. That's a staffing decision and a procurement decision, not a technology bet.

If you're still saying "not yet," make sure you're saying it because of an actual blocker, not because of a mental model that expired a year ago.

Related reading:

Self-Hosted LLM vs API: when the math actually works — the decision framework I use with clients (deutsche Version)
LLM API comparison 2026 — Claude vs GPT vs Gemini vs Mistral vs DeepSeek for production (deutsche Version)

I help teams navigate this decision. If your org is evaluating self-hosted inference and you want an honest assessment of readiness, reach out.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Your AI Content Tool Knows Your Strategy. Do You Know Where It Goes?

René Zander — Tue, 07 Apr 2026 06:59:24 +0000

Your team is using AI for content. Everybody is. LinkedIn posts, blog drafts, internal comms, maybe some customer-facing copy too.

And it works. The output is decent, the speed is real, nobody wants to go back to writing everything from scratch.

But have you thought about what you are actually pasting into these tools?

The Prompt Is the Product

Every time someone on your team writes a prompt, they are feeding context into a system they do not control. Brand voice guidelines. Competitive positioning notes. Messaging frameworks. That internal strategy deck someone summarized into a prompt last Tuesday.

This is not hypothetical. This is what good prompts look like. The more context you give, the better the output. So people give more context. They paste in the brief. They paste in the competitor analysis. They paste in the draft that legal has not approved yet.

The tool gets better because your data is better. And your data is sitting on someone else's infrastructure.

The Trust Model Is the Problem

Most AI content tools handle your data the same way: they promise not to train on it. That is the entire security model. A policy page. Maybe an enterprise agreement with a data processing addendum.

Your data still gets processed on shared infrastructure. It still passes through systems you cannot inspect. You are trusting that the vendor's internal controls work perfectly, that no employee has access they should not have, and that every subprocessor in the chain follows the same rules.

For most companies, this never becomes a visible problem. The data does not leak in a way anyone notices. The risk stays theoretical.

Until it does not.

A client asks where their data goes during your AI-assisted content process. Legal needs to document compliance for an audit. A competitor publishes something that looks suspiciously familiar. A new regulation drops that requires you to prove where personal data was processed, not just promise.

The Technology Already Exists

Here is what most people in the content space do not realize: the technology to solve this is not theoretical. It is production-ready. It has been running in cloud infrastructure for years. It just has not reached the content tooling layer yet.

Three capabilities change the game:

Client-side encryption. Your data gets encrypted before it leaves your browser. The server never sees plaintext. It processes encrypted inputs and returns encrypted outputs. The key stays with you. Not with the vendor. Not in their key management system. With you.

Confidential computing. Instead of shared servers where your workload runs alongside everyone else's, your data gets processed in an isolated hardware enclave. The cloud provider cannot see inside it. The vendor cannot see inside it. The operating system cannot see inside it. Your data exists in cleartext only inside a hardware boundary that nobody else can access.

Attestation. Cryptographic proof of what code is running in that enclave. Not a vendor's word that they are running the right version. A hardware-signed certificate that you can independently verify. You know exactly what software touched your data because the hardware tells you, not the vendor.

These are not research papers. AWS Nitro Enclaves, Azure Confidential VMs, and GCP Confidential Computing have been generally available for years. The infrastructure is there. The content tools just have not caught up.

Why This Matters Now

Two things are converging.

First, AI adoption in content workflows is no longer experimental. Teams are building real pipelines. They are feeding in real business data, not just test prompts. The volume and sensitivity of data flowing through AI tools is growing every quarter.

Second, regulation is catching up. GDPR already requires you to document where personal data is processed. The EU AI Act adds requirements around transparency and risk management for AI systems. Industry-specific regulations in finance, healthcare, and legal services are getting more specific about AI data handling. "We have a DPA" is becoming insufficient.

The companies that figure out verifiable AI data handling now will not be scrambling when their clients, their board, or their regulator asks how their AI content pipeline handles sensitive data.

What to Ask Your Vendors

You do not need to become a cryptography expert. But you should be asking three questions:

Where does my data exist in plaintext? If the answer is "on our servers," you are in the trust model. If the answer is "only inside a hardware enclave that we cannot access," you are in the proof model.

Can I verify what code processes my data? If the answer requires trusting the vendor's word, that is trust. If the answer involves a hardware attestation you can independently check, that is proof.

Who holds the encryption keys? If the vendor holds them, they can decrypt your data whenever they want, regardless of what the policy says. If you hold them, the vendor literally cannot access your plaintext data even if they tried.

The Shift from Trust to Proof

The content industry is going to go through the same transition that payments, healthcare, and financial services already went through. The question will shift from "do you promise to protect our data?" to "can you prove it?"

Right now, almost nobody in the AI content space is building with these guarantees. That gap will not last.

I am building Teedian, an AI content tool that uses exactly this architecture. Client-side encryption, confidential computing, attestation. Not as a roadmap item, but as the foundation.

If you work in a regulated industry, or you handle client data in your content workflows, or you want to understand what cryptographic privacy looks like in practice, I put together a short brief on teedian.com that walks through the architecture. Plain language, no jargon, 3 pages.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Spend Your Human Thinking Tokens Where They Compound

René Zander — Tue, 31 Mar 2026 08:32:22 +0000

More automations running. More agents deployed. More pipelines humming in the background.

I run about a dozen automated jobs. Daily briefings, proposal generation, content pipelines, data syncing, monitoring alerts. They handle a lot.

But the biggest improvement to my workflow this year wasn't adding more automation. It was getting honest about where my thinking actually matters.

You Have a Token Budget Too

LLMs have context windows. Feed in too much noise and the signal degrades. The output gets worse even though you gave it more to work with.

Human attention works the same way. I have maybe 4 good hours of focused thinking per day. When I spend those hours reviewing cron output or formatting documents or triaging alerts that resolve themselves, I'm burning tokens on low-value work.

The quality of my actual decisions goes down. Not because the decisions got harder, but because I already used up my thinking budget on stuff that didn't need me.

Where I Stopped Spending

I used to review my morning briefing line by line. Check every data point, verify every summary. Then I realized: if the briefing is wrong, I'll notice when the information doesn't match reality later that day. The cost of a slightly wrong briefing at 6:30 is near zero. The cost of spending 20 minutes checking it every morning is real.

Same with monitoring. I had alerts for everything. Cache refreshes, API response times, sync completions. Most of them were informational, not actionable. I stripped it down to alerts that require a decision: something broke, something is about to expire, something needs my approval before it touches an external system.

Data syncing runs on a schedule. If it fails, I get one alert. I don't watch it run. I don't check the logs unless the alert fires.

First drafts of anything. Cover letters, content outlines, research summaries. The AI produces a version. Sometimes it's good enough. Sometimes I rewrite half of it. But I never start from a blank page anymore, and that alone saves the hardest type of thinking: getting started.

Where I Still Spend Every Token

Scoping client work. An AI can research a company, summarize a job posting, draft a proposal. But deciding whether the project is actually worth pursuing? Whether the client's problem is what they say it is? That's pattern recognition built from years of seeing projects go sideways. No automation for that.

Choosing what to build next. I have a backlog of 50 things I could automate, improve, or ship. The AI can't tell me which one moves the needle this week. That decision depends on context it doesn't have: what conversations I had yesterday, what I'm optimizing for this month, what feels right.

Anything with my name on it that reaches another person. Proposals get edited. Posts get rewritten. Client messages get reviewed word by word. The AI drafts. I decide what actually represents me.

System design decisions. Where to draw the boundary between automatic and manual. What gets a human checkpoint and what runs unsupervised. These are the highest-leverage decisions in any AI system, and they're entirely human.

The Honest Ratio

Maybe 20% of my working hours involve focused, high-stakes thinking. The rest is execution, coordination, and maintenance.

Before I built these systems, that ratio was reversed. 80% thinking, 20% execution, and half the thinking was on tasks that didn't deserve it.

The goal was never "automate everything." It was "protect the 20% that matters and make sure I'm not exhausted when I get there."

The Shift

This isn't about working less. I work the same hours. But the distribution changed.

I spend less time on decisions that don't compound. I spend more time on the ones that do. Client relationships, system architecture, strategic bets. The stuff where being sharp at 10 in the morning instead of burned out from triaging alerts actually changes the outcome.

The question isn't how much your AI can do. It's whether you're spending your own thinking tokens on the right things.

Where are you still spending attention that you probably shouldn't?

I help teams figure out where AI should run unsupervised and where humans still need to be in the loop. If that's a question your team is working through, let's talk: cal.eu/reneza

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

AI Skills Are the New Boilerplate. They Solve Almost Nothing.

René Zander — Tue, 24 Mar 2026 11:13:17 +0000

Everyone's sharing their skill libraries right now. "Here are my 20 custom slash commands." "Check out my prompt template collection." "This skill saves me 2 hours a day."

I use skills too. I have about a dozen. They handle cover letters, content pipelines, code review, commit messages. Repeatable workflows where the input and output are predictable.

They cover maybe 10% of what my AI system actually does.

The other 90% is the part nobody shares on social media because it's ugly. It's API integrations that break when headers change. It's state management between sessions. It's error handling for when the third-party service returns garbage. It's monitoring that pages you at 6 AM because a cron failed. It's human-in-the-loop workflows where the AI proposes and you approve before anything touches production.

Skills can't solve this. Every client, every codebase, every problem has different infrastructure underneath. A skill is a template. The work is everything the template doesn't cover.

What Skills Actually Are

A skill is a saved prompt with some structure. Input goes in, the agent follows instructions, output comes out. It works when the task is the same shape every time.

"Generate a cover letter from this job posting." Same structure, different content. Perfect skill.

"Debug why the webhook stopped firing after the API provider changed their auth flow." No skill for that. Every instance is different. The agent needs to read logs, trace requests, understand the specific integration, and propose a fix that accounts for your deployment setup. That's infrastructure knowledge, not a prompt template.

The 90% Nobody Demos

Here's what actually keeps my system running day to day.

A server process that syncs data from multiple APIs, caches it locally, and exposes it to agents through a unified interface. When an API changes its response format, I fix the parser. No skill for that.

Scheduled jobs that run without any agent session. They pull data, generate reports, send notifications, and alert me when something fails. The agent isn't even involved. It's just cron, a script, and an alert channel.

Approval workflows where the AI researches options, presents them with rationale, and waits for a human decision before executing. The approval mechanism is buttons in a chat app. The execution layer calls APIs to star repositories, follow users, post comments. The plumbing between "AI suggested it" and "it actually happened" is custom for every use case.

State that persists between sessions. Not agent memory. Infrastructure state. Cache files with TTLs. Vector indexes that get rebuilt nightly. Configuration that lives in flat files because a database would be overkill.

None of this fits in a skill. It's bespoke infrastructure that exists because the specific problem required it.

Why This Matters

The skills hype creates a misleading impression of what production AI work looks like. Someone sees a collection of 30 slash commands and thinks: that's the system. It's not. It's the tip.

The system is the integration layer. The error handling. The monitoring. The state management. The human-in-the-loop controls. The deployment. The part where you wake up and the thing is still running, handling edge cases the skill never anticipated.

If you're evaluating someone's AI engineering capability, don't ask how many skills they have. Ask what happens when the skill fails. Ask what runs when nobody's in a session. Ask how state persists between interactions. That's where the actual engineering lives.

The Honest Ratio

I spend maybe 5% of my time writing new skills. I spend the rest building and maintaining the infrastructure that makes skills useful in the first place.

A skill that generates a cover letter is worthless without the task management system that tracks proposals, the message log that maintains conversation history, and the pipeline that routes everything to the right place.

A skill that creates a content draft is worthless without the publishing pipeline, the banner generation, the cross-platform distribution, and the editorial calendar that decides what to write next.

The skill is the last mile. The infrastructure is the entire road.

The Question

Next time you see someone demo their skill collection, ask yourself: what's underneath? What happens between sessions? What runs at 4 AM? What breaks, and who gets paged?

That's the 90%. That's the actual work.

I build production AI infrastructure, not prompt collections. If your team needs the 90% that skills don't cover, let's talk: cal.eu/reneza

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

How I Built a Business Email Agent with Compliance Controls in Go

René Zander — Sat, 21 Mar 2026 12:49:31 +0000

Every few weeks another AI agent product launches that can "handle your email." Dispatch, OpenClaw, and a dozen others promise to read, summarize, and reply on your behalf.

They work fine for personal use. But the moment you try to use them for business operations, three problems show up:

No spending controls. The agent calls an LLM as many times as it wants. You find out what it cost at the end of the month.
No approval flow. It either sends emails autonomously or it doesn't. There's no "show me the draft, let me approve it" step.
No audit trail. If a client asks "why did your system send me this?", you have no answer.

I needed an email agent for my consulting business that could triage inbound mail, draft replies, and digest threads. But I also needed to explain every action it took to a client if asked. So I built one.

The core constraint: AI never executes

The first decision was architectural. In most agent frameworks, the LLM decides what to do and does it. Tool calling, function execution, chain-of-thought action loops.

I went the other way. The LLM classifies and drafts. Deterministic code executes. Every side effect (sending an email, posting a notification, writing a log) is plain Go code with explicit control flow. The LLM is a function call that takes text in and returns structured JSON out. Nothing more.

This matters because it makes the system auditable. When something goes wrong, you read the pipeline definition, not a chat transcript.

Pipelines, not prompts

Every automation is a YAML pipeline. Here's a simplified email digest:

pipelines:
  - name: email-digest
    schedule: 30m
    steps:
      - name: fetch-unread
        type: deterministic
        action: gmail_unread

      - name: summarize
        type: ai
        skill: email-digest

      - name: report
        type: deterministic
        action: notify

Three step types:

deterministic: Plain code. Fetch emails, send notifications, write logs. No tokens, instant.
ai: Calls an LLM with a skill template. Returns structured output. Token-budgeted.
approval: Pauses the pipeline and asks a human to approve, skip, or edit.

The key insight: most of what an "email agent" does is not AI. It's polling an API, parsing MIME, filtering by label, formatting output. The AI part is a small step in the middle that classifies or summarizes. Treating it as a pipeline step instead of the orchestrator keeps costs down and behavior predictable.

Token budgets as circuit breakers

Every AI step runs inside three budget boundaries:

budgets:
  per_step_tokens: 2048
  per_pipeline_tokens: 10000
  per_day_tokens: 100000

If a step exceeds its token limit, it fails. If a pipeline exceeds its limit, it stops. If the daily limit is hit, the engine pauses all pipelines until midnight.

This is a circuit breaker, not a suggestion. I've seen agents get stuck in retry loops that burn through $50 in tokens before anyone notices. A hard ceiling at the engine level prevents that by design.

Human-in-the-loop on every outbound action

The email connector has three permission levels: read, draft, and send. Even at the send level, outbound emails go through an approval step first.

The approval flow works through a messaging bot. The operator gets the draft with three options:

Approve: Send immediately.
Skip: Cancel.
Adjust: Edit the text, then approve.

There's a 4-hour timeout. If no one approves, the action is logged and dropped. This is non-negotiable for business use. An autonomous email agent that sends on your behalf without approval is a liability.

The approval channel itself has security controls: allowed user IDs, rate limiting per user, input length caps, and markdown stripping to prevent prompt boundary injection through the operator interface.

The Gmail connector

The email connector is ~400 lines of Go. All outbound polling, no webhooks, no open ports. It handles:

OAuth 2.0 with automatic token refresh
Base64 MIME decoding with multipart handling
Thread reconstruction in chronological order
RFC 2822 compliant reply construction (In-Reply-To, References headers)
Body truncation for token efficiency

Permission scoping is enforced at the connector level. If the config says permission: read, the connector physically cannot call the send endpoint. It's not a policy check, it's a code path that doesn't exist.

type GmailConfig struct {
    TokenPath  string `yaml:"token_path"`
    Permission string `yaml:"permission"` // read | draft | send
}

Skills: reusable AI templates

AI steps reference skill files instead of inline prompts. A skill is a YAML template with a role, a prompt, and an optional output schema:

name: email-digest
role: classifier
prompt: |
  Summarize these unread emails. For each:
  [priority] Sender - Subject - what they need
  Priority: HIGH = needs response today,
  MED = this week, LOW = informational

Output schemas enforce structure. If the LLM returns JSON that doesn't match the schema, the step fails instead of passing garbage downstream. This matters more than people think. An unvalidated LLM response in an automation pipeline is a bug waiting to happen.

Why Go

Single binary. Cross-compile for the client's infrastructure. No runtime dependencies.

The entire engine runs on a 2-core VPS for under $5/month. There's no database server, no message queue, no container orchestration. SQLite for state, YAML for config, one binary for the engine. Deploy with scp and a systemd unit.

For a tool that runs on client infrastructure, operational simplicity is a feature. Every dependency is a support ticket waiting to happen.

How this compares to Dispatch and OpenClaw

	Dispatch	OpenClaw	FixClaw
Target	Personal productivity	Personal AI agent	Business operations
Token controls	None (subscription)	None	Per-step, per-pipeline, per-day budgets
Human approval	Pause on destructive	Optional	Every outbound action
Data residency	Cloud	Self-hosted	Self-hosted
Configuration	Natural language	Natural language	YAML (version-controlled, auditable)
Audit trail	None	None	Full action + decision logging

Dispatch and OpenClaw are good products for what they do. But "what they do" is personal productivity. The moment you need to explain to a client what your automation did and why, you need governance controls that personal tools don't have.

What I learned

Most agent complexity is accidental. The LLM doesn't need to decide what to do next. You know what to do next. Write it in a pipeline. Use the LLM for the one step that actually requires language understanding.

Token budgets change how you design prompts. When there's a hard ceiling, you stop sending full email bodies and start truncating aggressively. You pick smaller models for classification. You realize 90% of your use cases work fine with a fast, cheap model.

Human-in-the-loop is not a UX compromise, it's a product feature. Clients don't want autonomous agents sending emails on their behalf. They want a system that does the thinking and lets them press the button. The approval step isn't a limitation. It's the reason they trust it.

FixClaw is open source and written in Go. If you're building business automations that need compliance controls, check it out on GitHub.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Detecting When Smart Money Stops Being Smart

René Zander — Tue, 17 Mar 2026 10:26:14 +0000

Following a profitable wallet is easy. Knowing when to unfollow it is where the money is.

I learned this the hard way. Not in crypto, but in forex. Years ago I built trading systems that scraped signals from profitable traders and mirrored their positions. It worked until it didn't. The problem was never finding good traders to follow. The problem was staying too long after they stopped performing.

The same pattern repeats in on-chain copy trading. Someone finds a wallet with a 70% win rate, follows every move, and six weeks later wonders why they're bleeding. The wallet didn't get hacked. The market shifted and the strategy stopped working.

Nobody talks about this part.

The decay problem

Every trading strategy has a shelf life. A wallet that's profitable today is running a strategy tuned to current market conditions. When conditions change, the strategy breaks. Sometimes gradually, sometimes overnight.

In forex I watched this happen in real time. A signal provider would post a 40% return over three months. Followers would pile in during month four. By month six the drawdown had wiped most of the gains. The provider's edge was gone but the followers didn't know because they were looking at cumulative PnL, not recent performance.

On-chain it's the same dynamic but harder to detect because the data is messier. You're not looking at clean trade logs. You're parsing token swaps, liquidity events, and contract interactions, then trying to figure out if the pattern still holds.

What to actually measure

When I built a detection system for this, I focused on three metrics that signal regime change:

Win rate over a rolling window. Not all-time. A wallet with a 65% all-time win rate that's been running at 40% for the last three weeks is in trouble. The all-time number hides the decay.

Average return per trade, recent vs historical. If the average return is shrinking even while the win rate holds, the edge is thinning. The wallet is still picking winners but the magnitude is dropping. This is the earliest warning sign.

Trade frequency changes. A wallet that was trading daily and suddenly goes quiet for a week might be recalibrating. Or it might be done. Either way, the pattern you were following no longer exists.

None of these metrics work in isolation. A drop in win rate during a broad market pullback means nothing. But a drop in win rate while similar wallets maintain theirs, that's a signal.

The threshold problem

The hardest part isn't collecting the data. It's deciding what counts as a regime change versus normal variance.

Set the thresholds too tight and you're unfollowing wallets after every bad week. Too loose and you catch the decline three weeks after it started. I went through several iterations before landing on something useful: compare the wallet's recent performance against its own historical baseline, not against an absolute number.

A wallet that normally runs a 55% win rate dropping to 45% is a different signal than a wallet that normally runs 75% dropping to 65%. Both dropped 10 points, but the first one is within normal variance and the second one is a structural shift.

When to actually stop following

The detection is one thing. Acting on it is another.

The instinct is to wait. "Maybe it's just a bad week." That's the same instinct that kept me in losing forex signals for months. The data was clear but the hope was louder.

What worked for me: automate the alert, not the action. The system flags when a wallet crosses its regime change threshold. I review it, check the context (is the whole market down or just this wallet?), and decide. But the flag forces the decision. Without it, I'd never look.

Most people in DeFi are still following wallets based on a snapshot of past performance. They find a wallet on a leaderboard, follow it, and never re-evaluate. That's not a strategy. That's a bet that conditions never change.

The edge isn't in finding smart money. There are tools for that everywhere. The edge is in knowing when smart money stops being smart for the current market, and stepping aside before the losses compound.

If you're thinking about building something similar or want to talk through the detection approach, feel free to reach out: cal.eu/reneza.

Get the next field note in your inbox. A new mini case from real AI builds every two weeks. No theory, no pitches.

Subscribe at renezander.com

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Your Vector Database Decision Is Simpler Than You Think

René Zander — Tue, 17 Mar 2026 07:41:59 +0000

Every week someone asks which vector database they should use. The answer is almost always "it depends on three things," and none of them are throughput benchmarks.

I run semantic search in production on a single VPS. Over a thousand items indexed, embeddings generated on the same machine, queries return in under a second. But that setup only works because of the constraints I'm operating in. Change the constraints and the answer changes completely.

Here's how I think about it.

The overchoice problem

There are dozens of vector databases now. Every one of them publishes benchmarks showing millions of vectors queried in milliseconds. That's great if you're building a search engine for the entire internet. Most of us aren't.

The benchmarks test throughput at scale. What they don't test is: can this thing run on the same box as your application without eating all the memory? Can you set it up in ten minutes? Does it need a cluster?

Those are the questions that actually matter when you're picking one.

Scenario 1: Local device, small dataset, ephemeral

You have a CLI tool or a local application. Your data is a few hundred markdown files or JSON documents. The user runs it on their laptop.

You don't need a database. Load the vectors into memory on startup, compute cosine similarity, done. A flat array of float32 embeddings and a brute-force search will outperform any database at this scale because there's zero overhead. No process to manage, no port to configure, no persistence to worry about.

Pre-compute your embeddings at build time or on first run, store them alongside the source files. When the data changes, regenerate. At a few hundred items this takes seconds.

The mistake people make here is reaching for a database because it feels like the "proper" way. It's not. It's unnecessary complexity for a problem that fits in a single array.

Scenario 2: Small VPS, thousands of items, needs persistence

Now things change. Your data lives behind an API. It updates throughout the day. You need search results to reflect changes within minutes, not hours. The whole thing runs on a VPS with maybe 2GB of RAM, shared with other services.

This is where a lightweight vector database process makes sense. Something that runs as a single binary, stores vectors on disk, serves queries over a local HTTP API. You don't need clustering or replication. You need something that starts fast, uses a few hundred MB of RAM, and doesn't crash when you restart other services on the same box.

The key decisions here: embedding model runs locally or via API? If your VPS has enough RAM, running a small embedding model locally saves you per-request API costs and latency. If RAM is tight, use an embedding API and only store the resulting vectors locally.

Change detection matters at this scale. You don't want to re-embed everything on every sync. Hash the source content, compare to what's stored, only embed what changed. This keeps your sync jobs fast and your API costs predictable.

Scenario 3: Multi-service, millions of vectors, high availability

This is where the benchmarks actually apply. Multiple services querying the same vector index. Data measured in millions of items. Uptime requirements that mean you can't tolerate a single-process restart.

At this scale you need a managed service or a self-hosted cluster. Replication, sharding, automatic failover. The operational overhead is real but justified because downtime now costs money.

Most teams jump here first because it looks professional. But if your dataset fits in 2GB of RAM and you have one service querying it, you're paying for complexity you don't need.

The three questions

Before looking at any product page, answer these:

Where does the data live? If it's local files that rarely change, stay in-memory. If it's behind an API that updates constantly, you need a persistent store.

How much RAM can you spare? This determines whether you run embeddings locally or call an API, and whether your database runs in-process or as a separate service.

Do you need persistence or is ephemeral fine? If you can regenerate everything from source in seconds, skip the database. If regeneration takes minutes or hours, persist.

These three questions eliminate 80% of the options before you read a single benchmark.

Start from the environment

The pattern I keep seeing is people evaluating vector databases by features and benchmarks, then trying to fit the winner into their deployment. It works the other way around. Start from your environment, your constraints, your data volume. The right answer usually becomes obvious.

The comparison articles won't tell you this because they can't. They don't know if you're running on a laptop, a $5 VPS, or a Kubernetes cluster. You do.

If you're building something with semantic search and want to think through which approach fits your setup, I'm always up for a quick conversation: cal.eu/reneza.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

I Run 10 AI Agents in Production. They're All Bash Scripts.

René Zander — Thu, 12 Mar 2026 14:29:44 +0000

A week ago I wrote about shipping AI agents the right way. That piece was about the harness: quality gates, token economics, multi-model verification. The stuff that separates demos from production.

A lot of people resonated with it. But I left out the part that actually eats most of my time: keeping the boring stuff running.

So let me walk you through what production AI agents actually look like when the conference talk is over.

The stack

I run about ten agents in production. They handle things like daily briefings, team follow-ups, task classification, weekly status reports, and job screening. None of them are impressive. All of them are useful.

The architecture for every single one is the same:

Trigger > Fetch data > Scoped prompt > Write output > Notify

A scheduled job fires. The agent pulls data from a task system, a calendar, an inbox, whatever it needs. A scoped prompt with a token budget processes that data. The result gets written back or sent as a notification.

That's it. No framework. No orchestration layer. No agent-to-agent communication protocol. Each agent is a standalone script with a timeout and a budget cap.

The ones that run daily cost about $1 per run. The weekly planner, which needs more context, costs about $5. The job screener uses a smaller model and comes in at $0.25. Monthly total across everything: somewhere around $80.

State is the whole game

The interesting part of running agents isn't the LLM call. It's everything around it.

One of my agents tracks which emails it already processed. It maintains a list of the last 500 message IDs. If that list gets corrupted, every email gets reprocessed and I'm flooded with duplicate notifications. If it gets wiped, same thing.

Another agent maintains a vector index. Every night, a sync job pulls all tasks, checks which ones changed since the last run, re-embeds only those, and updates the index. Smart sync means it skips tasks where only metadata changed. Without that optimization, the job would take 30 minutes instead of 30 seconds.

A third agent prepends follow-up messages to task notes, separated by date markers. It never deletes old entries. The history is the context for the next run. Mess with the format and the agent loses its memory.

None of this is AI work. It's state management. The LLM call is the easy part.

The maintenance nobody warns you about

This is what I wish someone had told me before I started: the data your agents pull from and write to needs constant housekeeping.

Task descriptions get stale. Context windows bloat because old entries pile up. Duplicate state creeps in when two agents touch the same data. Embeddings drift as your task descriptions evolve.

I spend more time maintaining the data around my agents than I spend on the agents themselves. Trimming old entries, cleaning up state files, making sure the vector index stays in sync with the source of truth.

It's the same kind of work you do with any data pipeline. But nobody frames it that way because "AI agent" sounds more exciting than "scheduled ETL job with an LLM in the middle."

Token economics shape everything

Your architecture follows from your token budget, not the other way around.

I learned this the hard way. My first agent design was a single mega-prompt that tried to do everything: read tasks, check calendar, scan inbox, generate a plan, write follow-ups. It worked. It also burned through tokens like they were free and hallucinated more than I was comfortable with.

Now every agent has a single job. Small context, scoped prompt, hard token limit. The job screener doesn't need to know about my calendar. The weekly planner doesn't need to see my inbox. Isolation isn't just cheaper. It's more reliable.

The cost breakdown matters because it changes your design decisions:

A $0.25 agent can run four times a day. A $5 agent runs once a week.
A fast, cheap model handles screening. A slower, expensive model handles planning.
If an agent costs more than $2 per run, I look at whether the prompt can be tighter.

Framework vs. no framework

I tried frameworks. They lasted about three months.

The problem wasn't capability. The frameworks could do everything I needed. The problem was that every failure mode became a framework debugging session instead of a straightforward script fix.

A shell script fails: I read the log, find the error, fix the line. Done.

A framework agent fails: I dig through abstraction layers, figure out which middleware swallowed the error, check whether the state manager persisted something weird, and then fix the line.

Same fix at the end. Ten times the debugging surface.

My current stack is scheduled jobs, shell scripts calling an LLM API, and log files. It has fewer moving parts than most people's "hello world" agent demos. And it's been running stable for months.

What I'd build differently

If I started over:

State management first. Before writing a single prompt, I'd design how every agent tracks what it already did, what changed since last run, and where it writes output. This is the foundation everything else sits on.

One agent, one job. No multi-purpose agents. The overhead of running five cheap agents is lower than debugging one expensive agent that does five things.

Scheduled over event-driven. For anything that happens on a predictable cadence, a timer is simpler and more reliable than a webhook chain. Event-driven has its place, but most of my agents don't need it.

Budget caps from day one. Every agent gets a hard spending limit. If it hits the cap, it fails loudly instead of running up a bill.

I've been building and running these systems for a while now. If you're working through similar problems, or thinking about moving agents from demo to production, I'm always up for a conversation: cal.eu/reneza

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

DEV Community: René Zander

Claude Code with Local LLMs and ANTHROPIC_BASE_URL: Ollama, LM Studio, llama.cpp, vLLM

TL;DR cheat sheet

The local-Claude-Code rule of thumb

Recommended setup

1. Environment variables Claude Code reads

2. Context length: the hidden failure mode

Practical context tiers

Apple Silicon context presets

3. Claude Code Ollama setup (native, v0.14.0+)

4. Claude Code LM Studio setup (native, 0.4.1+)

5. Claude Code llama.cpp setup (Apple Silicon fast path for Gemma 4 26B-A4B)

6. Claude Code vLLM setup (native + tool parser)

7. LiteLLM — for fallbacks, not for translation

8. Common failures (the error strings developers google)

tool_use parse error / invalid tool call / tool_use is not supported

context length exceeded / model stopped mid-edit

empty assistant response

model not found / 404 the model X does not exist

messages: Extra inputs are not permitted (HTTP 422)

ANTHROPIC_BASE_URL ignored / Claude Code still calls the real API

9. Debug flow

10. Smoke test

11. Compatibility matrix (April 2026)

12. Hardware × model × context × backend (the cheat-sheet table)

13. Gemma 4 26B-A4B: the Apple Silicon sweet spot

Why 26B-A4B instead of 31B?

14. Other model picks for Claude Code (April 2026)

15. Setups I would avoid

16. Reusable startup script

17. Production recommendation

18. When local models are the wrong choice

Series

Sources

Reader contributions

Changelog

2026-04-28

2026-04-22

Voice AI in Production: From RunPod to Hosted Kubernetes

Single Pod Stops Working Around Four Concurrent Users

What Actually Breaks First

What the K8s Layer Actually Buys You

Hosted K8s Is the Service

Your Model Is the Value. Your Pod Isn't.

Ten CLAUDE.md rules for Claude Code - four edit-time, six runtime

Forrestchang's four (edit-time) — unchanged

Six runtime rules — lessons from fixclaw

5. Deterministic First

6. Declare Budgets, Halt On Breach

7. Human-In-The-Loop Is A First-Class Step Type

8. Validate AI Output Against A Schema

9. Sanitize Operator Input Before It Reaches A Prompt

10. Log Rejections Silently

The "working if" test

95% of PII Redaction Doesn't Need an LLM. The Other 5% Is Where Your Masker Leaks.

The 95% Where Deterministic Wins

Where a Fine-Tuned Model Earns Its Compute

The Hybrid Architecture

What I Won't Claim

What llama.cpp's Pace Tells You About On-Prem LLM Readiness

The Team That Celebrated Too Early

What Changed While You Were Waiting

The Decision You're Actually Making

The Question for Your Next Planning Cycle

Your AI Content Tool Knows Your Strategy. Do You Know Where It Goes?

The Prompt Is the Product

The Trust Model Is the Problem

The Technology Already Exists

Why This Matters Now

What to Ask Your Vendors

The Shift from Trust to Proof

Spend Your Human Thinking Tokens Where They Compound

You Have a Token Budget Too

Where I Stopped Spending

Where I Still Spend Every Token

The Honest Ratio

The Shift

AI Skills Are the New Boilerplate. They Solve Almost Nothing.

What Skills Actually Are

The 90% Nobody Demos

`tool_use parse error` / `invalid tool call` / `tool_use is not supported`

`context length exceeded` / model stopped mid-edit

`empty assistant response`

`model not found` / `404 the model X does not exist`

`messages: Extra inputs are not permitted` (HTTP 422)

`ANTHROPIC_BASE_URL` ignored / Claude Code still calls the real API