Kunal

Posted on Sep 8 Originally published at kunalganglani.com

5 Open Source Tools That Replaced My $20/mo AI Stack [2026]

#opensource #selfhosted #llm #ollama

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

5 Open Source Tools That Replaced My $20/mo AI Stack (Ollama, “9router”, Headroom, Diffy, OpenHands)

5 open source tools that replaced my 20/mo AI stack Ollama 9router Headroom Diffy OpenHands are a practical, self-hostable set of building blocks that cover the same five jobs most paid AI subscriptions try to bundle: local LLM inference, routing, meeting notes, evals/observability, and coding agents. The reason this matters in 2026 is simple. Small local models got good enough, gateways got boring and reliable, and subscription fatigue is real. Control and cost win.

Key takeaways

You don’t need “one AI app” anymore. You need five boring primitives wired together.
A gateway (router) is the difference between “local AI” as a toy and production AI you can actually operate.
Meeting transcription is the easiest subscription to kill because audio is your most sensitive data.
Evals and traces aren’t enterprise fluff. They’re how you stop prompt changes from silently breaking workflows.
Autonomous coding agents are useful, but only when they run in a sandbox with logging and a budget.

The best AI stack is five replaceable parts, not one magical app you can’t debug.

The $20/mo AI stack I replaced (and what these OSS tools map to)

Let’s name the jobs first. Tools come and go. Jobs don’t.

This post targets the ridiculous long-tail keyword on purpose: “5 open source tools that replaced my 20/mo AI stack Ollama 9router Headroom Diffy OpenHands.” The point is not that every one of those names is a canonical project. The point is that a typical $20/month stack usually includes:

Local chat / local inference (what Ollama does)
A router / gateway (what “9router” implies. I’m going to use LiteLLM because it’s verifiable and widely deployed.)
Meeting notes (what “Headroom” implies. I’m going to use Whisper as the transcription backbone.)
Evals and observability (what “Diffy” implies. I’m going to use Langfuse, plus a simple “golden test” mindset.)
Agentic coding (what OpenHands is explicitly about)

Here’s the cheat sheet:

Tool	Replaces (paid workflow)	What it does	Self-hosted?	Key downside
Ollama	Chat subscriptions / “local model app”	Runs open-weight models locally with a REST API	Yes	You’re now the “GPU scheduler”
LiteLLM	“LLM router” / gateway / usage dashboard	One OpenAI-style API for 100+ providers, load balancing, logging, cost tracking	Yes	Misconfig can leak spend fast
Whisper	Meeting transcription SaaS	Local speech-to-text you can build notes on top of	Yes	Diarization and cleanup are your job
Langfuse	Paid LLM observability + eval suites	Tracing, datasets, prompt management, evals	Yes	You need to instrument your apps
OpenHands	Paid coding agents	An AI-driven development agent with sandbox options	Yes	Requires strong sandbox + policy

Two quick credibility signals that this isn’t niche hobbyware:

Ollama is sitting around 180k GitHub stars and 17.8k forks right now.
OpenHands is around 86.8k stars and 11.4k forks.

Those numbers don’t mean quality. They do mean “you won’t be alone when it breaks.”

[Image: A simple diagram of the stack: IDE/Apps → LiteLLM → (Ollama or cloud APIs) + Langfuse tracing; Whisper pipeline feeding notes + summaries; OpenHands agent calling through gateway]

Ollama (Download, Get started, REST API): my default local LLM runtime

What is Ollama and how does it work? Ollama is a local model runtime. You pull an open-weight model, run it on your machine (CPU/GPU), and talk to it through a local server and REST API.

This is where most “local AI” content gets weirdly religious. My take is boring.

If your workload is interactive and private, local wins.
If your workload is batchy, spiky, or you need frontier quality, paid APIs still win.

Download

Ollama ships installers for macOS, Windows, and Linux, and it’s also straightforward in Docker. The canonical starting point is the repo: ollama/ollama.

Get started

My minimum viable setup looks like this:

Pick one model you can run fast.
Keep it pinned.
Don’t “model hop” mid-week and then wonder why results changed.

If you want model selection guidance, I maintain a benchmark database on this site. Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, throughput is usually the real trade on Apple Silicon. Unified memory lets you load big models, but tok/s and time-to-first-token decide whether it’s usable for daily work.

(That’s the part people learn after buying the fancy machine.)

REST API

The single reason I like Ollama more than most local runners is that it behaves like infrastructure. It has an API. That means I can put a gateway in front of it, log requests, and treat it like a service.

You don’t need code here to get the idea:

Your apps talk to http://localhost:11434 (or wherever you run it).
Everything else in this post assumes “models are reachable over HTTP.”

If you’re building anything agentic, also read my take on AI agents and why “just prompt it” dies the moment you add tool use.

LiteLLM (What is LiteLLM, Why LiteLLM, Features, Get Started): the “9router” equivalent that makes the stack sane

What is an LLM router/gateway and why would I need one? An LLM router is a policy layer that sits between your apps and multiple model backends (local and hosted). It gives you one stable API, then decides where to send each request based on latency, cost, privacy, or availability.

If you’ve ever shipped anything with third-party APIs, you already know why this matters. Without a gateway, “AI stack” means:

secrets scattered across laptops
no consistent retries
no centralized logging
no cost attribution

LiteLLM is the first OSS gateway I’ve used that feels like the obvious default for teams and serious solo builders. Their positioning is explicit: it’s an AI gateway that can call 100+ LLM APIs in an OpenAI-compatible format, with load balancing, guardrails, logging, and cost tracking (BerriAI/litellm).

Why LiteLLM

A gateway buys you three concrete things:

Fallbacks: when a provider flakes, you fail over.
Budgeting: route cheap prompts to cheap models.
Change control: you can roll model switches like a config change, not a repo-wide refactor.

I’ve seen this pattern pay off even in my own “tiny production”: running this blog’s multi-agent publishing pipeline taught me that deterministic gates and stable interfaces catch more breakage than “just upgrade the model.” That same mental model applies to LLMs. You want stable contracts.

If you care about model outages, I wrote up practical fallback patterns in ChatGPT down? 8 fallback patterns for API outages.

Features I actually use

OpenAI-compatible interface so tools don’t care if the backend is Ollama or Anthropic.
Load balancing when you run multiple local backends.
Logging hooks so you can ship traces to Langfuse.
Cost tracking so the “occasional API call” doesn’t quietly become $90.

Get started (home-lab friendly)

I’m not going to paste a 200-line docker-compose.yml here. Prose-first, remember.

The minimal topology is:

Ollama container (or host install)
LiteLLM proxy container
One config file mapping a “logical model name” to either ollama/... or a cloud provider

Once you do that, every app you use can point at the gateway. That includes your AI agents, your eval harness, and your coding agent.

How do you route between OpenAI/Anthropic/local models with fallbacks?

My routing rule of thumb:

Local first for anything that touches private text: meeting notes, internal docs, client data.
Cheap hosted model for bulk “summarize/transform” tasks when latency matters.
Frontier hosted model only when the task is high leverage: tricky refactors, deep debugging, or writing that has to ship.

Put differently: route by risk and value, not vibes.

Also, if you’re doing this in a team setting, read LLM cost and why per-task budgeting beats “monthly spend” dashboards.

[Image: A routing decision matrix: privacy × complexity → local/cheap/frontier]

Whisper: self-hosted meeting transcription and notes (Headroom replacement)

How can I self-host AI meeting transcription and notes? Split it into two stages: (1) local transcription, (2) local summarization. Don’t combine them into one magic “meeting notes app” unless you enjoy debugging black boxes.

Whisper is OpenAI’s open-source speech recognition model (openai/whisper). The repo is sitting around 109k stars and 13.2k forks, which tells you it’s the de-facto substrate for a ton of transcription products.

Is Whisper free and can I run it locally? Yes. You pay in compute time.

Here’s a realistic pipeline that doesn’t pretend diarization is solved:

Record audio locally (Zoom local recording, OBS, whatever)
Run Whisper on the file to produce a transcript
(Optional) run speaker diarization with a separate tool if you need attribution
Summarize the transcript with a local LLM through your gateway
Store the transcript and summary somewhere searchable

If you’re building a real knowledge system, pair this with my RAG and retrieval-augmented generation posts. Meeting notes are just another retrieval corpus. Treat them like one.

Privacy angle, blunt version: audio is one of the few data types that’s still meaningfully sensitive. Voices are biometric. If you can keep it local, keep it local.

For the threat-model side, see LLM data leakage playbook and AI security.

Langfuse: LLM evaluation + observability (Diffy replacement)

What are the best open-source tools for LLM evaluation and observability? Langfuse is the closest “one-stop” OSS answer I’ve found that doesn’t immediately shove you into a paid tier. It positions itself as an open source AI engineering platform for LLM evals, observability, metrics, and prompt management, and it integrates with OpenTelemetry (langfuse/langfuse).

This is the part most solo builders skip. Then they change a prompt and their agent gets 20% worse and they have no idea why.

I care about evals because I’ve already watched “invisible regressions” wreck systems. On this blog, I run a deterministic SEO quality gate inside a multi-agent publishing pipeline. The lesson was clear: deterministic checks before the LLM reviewer catch more issues than simply upgrading to a bigger model. That’s the same philosophy behind evals. Make failure observable.

What “Diffy” usually means in practice

“Diffy” in AI stacks is often shorthand for:

golden inputs + expected outputs
regression diffs across model/prompt versions
scoring with simple rubrics

If the exact “Diffy” project you meant isn’t available or verifiable, you can still get 80% of the benefit with two things:

Langfuse datasets + eval runs
a small set of golden tasks you run every time you change a prompt/model

Concrete numbers help. Start with 25 golden examples for your highest-value workflow. Add 5 per week. In a month, you have 45 examples and a trend line.

If you want a lightweight process for small teams, I wrote Agent evaluation roadmap for small teams and a more “systems” version in AI engineering evals: regression gates.

Observability is not optional for agents

If you’re building agentic systems, you need traces. Otherwise you’re debugging a stochastic program with print().

Langfuse plus OpenTelemetry is a clean path. I’ve gone deep on the instrumentation model in OpenTelemetry instrumentation for AI agents and How to build vendor-neutral LLM observability.

OpenHands (Quickstart, Architecture): the coding agent I actually trust more when it’s self-hosted

What is OpenHands and how is it different from other coding agents? OpenHands is an open-source AI-driven development agent. The difference isn’t “it writes code.” Everything writes code now. The difference is that OpenHands takes the agent loop seriously: execution environment, sandboxing options, and an architecture you can reason about.

The repo offers a Quickstart with multiple deployment options (no sandbox, Docker sandbox, from source) and has architecture docs (OpenHands/OpenHands).

Quickstart (what I would do first)

Start without a sandbox only if you’re experimenting on throwaway repos. The moment you point an agent at anything real, run it in a container/VM.

If you want a deeper security posture for agents, I’ve written a lot about this:

AI agent sandbox Linux VM
AI agent attack surface checklist
prompt injection and why it’s still the #1 footgun

Architecture (what matters operationally)

The architecture detail that matters most is separation:

agent brain (LLM calls)
tools (filesystem, git, shell)
sandbox boundary
logs/traces

When those are separate, you can:

rotate API keys without redeploying everything
block outbound network by default
record every tool call (and replay it)

If you’re comparing coding agents, I’ve already done the head-to-head framing in Aider vs Claude Code vs OpenHands.

[Image: An agent loop diagram: plan → call model → tool call → observe → retry → commit]

Cost, hardware, and the boring math (where the $20/mo savings actually comes from)

The “replace my $20/mo stack” headline is catchy. The real win is cost control.

There are three buckets of spend here:

Local compute (hardware + electricity)
Occasional API calls (frontier models for hard tasks)
Storage (logs, transcripts, traces)

If you already own a capable machine, bucket (1) feels “free” because it’s sunk cost. That’s honest for a lot of people.

If you’re buying hardware specifically for this, you need break-even math. I built a full model for that in Local LLM break-even math and a calculator-style version in Local LLM cost vs cloud API.

Concrete expectations that won’t insult your intelligence:

A workable laptop-tier setup for 7B–14B models wants 16–32 GB RAM.
For comfortable agentic coding on bigger models, you’re realistically looking at a GPU with 12–24 GB VRAM, or Apple Silicon with 32–64 GB unified memory.
If you run one local model server plus logging, expect a steady-state memory footprint that can sit north of 10–20 GB even before you open your IDE.

And yes, paid APIs still win sometimes. If the task is worth $2 to get right in 30 seconds, pay the $2. Just route it explicitly.

Security and privacy: the part SaaS stacks quietly make worse

Self-hosting doesn’t magically make you secure. It does change your threat model.

Keeping transcription local means:

raw audio doesn’t leave your machine
transcripts aren’t stored in a vendor’s retention policy you didn’t read

Routing through a gateway means:

one place to enforce redaction
one place to rotate keys
one place to log usage for incident response

If you want a practical set of defaults, start here:

One more thing: a lot of “self-hosted AI” guides ignore outbound network controls. If you run a coding agent, treat it like an untrusted junior engineer with sudo. Give it a sandbox, least privilege, and logs you can audit.

For container hardening, see Docker rootless mode security.

What exactly are “9router”, “Headroom”, and “Diffy” here?

This is the annoying but important part: some stacks circulate as screenshots and tweets, not as verifiable projects.

“9router”: I’m treating this as “an LLM router layer.” A verifiable replacement is LiteLLM because it’s explicitly a gateway/proxy and supports many providers in one OpenAI-style interface.
“Headroom”: there’s “Headroom” as a generic category of meeting-note tools. A verifiable open-source base layer is Whisper for transcription. The notes layer is just “summarize transcript with a model,” ideally through your gateway.
“Diffy”: often used as shorthand for diff-based eval workflows. A verifiable replacement is Langfuse for datasets, evals, and observability, plus a simple golden-test harness mindset.

If you do find a specific project named Diffy that you meant, the evaluation principles still hold. The workflow matters more than the logo.

My prediction: subscription AI will feel like cable TV by 2027

The industry is trying to bundle everything into one monthly plan again. Chat, notes, agents, search, “workspaces.” It’s cable TV with better UI.

I think the winning personal and small-team setup looks more like this post: a handful of replaceable primitives you can swap as the ecosystem shifts. Run local where privacy matters. Use a gateway so you can change your mind. Instrument and evaluate so you can trust changes.

If you’re building your own stack, my challenge is simple. Pick one workflow you currently pay for, replace it end-to-end in a weekend, and write down what broke. That list is your actual architecture backlog.

Originally published on kunalganglani.com

DEV Community