DEV Community: Rost

gstack: AI Software Engineering Stack

Rost — Mon, 28 Sep 2026 05:33:00 +0000

AI coding agents can already write functions, modify repositories, run tests, and open pull requests. The harder problem is getting an agent to follow a repeatable engineering process before, during, and after the code is written.

gstack, a project from Garry Tan originally built around Claude Code, takes a different route: instead of replacing your coding agent with another platform, it wraps the agent you already use with specialized skills, browser tooling, reviews, safety controls, and release processes. The project describes the result as a virtual engineering team -- twenty-three specialists and eight power tools, all slash commands, all Markdown, MIT licensed.

The comparison targets depend on which part of gstack you need: skill collections such as Superpowers, specification systems such as OpenSpec and GitHub Spec Kit, methodologies such as BMAD, orchestration platforms such as Ruflo, or your own maintained set of agent skills. Several of these combine with gstack rather than replace it, and the wider ecosystem they all belong to is mapped in the AI Developer Tools hub of this site.

What Is gstack?

gstack is an open-source collection of AI engineering workflows. In gstack's framing, software development consists of several different kinds of reasoning, and asking one generic coding prompt to perform all of them is a poor abstraction -- so the project exposes specialized skills instead:

product exploration
product and CEO review
architecture review
developer experience review
design review
implementation review
browser-based QA
investigation and debugging
security analysis
documentation
benchmarking
release preparation
deployment
retrospectives

The project presents these roles as a virtual engineering team: a CEO who rethinks the product, an eng manager who locks architecture, a designer who catches "AI slop", a reviewer who finds production bugs, a QA lead who opens a real browser, a security officer who runs OWASP and STRIDE audits, and a release engineer who ships the PR. The toolchain around the skill definitions is TypeScript and Bun: a setup script, generated skill documentation, session hooks, state under ~/.gstack/, a bundled browser, and a set of standalone CLIs.

gstack is not a foundation model and not a replacement for Claude Code; it is a process layer running on top of an agent harness:

flowchart TD
    A[LLM] --> B[Claude Code or another supported harness]
    B --> C[gstack skills and workflow rules]
    subgraph G[gstack process stages]
        D1[Planning]
        D2[Architecture review]
        D3[Design review]
        D4[Code review]
        D5[Browser QA]
        D6[Security]
        D7[Release and deployment]
        D8[Learning and memory]
    end
    C --> G
    G --> E[Git repository, browser, and development tools]

Two structural properties decide where gstack fits. First, the project describes it as a process rather than a collection of tools: the skills run in the order a sprint runs -- think, plan, build, review, test, ship, reflect -- and each skill hands its artifacts to the next, so /office-hours writes a design doc that /plan-ceo-review reads, and /plan-eng-review writes a test plan that /qa picks up. Second, gstack is not Claude Code-only: ./setup auto-detects the agents installed on the machine, and ./setup --host <name> targets Codex CLI, OpenCode, Cursor, Factory Droid, Kiro, Slate, OpenClaw, and Hermes, while a 2KB instruction-only digest in the repo covers rules-reading agents that need no install at all.

Why gstack Exists

A blank Claude Code session is extremely flexible, and that flexibility is also one of its weaknesses. Consider a feature request such as:

Add organization-level API tokens to the application.

A capable agent might immediately inspect the repository and start modifying authentication code, whereas a senior engineer would first ask who owns the tokens, whether users can belong to multiple organizations, how tokens are revoked, whether permissions are inherited, what happens to existing authentication, whether tokens should expire, how secrets are displayed, and which audit events are required. The agent may discover some of these questions eventually, but there is no guarantee it will discover them before implementation. gstack moves that discipline into reusable workflows: instead of idea -> coding agent -> code, the change passes product review, technical planning, architecture review, implementation, code review, browser QA, and release as named stages, each with its own command (the full sequence is in A Practical gstack Workflow below).

This does not make the AI correct; it changes the probability distribution of its mistakes. The agent is pushed to challenge assumptions earlier, inspect evidence, review its own work from several perspectives, and verify the application instead of stopping when the code compiles.

How gstack Turns Markdown Skills into Process

Most of gstack's capabilities originate in Markdown skill definitions. An agent skill can describe:

when it should run
what context it should inspect
what questions it should ask
which tools it may use
which commands it should execute
which evidence it must gather
which checks must pass
how the result should be structured

Such a skill acts somewhere between documentation, a reusable prompt, a standard operating procedure, and executable workflow configuration. The underlying mechanics are covered in Claude Skills and SKILL.md for Developers; gstack adds infrastructure on top of that primitive: generated skill definitions, startup and completion hooks, state management under ~/.gstack/, browser automation, safety mechanisms, repository inspection, opt-in telemetry, cross-session memory managed by /learn, and optional persistent knowledge through the separate GBrain project, which /setup-gbrain can stand up as a local PGLite database, a Supabase project, or a remote MCP endpoint.

The Most Important gstack Skills

The exact collection changes quickly, but several workflows illustrate how the system is intended to be used.

office-hours

/office-hours belongs near the beginning of a project or feature. It runs six forcing questions about the problem before any code is written, and in the README's worked example it reframes a request for a "daily briefing app" into a personal chief-of-staff AI, then writes the design doc that every downstream skill reads. For vague input like "we need better project search", the output is a requirement, not code.

plan-ceo-review

/plan-ceo-review examines the product-level assumptions behind a plan. It works in four scope modes -- Expansion, Selective Expansion, Hold Scope, Reduction -- and can challenge scope, identify missing opportunities, reduce unnecessary work, or suggest framing the problem differently. It runs before the requirements are fixed, a stage most coding-agent tools do not have.

plan-eng-review

/plan-eng-review shifts the perspective toward engineering: architecture, data flow, diagrams, edge cases, a test matrix, failure modes, and security concerns. It stays separate from the product review because merging both into one large prompt makes the model mix product decisions with implementation decisions.

plan-design-review and design-review

gstack treats visual and interaction design as a separate discipline. /plan-design-review rates each design dimension from 0 to 10, describes what a 10 looks like, and edits the plan to close the gap, with "AI slop" detection as a named check. The later /design-review runs the same audit against the actual implementation and fixes what it finds with atomic commits and before/after screenshots. For web applications, both pair with gstack's browser automation.

review

/review performs engineering review against repository changes from a staff-engineer perspective: it auto-fixes the obvious findings, flags the rest for approval, and keeps an advisory simplification lens for over-built code. Code written successfully is not necessarily code that should be merged.

investigate

/investigate enforces a systematic debugging rule the project calls the Iron Law: no fixes without investigation. It traces data flow, tests hypotheses, and stops after three failed fix attempts instead of continuing to thrash. It also auto-activates /freeze, which locks edits to the module under investigation.

qa and qa-only

/qa has the agent operate a browser, interact with the application, find bugs, fix them with atomic commits, re-verify, and generate a regression test for every fix. /qa-only runs the same methodology report-only. Many coding agents stop verification at "tests passed"; for a web application the browser is where integration errors, layout problems, incorrect flows, authentication failures, and JavaScript exceptions usually become visible.

ship, land-and-deploy, and canary

The release chain is three skills rather than one. /ship syncs main, runs tests, audits coverage, pushes, and opens the pull request, bootstrapping a test framework if the project has none. /land-and-deploy merges, waits for CI and the deployment, and verifies production health. /canary then runs a post-deploy monitoring loop that watches for console errors, performance regressions, and page failures.

autoplan, spec, learn, and retro

/autoplan runs the CEO, design, DX, and engineering review pipeline automatically -- engineering always last, so the shipping gate reviews the final amended plan -- and surfaces only taste decisions for approval. /spec turns vague intent into a precise, executable spec in five phases (why, scope, technical with mandatory code-reading, draft, file) with an outside-review quality gate before filing. /learn manages what gstack has learned across sessions -- patterns, pitfalls, and preferences -- with review, search, prune, and export. /retro produces a team-aware weekly retro; /retro global runs it across all your projects and AI tools.

Browser Automation in gstack

On supported macOS systems (macOS 15+), gstack drives the Aside browser first -- your real browser, with your real logged-in sessions, in tabs the agent opens for itself and closes when done. When Aside is unavailable, gstack falls back to its own Chromium-based engine, which ./setup builds and which runs a persistent daemon rather than launching a fresh browser for every command:

flowchart LR
    A[Coding Agent] --> B[gstack Browser CLI]
    B --> C[Local Browser Service]
    C --> D[Aside or bundled Chromium]
    D --> E[Application]

Persistent browser state lets cookies, authentication sessions, and tabs survive between operations, which makes browser-based QA practical. /open-gstack-browser exposes the fallback engine headed, with a sidebar agent that routes fast actions (click, navigate, screenshot) to Sonnet and reading or analysis to Opus. When the agent hits a CAPTCHA, an auth wall, or an MFA prompt, $B handoff opens a visible browser at the same page with cookies and tabs intact; you solve it, and $B resume continues where the agent left off. The agent suggests a handoff automatically after three consecutive failures. /pair-agent shares the browser with other agents -- OpenClaw, Hermes, Codex, Cursor, or anything that can curl -- with scoped tokens, tab isolation, rate limiting, and per-tab activity attribution.

The persistent engine also increases the security surface, since an agent with access to authenticated sessions holds a meaningful privilege. gstack ships a layered prompt-injection defense for this: content filters (datamarking, hidden-element stripping, ARIA scrubbing, URL blocklist) on every page read, plus a local ML classifier in a sidecar subprocess that scans page-derived content before the agent sees it, with a verdict combiner that requires classifier agreement before blocking. Page content is treated as untrusted input -- the agent takes syntax from a page, never instructions. Checks before and while using browser-driven QA:

Decide in advance which authenticated environments the agent may operate in, and prefer an isolated profile for QA work when the fallback engine is in use.
Know the emergency kill switch: GSTACK_SECURITY_OFF=1 disables the security layer -- do not leave it set.
The persistent daemon retains cookies and sessions between runs, so stop it when you are done and confirm nothing is left running, for example ps aux | grep -i chrom.
Read the hooks and safety mechanisms in the cloned repository before enabling them -- they are plain files, so review them the way you would review CI configuration.
After updating the clone, re-run ./setup so generated components stay in sync with the skill definitions.

Safety Guardrails and Second Opinions

Three power tools act as session-level safety switches. /careful warns before destructive commands -- rm -rf, DROP TABLE, force-push, git reset --hard -- and activates by saying "be careful"; recursive deletes of the root or home directory and force-pushes to the default branch are hard-denied. /freeze restricts file edits to one directory so the agent cannot "fix" unrelated code while debugging, and /guard activates both at once.

Second-opinion reviews cross harnesses: on Claude Code, /codex sends the work to OpenAI Codex CLI for an independent review, challenge, or consultation; on the other harnesses, /claude-code does the reverse. Each report identifies the provider that actually completed the review.

A Practical gstack Workflow

You do not need every gstack skill for every change. A reasonable feature workflow:

flowchart TD
    A[Feature idea] --> B[office-hours]
    B --> C[plan-ceo-review]
    C --> D[Create implementation plan]
    D --> E[plan-eng-review]
    E --> F[Implement]
    F --> G[review]
    G --> H[qa]
    H --> I[ship]

Which review skills to add depends on who the software is for:

Building for	Plan stage (before code)	Live audit (after shipping)
End users (UI, web app, mobile)	`/plan-design-review`	`/design-review`
Developers (API, CLI, SDK, docs)	`/plan-devex-review`	`/devex-review`
Architecture (data flow, perf)	`/plan-eng-review`	`/review`
All of the above	`/autoplan`	--

For a trivial bug fix, going straight to investigate, implementation, review, and tests is often enough. A tool-neutral version of the same shape -- spec, design, tasks, implement, validate -- is in Spec-Driven Development Workflow From Requirements to Code. The project's README describes running ten to fifteen of these sprints in parallel, each in its own isolated workspace; the sprint structure is what the project says keeps parallel agents from becoming sources of chaos.

Installing gstack

The current installation expects a working Claude Code setup, Git, Bun v1.0+, and, on Windows, Node.js -- Bun has a known bug with Playwright's pipe transport on Windows, so the browse server falls back to Node.js there. If you have not configured Claude Code yet, start with the Claude Code overview first. On macOS, the Aside browser (macOS 15+) is recommended for the browser skills; without it, the bundled Chromium daemon is used.

Verify your prerequisites: git --version and bun --version (the toolchain is Bun-based).
Clone gstack into the Claude skills directory:

   git clone --single-branch --depth 1 \
     https://github.com/garrytan/gstack.git \
     ~/.claude/skills/gstack

Run the setup script from the cloned directory:

   cd ~/.claude/skills/gstack
   ./setup

Setup installs and generates the components required by the supported skills and builds the bundled browser; a Chromium install failure is best-effort, setup records the reason, finishes registering every skill, and prints which skills are affected.

Add a ## gstack section to the project's CLAUDE.md. The project's install instructions include this step, and it is what makes Claude Code route the skills: use /browse from gstack for all web browsing, never use mcp__claude-in-chrome__* tools, and list the available skills.
Verify the install: check that the generated files are present in the cloned directory, start a Claude Code session, and run /office-hours on a scratch project to confirm the skill is recognized.

Team mode

For repositories, gstack offers a team-oriented setup where developers share one workflow instead of individually configured environments:

(cd ~/.claude/skills/gstack && ./setup --team) && \
  ~/.claude/skills/gstack/bin/gstack-team-init required && \
  git add .claude/ CLAUDE.md && \
  git commit -m "require gstack for AI-assisted work"

required blocks AI-assisted work in the repo without gstack; swap it for optional to nudge teammates instead of blocking them. No files are vendored into the repo: every Claude Code session starts with a fast auto-update check (throttled to once per hour, network-failure-safe, silent), which removes version drift across the team. Personal configuration improves one developer; repository-level configuration creates a shared engineering convention.

Other harnesses, upgrades, and uninstall

Other agents: ./setup --host codex, --host opencode, --host cursor, --host factory, --host kiro, --host slate, --host openclaw, and --host hermes install the skills into each agent's own skills directory. The 2KB instruction-only digest at agents-digest/gstack-AGENTS.md covers agents that only read rules files.
Command naming: skills register with short names by default (/qa, /review); ./setup --prefix switches to namespaced names (/gstack-qa), which matters when you run other skill packs alongside gstack.
Upgrades: re-run ./setup after a git pull (required on Windows, where installs are file copies), or use the /gstack-upgrade skill; setting auto_upgrade: true in ~/.gstack/config.yaml keeps the install current automatically.
Telemetry is off by default and asks for opt-in on first run. If you opt in, it sends skill name, duration, success/fail, gstack version, and OS -- never code, file paths, repo names, or prompts. gstack-config set telemetry off disables it at any time.
Uninstall: ~/.claude/skills/gstack/bin/gstack-uninstall removes skills, symlinks, ~/.gstack/ state, project-local state, browse daemons, and hook registrations.

Verify the install and fix common failures

./setup fails -- confirm Bun is on your PATH with bun --version; the generated components are built by the Bun toolchain.
Skills not recognized by Claude Code -- confirm the clone actually lives in ~/.claude/skills/gstack, that the project's CLAUDE.md has a gstack section, and re-run ./setup.
/browse reports NEED_ASIDE or ASIDE_NOT_RUNNING -- the probe is telling you it will use the fallback browser. That is normal on Linux and Windows; on macOS it means Aside is not open or signed in.
The fallback browser fails -- cd ~/.claude/skills/gstack && bun install && bun run build.
Stale install after an update -- run /gstack-upgrade, or set auto_upgrade: true in ~/.gstack/config.yaml.

Trying gstack Without Adopting Everything

Use gstack on a real but non-critical feature rather than migrating your development process. The project's quick start is the same trial, and it ends with "stop there":

/office-hours -- problem definition
/plan-ceo-review -- product reasoning
/review -- engineering verification, after implementation
/qa -- runtime verification, for web projects

If those stages surface findings your normal Claude Code workflow misses, the rest of the system is worth exploring; if they mostly produce additional text without changing engineering decisions, adopting the entire stack probably will not help.

What gstack Does Well

Separated engineering roles. Instead of one giant "be a senior engineer" instruction, product strategy, architecture, UX, QA, security, and release engineering each get their own reasoning mode.

Verification, not just generation. Review, browser QA with regression-test generation, security audits, benchmarking, and the ship-deploy-canary chain are first-class workflows in gstack rather than optional afterthoughts.

Inspectable. Much of the behavioral layer is plain Markdown files developers can read and modify, unlike the internal workflows of a proprietary autonomous agent. The repo also ships audit tooling for the stack itself: gstack-context-bill reports what an installed skill tree costs in tokens, and gstack-egress writes a hash-chained receipt for every off-machine send, telemetry included.

Team infrastructure. Skills can encode engineering conventions -- instead of typing

Remember to check API compatibility, run integration tests,
inspect the browser console, and update the changelog.

in every session, the requirements live in a reusable workflow, and team mode makes that workflow a repository requirement.

Where gstack Can Be Too Much

gstack is intentionally opinionated, and that limits its fit: a mature organization may already have architecture review procedures, release tooling, CI gates, QA automation, security scanning, ADR conventions, specification templates, and code review policies, and adding another complete methodology on top creates overlap instead of clarity.

There is also a context and token cost: each additional review stage adds repository inspection, model reasoning, and potentially more external model calls. The goal is the minimum reliable process needed to ship correct software, not a maximum number of AI reviews; gstack-context-bill can quantify what your installed skill set actually costs per session before you decide how much of it to keep.

gstack works best as a toolbox whose workflows you select and adapt, not as ceremony for every commit.

gstack Alternatives and Combinations

The closest alternatives, and the layer each occupies:

System	Primary focus	Workflow style	Agent portability	Best fit
gstack	Full engineering workflow	Role-oriented skills and tools	10 agents via `./setup --host`	End-to-end AI-assisted engineering
Superpowers	Engineering methodology	Automatic composable skills	High	Disciplined coding and TDD
OpenSpec	Change specifications	Lightweight spec artifacts	High	Brownfield feature development
GitHub Spec Kit	Spec-driven development	Structured multi-stage workflow	High	Formal requirements-to-code process
BMAD Method	AI-driven agile development	Adaptive roles and workflows	High	Larger end-to-end projects
Ruflo	Multi-agent orchestration	Agents, swarms, memory	Platform-oriented	Parallel autonomous agent systems
Custom skills	Your own process	Fully customizable	Potentially very high	Mature teams with established practices

gstack + Superpowers: implementation discipline inside the roles

Both are skill frameworks, so they overlap the most. The division of labor when combining them: gstack supplies the surrounding roles -- product, design, QA, release -- while Superpowers supplies the discipline inside the implementation phase (TDD, planning before implementation, systematic debugging, subagent review). Install both skill sets, then trim the overlapping skills so the agent never sees two conflicting instructions for the same phase; if command names collide, install gstack with ./setup --prefix so its skills register as /gstack-* and coexist with the other pack. Install and workflow details are in the Superpowers quickstart.

gstack + OpenSpec: durable specs, live reviews

OpenSpec keeps human and agent aligned around explicit change specifications -- artifacts for the proposed change, specifications, design decisions, and implementation tasks. The key property is persistence: a chat conversation disappears into context history, but a specification remains in the repository where humans and future agent sessions can review it. gstack adds the product review before the spec exists and the review and QA after it is implemented:

flowchart LR
    A[Feature request] --> B[gstack product review]
    B --> C[OpenSpec change]
    C --> D[Implementation]
    D --> E[gstack review]
    E --> F[gstack QA]

A concrete sequence: run /office-hours and /plan-ceo-review, capture the outcome as an OpenSpec change, implement against it, then run /review and /qa. Note that gstack also ships its own /spec skill, which archives specs under ~/.gstack; if OpenSpec owns the specification, keep gstack's /spec out of the loop so the two do not diverge. The OpenSpec quickstart covers the explore-propose-apply-archive loop in detail.

gstack + GitHub Spec Kit: pick one planning backbone

Spec Kit's core workflow is a sequence of explicit stages -- constitution, specify, plan, tasks, implement, converge -- and it has expanded into bug-fixing, idea-assessment, extensions, presets, and integrations. Since both Spec Kit and gstack center the planning stage, running both full flows duplicates work. If requirements traceability and formal stages matter, let Spec Kit own the specification backbone and use gstack for the layers Spec Kit does not enforce -- product review, design review, browser QA, and shipping. A broader comparison of spec-driven setups, including Kiro and Claude Code, is in GitHub Spec Kit vs Kiro vs Claude Code SDD Workflows.

BMAD and Ruflo: different axes

BMAD is a broader AI-driven development methodology whose adaptive workflows cover product thinking, specifications, architecture, and implementation, scaling the ceremony to the size of the work. It and gstack both play the process-backbone role, so choose one as the backbone rather than running both in full; gstack's individual skills can still be selected alongside a methodology.

Ruflo targets multi-agent orchestration: coordinated workers, shared memory, swarms. gstack applies multiple specialist perspectives to one engineering workflow; an orchestration platform applies multiple executing agents to one engineering objective. The boundary blurs -- gstack can call external tools and additional models, and orchestrators can implement structured engineering roles -- but the decision is independent: if the problem is that the agent skips engineering discipline, a skill framework is the direct fix; if it is running ten agents concurrently across many tasks and repositories, an orchestrator sits above a workflow like gstack rather than replacing it.

Custom skills: the most customizable layer

You can also skip the framework entirely and create a small collection of skills for the procedures your team already follows:

skills/
  architecture-review/
  api-review/
  database-migration-review/
  incident-analysis/
  release-check/
  security-review/

Each skill encodes organization-specific knowledge a generic framework cannot know. A database migration skill can require rollback analysis, table-lock analysis, index impact review, migration duration estimation, deployment ordering, and compatibility with the previous application version; an API review skill can require backwards compatibility, authentication checks, pagination consistency, idempotency analysis, rate-limit behavior, and OpenAPI changes. A practical path: start from the gstack skills you actually use, copy their structure into your own skills/ directory, and rewrite the checks around your conventions.

Four Layers: Skills, Specifications, Methodologies, Orchestrators

Four layers cover most of these tools, and they show how the combinations above fit together:

Skills answer "how should the agent behave?"

gstack, Superpowers, and custom agent skills.

Specification systems answer "what exactly are we building?"

OpenSpec and GitHub Spec Kit; the underlying spec-driven concepts and terminology are defined in What Is Spec-Driven Development?.

Methodologies answer "how should the project move from idea to software?"

BMAD, Superpowers, and parts of gstack.

Orchestrators answer "how should multiple agents execute work?"

Ruflo and other multi-agent runtimes.

The layers compose; a development environment can contain all four:

flowchart TD
    A[Product requirement] --> B[Specification system]
    B --> C[Engineering workflow]
    C --> D[Agent orchestrator]

    D --> E[Implementation agent]
    D --> F[Test agent]
    D --> G[Review agent]
    D --> H[QA agent]

    E --> I[Repository]
    F --> I
    G --> I
    H --> I

gstack already spans several of these boundaries.

Should You Use gstack?

The project's README describes the audience as technical founders and CEOs who still want to ship, first-time Claude Code users who want structured roles instead of a blank prompt, and tech leads and staff engineers who want rigorous review, QA, and release automation on every PR. gstack is worth trying if you use coding agents extensively and the limiting factor is no longer code generation itself. Typical symptoms:

the agent starts implementing before understanding the problem
implementation plans miss architectural implications
generated code passes tests but fails in the browser
reviews are inconsistent between sessions
release steps are repeatedly forgotten
different developers prompt the agent in completely different ways
useful engineering instructions remain buried in CLAUDE.md files
you repeatedly type the same review prompts manually

If your automation already provides strong deterministic gates and the agent only handles small, well-specified tasks, gstack adds little.

gstack and the Direction of AI Software Development

The shift gstack sits in tracks in generations: code completion (2022-2023), coding agents (2024-2025), specifications and agent workflows (2025-2026), and programmable AI engineering organizations. The products will change, but the model remains one component; engineering quality increasingly depends on the surrounding system:

persistent specifications
reusable skills
repository knowledge
browser access
tests
deterministic tools
review loops
security controls
memory
human approval boundaries
orchestration

Conclusion

gstack is the engineering process around a coding agent, packaged as inspectable, version-controlled skills, and its value is in forcing that process onto the agent rather than in any single skill.

Start from the trial sequence and keep only the skills that earn their place. Beyond it, the direction is composable layers -- specification, skills, deterministic verification, orchestration -- each doing what the others cannot.

References

gstack repository -- source, skill definitions, and setup
gstack skill deep dives -- philosophy, examples, and workflow for every skill
Aside browser -- the browser gstack drives first on macOS
Superpowers repository -- source, skills, and plugin manifests
OpenSpec repository -- source, docs, and the CLI package
GitHub Spec Kit documentation -- official Spec Kit workflow reference
BMAD-METHOD repository -- the adaptive AI-driven agile methodology
Ruflo repository -- the multi-agent orchestration platform

Self-Hosted Deep Research Systems: 12 Tools Compared

Rost — Fri, 25 Sep 2026 23:30:04 +0000

Deep Research has become its own category of software, not just a model pointed at a search box. This article compares twelve self-hosted systems and the research architectures behind them.

The line that actually matters is not whether a product ships a button labelled Deep Research, but what happens after the first round of retrieval. A genuine system notices that its plan was incomplete, chases a newly discovered lead, weighs conflicting sources, and only then writes the report.

Below I compare twelve open-source and self-hosted projects that implement this loop in different ways: recursive research trees, planner-plus-subagent designs, evidence-gap loops, perspective-driven question generation, and model-driven agentic search. For each one I cover the architecture, local-LLM support, RAG or private-document access, deployment complexity, and the license you actually inherit if you self-host. Where a system is also a full product (Open WebUI, Vane), I keep the focus on how it researches and link out to the dedicated guide for installation and configuration. Deep Research is one of the more demanding applied workloads in the AI Systems - it stresses retrieval, planning, and multi-step orchestration all at once, rather than any single layer in isolation.

What Is Deep Research?

A conventional AI web-search workflow is mostly linear. Even when several searches are performed, the model usually just creates related queries, retrieves documents, and summarizes what it finds:

Question
  |
Search
  |
Retrieve pages
  |
Summarize
  |
Answer

Deep Research adds a layer: the research process itself becomes adaptive. After the first pass, the system can branch, re-check, and keep going until the evidence is sufficient.

flowchart TD
    A[Research question] --> B[Build research plan]
    B --> C1[Investigate topic A]
    B --> C2[Investigate topic B]
    B --> C3[Investigate topic C]

    C1 --> D1[Discover new question]
    C2 --> D2[Find conflicting evidence]
    C3 --> D3[Identify missing information]

    D1 --> E1[Research new question]
    D2 --> E2[Verify competing claims]
    D3 --> E3[Search for missing evidence]

    E1 --> F[Combine evidence]
    E2 --> F
    E3 --> F

    F --> G[Evaluate remaining gaps]
    G -->|More research needed| B
    G -->|Enough evidence| H[Generate cited report]

The distinction matters. A system that searches five times is not necessarily performing Deep Research; a stronger system starts with one question, discovers an unexpected implementation detail, opens a new research branch around it, compares primary and secondary sources, and revises its original assumptions. For the broader Search vs Deep Search vs Deep Research distinction and how cloud offerings frame the same idea, see Search vs Deep Search vs Deep Research in 2026.

There is no single Deep Research architecture. Current self-hosted implementations generally fall into five groups:

Recursive research trees.
Planner and subagent architectures.
Evidence-gap-driven research loops.
Perspective and question-driven research.
Agentic iterative search.

The first four provide more explicit research structure. The fifth can still perform surprisingly deep investigation when paired with a strong reasoning and tool-calling model, but much of the strategy is delegated to the model itself. The evidence-gap loop in particular is a system-level cousin of the self-reflective retrieval used in Self-RAG-style pipelines — deciding whether to retrieve again, judging relevance, and critiquing the draft before answering. See Advanced RAG: LongRAG, Self-RAG and GraphRAG for that pattern at the retrieval-pipeline level.

Self-Hosted Deep Research Systems Compared

The table below summarizes the major systems. "Recursive depth" does not mean that multiple web searches are possible; it means the system has some mechanism for deriving additional investigation from intermediate findings.

System	Local LLM	Web Research	Private Docs / RAG	Planning	Recursive / Adaptive Depth	UI	Research Style
GPT Researcher	Yes	Yes	Yes	Yes	Excellent	Web UI	Recursive breadth/depth research tree
Unsloth Studio	Excellent	Yes	Yes	Yes	Very good	Excellent	Planned evidence-driven research
Local Deep Research	Excellent	Yes	Yes	Yes	Excellent	Web UI	Multiple strategies plus autonomous agent
STORM / Co-STORM	Yes	Yes	Custom corpus possible	Yes	Very good	Basic / demo UI	Perspective and follow-up-question research
DeerFlow	Yes	Yes	Yes	Excellent	Excellent	Good	Planner plus subagents and long-horizon agents
Onyx	Yes	Yes	Excellent	Yes	Excellent	Excellent	Multi-step enterprise Deep Research
Open Deep Research	Yes	Yes	Via tools / MCP	Excellent	Excellent	LangGraph oriented	Planner plus parallel researchers
Open WebUI	Excellent	Yes	Excellent	Model driven	Good	Excellent	Agentic iterative search and link following
Khoj	Yes	Yes	Excellent	Yes	Moderate	Good	Personal knowledge plus autonomous research
SurfSense	Yes	Yes	Excellent	Yes	Good	Excellent	Web/data research plus knowledge workspace
Vane	Yes	Yes	File search	Limited	Limited	Excellent	Search-first answering engine
Deep Research by lukeswade	Excellent	Yes	Research library	Yes	Excellent	Web UI	Gap-driven iterative investigation

One point stands out: there is no direct relationship between UI sophistication and research depth. Open WebUI and Vane provide polished interfaces, while GPT Researcher and STORM are centered more on the research algorithm. Conversely, Onyx and Unsloth Studio try to provide both a strong user experience and a substantial research workflow. Most of these systems run against the same local inference backends covered in the LLM Hosting guide.

Deep Research Architectures

Before comparing individual products, it is useful to understand the architectural differences.

Style	Representative Systems	Main Idea
Recursive research tree	GPT Researcher	Explicit breadth and depth generate new research branches
Planner plus subagents	DeerFlow, Open Deep Research	Planner decomposes work and independent agents investigate pieces
Evidence-gap driven	Unsloth Studio, Local Deep Research, lukeswade/deep-research	Findings are evaluated and missing evidence triggers another research round
Perspective driven	STORM / Co-STORM	Research is expanded by generating perspectives and follow-up questions
Multi-step research workflow	Onyx	Multiple research tasks gather and synthesize web and private knowledge
Agentic iterative search	Open WebUI	Model decides when to search, read, verify, and search again
Knowledge-first research	Khoj, SurfSense	Research combines private information with external sources
Search-first answering	Vane	Search and retrieval are optimized primarily for cited answers

The categories overlap. Local Deep Research offers several research strategies, and DeerFlow 2.0 is a general-purpose agent platform that can perform research rather than a research-only application. The distinction is nevertheless useful when choosing a system: a recursively branching researcher behaves differently from a chat interface whose model simply has a search_web tool. Systems that retrieve private documents alongside the web lean on the same retrieval patterns described in the RAG cluster.

License Comparison

Licensing is particularly important if the system will become part of an internal platform, commercial service, or redistributed product.

System	License	Licensing Notes
GPT Researcher	MIT	Current `pyproject.toml` declares MIT; some older package metadata still reports Apache-2.0
Unsloth Studio	AGPL-3.0	Studio UI is AGPL-3.0; core Unsloth remains Apache-2.0
Local Deep Research	MIT	Permissive open-source license
STORM / Co-STORM	MIT	Permissive open-source license
DeerFlow	MIT	Applies to current DeerFlow 2.0 repository
Onyx	MIT plus Enterprise License	Core is MIT; `ee` directories use the Onyx Enterprise License; `onyx-foss` is 100 percent MIT
Open Deep Research	MIT	Repository was archived in August 2026
Open WebUI	Open WebUI License	Current versions include branding restrictions; older code has MIT/BSD history
Khoj	AGPL-3.0-or-later	Network copyleft should be considered for modified hosted deployments
SurfSense	Apache-2.0	Current repository declares Apache-2.0
Vane	MIT	Formerly known as Perplexica
Deep Research by lukeswade	MIT	Permissive open-source license

For private self-hosting, none of these licenses prevents normal use. The differences matter when modifying the software, offering it to other users, embedding it into another commercial application, or redistributing derivatives. MIT and Apache-2.0 are generally the simplest options for integration. AGPL-3.0 deserves closer review for network-accessible modified deployments, and Open WebUI's current license adds its own branding conditions.

GPT Researcher

GPT Researcher is developed by Assaf Elovic and contributors as an autonomous research agent focused specifically on comprehensive online investigation. It is one of the clearest reference implementations of what "Deep Research" means when the term describes an algorithm rather than a user-interface feature.

Its strongest feature is explicit breadth and depth. Deep Research mode exposes parameters such as deep_research_breadth, deep_research_depth, and concurrency, allowing one investigation to generate several branches and those branches to generate additional research. This creates a real research tree rather than a fixed collection of search queries.

That approach also has costs. Recursive expansion can produce many retrieval and LLM operations, and the quality of the final result depends heavily on the model's ability to formulate useful research questions, extract evidence, and avoid propagating weak assumptions into deeper levels. GPT Researcher is also more research-engine oriented than applications such as Open WebUI or Unsloth Studio.

Installation is moderate rather than trivial: the project uses Python and ships a web application, while useful deployments also require suitable model and search providers. Current project metadata declares the MIT license. Choose GPT Researcher when explicit research depth, configurable recursion, and a research-first architecture matter more than an all-purpose local AI workstation.

Unsloth Studio

Unsloth Studio is developed by the Unsloth team as part of the broader Unsloth ecosystem. Originally best known for efficient model fine-tuning, Unsloth has expanded Studio into a local AI environment for inference, chat, tools, RAG, model management, and now Deep Research.

The interesting aspect of Studio is how tightly research is integrated with local model operation. Its Deep Research workflow includes a planning stage, plan review, evidence gathering, report generation, document handling, and failure handling when research steps fail to gather evidence. For users already running GGUF or other local models, this makes Studio considerably more convenient than assembling a separate research framework, inference server, and frontend.

Studio does not expose the same simple breadth/depth research-tree abstraction as GPT Researcher. Much of the workflow is organized around a research plan and evidence collection rather than arbitrary recursive expansion, and the feature is newer than some dedicated research projects. The quality of local research also remains sensitive to context length, output limits, tool use, and the reasoning quality of the selected model.

Installation is relatively friendly because Unsloth now provides Studio and desktop-oriented workflows across major platforms, although GPU and model configuration can still become substantial for advanced local deployments. The Studio component is AGPL-3.0, while the core Unsloth package remains Apache-2.0. Choose Unsloth Studio when Deep Research should be part of a broader local-model workstation rather than a standalone research service.

Local Deep Research

Local Deep Research is maintained by LearningCircuit and contributors as a privacy-oriented open-source research assistant. Its explicit goal is systematic research using web sources, academic databases, private documents, and local language models.

Its major advantage is architectural flexibility. Rather than enforcing one research algorithm, Local Deep Research supports pipeline-oriented strategies as well as a LangGraph agent strategy in which the model can decide what to search, which specialist sources to use, and when enough evidence has been collected. Academic sources such as arXiv, PubMed, Semantic Scholar, and other search mechanisms make it particularly attractive for technical and scientific research.

The downside of flexibility is complexity. Different strategies can behave substantially differently, which makes results harder to characterize with one simple "depth" parameter. It also has more moving pieces than a conventional chat UI, and users looking only for fast AI-assisted web answers may find it unnecessarily elaborate.

The project supports local operation and has developed a substantial application around the underlying research engine. Its license is MIT. Choose Local Deep Research when privacy, local inference, multiple research strategies, academic information sources, and control over the research process are more important than a minimal setup.

STORM and Co-STORM

STORM, developed by Stanford OVAL, stands for Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking. It was designed around knowledge curation and long-form report generation rather than general AI chat.

STORM's distinctive technique is perspective-driven research. It tries to discover different perspectives on a subject and uses question asking to broaden the information collected before writing an outline and article. Co-STORM extends the concept toward collaborative human-AI knowledge curation. This can uncover dimensions of a topic that a conventional list of keyword searches may overlook.

STORM is less suitable as a general local AI frontend. Its workflow is strongly oriented toward researching and writing structured, Wikipedia-like articles, and the project's own documentation notes that generated output should not automatically be considered publication-ready. It is therefore better understood as a specialized research and knowledge-curation engine than a replacement for Open WebUI.

Installation is Python oriented, and the pipeline can be customized with different models and retrievers. STORM uses the MIT license. Choose it when the objective is broad topic exploration, perspective discovery, structured outlines, and long-form knowledge synthesis.

DeerFlow

DeerFlow is developed by ByteDance and the DeerFlow community. Its name originally stood for Deep Exploration and Efficient Research Flow, but an important distinction now exists between the original 1.x Deep Research framework and DeerFlow 2.0.

DeerFlow 1.x was specifically designed around Deep Research. DeerFlow 2.0 is a ground-up rewrite into a more general SuperAgent harness capable of orchestrating subagents, memory, sandboxes, tools, and skills. For research, this architecture is powerful because a coordinator can delegate different parts of a problem to separate agents and later synthesize their findings - the same planner-plus-subagent decomposition covered in Multi-Agent Orchestration Patterns.

The trade-off is that DeerFlow 2.0 is no longer a narrowly optimized research engine. It is closer to a general long-horizon agent platform in which research is one workload among coding, artifact generation, and other tasks. If the requirement is a small dedicated research service, this additional machinery can be unnecessary.

Deployment is consequently more involved than a simple search UI, although the architecture provides much more room for customization and extension. DeerFlow is MIT licensed. Choose DeerFlow when Deep Research is expected to become one capability inside a broader multi-agent automation environment.

Onyx

Onyx, originally known as Danswer, is developed by DanswerAI and positioned as a self-hostable AI application layer for organizations. It combines chat, agents, web search, RAG, MCP integration, many enterprise data connectors, and a dedicated Deep Research capability.

Onyx stands out because research can span both the public web and a substantial private knowledge environment. Its Deep Research implementation is a real multi-step research flow rather than merely a search-result summarizer, and the project has published results and execution logs for DeepResearch Bench. For organizations that need research across internal documentation, indexed applications, and external sources, this is a particularly strong combination.

The cost of those capabilities is infrastructure complexity. Onyx is a larger platform than GPT Researcher or a lightweight local research project, and many of its strengths only matter when connectors, indexing, authentication, document stores, and organizational data are actually being used.

Onyx can be deployed self-hosted, including in restricted environments. Most of the main repository is MIT licensed, while code under ee directories uses the Onyx Enterprise License; a separate onyx-foss repository is maintained as a fully MIT-licensed variant. Choose Onyx when Deep Research must coexist with serious enterprise RAG and organizational knowledge retrieval.

Open Deep Research

Open Deep Research was developed by LangChain as an open implementation of a configurable Deep Research agent. It combines planning, research, report generation, multiple model providers, search tools, and MCP integrations using LangGraph.

Its architecture is particularly interesting for developers. Research work can be decomposed and parallelized, making it a useful reference for planner-researcher-synthesizer designs. Because the project was built around LangGraph rather than a monolithic UI, it is also easier to study as an implementation pattern for building custom research agents.

There is one major problem for new deployments: LangChain archived the repository on August 21, 2026, and it is now read-only. The code remains useful, but starting a production system around an archived reference project creates an obvious maintenance risk.

The project is Python based and uses an MIT license. Choose it today mainly for architectural study, experimentation, or as a source of implementation ideas rather than as the default foundation for a new long-lived installation.

Open WebUI

Open WebUI is one of the most popular general-purpose self-hosted interfaces for local and remote LLMs. Its recent agentic tool architecture gives models access to web search, URL fetching, knowledge bases, files, memory, code execution, and other tools. For installation, RAG setup, and the full feature set, see the Open WebUI guide.

Open WebUI's research model is interesting because the research loop is largely controlled by the language model. In native agentic mode the model can search, inspect snippets, fetch full pages, identify missing information, follow newly discovered URLs, cross-check sources, and repeat the process before generating an answer. With a capable reasoning and tool-calling model, this can produce genuine investigative behavior without a dedicated fixed research tree.

The limitation is precisely that this structure is model driven. Open WebUI does not provide the same explicit breadth/depth research topology as GPT Researcher, and there is less deterministic control over how many independent branches will be explored. A weak tool-calling model can stop too early, search poorly, or fail to follow important leads.

Installation is among the easiest in this comparison, particularly for users who already run Ollama, llama.cpp, vLLM, or another OpenAI-compatible inference server. Current releases use the Open WebUI License, which retains substantial permissive characteristics but adds branding restrictions; earlier portions of the project have MIT and BSD-3-Clause history. Choose Open WebUI when you want excellent local-LLM integration and a general AI interface in which research is one of many agentic capabilities.

Khoj

Khoj is developed as a self-hostable personal AI and "second brain." It combines local or cloud language models with web retrieval, personal documents, semantic search, custom agents, automations, and an experimental /research mode.

Its strongest use case is research that crosses the boundary between public information and a user's existing knowledge base. A question can be investigated in the context of PDFs, Markdown files, notes, office documents, or connected information rather than treating every task as web research from scratch. This makes Khoj useful for ongoing personal or team knowledge work.

Khoj is not primarily designed around a visible recursive research tree. Its research functionality is better understood as autonomous investigation within a larger personal knowledge system. Users seeking explicit breadth/depth controls or a dedicated research-engine API may prefer GPT Researcher or Local Deep Research.

Self-hosting is supported and the system can work with local models including the Llama, Qwen, Gemma, and Mistral families. Khoj is licensed under AGPL-3.0-or-later. Choose it when Deep Research should be closely integrated with a long-lived personal knowledge base rather than treated as an isolated web research job.

SurfSense

SurfSense is an open-source research workspace that evolved from a NotebookLM-style knowledge system toward an agent-oriented open-web research platform. It combines a searchable knowledge base with web and platform-specific data connectors, reports, automations, MCP access, and local model support.

SurfSense's distinctive advantage is its data surface. It is designed to give agents structured access not only to ordinary web pages and search results but also to sources such as Reddit, YouTube, Google Maps, and other live information services. Research findings can then remain in the same environment as uploaded documents and previously collected knowledge.

It is less purely focused on the research algorithm than GPT Researcher or STORM. A significant part of SurfSense's value comes from retrieval infrastructure, connectors, knowledge management, and downstream artifacts rather than an explicit recursively expanding research graph.

Self-hosting is supported through Docker-oriented installation, and local models can be connected through common local inference interfaces. The current repository is Apache-2.0 licensed. Choose SurfSense when the difficult part of research is obtaining, structuring, retaining, and reusing information from many different data sources.

Vane, Formerly Perplexica

Vane, formerly known as Perplexica, is an open-source AI answering engine designed as a self-hosted alternative to search-first products such as Perplexity. It combines an AI chat interface, search backend, citations, local model support, and semantic search over uploaded files. For the Docker quickstart, SEARXNG_API_URL wiring, and Ollama/llama.cpp setup, see Vane (Perplexica 2.0) Quickstart With Ollama and llama.cpp.

Vane is good at the search-and-answer experience. It classifies questions, conducts web research, retrieves useful information, and generates cited responses through a polished interface. Most deployments back it with SearXNG as the search layer. For users who mainly want a private AI search engine backed by SearXNG and local models, it provides a much more focused experience than a large general agent platform.

Its limitation in this comparison is research depth. Although the system can run research operations, its architecture is still primarily that of an answering engine rather than a recursively branching Deep Research framework. It should therefore not be treated as equivalent to GPT Researcher merely because both can perform multiple searches before responding.

Vane is relatively straightforward to deploy with Docker and supports common model providers and local inference systems. It is MIT licensed. Choose Vane when the primary requirement is high-quality self-hosted AI search with citations rather than long-running autonomous investigation.

Deep Research by lukeswade

The lukeswade/deep-research project is a smaller self-hosted research system, but it implements one of the more interesting workflows in this comparison. It supports both cloud models and local OpenAI-compatible endpoints including llama.cpp, LM Studio, Ollama, vLLM, and MLX - for the server side of that endpoint, see llama.cpp Quickstart with CLI and Server.

Its research loop is explicitly gap driven. A run decomposes the question into targeted searches, reads relevant pages, produces per-source notes containing evidence, and analyzes what remains unknown before deciding what should be searched next. Higher depth settings allow several rounds and progressively larger source budgets, while saturation detection can stop research early when new searches cease producing useful information.

It does not have the ecosystem, organizational connectors, or general AI workstation capabilities of Onyx, Open WebUI, or Unsloth Studio. It is much more narrowly focused on doing one thing: researching a question deeply and storing the resulting research in a searchable local library.

That focus also makes deployment relatively understandable. The system has a web UI and is particularly friendly to local OpenAI-compatible model servers; the documentation recommends capable models and supports using a smaller fast model for high-volume note processing. It is MIT licensed. Choose it when local inference, transparent evidence collection, and information-gap-driven research matter more than a large surrounding platform.

Which Deep Research System Should You Choose?

There is no single winner, because these systems solve somewhat different problems.

For an explicit research algorithm with understandable depth controls, GPT Researcher remains one of the clearest starting points. Its breadth/depth model makes it easy to reason about why research expands and how expensive a run may become.

For a strongly local workflow, Local Deep Research and Unsloth Studio are particularly attractive. Local Deep Research provides more research-strategy flexibility, while Unsloth Studio integrates research with model management, inference, RAG, and the broader local-model workflow.

For long-form knowledge curation, STORM remains unusually interesting because its multi-perspective question-generation technique attacks a problem that many research systems ignore: discovering the questions that the original user did not know to ask.

For multi-agent systems, DeerFlow represents a different direction. Instead of building a specialized research loop, it treats research as a long-running agent task that can be delegated among subagents and combined with tools, memory, code execution, and other capabilities.

For organizations, Onyx has one of the strongest combinations of Deep Research, RAG, web investigation, and enterprise knowledge connectors. Its heavier infrastructure is justified when internal information sources matter as much as the public web.

For an existing local AI installation, Open WebUI may be all that is necessary. A sufficiently capable local model with native tool calling can repeatedly search, read pages, follow links, verify information, and fill gaps without installing a separate research engine.

Finally, lukeswade/deep-research is worth watching precisely because it is smaller. Its gap-driven workflow is conceptually clean, supports llama.cpp directly through an OpenAI-compatible endpoint, and separates expensive planning and synthesis from high-volume per-source processing.

How to Tell Real Deep Research from Repeated Search

The most useful way to evaluate these systems is not to ask whether they have a button labelled "Deep Research." Instead, inspect what happens after the first round of information is collected.

A genuine research system should be able to discover that its original plan was incomplete. It should recognize a contradiction, missing source, unexpected implementation detail, or newly relevant subtopic and change its subsequent investigation accordingly. That is the line separating sophisticated retrieval from research.

For self-hosting, the ecosystem is now broad enough that the choice is no longer simply between a cloud Deep Research service and a home-grown script. There are dedicated recursive researchers, academic knowledge-curation systems, enterprise research platforms, local-model workstations, general agent frameworks, and lightweight gap-driven tools.

The right choice therefore depends less on which project advertises the most features and more on the research architecture you want to operate: explicit recursion, multi-agent delegation, evidence-gap analysis, perspective discovery, or model-driven autonomous search.

References

The Efficient Frontier of Open Models: Finding the Sweet Spot in 2026

Rost — Thu, 24 Sep 2026 10:19:42 +0000

In September 2026 the open-model efficient frontier sits between 25B and 34B parameters: near-frontier agentic work on one 24 GB GPU, at a fraction of 70B-class hardware.

That band shows up in deployments, not in a parameter-count slogan. Public leaderboards, single-GPU runs, and monthly API bills converge on models that stay close to frontier tool-use while the weights and the KV cache still fit hardware you can rent or own.

This page is the decision hub for that choice, inside the LLM performance cluster. It walks from small instruct models to 70B-class dense weights, then stops on the two architectures that define the frontier right now: Qwen3.8-27B, which is comfortable on one 24 GB card, and Llama 4 Scout, whose compute looks modest only because most of its experts stay idle. Token-per-second tables stay on the benchmark pages linked from each section.

By the end you should know which size to test first, how much VRAM the weights and the cache actually take, how a rented RTX 4090 compares with current Claude Sonnet token prices, and when a larger model is still the right call.

Why public benchmarks overstate production transfer

A model can sit within a couple of points of a frontier API model on MMLU or HumanEval and still fail the job you are shipping. Multi-tool flows drop the second tool call, refusal calibration drifts toward over-refusal on medical-adjacent queries, and P99 latency under burst load doubles. The leaderboard gap looked small because those benches measure capability on a fixed prompt set. They do not measure transfer onto your tools, your documents, and your latency budget.

The efficient frontier is the set of models that hold up on your workload at a cost you can keep paying. That takes a paired comparison on production-like tasks, with three signals recorded the whole way: tool-call success, refusal behaviour, and P99 latency. Future AGI's write-up on cheap-frontier substitution makes the same point from the evaluation side: a small public-benchmark gap is not a licence to swap models.

Treat the scores below as a map of where to start the bake-off. They are dated to September 2026, and several headline coding numbers are vendor-run rather than independent.

Where 8B, 30B, and 70B actually sit

At the low end, an 8B-class instruct model will follow a short, explicit instruction and write a small function. Ask it to debug a failing CI pipeline across several services, recover when the first tool call returns an error, and keep the plan intact, and the run falls apart. That is a workload failure, not a leaderboard failure.

The middle of the range is where general-purpose agentic work currently clears a single prosumer GPU. Qwen3.8-27B is the reference dense model. Architectures in the same size band, including the Qwen 30B-class MoE compared in the Qwen3 30B vs GPT-OSS 20B head-to-head, are the ones to test before you spend on anything larger.

At the high end, a dense 70B model at 4-bit is on the order of 35–40 GB of weights before any KV cache. It does not fit a 24 GB card. You are into a 48 GB GPU, an 80 GB data-center card, or a two-GPU split, and the extra quality over a well-tuned 27B model is often marginal on coding agents, document pipelines, and support bots. Closed frontier APIs such as Claude Opus sit in a different budget entirely: you pay per token, and you do not carry the weights at all.

Before you pick a checkpoint, confirm the card and the free memory. Model weights are only part of the footprint. Context (the KV cache) sits on top.

# GPU name and free VRAM before picking a quant
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

# after loading: confirm the process stayed on the GPU
nvidia-smi

If layers spill to CPU, generation throughput collapses. You can see the split in memory.used on the GPU and in the runtime's offload log. For what dense and MoE checkpoints actually deliver on a 16 GB card, the measurements live in the 16 GB VRAM llama.cpp benchmarks and the Ollama comparison on a 16 GB RTX 4080. Those pages own the speed tables. This one only needs the fit decision.

Qwen3.8-27B: the 24 GB reference point

Alibaba's Qwen3.8-27B is a dense model, not a mixture of experts. On Artificial Analysis it scores about 51 on the Agentic Index (the rounded figure; the underlying value is 50.9) and about 52 on the Intelligence Index at maximum reasoning effort. Those are different measurements. The Agentic Index is tool-use and multi-step tasks. The Intelligence Index is a broader capability blend. MindStudio's hands-on bench frames the agentic score as just behind Kimi K2, a vastly larger mixture-of-experts model. Qubrid's breakdown is the one to read next to it: the margin over the next model is under a point, and Qwen's own SWE-bench Pro figure of 61.7 has not been reproduced as an independent run. The interesting result is the shape of the scores. A 27B model converts a compact budget into planning and tool use unusually well, and it does not lead a broad knowledge suite.

The architecture is why the long context is affordable. Of 64 layers, only 16 use full attention. The other 48 are Gated DeltaNet layers, a linear-attention design from the Gated Delta Networks line of work, and they keep a fixed recurrent state instead of a per-token key/value history. At FP16 the full-attention layers cost about 64 KB of KV cache per token. A conventional stack with the same layer count would cache roughly four times that. Locally Uncensored's quant table puts Q4_K_M weights at 17.1 GB and IQ4_XS at 15.7 GB. Hardware Corner's llama.cpp measurements on a 24 GB GPU landed near 18 GB at short context and about 22 GB at 64K, which is the practical target on an RTX 4090. A 16 GB card can hold an IQ4-class quant of the weights and almost no context. If that is your machine, stay with the Qwen 3.5 and 3.6 runs already measured on 16 GB, including Qwen 3.6 27B MTP versus standard decoding. How to budget the cache itself is covered in KV cache on 16 GB GPUs.

Hands-on reports, MindStudio's among them, also find the model usable for coding, image analysis, and agent loops, which is the part a single index cannot show. Vision adds a small projector file on top of the text weights. Plan for it if the workload includes screenshots.

Llama 4 Scout: active parameters are not VRAM

Meta's Llama 4 Scout is a different kind of efficiency. The Llama 4 model card lists 17 billion active parameters and 109 billion total, with 16 experts, native image input, and a 10 million token context window. Meta's launch post says the INT4 checkpoint fits on a single H100. Decode compute tracks the 17B that fire for each token. Memory does not. Every expert still has to be resident, so the VRAM bill looks like a 100B-class model even though the FLOPs look like a 17B model.

That is the correction worth making before you budget a machine. Scout does not "run like a 3B model." Active parameters set speed. Resident parameters set whether the card can hold the weights. On a 24 GB GPU, Qwen3.8-27B is the model that fits. Scout is the model that makes sense once you already have an 80 GB card and want 17B-class compute with a much larger expert pool and a very long context.

A June 2025 paper, Can Mixture-of-Experts Surpass Dense LLMs Under Strictly Equal Resources?, argues that MoE can beat a dense model when the comparison holds total compute and data fixed and the activation rate sits in a workable band. The practical reading is narrower than the headline. MoE wins on cost only after you have paid for the memory that stores the idle experts. If you are already serving a model of this size and want more tokens per second without changing the weights, speculative decoding is the neighbouring technique: a draft path proposes tokens and the main model verifies them.

Phi-4: the small-model exception for math

Microsoft's Phi-4 is a 14B dense model. In the December 2024 technical report, simple-evals scores it at 80.4 on MATH, ahead of the GPT-4o figure in the same table, with a context window that the report extends to 16K. A Q4_K_M quant is commonly run in about 8 GB. That is real efficiency, and it is narrow. Phi-4 was trained for STEM question-answering. It is the model to try when the job is math or short technical reasoning and the machine is small. It is not the 2026 default for multi-tool agents, long documents, or vision. The frontier for that work is still the 25–34B band above, or Scout when the GPU is an H100-class card.

Self-hosted GPU hours versus per-token API prices

Prices move. These are mid-September 2026 figures, and they will not survive unchanged into 2027.

GPU Finder, checked on 18 September 2026, showed an in-stock RTX 4090 at about $0.26 per GPU-hour on Vast, with RunPod community cloud listed at $0.34. The six-month on-demand floor for a single 4090 sat between $0.11 and $0.60. At $0.34 and 730 hours in a month, a card that stays up all month is about $248 before tax, storage, and egress. At the $0.26 in-stock quote the same month is about $190. An earlier planning figure of $0.53 per hour is inside that six-month range, but it overstates what marketplace stock was actually asking in the second half of September.

On the API side, Anthropic's Sonnet 5 pricing notes list $2 per million input tokens and $10 per million output tokens. Sonnet 4.6 remains at $3 and $15. BenchLM's rate card, updated 3 September 2026, shows the same split. A workload of 100 million input tokens and 10 million output tokens is $300 on Sonnet 5 and $450 on Sonnet 4.6.

Put next to a 4090 rented all month at $248, that example does not say "self-hosting is always cheaper." It says the two bills meet in the same neighbourhood for this volume. Sonnet 5 at a 10:1 input-to-output ratio costs $30 per 10 million input plus 1 million output. The $248 rental covers about eight of those blocks: roughly 80 million input tokens and 8 million output tokens. Above that, the rented card pulls ahead, provided you can keep it busy. Below it, the API is the cheaper line, and you are not paying for an idle GPU. Owned hardware replaces the rental with electricity and depreciation. The strategies around that bill — budgeting, caching, and fallbacks — are in cost optimization for LLM systems. The reason a lower token price can still be the wrong architecture is data gravity and API lock-in.

When a larger model is still the right call

The 25–34B band is the default for production workloads that repeat: coding agents, support bots, document processing, extraction, and automation. It is the wrong default in a few situations.

Frontier reasoning on problems the smaller model has already failed in a paired test.
Prose where the difference in voice is the product, and volume is low enough that token price is noise.
Work that needs broad coverage across several specialist domains at once, where the 27B model keeps missing the same class of fact.
Low-volume, high-stakes decisions, where the cost of a wrong answer exceeds a year of model rental.

In those cases, move up to a 70B-class open model or a frontier API, and keep the same three production signals. If the larger model does not move tool-call success, refusals, or P99 in the direction you care about, the extra size is not buying anything you can use.

How to pick a point on the frontier

Use this order. It is cheaper to discover that a 27B model is enough than to discover that an 80 GB card was unnecessary.

Start from the workload, not from the largest open weights you can download. Single-shot instructions and math can start at 8–14B. Multi-step tool use starts at Qwen3.8-27B if you have 24 GB, or at the best 16 GB quant you have already measured if you do not.
Separate active parameters from resident parameters. Scout's 17B active figure is a compute claim. The 109B total is the VRAM claim. Budget the second number.
Quantize, then re-check memory. Q4_K_M of Qwen3.8-27B is a 24 GB conversation, with about 64K of context before the card is full. Run nvidia-smi after the load, at the context length you will actually serve.
Price the month, not the token. Compare 730 hours of the GPU you would rent with the input and output mix you already log. If you are under the break-even above, stay on the API until the volume arrives.
Decide on your own tasks. A paired run that records tool-call success, refusal behaviour, and P99 under burst is the test that public benches cannot replace.

flowchart TD
    A[Define the workload] --> B{Multi-step tool use?}
    B -- No: single-shot or math --> C[Start at 8-14B]
    B -- Yes --> D{One 24 GB GPU?}
    D -- Yes --> E[Qwen3.8-27B near Q4, then check KV cache]
    D -- No: 80 GB class card --> F[Llama 4 Scout: 17B active, 109B resident]
    C --> G[Paired run on your tasks]
    E --> G
    F --> G
    G --> H{Tool calls, refusals, and P99 pass?}
    H -- Yes --> I[Deploy and keep those three signals]
    H -- No, volume is low --> J[70B-class weights or a frontier API]
    H -- No, volume is high --> K[Next size up, then reprice the month]

The useful question in 2026 is which weights fit the card, the context, and the monthly bill while still clearing your own tasks. For a lot of agentic work, that point is a 27B hybrid model on one 24 GB GPU. Scout is the same idea at data-center memory: less compute per token than the parameter total suggests, and no discount on the VRAM.

References

MindStudio. "Qwen 3.8 27B Benchmarked: Agentic Index, Vision, and Reasoning Tests." https://www.mindstudio.ai/blog/qwen-3-27b-local-benchmark
Qubrid AI. "Qwen3.8-27B Benchmarks: Official and Independent Results." https://www.qubrid.com/blog/qwen38-27b-benchmarks-official-and-independent-results
Hardware Corner. "We Tested Qwen3.8 27B: How Much GPU and VRAM Do You Really Need?" https://www.hardware-corner.net/qwen3-8-27b-hardware-tests/
Locally Uncensored. "How to Run Qwen 3.8 27B Locally: VRAM, Quants and the Template Trap." https://locallyuncensored.com/blog/how-to-run-qwen-3-8-27b-locally.html
Malik, Umesh. "Qwen3.8 27B VRAM: how to fit 262K context in 16 GiB, not 64." https://umesh-malik.com/blog/qwen3-8-27b-vram-kv-cache-math
Meta AI. "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation." https://ai.meta.com/blog/llama-4-multimodal-intelligence
Meta. "Llama 4 model card." https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md
arXiv. "Can Mixture-of-Experts Surpass Dense LLMs Under Strictly Equal Resources?" June 2025. https://arxiv.org/html/2506.12119v1
Yang, S., Kautz, J., Hatamizadeh, A. "Gated Delta Networks: Improving Mamba2 with Delta Rule." ICLR 2025. https://jankautz.com/publications/GatedDeltaNet_ICLR25.pdf
Abdin, M., et al. "Phi-4 Technical Report." arXiv:2412.08905, December 2024. https://arxiv.org/html/2412.08905v1
GPU Finder. "RTX 4090 GPU Rental Prices & Live Availability." Checked 18 September 2026. https://gpufinder.dev/gpu/rtx-4090
Anthropic. "What's new in Claude Sonnet 5" (pricing: $2 / $10 per million tokens). https://platform.claude.com/docs/en/models/sonnet-5/whats-new-sonnet-5
BenchLM. "Claude API pricing." Updated 3 September 2026. https://benchlm.ai/anthropic/api-pricing
Future AGI. "Evaluating Cheap Frontier Models in 2026: Substitution Without a Quality Cliff." https://futureagi.com/blog/evaluating-cheap-frontier-models-2026
Northflank. "Qwen3.8-27B: Performance, benchmarks, GPU requirements & how to run it." 17 August 2026. https://northflank.com/blog/qwen3-8-27b-performance-benchmarks-gpu-requirements-and-how-to-run-it

Mnemosyne for Hermes Agent: Local Memory Quickstart

Rost — Tue, 22 Sep 2026 10:07:07 +0000

Mnemosyne is a local-first memory provider for Hermes Agent, storing working memory, structured facts, temporal data, and episodic history in local SQLite — no hosted service, no mandatory network calls, and unusually granular write control.

Its most useful property is not raw recall quality. It is the amount of control it exposes over the write path: conversation autosave can be restricted by role or disabled outright, tool-result logging defaults off, explicit remember and forget operations stay available regardless, and newer releases add opt-in self-echo suppression around context-compression boundaries. That combination makes it a reasonable choice when you want persistent memory without automatically turning every conversation into permanent knowledge.

That write-path discipline matters because agent memory has a well-documented failure mode: a model's own inference can be captured, retrieved later as if it were an observation, and used to justify an even stronger version of itself. Self-Reinforcing Memory Loops in AI Agents covers that failure mode in depth; this guide focuses on the concrete Mnemosyne configuration that limits it in practice. For where Mnemosyne sits relative to the other Hermes memory backends, see Agent Memory Providers Compared.

Mnemosyne in one minute

A typical memory provider does some version of capture, extract, store, retrieve, then inject into a future prompt. Mnemosyne adds several distinct layers around that basic loop: working memory, semantic and lexical recall, structured facts, temporal information, entity links, episodic memory, consolidation, canonical facts, and memory validation. Storage is local SQLite with FTS5 and optional vector retrieval, which makes it considerably more inspectable than a cloud-only memory product and more capable than a plain MEMORY.md file.

Very briefly, relative to the rest of the Hermes provider ecosystem: Holographic is simpler and deliberately fact-store oriented; Hindsight emphasizes hybrid retrieval, knowledge graphs, and reflection; Honcho emphasizes peer and user modeling with dialectic reasoning; Mem0 emphasizes automatic LLM-based fact extraction; and Mnemosyne combines local SQLite storage, hybrid recall, consolidation, structured facts, and unusually granular retention controls. The full breakdown, including infrastructure requirements and self-hosting notes for every provider, is in Agent Memory Providers Compared.

Current versions

As of September 2026, the stable PyPI release is mnemosyne-memory 3.15.1, with the 4.0 branch available as a pre-release. For a production Hermes installation, start with the stable version unless you specifically need a 4.0 fix or feature and are prepared to test the database migration and behavior change. Check your installed version with:

hermes mnemosyne version

Installing Mnemosyne into Hermes

Activate Hermes' own virtual environment first if you used the standard local installation:

source ~/.hermes/hermes-agent/venv/bin/activate

For local embedding support, install the core package with the embeddings extra plus the Hermes plugin wrapper:

python -m pip install \
  "mnemosyne-memory[embeddings]" \
  mnemosyne-hermes

Then register the plugin:

mnemosyne-hermes install

If you are replacing an existing plugin registration:

mnemosyne-hermes install --force

Activate the provider and restart the gateway:

hermes config set memory.provider mnemosyne
hermes gateway restart

Verify with:

hermes memory status

Expected output looks similar to:

Provider: mnemosyne

Plugin: installed
Status: available

Docker and persistent-server installs

If Hermes runs inside a persistent Docker or image-based deployment, install into a side virtual environment on the mounted Hermes home instead of the container's rebuildable Python environment, so the plugin survives image rebuilds:

export HERMES_HOME=/opt/data
VENV="$HERMES_HOME/.mnemosyne/venv"
python3 -m venv "$VENV"
"$VENV/bin/python" -m pip install --upgrade "mnemosyne-memory[embeddings]" mnemosyne-hermes
"$VENV/bin/mnemosyne-hermes" install --mode wrapper --python "$VENV/bin/python"
hermes config set memory.provider mnemosyne

The side venv must use the same Python major/minor version as the running Hermes gateway — do not point it at an unrelated python3 from PATH. Restart the actual container or service afterward and verify with "$VENV/bin/mnemosyne-hermes" status alongside hermes memory status.

Do not disable the whole Hermes memory toolset

Keep two concepts separate: Hermes' own built-in memory (MEMORY.md / USER.md, covered in full in Hermes Agent Memory System) and the external provider (Mnemosyne). Do not casually run hermes tools disable memory when configuring an external provider — depending on the Hermes version, that command can also hide external memory-provider tools. Use provider configuration instead, as shown below.

Basic status and inspection

hermes memory status
hermes mnemosyne stats
hermes mnemosyne stats --global
hermes mnemosyne inspect "query"

Export a portable backup:

hermes mnemosyne export \
  --output ~/mnemosyne-backup.json

The backing database normally lives under ~/.hermes/mnemosyne/data/mnemosyne.db. Because it is SQLite, inspection and backup are straightforward with standard tools. For the rest of the gateway, session, and diagnostics commands referenced throughout this guide, the Hermes Agent CLI cheat sheet is a faster reference than digging through --help output.

The default retention policy deserves attention

The first control worth understanding is sync_roles. Current Mnemosyne defaults are already more conservative than early releases — automatic Hermes synchronization defaults to user turns rather than both user and assistant turns — but for strict explicit-only retention, disabling turn autosave completely is worth the extra step. Edit ~/.hermes/config.yaml:

memory:
  provider: mnemosyne

  mnemosyne:
    sync_roles: []

An empty list means ordinary conversation turns are not automatically saved by sync_turn(). Explicit mnemosyne_remember operations continue to work regardless — normal conversation stops flowing into memory automatically, while an explicit "remember this" still reaches Mnemosyne.

Disable automatic tool-result logging

Mnemosyne can also log tool executions as memory. For a conservative setup, leave that disabled in ~/.hermes/.env:

MNEMOSYNE_LOG_TOOLS=0

This is already the default, but setting it explicitly documents the policy rather than relying on an assumption about defaults. Restart Hermes afterward:

hermes gateway restart

With sync_roles: [] and MNEMOSYNE_LOG_TOOLS=0 together, both major automatic write paths — conversation autosave and tool-result autosave — are off.

Keep automatic recall

Disabling automatic writes does not require disabling recall. A useful policy keeps automatic retention off while automatic recall, explicit remember, and explicit forget all stay on — memory should be easy to read and difficult to write, which is close to the opposite of a "capture everything and sort it out later" default.

Add a durable agent instruction

Provider configuration blocks automatic provider-level capture, but the model can still decide to call an explicit write tool on its own initiative. Add an explicit policy to SOUL.md:

## Long-term memory policy

Mnemosyne is the long-term memory provider.

Do not write anything to Mnemosyne unless the user explicitly asks you to
remember, save, retain, or store that information.

If information appears useful for future sessions but the user did not
explicitly request that it be remembered, ask for permission before calling
mnemosyne_remember or another Mnemosyne write tool.

Do not create durable memories from your own reasoning, assumptions,
summaries, interpretations, conclusions, or inferred preferences.

Do not create durable memories from tool output unless the user explicitly
asks for that result to be remembered.

When storing an approved memory, preserve what the user actually stated.
Do not embellish it with inferred context or conclusions.

Reading and recalling Mnemosyne memories is allowed without asking for
permission.

Restart the gateway and start a fresh session afterward:

hermes gateway restart

/new

This is a model-enforced policy, not a hard permission boundary — it complements the provider-level configuration above rather than replacing it.

What about `memory.write_approval`?

Hermes supports memory.write_approval: true for built-in MEMORY.md / USER.md writes, and Mnemosyne implements its own provider-specific staging for explicit writes in newer releases. This is promising, but there is an architectural caveat worth taking seriously: Hermes does not yet expose one uniform, provider-neutral approval contract across all external memory providers, and Mnemosyne's pending/apply implementation is provider-specific rather than part of a shared standard. Do not assume approval works correctly just because the configuration key is present — test it against your exact Hermes and Mnemosyne versions. Until provider-independent approval matures, combining sync_roles: [], MNEMOSYNE_LOG_TOOLS=0, and the explicit-write SOUL.md policy above gives you a dependable baseline, with the approval path tested separately if you intend to rely on it.

Enable self-echo suppression

Current Mnemosyne also offers optional self-echo suppression:

MNEMOSYNE_SELF_ECHO_ENABLED=1

Put this in ~/.hermes/.env, then restart:

hermes gateway restart

Self-echo suppression targets context-compression boundaries specifically — its purpose is to reduce cases where memory the provider just created gets immediately fed back into the agent as if it were independent context. It is intentionally best-effort and does not replace write filtering: write controls stop questionable memories from entering in the first place, while self-echo controls stop recent provider output from bouncing straight back. Both matter, and neither substitutes for the other.

A conservative Mnemosyne configuration

Putting the pieces together, a starting configuration for a self-hosted personal engineering agent looks like this. In ~/.hermes/config.yaml:

memory:
  provider: mnemosyne

  mnemosyne:
    sync_roles: []

In ~/.hermes/.env:

MNEMOSYNE_LOG_TOOLS=0
MNEMOSYNE_SELF_ECHO_ENABLED=1

And in SOUL.md, at minimum:

Only store long-term memory when the user explicitly requests it.
Do not promote model-generated conclusions or tool output into durable memory
without explicit permission.

flowchart LR
    U[User conversation] -.->|blocked| M[(Mnemosyne)]
    T[Tool results] -.->|blocked| M
    R["Explicit: remember this"] -->|mnemosyne_remember| M
    Q[Future question] -->|recall| M

Test that ordinary conversation is not retained

Check the baseline count first:

hermes mnemosyne stats

Start a new Hermes session and say a plain factual statement without asking the agent to remember it, for example:

PurpleOtter uses port 48123.

Afterward, search for it:

hermes mnemosyne inspect "PurpleOtter"

Expected: Results for 'PurpleOtter': 0. Also re-check hermes mnemosyne stats — the working-memory count should not have increased because of that ordinary turn.

Test explicit memory

Now say the same kind of statement, but explicitly ask for retention:

Remember that BlueKoala uses port 17321.

Inspect it, then start a new session and ask for it back:

hermes mnemosyne inspect "BlueKoala"

/new

What port does BlueKoala use?

Hermes should retrieve the value correctly — this pair of tests isolates the write-path policy (nothing gets in without asking) from the retrieval mechanism (what gets in comes back out reliably).

Test tool logging

With MNEMOSYNE_LOG_TOOLS=0 set, ask Hermes to run a distinctive, unique command:

Use the terminal tool to run:
echo tool-canary-834729

Then search for the canary string:

hermes mnemosyne inspect "tool-canary-834729"

Expected: 0 results. This is a much stronger test than simply trusting that the environment variable is honored everywhere.

Inspecting the database

Because storage is SQLite, the internal schema is directly inspectable:

sqlite3 ~/.hermes/mnemosyne/data/mnemosyne.db '.tables'

Depending on version, you may see tables such as working_memory, episodic_memory, facts, consolidated_facts, gists, graph_edges, memoria_facts, and memory_embeddings. This matters when testing deletion — a memory system can successfully remove a working-memory row while leaving a derived fact, gist, or graph object behind. Mnemosyne has had real bugs in this area involving orphaned derived records, and newer releases have tightened both deletion and diagnostics accordingly. Prefer the provider's supported delete and doctor/repair paths over manually deleting SQLite rows unless you fully understand the current schema.

Deleting session-scoped working memory

One subtlety: Mnemosyne working memories can be session-scoped, so a row with scope = session may not be visible to a standalone delete operating in the default session. When debugging, inspect scope directly:

SELECT id, session_id, scope, content
FROM working_memory;

The provider or API needs the correct session scope to mutate session-local records — another reason to prefer supported administration tools over raw SQL edits.

Consolidation: do not rush to `sleep()`

Mnemosyne can consolidate working memory into longer-lived representations, which is useful but is a mutating operation. Before enabling aggressive automatic consolidation, inspect what is actually being captured, verify that ordinary turns are not entering memory unexpectedly, verify deletion end to end, and back up the database. Then experiment with:

hermes mnemosyne sleep

Recent Mnemosyne changes made conflict handling more conservative — semantic similarity alone no longer proves that one memory should invalidate another, which is exactly the direction a durable agent-memory system should move in, as covered in Self-Reinforcing Memory Loops in AI Agents.

Backup before upgrades

Create a portable export before any significant change:

hermes mnemosyne export \
  --output ~/mnemosyne-backup.json

For important installations, also copy the local database or data directory before major upgrades. Mnemosyne 4.x is currently a pre-release line, so a major-version upgrade deserves more caution than a routine patch update.

Final recommended setup

For a long-running Hermes installation where memory accuracy matters more than remembering everything, the durable configuration is: Mnemosyne local storage on, automatic recall on, conversation autosave off, assistant-message autosave off, tool-result logging off, explicit remember and forget on, self-echo suppression on, session search on, and human review for sensitive writes desirable once the approval path is tested. That makes Mnemosyne function primarily as a curated long-term memory store rather than a transcript archive — the goal is not to make Hermes remember everything it has ever said, but to make it remember the things that will still be true when the next session begins. If you run several profiles with different providers or retention policies, Hermes Agent production setup covers the profile-level wiring for keeping them consistent.

Self-Reinforcing Memory Loops in AI Agents: Causes and Fixes

Rost — Sat, 19 Sep 2026 09:56:04 +0000

Persistent memory turns an agent from a re-explained tool into one that carries context forward — but it opens a failure mode stateless chat avoids: an interpretation can become memory, retrieved as fact, and justify a stronger version of itself.

That is a self-reinforcing memory loop. The mechanism does not require malice, a broken plugin, or an unusual prompt — a normal capture pipeline stores the assistant's output, a normal retrieval pipeline surfaces it as context, and the model treats retrieved text as evidence because that is what retrieved text usually is.

This is different from a hallucination in one important way: a hallucination disappears when the conversation ends, but a hallucination that gets promoted into durable memory can outlive the session that created it, resurface weeks later in an unrelated context, and gain apparent credibility purely from repetition. For the broader memory model this problem sits inside — working memory, structured state, and retrieval memory as three separate contracts, part of the AI Systems Memory hub — see Memory Systems in AI Assistants, which already flags stale and contradictory memory as the most common production failure. This article goes one level deeper into why that specific failure keeps recurring.

The question worth asking about any memory system is not simply whether it remembers. It is what is allowed to become evidence for future reasoning. Answering that well requires distinguishing something the user explicitly stated, something a tool actually observed, something an external document reported, and something the model merely inferred or summarized. Once those categories collapse into a single undifferentiated pool called "memory", a generated conclusion becomes indistinguishable from an observation — the memory system has effectively laundered an inference into a premise. For a practical walkthrough of running one provider with that distinction enforced, see Mnemosyne for Hermes Agent: Local Memory Quickstart; for how the eight-plus mainstream providers differ on this exact axis, see Agent Memory Providers Compared.

What is a self-reinforcing memory loop?

The simplest version has a fixed shape: a user states something, the agent infers a conclusion from it, memory stores that conclusion, a future session recalls it, the agent treats the recalled statement as evidence, derives a stronger conclusion, and writes that back to memory. The cycle then repeats with a slightly more confident claim each time, without a single new observation ever entering the pipeline.

flowchart LR
    A[User states X] --> B[Agent infers Y]
    B --> C[Memory stores Y]
    C --> D[Future session recalls Y]
    D --> E[Agent treats Y as evidence]
    E --> F[Agent derives stronger Y2]
    F --> C

Consider a developer who tells an agent that a deployment failed after a cache configuration change. A reasonable inference is that the cache configuration probably caused the failure — useful reasoning if it stays inside the current context. The damage begins when an automatic memory extractor stores the flat claim "the cache configuration caused the deployment failure" as a fact. A week later, a second unrelated deployment fails; the agent retrieves the stored claim and reasons that the cache layer has a history of instability, which gets written back as an even more general belief. By the third pass, the stored memory reads as "the cache layer is known to be unreliable and should be replaced" — a confident institutional claim built from zero new evidence.

Why persistent agent memory makes this worse than a database

A conventional application database has an explicit write path: a field changes because a known user, API call, or transaction changed it. Agent memory systems typically have many more writers — the user, the assistant, tool results, an automatic turn-capture hook, a fact extractor, a session summarizer, a reflection pass, a consolidation process, and sometimes another agent — and just as many readers, including automatic prompt injection, semantic retrieval, and sub-agent tools. Once the output of one reader can become the input to another writer, the system is a feedback loop rather than a simple store, and ordinary database intuitions about "who wrote this and when" stop applying.

The main forms of memory feedback

Self-reinforcement is not one mechanism — it shows up in at least seven related but distinct patterns, and a memory provider can be resistant to one and vulnerable to another.

Assistant self-echo

The simplest case occurs when assistant messages are automatically retained: the model's own previous answer becomes contextual evidence for its next answer. That does not automatically make the answer wrong, but it does change its epistemic status — generated language has become persistent context. The safest general rule is that user and tool observations may be memory candidates, but assistant conclusions should not automatically become facts without a separate promotion step.

Summary-of-summary drift

Long-running agents compress conversations repeatedly — raw conversation to summary, summary to long-term memory, memory to a user profile — and each transformation can quietly discard a qualifier. "I usually use PostgreSQL, but SQLite is fine for small tools" can become "User prefers PostgreSQL," then "User uses PostgreSQL," then "User's projects use PostgreSQL," at which point a future SQLite suggestion gets flagged as violating the user's architecture preference. No single step in that chain is dramatic; the cumulative effect is a wrong belief with a completely plausible paper trail.

Reflection amplification

Some providers intentionally perform higher-order reasoning over stored memories — Hindsight's reflect is a documented example — and this is genuinely useful because agents need synthesis, not only retrieval. The risk begins when a derived conclusion is stored alongside the raw observations it came from with no marker distinguishing them, so a later reader sees four apparently independent facts instead of three observations and one interpretation of those observations.

Retrieval amplification

Retrieval itself introduces bias without any reflection step at all: a memory that gets retrieved often appears in more prompts, gets mentioned more often, gets recaptured more often, and produces more related memories, which in turn get retrieved even more often. The memory becomes prominent partly because it was already prominent — a popularity loop rather than an evidence loop.

Contradiction collapse

A dangerous pattern appears when a memory system decides which of two conflicting statements is true using similarity alone. Mnemosyne provides a concrete real-world example: a production audit found that similarity-based conflict handling had invalidated 142 of 243 stored items across consolidation passes, because the system treated "these two statements look alike" as proof that one superseded the other. Newer Mnemosyne releases now treat similarity as a candidate contradiction rather than proof — actual invalidation requires a successful validation step — which is the right general direction for any provider with a consolidation pass. The underlying lesson generalizes well beyond Mnemosyne: semantic similarity is not evidence of contradiction, since two statements can differ because of date, environment, branch, or deployment rather than because one is wrong.

User-model reinforcement

Systems that maintain a running model of the user, not just a list of facts, face a sharper version of the same problem. "User prefers concise answers" or "user deploys to AWS" are useful and low-risk; "user dislikes technology X" or "user always chooses architecture Y" are inferred traits that, if partly based on the agent's own earlier interpretations, can progressively turn a real person into a caricature of one interaction.

Agent self-model reinforcement

The most subtle case is an agent that models itself: it performs an action, explains that action, and a memory system builds a self-model from the explanation that gets fed into the next session, which then behaves according to that self-model and reinforces it further. A useful self-model can stabilize an agent's behavior over time. A wrong one stabilizes the agent around the wrong behavior just as effectively — the loop does not care which direction it locks in.

Why confidence tends to increase along the way

Large language models do not automatically know that a retrieved sentence was originally generated by another instance of themselves. "User: I think server X might have a networking problem" reads as tentative; "Relevant memory: Server X has a networking problem" reads as settled, even though both may trace back to the same uncertain guess. That syntactic shift from a hedged claim to a declarative memory object is source laundering, and it compounds when several derived memories happen to agree with each other — three semantically similar memories can look like independent corroboration even when all three originated from a single conversation.

Consequences that show up in production systems

The practical damage takes a handful of recognizable shapes. False certainty means the agent stops checking an assumption because memory presents it as already settled. Preference drift means a tentative preference gradually hardens into an absolute instruction. Wrong user profiles mean one unusual interaction gets generalized into a long-term behavioral trait. Tool-action cascades are the most expensive version: a false remembered premise drives a wrong diagnosis, which drives a tool call, which drives a real configuration change — persistent agents raise the cost of a memory error precisely because the error can reach the outside world. Duplicate-memory inflation and stale-state lock-in both waste prompt budget and retrieval quality over time, and destructive consolidation can let an inferred, newer-looking statement quietly supersede an older but more authoritative observation.

Deletion deserves a specific warning here. Modern providers frequently build several derived structures from one captured item — working memory, extracted facts, summaries, embeddings, graph edges, canonical facts, and profile entries — and deleting the original memory does not guarantee every derived representation disappears with it. Memory deletion needs to be tested end-to-end, not assumed to work because an API returned success.

Provenance matters more than embedding quality

Most memory engineering effort goes into retrieval — vector similarity, BM25, hybrid search, rerankers, graph traversal, temporal weighting — and all of that is genuinely useful, but none of it addresses the fundamental problem, because retrieval quality only affects which memories surface, not whether a surfaced memory deserves the confidence it is given.

A production memory object should carry metadata beyond its content: source, source type, timestamp, scope, confidence, what it was derived from, validation status, and whether it has been superseded. A rough tiering that works in practice ranks explicit user statements and direct tool observations highest, trusted external data next, deterministic extraction below that, then summaries, with model inference and reflection output at the bottom of the trust hierarchy — not because inference is worthless, but because it should never silently inherit the trust level of the observation it was built from. Retrieval and consolidation can then respect that hierarchy instead of ranking purely by semantic similarity.

A safer architectural pattern

For most personal and engineering agents, a deliberately boring memory pipeline outperforms a fully automatic one. The key design decision is that the agent does not turn every conversation into durable truth by default — a candidate memory gets classified before it is retained, with observations retained, inferences kept transient, and uncertain cases routed back to the user rather than silently written.

flowchart TD
    A[Conversation] --> B[Candidate memory]
    B --> C{Classification}
    C -->|Observation| D[Retain]
    C -->|Inference| E[Keep transient]
    C -->|Uncertain| F[Ask user]

For high-value environments, a human approval gate is worth the friction: a candidate memory moves to pending status, a human reviews it, and only an explicit approval commits it to durable memory while a rejection discards it. Hermes' own memory.write_approval: true setting stages built-in MEMORY.md writes for exactly this reason, and the same idea shows up as provider-specific staged writes in Mnemosyne — though it is worth noting that no uniform, provider-independent approval contract exists across Hermes external memory plugins yet, so this path should be tested against the exact versions you run rather than assumed to work everywhere.

How current providers address the problem

No provider eliminates feedback loops completely; each makes a different trade-off between convenience and control.

Hermes' own built-in MEMORY.md and USER.md files are intentionally small and human-readable, which makes them easy to audit even without special tooling — the trade-off is scale, since this is not a semantic long-term memory database. Hermes Agent Memory System covers that bounded design in full.

Mnemosyne is one of the more governance-oriented external providers precisely because it exposes independent controls over what gets written: conversation autosave can be disabled entirely with sync_roles: [] while explicit memory operations stay available, tool-result logging defaults off, and newer builds add opt-in self-echo suppression around context-compression boundaries. Mnemosyne for Hermes Agent: Local Memory Quickstart walks through a conservative configuration end to end.

Hindsight's default Hermes integration is comparatively automatic — autoRecall and autoRetain both default to true — which is convenient but increases the number of feedback paths; setting auto_retain=false while keeping recall on is worth considering if provenance matters more than convenience. Holographic and ByteRover both default auto_extract to off, which means they can operate primarily as explicit fact stores rather than automatic transcript-to-memory pipelines, an advantage if feedback loops are your main concern. Honcho's unified observation mode is more conservative than its directional default because it lets the AI model the user without building the matching self-observation loop from its own messages — worth serious consideration for anyone specifically worried about agent self-model reinforcement. Agent Memory Providers Compared has the full provider-by-provider comparison, including capture policy and approval support for each.

Configuration questions that matter more than benchmarks

Recall benchmarks measure whether an agent can retrieve the right information. Production systems need answers to a different set of questions: what gets written automatically, can assistant output become memory, are tool results retained automatically, are summaries stored as facts, are derived facts marked as derived, can old memories be superseded automatically, can a user inspect everything retained, does delete remove derived representations too, can automatic recall be disabled independently from automatic retention, is there a human approval gate, and can the agent itself bypass that gate. Those eleven questions are usually more diagnostic than another five points on a long-term recall benchmark.

My preferred policy for personal engineering agents

For a self-hosted engineering assistant, automatic conversation retention, automatic assistant retention, and automatic tool-result retention should all default off, while automatic recall stays on or selective, explicit remember stays on, session history search stays on, and derived conclusions stay transient by default rather than durable. The durable store should contain facts worth carrying into another session; the original session history should remain separately searchable when the agent actually needs evidence rather than a summary of it. Memory becomes concise retained knowledge, and session search becomes the original evidence — the two should never be conflated into one undifferentiated pool.

How to test a memory provider

Testing whether a provider remembers is the easy half. The harder and more useful half is testing whether it refuses to remember and whether it forgets completely when asked.

Tell the agent an ordinary fact without asking it to remember anything, start a new session, and confirm the value does not appear if automatic capture is supposed to be disabled. Then explicitly ask it to remember a different fact, start a new session, and confirm that one does retrieve correctly — this pair of tests isolates the write-path policy from the retrieval mechanism. Separately, give the agent enough information to make an inference but never state that inference yourself, then inspect the memory database directly; the inference should not silently appear as a standalone fact. Run a distinctive, unique tool command and search memory for it afterward to confirm tool-result logging behaves as configured. Store a fact, delete it, and then check every layer a provider might use — working memory, semantic recall, fact tables, graph nodes, summaries, embeddings, and profile context — because a successful delete API response is not sufficient proof that the data is actually gone. Finally, store two contradictory facts and inspect whether the provider keeps both with timestamps, marks one superseded, destroys the old record, or asks for validation — that single test reveals more about a provider's epistemic model than any feature list.

The central design rule

A model-generated conclusion must not become stronger evidence merely because the same model remembered it. Memory systems need provenance, controlled write paths, explicit treatment of derived knowledge, and deletion that actually reaches every derived representation, not just the record a user can see. The most advanced memory provider is not necessarily the one that remembers the most — for long-running agents, the better provider is often the one that knows when not to remember.

OpenSpec Rejected Proposals: A Decision Memory Convention

Rost — Fri, 18 Sep 2026 09:09:18 +0000

An agent that proposed and shipped "move persistence into a shared library" six months ago will happily propose it again next quarter unless something durable tells it the idea was already investigated and rejected -- and OpenSpec has no built-in state for that today.

/opsx:archive is built for one outcome: a change that shipped. It syncs the delta specs into openspec/specs/ and moves the folder to openspec/changes/archive/YYYY-MM-DD-<name>/ as a record of what changed and why. There is no /opsx:reject or /opsx:abandon counterpart, and nothing in the archive format tells a future proposal "this exact idea was investigated and turned down." That gap matters most in exactly the codebases where OpenSpec is otherwise a good fit: brownfield systems with a small number of contributors and agents that periodically re-explore the same architectural questions -- merge these two services, share this persistence layer, replace this HTTP boundary with a direct import.

This is not a hypothetical gap. It was raised directly with OpenSpec's own maintainers as a feature request, and the way that conversation played out is worth knowing before you improvise your own fix: what the project actually concluded shapes which convention is worth adopting. This guide walks through what happens if you rely on /opsx:archive alone, the real discussion that already happened in OpenSpec's issue tracker, and a lightweight decision.md pattern you can adopt today without waiting for -- or needing -- core support.

Why Archiving Alone Doesn't Record a Rejected Decision

Archiving a change you decided not to build technically works -- the folder moves out of your active list either way. The problem is what that archived folder fails to communicate once it is sitting next to dozens of shipped changes:

No status field. An archived change looks identical whether it shipped or was abandoned three messages into /opsx:propose. A teammate or an agent scanning openspec/changes/archive/ cannot tell the difference without opening every proposal folder and reading the artifacts inside it.
No signal to check first. Nothing in the default workflow instructs an agent to search the archive before drafting a new proposal. /opsx:propose drafts from your current request and the state of the codebase, full stop -- it does not cross-reference prior rejected changes unless you tell it to.
Delta specs you don't want synced. If a rejected change already has draft delta specs and you archive it the ordinary way, /opsx:archive will offer to sync those deltas into openspec/specs/ first. Accepting that offer teaches your canonical specs to describe behavior you decided not to build, which quietly corrupts the "what does the system currently do" record that every other proposal reads before it plans anything.

None of this is a bug. /opsx:archive is doing exactly what its documentation says it does: complete a change that shipped. The rejection case sits outside that documented scope on purpose, and OpenSpec's own team-workflow guide is explicit that most of what it recommends -- branch conventions, PR review order, when to archive -- is convention layered on top of the tool, not something OpenSpec enforces for you. Handling a rejection is one more convention you get to define yourself, and the CLI already gives you the flag you need to do it cleanly: pass --skip-specs when you archive a change you are not shipping, so openspec archive investigate-shared-persistence --skip-specs files the folder away without touching openspec/specs/ at all.

What OpenSpec's Maintainers Actually Decided About ADR Support

Before inventing a house convention, it is worth reading how this exact question played out in public, because the resolution is more specific -- and more interesting -- than "no." GitHub issue #557 opened in January 2026 with a request for first-class Architecture Decision Record support: durable records that persist independently of any single change's lifecycle, so a rejected or superseded decision stays visible to every future proposal. A contributor even opened a pull request implementing it.

What followed was seven months of genuinely substantive back-and-forth involving lead maintainer Tabish Bidiwale (@TabishB) and several deeply engaged community members, covering immutable versus mutable records, whether an ADR belongs to the research phase or the design phase, cross-change ownership when one decision cascades into a dozen later changes, and how ADRs relate to specs as the "authoritative" description of the system. Tabish Bidiwale's early framing set the direction the thread ultimately settled on: OpenSpec should stay lightweight by default and make specialized workflows like ADRs configurable through its schema system rather than baking them into core. A community member later summarized where the discussion landed:

ADR workflows are valuable, but OpenSpec does not currently have first-class/native ADR support... the direction discussed here is to keep the default workflow lightweight and make specialized workflows configurable.

Maintainer Clay Good (@clay-good) closed the issue on that basis in August 2026 and moved it into GitHub Discussion #1553 so the conversation could keep evolving without staying open as an unresolved bug. That is a reasonable call for a tool whose entire pitch is avoiding Spec Kit-style ceremony by default. It also means the fix lives one layer up, in one of two places:

A community schema. The spec-driven-with-adr schema, built by OpenSpec technical advisor Hari Krishnan (@harikrishnan83) and documented on intent-driven.dev, adds a fifth artifact to OpenSpec's default four-artifact pipeline. It exists because the default schema loses design.md's reasoning the moment a change is archived -- only the spec deltas get synced forward, so the "why" behind a decision disappears with the change unless something else preserves it.
A repository-level convention. A small, hand-rolled decision.md file plus a naming rule, which costs nothing to adopt and does not require installing a custom schema.

The rest of this guide covers option two in depth, since it is the lower-friction starting point for most teams -- and, as the section on the community schema below shows, it is compatible with switching to that heavier tooling later if your rejection log grows large enough to earn it.

The `decision.md` Convention for Recording a Rejected Change

Structure a rejected investigation the same way you would a shipped one, but stop before syncing any deltas, and add one file that states the outcome plainly:

openspec/
  changes/
    archive/
      2026-09-16-rejected-shared-persistence-layer/
        proposal.md
        decision.md

decision.md answers the same four questions a proper Architecture Decision Record does -- what was decided, why, what alternatives existed, and what would change the answer:

# Decision

Status: Rejected

## Decision

Do not replace the service-to-service HTTP boundary with a direct
package import between the two Go services.

## Reasons

- Increases compile-time coupling between independently deployed services.
- Makes the persistence layer an implicit, undocumented contract.
- The measured benefit (latency, code duplication) was smaller than
  the coupling cost in this codebase.

## Alternatives considered

- Shared internal Go module -- rejected for the same coupling reason.
- gRPC instead of HTTP -- deferred, not rejected; revisit if HTTP
  overhead becomes a measured bottleneck.

## Reconsider only if

- The two services are intentionally merged into one deployable, or
- Latency measurements show the HTTP hop is a proven bottleneck.

## Related

- Architecture rule: services communicate over HTTP, not shared packages.

The one hard rule that makes this whole convention work: do not run the sync step for a rejected change. If /opsx:propose already drafted delta specs before you decided against the change, use the flag the CLI already gives you for exactly this situation:

openspec archive investigate-shared-persistence --skip-specs

--skip-specs tells openspec archive to file the change away without touching openspec/specs/ at all, which is the safest default for anything you are archiving without shipping. Accepting the ordinary sync prompt instead would merge the rejected idea's delta specs into your canonical specs, and canonical openspec/specs/ should describe what the system currently does, not every idea that was drafted and turned down. If a change permanently produces no spec changes for a structural reason -- a pure investigation folder, say -- OpenSpec also supports declaring skip_specs: true in that change's .openspec.yaml so it archives cleanly without the flag every time.

Naming Rejected Changes So Humans and Agents Can Scan the Archive

A decision.md file only helps if someone opens the folder. Prefix the folder name with the outcome so both a human skimming ls openspec/changes/archive/ and an agent listing changes can tell status without opening a single file:

2026-09-16-rejected-shared-persistence-layer/
2026-09-20-abandoned-react-router-migration/
2026-10-01-superseded-old-auth-design/
2026-10-10-add-project-filtering/          # shipped, no prefix needed

This mirrors the status vocabulary already recommended for standalone decision records -- proposed, accepted, superseded, deprecated -- applied to OpenSpec's own archive instead of a separate docs/decisions/ folder. Keep the vocabulary small. Three or four consistent prefixes beat a free-text status line that every proposal spells slightly differently.

How to Make Your Agent Check the Archive Before Proposing Again

Naming and a decision.md file solve discoverability for a human skimming the folder. They do nothing on their own to make an agent search the archive before drafting a new proposal -- that has to be an explicit instruction, because /opsx:propose does not do it by default, and no amount of tidy file naming changes that on its own.

Two places to put that instruction, matching how OpenSpec already expects project-specific guidance to be injected:

In openspec/config.yaml, under the context: field that gets injected into every planning request (mind the 50KB cap covered in the OpenSpec quickstart):

context: |
  Before proposing a change, search openspec/changes/archive for folders
  prefixed "rejected-" or "abandoned-" that describe a materially similar
  idea. If one exists, summarize its decision.md and state what has
  changed before proposing the idea again. Do not re-litigate a rejected
  decision without new evidence.

In AGENTS.md or your project's own agent instructions, as a standing rule rather than a per-request context blob:

## Rejected OpenSpec changes

When a proposal is investigated and rejected:

1. Do not sync or apply its delta specs.
2. Add `decision.md` with Status, Decision, Reasons, Alternatives
   considered, and Reconsider only if.
3. Prefix the archived folder name: `rejected-<name>` or `abandoned-<name>`.
4. Before proposing a materially similar change, search
   `openspec/changes/archive/` and reference the prior decision.
5. Do not reopen a rejected decision unless its documented
   reconsideration conditions have actually changed.

flowchart TD
  A[New idea worth a change] --> B{Search openspec/changes/archive}
  B -->|Similar rejected decision found| C[Summarize prior decision.md]
  C --> D{Reconsideration conditions changed?}
  D -->|No| E[Do not re-propose. Reference the decision.]
  D -->|Yes| F["/opsx:propose with the changed context stated"]
  B -->|Nothing similar found| F

Neither instruction guarantees compliance -- an agent can still skip the search step, the same way it can skip reading any other context you inject. But it is the difference between "the information exists somewhere in the repo" and "the agent is told, every time, to go look for it," and only the second one actually reduces repeated investigations in practice.

A Worked Example: Rejecting a Proposal, Then Correctly Reconsidering It

Put the pieces together on a concrete case. Say a teammate asks an agent to look at replacing a service-to-service HTTP call with a direct Go package import, to shave off network latency.

Explore, then propose. /opsx:explore reads both services, and /opsx:propose replace-http-with-direct-import drafts a proposal, a design doc weighing the latency win against the coupling cost, and a draft delta spec.
Investigate and reject. After reviewing the design doc, the team decides the coupling cost -- two independently deployed services now sharing a compile-time dependency -- outweighs a latency win nobody has actually measured as a problem. Nothing gets built.
Archive without syncing. Rather than deleting the folder, run openspec archive replace-http-with-direct-import --skip-specs, then add decision.md to the archived folder with Status: Rejected, the reasons above, and a Reconsider only if clause naming the condition that would change the answer -- for example, "latency measurements show the HTTP hop is a proven bottleneck." Rename the folder with a rejected- prefix so it reads as openspec/changes/archive/2026-09-16-rejected-replace-http-with-direct-import/.
Months later, someone re-raises it. A different contributor, or the same agent in a fresh session, gets asked to "speed up the checkout-to-inventory call" and starts drafting a proposal that looks a lot like the same idea. Because openspec/config.yaml instructs the agent to search the archive first, it finds the rejected folder, reads decision.md, and reports back: "A materially similar change was proposed and rejected on 2026-09-16 for coupling reasons. The reconsideration condition was 'latency measurements show the HTTP hop is a proven bottleneck.' Do you have new measurements, or is this a different problem?"
The team supplies new evidence. If profiling now shows the HTTP hop genuinely dominates checkout latency, that is exactly the changed circumstance the original decision.md asked for. The agent proceeds with /opsx:propose, and the new proposal's decision.md -- once this one is also archived, accepted or rejected -- references the prior one under Related, so the archive reads as a continuous decision history rather than two unconnected folders that happen to describe the same idea.

That fifth step is the entire point of the convention. Without it, step 4 either does not happen at all -- the agent just re-investigates from zero -- or it happens by luck, because a human remembered the prior conversation. The decision.md file and the archive-search instruction turn "someone might remember" into something the workflow actually checks.

OpenSpec's Archive vs. a Dedicated ADR Log: Who Owns What

Once you are maintaining decision.md files inside the archive, it is worth being explicit about which artifact answers which question, so the convention does not quietly turn into duplicate documentation:

Artifact	Answers
`openspec/specs/`	What does the system currently do?
`openspec/changes/<name>/` (active)	What are we proposing to change, right now?
`openspec/changes/archive/<name>/`	What changed (or was rejected) in the past, and why?
`docs/adr/` (standalone, tool-neutral)	What durable architectural rule did we learn, independent of any single change?

For a decision narrow enough to belong to one investigation -- "we looked at sharing this persistence layer and said no" -- the decision.md-inside-the-archive convention above is enough. For a decision that should outlive and constrain many future changes -- "services communicate over HTTP, never shared packages" -- promote it to a standalone Architecture Decision Record in docs/adr/, and have the rejected change's decision.md reference it under Related. That split keeps OpenSpec's archive focused on individual investigations while the ADR log holds the small number of rules that should survive any single tool's lifecycle -- including a future migration off OpenSpec entirely.

When to Adopt the `spec-driven-with-adr` Schema Instead

The hand-rolled convention above costs nothing and fits inside fifteen minutes of setup, which makes it the right default. But it is worth understanding what the more structured alternative actually does before you decide you have outgrown a naming prefix.

spec-driven-with-adr inserts a fifth artifact, adr, between design and tasks in OpenSpec's pipeline. Rather than writing ADR content directly into the change folder, the adr step produces a short change-local adr.md review manifest and, when the change introduces a genuinely durable architectural commitment, a numbered record at the repository root -- /adr/0042-use-postgres-for-catalog.md, sibling to openspec/, not nested inside it. Every ADR the schema creates is immutable once accepted: the schema's own instructions call this out as an "iron rule" -- you never edit an accepted record's status, body, or date. To change a previous decision, you write a new ADR whose Supersedes: field names the old one, and future designs walk that supersession chain to know which decisions are still in force. That is a more rigorous version of exactly the "reconsider only if" idea in the decision.md convention above, enforced by the schema rather than left to a human remembering to write it down.

It is worth being precise about what this schema does and does not solve. It is built for decisions that get accepted and need to survive archiving -- Postgres over DynamoDB, JWT over session cookies -- not for proposals that were investigated and rejected without shipping anything. A rejected investigation still has nowhere obvious to live under this schema either; you would layer the same decision.md-and-naming convention from this guide on top of it, just referencing /adr/ records instead of a standalone docs/adr/ folder.

Reach for it once you notice any of these:

Your rejected-decision count is large enough that grepping openspec/changes/archive/ for prefixes stops being fast.
You want durable architectural decisions validated and cross-referenced against every new design automatically, rather than by convention and grep.
Multiple contributors keep inventing slightly different decision.md shapes, and you want a schema to enforce one immutable, numbered format instead.

Installing a custom schema is a bigger commitment than a naming convention -- it changes what /opsx:propose generates for every future change, not just rejected ones -- so treat it as a step up once the lightweight version is visibly straining, not a default first move.

Conclusion

OpenSpec's archive was designed around one outcome -- a change that shipped -- and its own maintainers have been explicit, after a seven-month public discussion, that first-class rejection or ADR support is not coming to the core workflow soon. That leaves the fix where OpenSpec already puts most of its team conventions: in your repository, not in the tool. A decision.md file, the --skip-specs flag on archive, a rejected-/abandoned- naming prefix, and an explicit instruction telling the agent to search the archive before proposing are enough to stop most repeated investigations. Reach for the spec-driven-with-adr schema only once that lightweight convention is genuinely straining under the number of decisions you are tracking -- and even then, keep the distinction clear: it manages decisions you accepted and want to survive archiving, not the ones you turned down.

Useful Links

OpenSpec Quickstart: Install, Workflow, and Common Pitfalls -- install, the explore-propose-apply-archive loop, and everyday pitfalls
Decision Records for AI-Driven Software Development -- the general ADR/PDR/DDR format, status lifecycle, and AI-reading instructions this convention borrows from
GitHub issue #557: architecture decision records support -- the full seven-month discussion of why ADRs are not core to OpenSpec
GitHub Discussion #1553 -- where that conversation continues after the issue was closed
spec-driven-with-adr schema -- the community schema that keeps ADRs alive outside the change lifecycle, by OpenSpec advisor Hari Krishnan
OpenSpec team-workflow docs -- how archiving, branches, and PR review are meant to fit together
GitHub Spec Kit vs Kiro vs Claude Code SDD Workflows -- how OpenSpec compares to heavier SDD tooling overall

OpenSpec Quickstart: Install, Workflow, and Common Pitfalls

Rost — Thu, 17 Sep 2026 12:20:00 +0000

OpenSpec is a free, open-source CLI from Fission AI that gets you and your coding agent to agree on a change in plain Markdown before any code gets written, without the phase-gated ceremony of heavier spec-driven frameworks.

Most teams who try Spec-Driven Development stall on the same trade-off: enough process to stop an agent from guessing, without so much scaffolding that a fifty-line bugfix needs a proposal document. OpenSpec's answer is to skip the "document the whole system first" instinct entirely and write specs only for what a change actually touches, using ADDED, MODIFIED, and REMOVED deltas instead of a full rewrite every time.

That change-centric design is also why OpenSpec keeps coming up next to GitHub Spec Kit, Kiro, and Superpowers in the comparison of SDD tool categories -- it is usually the pick when a team wants reviewable specs without an 800-line planning phase. This guide covers installing the CLI, the four-command workflow you actually use day to day, what a change looks like on disk, and the questions and complaints that show up most often on Reddit and in OpenSpec's own issue tracker.

What Is OpenSpec?

OpenSpec describes its own philosophy in four lines: fluid not rigid, iterative not waterfall, easy not complex, built for brownfield not just greenfield. In practice that means there are no locked phases -- you can edit a proposal, a spec, or a task list at any point in a change, rather than being forced through specify-then-plan-then-implement in strict order the way the tool-neutral SDD workflow describes it.

A change in OpenSpec produces up to four Markdown artifacts in its own folder:

Artifact	Purpose
`proposal.md`	Why the change exists and what it changes, in plain language
`specs/`	Delta requirements and scenarios -- the testable spec for this change
`design.md`	Optional technical approach, for changes that need one
`tasks.md`	The implementation checklist the agent works through

Once a change is implemented and archived, its delta specs merge into openspec/specs/, which becomes the durable, current-state description of your system -- the same "spec as source of truth" idea covered in What Is Spec-Driven Development?, just scoped one change at a time instead of written all at once.

Installing OpenSpec

OpenSpec is a Node.js CLI, so you need Node 20.19.0 or newer on your machine.

node --version

Install the CLI globally with npm, then verify it landed on your PATH:

npm install -g @fission-ai/openspec@latest
openspec --version

Deno, pnpm, yarn, bun, and nix are also supported install paths if that fits your setup better than npm. Once installed, initialize it inside a project:

cd your-project
openspec init

openspec init asks which AI tools you use and writes the matching skill and command files -- OpenSpec supports 30+ assistants, including Claude Code, Cursor, GitHub Copilot, Gemini CLI, Codex, Kiro, and OpenCode. For CI or scripted setup, skip the picker entirely:

openspec init --tools claude,cursor   # set up specific tools
openspec init --tools all             # every supported tool
openspec init --tools none            # openspec/ structure only, no tool files

Restart your IDE afterward so it picks up the newly written skills and commands. If you would rather have your assistant do the whole install for you, OpenSpec ships a setup prompt you can paste into Claude Code or another agent, which runs the install, executes openspec init, and reports back what it configured.

The Core Workflow: Explore, Propose, Apply, Archive

This is the one thing that trips up almost everyone on their first day: openspec commands run in your terminal, but /opsx: commands run in your AI assistant's chat window. There is no separate "interactive mode" to enter -- typing the slash command in chat is how you start.

flowchart LR
  A["/opsx:explore (optional)"] --> B["/opsx:propose change-name"]
  B --> C["/opsx:apply"]
  C --> D["/opsx:archive"]
  D -->|specs merged| E["openspec/specs/"]

/opsx:explore is a no-stakes thinking partner. It reads the relevant part of your codebase, lays out options, and shapes a plan before anything is written to disk -- worth forming as a habit specifically because it stops an eager agent from confidently building the wrong thing.
/opsx:propose <name> creates openspec/changes/<name>/ and drafts the proposal, delta specs, optional design, and task list in one step. You review the plan here, before implementation starts.
/opsx:apply works through the task list, checking items off as it goes. Because progress lives in files rather than only in chat history, you can clear your context window or start a fresh session and pick up exactly where /opsx:apply left off.
/opsx:archive files the completed change to openspec/changes/archive/YYYY-MM-DD-<name>/ and merges its delta specs into the canonical openspec/specs/ tree.

The default core profile installs exactly those four commands plus update and sync. An expanded profile adds new, continue, ff, verify, bulk-archive, and onboard for teams who want to create one artifact at a time instead of all at once -- switch to it with openspec config profile followed by openspec update.

Each tool spells the same command differently depending on how it loads custom instructions: /opsx:propose in Claude Code, /opsx-propose in Cursor and GitHub Copilot, @opsx-propose in Amazon Q, or $openspec-propose in Codex. openspec init prints the exact form for the tools you picked, so the fastest fix for "nothing happened when I typed the command" is usually to re-read that printed hint rather than guess.

What a Change Looks Like on Disk

A change folder under openspec/changes/add-dark-mode/ typically contains a proposal, a delta spec, and a task list like this:

## ADDED Requirements

### Requirement: Theme selection
The app SHALL let users switch between light and dark themes,
defaulting to the system preference.

#### Scenario: User toggles dark mode
- **WHEN** the user clicks the theme toggle
- **THEN** the app switches to dark mode and persists the choice

That ADDED/MODIFIED/REMOVED delta format is the mechanism that lets OpenSpec avoid rewriting an entire spec file for a one-field change. It is also why OpenSpec is explicitly brownfield-first rather than greenfield-first: you never document your whole application before getting value, you just document the slice each real change touches, and openspec/specs/ fills in naturally over months of normal work.

Useful CLI commands for checking on that state without leaving the terminal:

openspec list                 # active changes
openspec show add-dark-mode   # view a change's artifacts
openspec validate --all       # check spec formatting across the project
openspec view                 # interactive dashboard

Commit the whole openspec/ folder to git. The active changes and the archive are meant to become a durable, versioned record of what your system does and why it changed -- not a scratch pad you delete after merging.

Adopting OpenSpec on an Existing Codebase

The most common worry from teams evaluating OpenSpec on a real project is some version of "my app is 80,000 lines old, do I have to spec all of it first?" You do not. OpenSpec's own guidance is blunt about this: pick something small and real that you were already going to build this week, run /opsx:explore on the area you are about to touch so the agent maps how things actually work first, then /opsx:propose a change scoped to just that slice.

If you already have PRDs, SRS documents, or design docs sitting in Notion or Confluence, treat them as source material for exploration rather than something to bulk-convert into specs. Paste the relevant section into an /opsx:explore session and let the agent shape a focused delta from it; a one-time mechanical conversion of a forty-page PRD tends to produce a spec nobody trusts six months later. For teams that want a guided, narrated first run instead of jumping straight into a real change, the expanded /opsx:onboard command scans your codebase for a small, safe improvement and walks through the full loop on it.

Common Questions and Problems

These are the issues that show up repeatedly across OpenSpec's Discord, GitHub issues, and Reddit threads in subreddits like r/cursor, r/RooCode, and r/opencodeCLI.

"I typed the slash command and nothing happened." Almost always one of: you typed it in the terminal instead of your assistant's chat, your IDE has not restarted since openspec init ran, or the CLI version is old enough that openspec update reports everything current without ever writing the newer workflow files. Run openspec update, restart the IDE, and confirm the skill folders exist (.claude/skills/openspec-* for Claude Code, or your tool's equivalent from the supported-tools list).

"The AI generates way more spec than I need." This is the most-cited complaint in longer write-ups: an agent can turn a thirty-minute feature into an 800-line spec. OpenSpec caps the context: field injected into every request at 50KB specifically to force discipline, but the delta specs themselves have no hard limit, so trimming generated specs down to what is actually load-bearing is a habit you have to maintain yourself, not something the tool enforces for you.

"Two changes touched the same requirement and one silently dropped the other's scenario." This is a real, documented edge case: archiving applies a MODIFIED delta as a whole-block replace keyed by requirement name, so if two in-flight changes both modify the same requirement, archiving the second one used to overwrite the first's scenarios without warning. Current versions add a drift check that aborts the archive and tells you to refresh the change's spec first -- but it is still worth knowing the failure mode exists if you run several changes on the same area in parallel.

"Which AI model should I actually use with it?" OpenSpec's own docs recommend high-reasoning models for both planning and implementation -- Opus-class and Codex-class models are called out specifically -- and clearing your context window before implementation, since a clean context produces measurably better results than a long, accumulated session.

"How is this different from Spec Kit, Kiro, Superpowers, or BMAD?" This is the single most frequent Reddit question, and the honest answer is "process weight." OpenSpec's own README frames the comparison directly: Spec Kit is thorough but heavier, with more Markdown and rigid phase gates; Kiro is powerful but locks you into AWS's IDE and Claude models; OpenSpec trades some of that upfront structure for the ability to iterate freely and work with whatever assistant you already have open. For the full breakdown against Spec Kit, Kiro, Claude Code skills, BMAD-METHOD, and Superpowers, see the dedicated SDD tool comparison.

"Does the AI actually follow the spec it just wrote?" Not always, and this is a documented problem across SDD tools generally, not unique to OpenSpec -- a large context window does not mean the agent attends equally to every part of it. The /opsx:verify command exists specifically to catch generated code that contradicts its own spec, and it is worth running on anything non-trivial rather than trusting the implementation blindly.

"Do I need this for a one-line fix?" No. OpenSpec's own FAQ says as much: use it where agreement matters, which is most non-trivial, multi-file work, and skip it for a typo fix or a throwaway prototype you will delete in a week.

When OpenSpec Fits and When It Doesn't

Good fit:

Brownfield codebases where you want reviewable specs without documenting the entire system upfront.
Solo developers and small teams who want lighter ceremony than Spec Kit while still getting a written plan before code.
Work that spans several files, a schema change, or anything a junior engineer would reasonably want a short design doc for.
Teams already committed to reviewing plans in pull requests -- delta specs diff cleanly since they only describe what changed.

Weaker fit:

One-line bug fixes and throwaway prototypes, where the proposal-review step costs more than it saves.
Teams that need the heavier, more prescriptive structure of Spec Kit or an AWS-native, IDE-integrated experience like Kiro -- see the decision framework in the tool comparison for where each tool wins.
Cross-repo features today, unless you are willing to try OpenSpec's beta stores feature, which moves planning into its own shared repository so multiple codebases and agents can read the same plan.
Anyone still deciding whether a given feature deserves a spec at all -- read Spec-Driven Development vs Vibe Coding first, since OpenSpec only helps once you have already decided structure is worth the overhead.

Conclusion

OpenSpec's bet is that most Spec-Driven Development pain comes from ceremony, not from the underlying idea of agreeing on a plan before code exists. Deltas instead of full rewrites, no locked phases, and a brownfield-first workflow make it noticeably lighter than Spec Kit or Kiro to adopt on a codebase you did not build from scratch. The trade-offs are real too -- spec bloat is a genuine risk without discipline, the conflict handling around simultaneous changes to one requirement is still maturing, and the ecosystem is younger than GitHub's own tooling. Install it on one real project, run a small change through explore-propose-apply-archive end to end, and decide from there whether the lighter ceremony earns its keep against your actual workload.

Useful Links

OpenSpec repository -- source, docs, and the CLI package
OpenSpec documentation home -- getting started, concepts, FAQ, and troubleshooting
GitHub Spec Kit vs Kiro vs Claude Code SDD Workflows -- full tool comparison and decision framework, including OpenSpec
Superpowers Quickstart: Install, Workflow, and Tryout -- the enforced-skills alternative to OpenSpec's lighter-touch workflow
Spec-Driven Development Workflow From Requirements to Code -- the tool-neutral five-phase process OpenSpec implements more fluidly
What Is Spec-Driven Development? The Spec as Source of Truth -- core SDD concepts and terminology
Spec-Driven Development vs Vibe Coding: Waterfall? -- deciding whether a feature deserves a spec at all

How to Migrate from OpenClaw to Hermes Agent Safely

Rost — Tue, 15 Sep 2026 11:32:34 +0000

Migrating an AI assistant is not the same as copying an application config. The hard part is preserving identity, memory, tool behavior, scheduled work, and messaging access without two gateways acting as the same bot.

Hermes Agent now includes hermes claw migrate, a real migration planner rather than a cosmetic import command. It can map more than 30 categories from OpenClaw, detect conflicts, create a Hermes restore point, and archive incompatible state for manual review. That makes the move practical, but it does not make the move automatic.

The approach below is a staged cutover: back up OpenClaw, dry-run the full migration, import without secrets, validate Hermes from the terminal, and transfer messaging credentials only after the new agent behaves correctly. Do not begin with --overwrite --migrate-secrets --yes; those flags are useful for automation after a rehearsed migration, not for discovering what your assistant actually depends on.

The OpenClaw to Hermes migration runbook

Phase	Command or action	Exit condition
Inventory	Record versions, workspaces, plugins, channels, cron jobs, and providers	Every non-file dependency has an owner
Backup	`openclaw backup create --verify`	A verified archive exists outside OpenClaw state
Preview	`hermes claw migrate --dry-run --preset full`	No unexplained conflicts or skipped critical data
Import	Run the full preset without secrets	Hermes config, persona, memory, skills, and MCP entries exist
Local test	Run Hermes in the terminal	Model, tools, memory, approvals, and workspace pass tests
Channel cutover	Stop OpenClaw, migrate or set secrets, start Hermes gateway	Only Hermes owns each bot token or account
Soak	Keep OpenClaw stopped but recoverable	Scheduled and inbound work behaves correctly
Cleanup	Archive old OpenClaw state only after acceptance	Rollback window is closed intentionally

flowchart TD
    A[Inventory the live system] --> B[Verified OpenClaw backup]
    B --> C["Dry-run: hermes claw migrate --dry-run --preset full"]
    C --> D[Import without secrets]
    D --> E["Terminal validation in a new session"]
    E --> F["Controlled channel cutover"]
    F --> G[Soak with OpenClaw stopped]
    G --> H["Cleanup after acceptance"]

The command is short because the judgment has moved into the preview and verification stages. Treat the generated migration report as a change plan, not as reassuring console output.

What `hermes claw migrate` actually reads

The migrator reads ~/.openclaw/ by default. It also detects the older ~/.clawdbot/ and ~/.moltbot/ directories, along with legacy config filenames, so an older installation does not need to be renamed before migration.

OpenClaw has used several workspace layouts. Hermes checks workspace/, workspace.default/, and workspace-main/, and it recognizes per-agent directories such as workspace-<agentId>. If you use custom agent roots or multiple profiles, verify every resolved path in the preview rather than assuming the default workspace represents the whole system.

The destination is normally ~/.hermes/. A pre-existing Hermes installation is not treated as an empty bucket: the planner reports conflicts and refuses to apply by default when it cannot preserve both sides safely.

What migrates and what does not

The useful distinction is not "supported" versus "unsupported." Some OpenClaw state maps directly, some must be transformed, and some can only be archived because the two agents use different execution models.

Direct or transformed migration

OpenClaw source	Hermes destination	Migration behavior
`workspace/SOUL.md`	`~/.hermes/SOUL.md`	Direct persona copy
`workspace/MEMORY.md`	`~/.hermes/memories/MEMORY.md`	Parsed, merged, and deduplicated
`workspace/USER.md`	`~/.hermes/memories/USER.md`	Parsed, merged, and deduplicated
`workspace/memory/*.md`	Main Hermes memory	Daily files are merged into entries
`workspace/AGENTS.md`	Chosen project directory	Requires `--workspace-target`
OpenClaw skill directories	`~/.hermes/skills/openclaw-imports/`	Copied with an explicit conflict policy
`agents.defaults.model`	Hermes model configuration	Primary and fallback forms are interpreted
`models.providers.*`	Hermes provider configuration	Base URL and API type are mapped
`mcp.servers.*`	`mcp_servers.*`	Stdio and HTTP/SSE definitions are mapped
Channel tokens and allowlists	Hermes `.env`	Only with `--migrate-secrets`
Session reset policy	`session_reset`	Daily and idle modes are translated
Exec approvals	Hermes approvals and command allowlist	Modes and patterns are transformed
Browser, TTS, sandbox, and timeout settings	Related Hermes config	Supported fields are mapped

Memory is not copied as one opaque document. The migrator parses OpenClaw memory and user-profile files, merges them with existing Hermes entries, and deduplicates them. This is safer than replacing an established Hermes memory file, but it also means you should compare meaning and structure, not only file sizes.

Archived for manual reconstruction

OpenClaw feature	Why it is not directly portable	Hermes direction
Cron jobs	Schedulers and delivery models differ	Recreate with `hermes cron create`
Plugins	Plugin APIs are product-specific	Replace with a Hermes plugin, skill, MCP server, or built-in tool
Hooks and webhooks	Event and permission contracts differ	Recreate with Hermes webhooks or gateway hooks
Advanced memory backend	Databases and recall semantics differ	Configure a Hermes memory provider separately
Skills registry settings	Registry implementation differs	Configure with `hermes skills config`
Multi-agent list and bindings	Routing and profile models differ	Rebuild with Hermes profiles and gateway configuration
`IDENTITY.md`	Hermes uses a different identity split	Merge relevant identity into `SOUL.md`
`HEARTBEAT.md`	No direct file-driven heartbeat equivalent	Express periodic work as cron jobs
`TOOLS.md`	Hermes supplies its own tool instructions	Move only genuine workflow rules into a skill or context file
`BOOTSTRAP.md`	Bootstrap semantics differ	Use context files, setup, or a skill

These items are saved below ~/.hermes/migration/openclaw/<timestamp>/archive/. A successful migration with a non-empty archive is therefore not finished; the archive is the remaining work queue.

Step 1: Inventory the live OpenClaw system

Before installing anything, write down which behaviors are actually in use. Config files alone may not reveal a plugin's external database, a manually supervised gateway, a custom agent directory, a local model process, or the account that owns a webhook endpoint.

At minimum, record:

OpenClaw and Hermes versions.
The active OpenClaw state directory and config path.
All agent and workspace directories.
Model providers, fallback models, and local endpoints.
Installed and enabled plugins, including their persistent data.
Skills from workspace, managed, personal, and project directories.
MCP servers, environment variables, working directories, and credentials.
Telegram, Discord, Slack, WhatsApp, Signal, Matrix, and Mattermost accounts.
Cron jobs, hooks, webhooks, heartbeat behavior, and external supervisors.
Approval rules, command allowlists, sandbox backend, and browser access.

This inventory becomes the acceptance checklist later. Without it, a migrated assistant can look healthy because it answers messages while silently missing the weekly backup, a memory provider, or a restrictive approval rule.

Step 2: Create a verified OpenClaw backup

OpenClaw 2.0 includes a backup command that understands its current SQLite state, configured agent roots, credentials, plugins, and workspaces. Use it instead of copying live database files and hoping their WAL sidecars were captured consistently.

mkdir -p ~/Backups
openclaw gateway stop
openclaw backup create --output ~/Backups --verify

Keep the resulting archive outside ~/.openclaw/. The --verify option validates the archive immediately, including path safety and supported SQLite integrity checks. OpenClaw-owned databases are captured through SQLite's online backup API, owner-verified, and compacted, rather than copied as raw files. If your workspaces are large, you can use --no-include-workspace, but then back up those repositories and non-Git files separately; agent directories stay included either way.

The pre-2.0 transcript trap

OpenClaw 2.0 moved sessions and transcripts out of sessions.json and JSONL files into SQLite, by default at ~/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite. That matters here for one non-obvious reason: the portable backup create archive omits legacy JSONL transcripts and logs even when they are no longer being written.

So if your OpenClaw install predates 2.0 and you care about the old conversation history, a verified archive alone does not protect it. Stop the gateway and take a filesystem, volume, or VM snapshot before you migrate, or use OpenClaw's per-database snapshot commands for the databases you want a compact, independently verifiable copy of:

openclaw backup sqlite create --global --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite create --agent main --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite list --repository ~/Backups/openclaw-sqlite
openclaw backup sqlite verify ~/Backups/openclaw-sqlite/<snapshot-id>

Treat those snapshot repositories with the same permissions and retention policy as live state — they can contain auth profiles, session state, and plugin data. For a continuously replicated setup rather than periodic archives, OpenClaw documents Litestream against the same databases; that is a better answer than hand-rolled cp jobs if the migration is going to take days.

Also create a Hermes backup if Hermes already contains useful state:

hermes backup

The migration normally creates its own pre-migration Hermes archive under ~/.hermes/backups/. Do not pass --no-backup during the first cutover; saving a few seconds is not worth removing the simplest rollback path.

Step 3: Install and test an empty Hermes Agent

Install Hermes, select a model, and prove that the basic terminal agent works before importing OpenClaw state. This separates installation and provider failures from migration failures. The Hermes AI Assistant guide covers provider selection and gateway configuration in depth; for the migration you only need a working terminal baseline.

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
source ~/.bashrc
hermes setup
hermes status
hermes doctor

If you already installed Hermes, update it before relying on current migration behavior:

hermes update
hermes --version

That version check is not a formality. The claw migrate safety posture changed materially during 2026: current builds refuse to apply a conflicted plan, write a pre-migration restore point by default, redact secrets in the reports they save to disk, and require --migrate-secrets explicitly even under --preset full. Older builds did none of those things — notably, --preset full used to pull in API keys silently, and a conflicted plan would report "migrated 0" after you had already confirmed. If you are following an older tutorial, the flags may look identical while the behavior differs in exactly the places that matter.

Do not configure the old bot tokens yet. Terminal-only validation allows OpenClaw to remain live while you prepare Hermes, and it avoids two gateway processes competing for the same messaging identity.

Step 4: Run the dry-run before choosing flags

Start with the full preset because it reveals the largest possible mapping surface, but keep secrets excluded:

hermes claw migrate --dry-run --preset full

The migration always presents a preview before applying, even without --dry-run. The explicit flag is still valuable because it makes your intent unambiguous and gives you time to inspect source paths, destinations, transforms, conflicts, skipped items, archives, and secret warnings without an impatient confirmation prompt. The complete flag set for claw migrate and its neighbours is summarised in the Hermes Agent CLI cheat sheet.

Use a custom source when OpenClaw state is not in the default location:

hermes claw migrate \
  --dry-run \
  --preset full \
  --source /srv/openclaw-state

If AGENTS.md should apply to a particular repository, say so explicitly:

hermes claw migrate \
  --dry-run \
  --preset full \
  --workspace-target /srv/projects/my-project

Without --workspace-target, workspace instructions are not placed into an arbitrary current directory. That is the correct behavior: an instruction file belongs to a scope, and guessing its scope can change every Hermes session launched below the wrong directory.

Full or user-data preset?

The full preset includes compatible infrastructure and behavior settings. The user-data preset focuses on persona, memories, skills, and related user content while excluding infrastructure configuration.

Use user-data when Hermes already has a carefully built provider, gateway, security, or sandbox configuration. Use full when Hermes is new and OpenClaw is the authoritative setup, but still inspect every transformed behavior setting. Neither preset imports secrets unless --migrate-secrets is added.

Step 5: Resolve conflicts without destroying provenance

The default conflict behavior is conservative: the migration refuses to apply a plan with unresolved file conflicts unless --overwrite is set. That is preferable to an apparently successful cutover that overwrites a newer Hermes persona or skill — and preferable to the older behavior, where confirming a conflicted plan produced a "migrated 0" result that looked like a no-op but was really a silent skip.

Skill conflicts are handled separately, and the default there is skip, which quietly keeps the existing Hermes version and discards the incoming one. For a first migration I recommend rename instead:

hermes claw migrate \
  --preset full \
  --workspace-target /srv/projects/my-project \
  --skill-conflict rename

Imported skills are placed under ~/.hermes/skills/openclaw-imports/. With rename, a name collision produces an imported sibling rather than hiding either version. Review the two implementations, test the chosen one, and remove the redundant copy later.

Use --overwrite only after reviewing the preview or when rebuilding a disposable Hermes profile. It applies more broadly than skill conflict handling and can replace existing Hermes files. The presence of a backup makes overwriting recoverable, not desirable.

Step 6: Migrate configuration and user data without secrets

Apply the reviewed plan and leave credentials for the cutover stage:

hermes claw migrate \
  --preset full \
  --workspace-target /srv/projects/my-project \
  --skill-conflict rename

After completion, save the printed counts for migrated, skipped, conflicting, and archived items. Open the timestamped migration directory and read its summary before starting a new Hermes session. Current builds redact detected secret values in the report.json and summary.md they write, so those files are safe to keep alongside your change notes — but confirm that on your version rather than assuming it, because earlier builds wrote raw API keys into the same reports.

New sessions matter. Imported skills and memory entries are loaded when a session begins, so testing inside a session that predates the migration can produce a false "skill not found" or stale-memory result.

Step 7: Validate behavior before channel cutover

Run the post-migration checks from the terminal:

hermes status
hermes doctor
hermes config show
hermes gateway status

If memory recall looks incomplete, rebuild the index before concluding the import failed:

hermes memory reindex

Then start a new Hermes conversation and test observable behaviors, not just file presence. Ask for a known user preference from memory, invoke one imported skill, call an MCP tool, run a harmless terminal command that should be allowed, and try one that should require approval.

A useful acceptance matrix looks like this:

Area	Test	Failure usually means
Persona	Ask a question where tone and boundaries are obvious	`SOUL.md` was not found, was overwritten, or needs identity content merged
User memory	Ask for a known stable preference	Memory entries were not imported, deduplicated unexpectedly, not reindexed, or not loaded in a new session
Skill	Invoke a distinctive imported workflow	Name conflict, invalid metadata, missing dependency, or stale session
Provider	Run a normal and long response	Wrong model mapping, missing credential, or incompatible API type
MCP	Call one read-only tool from each server	Missing environment, wrong `cwd`, transport mismatch, or tool filter issue
Terminal	Test allowed and approval-required commands	Approval mode or allowlist mapping changed policy
Browser	Open a harmless test page	CDP URL, browser backend, or sandbox access differs
Compression	Run a long disposable session	Summary model or compaction behavior was not mapped as intended
Session reset	Inspect config and test on a disposable profile	Daily/idle interpretation differs from OpenClaw rules

The migration maps timeoutSeconds to an estimated maximum-turn value, translates reasoning levels, and converts approval modes. Those are semantic mappings rather than byte-for-byte copies. Check that the resulting behavior matches your intent, especially for long autonomous tasks and command execution.

Step 8: Handle secrets as a separate security change

--migrate-secrets can collect allowlisted keys from OpenClaw config values, ~/.openclaw/.env, config environment objects, and per-agent auth profiles (~/.openclaw/agents/<agent>/agent/auth-profiles.json). It understands plain strings, environment templates, and environment-backed SecretRef objects.

It intentionally does not copy arbitrary secret names. File-backed and command-backed SecretRefs cannot be resolved automatically, and values outside the supported allowlist remain for manual setup. Treat every warning here as a control working as designed, not as a reason to paste the entire OpenClaw environment into Hermes.

For a first migration, I prefer to configure provider credentials through Hermes after the data import. If you do use automated secret migration, preview it and run it only when you are ready to transfer channel ownership:

hermes claw migrate \
  --dry-run \
  --preset full \
  --migrate-secrets

Then verify presence without printing values:

hermes status
hermes auth status

Rotate credentials if they were exposed in shell history, pasted into migration notes, or stored with weaker permissions than intended. Migration preserves access; it does not prove that the old secret-handling practice was safe.

Step 9: Perform a controlled messaging cutover

There is no honest zero-downtime handoff when two processes would poll, subscribe, or respond as the same bot account. The safe pattern is prepare in parallel, stop OpenClaw, start Hermes, test each platform, and keep the rollback commands ready.

First stop the OpenClaw gateway and confirm it is stopped:

openclaw gateway stop
openclaw gateway status

Now migrate or manually set the messaging secrets, configure the Hermes gateway, and start it:

hermes gateway setup
hermes gateway install
hermes gateway start
hermes gateway status

Send a direct message from an allowed user on each platform. Test inbound text, a reply, an attachment if used, a slash command, a long-running task, interruption, and an outbound scheduled or manual send. A green service status proves that a process is running; it does not prove that allowlists, thread routing, delivery, and formatting survived the move.

WhatsApp requires re-pairing because the migration does not transfer the Baileys session as a reusable token. Run hermes whatsapp and complete the QR flow. Other channels may reuse tokens, but account layouts and multi-account bindings still deserve explicit testing.

Skills, plugins, and MCP servers are not interchangeable

OpenClaw skills from four locations can be imported, but an imported directory is only useful if its assumptions remain true. Check command names, filesystem paths, environment variables, platform-specific tools, and references to OpenClaw-only APIs. The OpenClaw skills guide explains the source formats; the Hermes skill authoring guide covers the destination behavior.

OpenClaw plugins do not become Hermes plugins. Reconstruct the capability at the narrowest suitable layer:

Use a Hermes skill for procedure, tool selection, and reusable instructions.
Use an MCP server for live data or an external service boundary.
Use a built-in Hermes tool when it already provides the capability.
Use a Hermes plugin only when code must participate in the agent runtime itself.

This is a good moment to remove architectural sediment. A plugin installed to compensate for an old OpenClaw limitation may have no reason to survive in Hermes, while a plugin holding a durable database needs a deliberate export or replacement plan.

MCP definitions migrate more directly, including commands, arguments, environments, working directories, URLs, and include/exclude tool filters. Still test every server separately: a correct YAML mapping cannot install a missing executable, renew OAuth, or make a path from the old host exist on the new one.

Memory needs a quality check, not a line-count check

Hermes imports MEMORY.md, USER.md, and daily memory files into its memory structure. That preserves useful facts, but OpenClaw memory plugins, long-context databases, embedding indexes, and recall policies are archived rather than translated into an equivalent cognitive system.

Review imported memory in three passes:

Identity and stable preferences: preserve concise facts that should influence many sessions.
Operational knowledge: move repeatable procedures into skills or project context instead of global memory.
Historical residue: archive completed incidents, stale plans, and self-referential agent commentary rather than injecting it forever.

Do not import every transcript as durable memory. More remembered text can make an agent less coherent by repeatedly retrieving obsolete constraints and its own earlier guesses. The Hermes memory system guide explains where the imported entries will live, and the agent memory provider comparison is the better place to choose a new long-term backend.

Recreate cron jobs, heartbeats, hooks, and multi-agent routing

Cron jobs are archived because scheduled execution is not just a cron expression. A job also has a prompt or command, working directory, model, timeout, delivery destination, permissions, retry behavior, and expectations about session state.

For every archived OpenClaw job, write down those fields and recreate it with Hermes:

hermes cron create
hermes cron list

Run each job once manually before enabling its schedule. Verify both the work and the delivery path, especially when the old job posted to a Telegram chat, Slack channel, or Discord thread.

Translate HEARTBEAT.md into explicit scheduled jobs only when periodic execution is truly required. A vague heartbeat that asks the agent to inspect everything every few minutes is expensive and difficult to verify; separate named jobs with observable outcomes are easier to operate.

Multi-agent definitions and channel bindings also require manual design. Hermes profiles provide isolated state and gateways, but they are not a syntactic rewrite of OpenClaw's agent list. Map each agent by responsibility, workspace, credentials, channel, and security boundary rather than reproducing names first; the profile-first reasoning behind that mapping is worked through in the Hermes production setup guide.

Troubleshooting the failures that matter

"OpenClaw directory not found"

The command searches the current OpenClaw, Clawdbot, and Moltbot default directories. If your state lives elsewhere, point to the directory that contains the OpenClaw config and related state:

hermes claw migrate --dry-run --source /path/to/openclaw

Do not point --source at only the workspace unless that is genuinely the complete source tree. The preview should show config, workspace, and recognized categories.

The migration refuses because of conflicts

This is the safe default, not a crash. Back up Hermes, identify which side is authoritative for each conflict, use --skill-conflict rename for skills, and reserve --overwrite for a reviewed plan.

If the existing Hermes configuration is valuable, consider the user-data preset. It imports the assistant's user-owned content without trying to replace established infrastructure.

Imported skills do not appear

Start a new session and inspect the imported directory below ~/.hermes/skills/openclaw-imports/. Use /skills inside Hermes to confirm discovery. If the skill exists but cannot run, inspect its dependency and tool assumptions rather than repeating the migration.

Provider keys were not found

The key may be stored in an OpenClaw environment file, config environment object, auth profile, file-backed SecretRef, command-backed SecretRef, or unsupported variable name. The migrator resolves the supported forms and warns about the rest. Add unresolved values through Hermes configuration or authentication commands instead of converting secure references into plaintext merely to satisfy the importer.

The bot is running but messages are missing or duplicated

Confirm that the OpenClaw gateway is stopped and that only one Hermes profile owns the token. Then inspect hermes gateway status and gateway logs, followed by channel allowlists and account selection. Duplicate consumers and incorrect allowlists are more common than a broken language model.

The personality is present but recall is poor

SOUL.md and memory are different layers. Confirm that the persona copied to ~/.hermes/SOUL.md, memory entries reached ~/.hermes/memories/, and the test uses a new session. Run hermes memory reindex before deeper debugging. If OpenClaw depended on an external memory plugin, configure a Hermes memory provider rather than expecting Markdown import to recreate its retrieval behavior.

Roll back Hermes

Stop the Hermes gateway before restoring the pre-migration Hermes backup:

hermes gateway stop
hermes import ~/.hermes/backups/pre-migration-<timestamp>.zip

hermes import overwrites files in Hermes home with the archive contents, so inspect the exact filename and understand that post-migration Hermes sessions may be replaced. Then keep Hermes stopped, restart OpenClaw, and verify its gateway and channel health.

Manual migration when the command cannot model your setup

A manual fallback is slower but sometimes clearer for heavily customized installations. Build a clean Hermes profile and migrate by responsibility:

Copy or rewrite persona content into ~/.hermes/SOUL.md.
Curate stable user facts into Hermes MEMORY.md and USER.md rather than copying all history.
Place project instructions in the correct repository-level AGENTS.md.
Copy compatible skills into a named import directory and test them individually.
Translate provider and MCP definitions into ~/.hermes/config.yaml without printing secrets.
Configure credentials through Hermes auth or secret management.
Recreate approvals, sandboxing, browser access, cron jobs, webhooks, and channels.
Replace each OpenClaw plugin with an explicit Hermes capability or retire it.

The manual route is especially appropriate when the source contains several OpenClaw agents with different workspaces, memory plugins, and channel bindings. An automatic union can preserve files while erasing the isolation that made the setup safe.

Do not clean up OpenClaw immediately

After Hermes has passed local and messaging tests, keep OpenClaw installed but stopped for a soak period. Preserve the verified OpenClaw backup, migration archive, pre-migration Hermes backup, and a copy of the acceptance checklist.

Hermes documents hermes claw cleanup for renaming leftover OpenClaw directories to .pre-migration/, and hermes claw cleanup --dry-run to preview what would be archived. Use it only after the OpenClaw gateway is stopped, the current Hermes version includes process guards, and you have decided not to roll back. Older 2026 builds had a reported cleanup path that could move state while an OpenClaw gateway was still running; current code marks the guard as implemented, but a verified backup and stopped source service remain the sensible boundary.

Cleanup is not required to prove that Hermes works. It exists to reduce future state confusion, so postponing it during a rollback window is good operations, not untidiness.

When to stay on OpenClaw 2.0

OpenClaw 2.0 is not an abandoned baseline. The v2026.8.1 release landed over 16,000 pull requests from more than 900 contributors — roughly half of the project's total merge history — and substantially changed onboarding, the web Control UI, session storage, backups, channels, memory, plugins, automations, browser and computer use, security, and service reliability. If those platform features are central to your deployment, migration may remove more working capability than it simplifies.

Stay on OpenClaw when you depend on:

Its rebuilt Control UI, with docked file editor, git-backed Changes panel, browser panel, and in-conversation approvals.
Session presets, transcript search, groups, status views, and batch actions.
A product-specific plugin with no Hermes equivalent.
Complex multi-user, mobile, device, or channel routing already working in production.
OpenClaw-specific browser, computer-use, or Gateway administration.
A memory or session database that cannot be exported with acceptable loss.
Operational controls your team already knows and monitors.

Move to Hermes when its simpler terminal-first workflow, profiles, learning-oriented skills, memory model, scheduled tasks, provider flexibility, or delegation model better matches what you actually operate. The OpenClaw and Hermes comparison discusses that decision with current numbers; this page is about executing the cutover once the decision is made.

Final migration checklist

[ ] OpenClaw version and resolved paths recorded.
[ ] Verified OpenClaw backup stored outside live state.
[ ] Pre-2.0 JSONL transcripts snapshotted separately if they matter.
[ ] Existing Hermes backup created.
[ ] Hermes version checked against current claw migrate safety behavior.
[ ] Full dry-run reviewed.
[ ] Every conflict assigned a resolution.
[ ] Archive contents added to the manual work list.
[ ] Persona, user memory, and skills tested in a new session.
[ ] Provider, fallback model, MCP, browser, and terminal tested.
[ ] Approval and sandbox behavior tested, including a denied action.
[ ] Cron jobs, plugins, hooks, memory backend, and multi-agent bindings rebuilt or retired.
[ ] OpenClaw gateway stopped before channel credentials moved.
[ ] Every messaging channel tested from an allowed account.
[ ] WhatsApp re-paired if used.
[ ] Rollback commands and archive names recorded.
[ ] OpenClaw cleanup deferred until the soak period ends.

Final verdict

hermes claw migrate is good enough to make an OpenClaw-to-Hermes move routine, but only if "routine" means planned and reversible. Its strongest feature is not the number of files it copies; it is the preview that tells you which parts of the old assistant have a real Hermes equivalent and which parts still require engineering judgment.

Use the full preset to discover the scope, keep secrets out of the first pass, rename skill conflicts, test from the terminal, and transfer channel ownership as a separate event. Most importantly, preserve the old system until Hermes has completed real scheduled work and real conversations, not merely returned a successful status command.

References

llama.cpp vs Ollama in 2026: Which Runtime Should You Run?

Rost — Mon, 14 Sep 2026 10:14:29 +0000

Ollama and llama.cpp are often compared as if they were rival inference engines. The real choice is between a managed model service and a toolkit you operate directly.

Ollama wraps a pinned and patched llama.cpp inside a scheduler, a model store, and an API, so the named model becomes the unit you operate. Direct llama.cpp inverts that: the llama-server process and its flags are the unit, and every choice about context, KV cache, and GPU placement is one you make and can show in a command.

This guide compares the two the way the decision actually happens: installation and daily commands, model management and lifetime, runtime control, APIs, performance, failure modes, and security. It ends with concrete triggers for keeping Ollama, moving to llama-server, and a low-risk migration path between them. If you are still deciding between local, self-hosted, and cloud approaches at a higher level, start with the LLM Hosting overview; for the wider landscape of local tools beyond this pair, the local LLM hosting comparison covers vLLM, LM Studio, LocalAI, and more.

llama.cpp vs Ollama: the short answer

Requirement	Better default	Why
First local chat model	Ollama	One command pulls, configures, and runs a named model
Reusable model catalog	Ollama	Tags, manifests, a registry, and `Modelfile` recipes
Exact GGUF file control	llama.cpp	The server can run the file directly without importing it
Fine-grained GPU placement	llama.cpp	Explicit layer offload, device selection, and multi-GPU split modes
Per-server KV cache tuning	llama.cpp	Separate K and V cache types and many cache controls
Automatic model loading and expiry	Ollama	Built-in scheduler and `keep_alive` behavior
OpenAI-compatible local endpoint	Either	Both support common routes, but neither promises perfect compatibility
Metrics and slot inspection	llama.cpp	Native Prometheus metrics and server slot endpoints
Native SDKs and tool integrations	Ollama	Polished Python and JavaScript clients plus named integrations
New llama.cpp feature immediately	llama.cpp	No wait for Ollama to update its pinned and patched revision
Multiple GGUF models behind one endpoint	Ollama, usually	Mature lifecycle management; llama.cpp router mode is now a credible alternative

If you only need a dependable backend for Open WebUI, a coding assistant, or a few local scripts, Ollama is usually the less distracting choice. If you keep asking what Ollama selected, allocated, changed, or hid, you have probably reached the point where llama-server is the cleaner system.

What the comparison actually means in 2026

llama.cpp is a C and C++ inference project with CPU and GPU backends, GGUF model tooling, command-line programs, and an HTTP server. Its direct serving program supports OpenAI-compatible Chat Completions, Responses, embeddings, multimodal requests, function calling, structured output, continuous batching, speculative decoding, monitoring endpoints, and a built-in web UI.

Ollama is a higher-level service. It maintains a local model store, gives models stable names, downloads and imports artifacts, applies templates and defaults, selects an available backend, schedules model processes, and unloads idle models. Its native API also reports timing and loading information that is convenient for local applications.

The frequently repeated statement that "Ollama is just a wrapper around llama.cpp" is directionally useful but technically incomplete. Ollama pins llama.cpp source, applies compatibility patches, and launches a server through its own scheduler, but it also has product behavior that llama.cpp does not define; on Apple silicon, Ollama can use its MLX engine as well. The request path makes the difference concrete:

flowchart LR
    subgraph o["Ollama stack"]
        A[Client] --> B[Ollama API on port 11434]
        B --> C[Scheduler and model store]
        C --> D[Pinned and patched llama.cpp]
        D --> E[GPU or CPU]
    end
    subgraph l["Direct llama.cpp"]
        F[Client] --> G[llama-server on port 8080]
        G --> H[llama.cpp runtime with explicit flags]
        H --> I[GPU or CPU]
    end

This leads to the most useful mental model:

With Ollama, the named model is the unit you operate.
With direct llama.cpp, the server process and its flags are the unit you operate.

Installation and the daily command surface

Ollama optimizes the first five minutes. After installation, pulling and starting a model is intentionally terse:

ollama run qwen3:8b

The model name represents more than its weights. Ollama can associate a template, parameters, system prompt, license, adapter, and minimum runtime version with that name. ollama list, ollama show, ollama ps, and ollama stop provide a coherent management surface.

Direct llama.cpp starts closer to the metal. You can download a release binary, build a backend-specific version, use a container, or use the newer Hugging Face download path, as the llama.cpp quickstart details. A local GGUF server might start like this:

llama-server \
  --model /srv/models/qwen3-8b-q4_k_m.gguf \
  --alias qwen3-8b \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 32768 \
  --n-gpu-layers all \
  --flash-attn on

Current llama.cpp documentation also shows the unified llama serve command in its quickstart. Packaged executable names can vary by distribution, so check the release or package you installed rather than copying a service file blindly.

The longer command is not automatically a disadvantage. It is an executable record of the runtime you intended to create. Put it in a systemd unit, Compose file, or shell script, and configuration becomes reviewable instead of being spread across a model manifest, environment variables, API options, and scheduler defaults.

The practical installation tradeoff

Ollama is easier to install consistently across developer machines. It is also easier to explain to someone who should use a model but should not need to understand tensor offload, chat templates, or KV memory.

llama.cpp is easier to make exact. You choose the build, backend, version, file, and flags, which is valuable when a new GPU kernel fixes your workload or a recent commit breaks it. That freedom also means you own upgrades, service supervision, and regression testing.

Model management: library names or ordinary files

Ollama treats models rather like container images. A familiar name points to a manifest and content-addressed blobs, and ollama pull resolves the required layers. This is excellent for repeatable workstation setup and for applications that should refer to qwen3:8b rather than a long filesystem path.

A Modelfile makes customization reproducible:

FROM ./qwen3-8b-q4_k_m.gguf
PARAMETER num_ctx 32768
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM You are a precise technical assistant.

ollama create qwen3-8b-local -f Modelfile
ollama run qwen3-8b-local

Ollama can import a local GGUF, so choosing Ollama does not restrict you to the public Ollama library. The import step does, however, hand the artifact to Ollama's model store. If you also retain the original GGUF for llama.cpp or LM Studio, account for the additional managed copy unless your storage layer deduplicates it.

llama.cpp can simply point at the GGUF you already have. It can also download a selected quantization from Hugging Face:

llama-server -hf ggml-org/Qwen3-8B-GGUF:Q4_K_M

This file-oriented approach works especially well for testing fresh quantizations. Download a file, change one path, and start it; there is no create step and no question about which blob a model name resolves to.

Templates are part of the model, even when they look like configuration

The weights do not define the entire chat behavior. The chat template controls how system, user, assistant, thinking, and tool messages become tokens. Stop sequences and parser behavior can change the result again.

Ollama's curated library reduces this risk because its named models carry tested metadata, and current releases (Ollama moved from 0.30 to 0.33.3 between June and September 2026) increasingly honor GGUF-defined default parameters directly rather than requiring you to restate them in a Modelfile. A hand-imported GGUF may still need a correct TEMPLATE, parser, or renderer, while llama.cpp normally reads the embedded GGUF chat template and lets you override it. Neither runtime can repair incorrect or missing model metadata by magic.

If the same quantized model feels noticeably worse after a runtime change, do not conclude that one engine has damaged the weights. First compare the template, context limit, sampling values, thinking mode, tool parser, and runtime revision, one variable at a time, before touching the model itself.

Model lifetime and switching

Ollama's scheduler is one of its strongest reasons to exist. By default, an idle model remains loaded for five minutes; a request-level keep_alive value can keep it resident indefinitely, change the duration, or unload it immediately. ollama ps shows loaded models, processor placement, context allocation, and expiry.

# Keep a model loaded.
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "keep_alive": -1
}'

# Unload it immediately.
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "keep_alive": 0
}'

A traditional llama-server --model ... process loads one model and keeps it until the process exits. That behavior is wonderfully predictable for a dedicated service: there is no surprise cold load after an idle timeout and no scheduler deciding that another model deserves the memory.

llama.cpp now also has router mode. Starting llama-server without a model can expose cached models, a GGUF directory, or INI presets and dynamically load instances according to the requested model name. It closes the old lifecycle gap only partially: only one model is resident per worker at a time, a switch is a full unload-and-reload rather than instant, and there is no eviction policy or warm pool — every alternating request between two models pays a full reload. This narrows the gap versus never having router mode at all, but it does not make the two products identical; Ollama still provides the smoother registry, warm-pool, and administration experience. For the full configuration walkthrough, current limitations, and an honest comparison to both Ollama and llama-swap, see the llama-server router mode guide. If you need one endpoint across llama.cpp, vLLM, SGLang, and other engines, llama-swap is a more appropriate abstraction than asking either runtime to become a universal model proxy.

Runtime control: where llama.cpp earns the extra work

The decisive llama.cpp advantage is not that it is always faster. It is that you can express the memory and execution plan directly, inspect it, and change one variable at a time.

Context and KV cache precision

For direct llama.cpp, context size and K/V cache types can be set per server process:

llama-server \
  --model model.gguf \
  --ctx-size 65536 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --parallel 2 \
  --flash-attn on

K and V can use different types, and llama.cpp exposes additional controls for unified KV allocation, per-slot context limits, prompt cache reuse, and cache persistence. These flags are not decorative on a 16 GB or 32 GB GPU — they determine whether long-context requests fit and how many slots can remain useful, and the underlying VRAM budget math is the same regardless of which runtime enforces it; see KV Cache on 16 GB GPUs for the formula and per-engine cache-type tables.

Ollama exposes the important common case with OLLAMA_CONTEXT_LENGTH, the num_ctx option, and OLLAMA_KV_CACHE_TYPE. Its KV cache type is a server-wide setting, however, rather than a per-named-model choice. Ollama also scales memory with configured parallelism and context length, which can make a harmless-looking concurrency change consume much more VRAM.

That behavior belongs primarily in the Ollama parallel requests guide. For this comparison, the decision is simpler: use Ollama when a global cache policy is acceptable; use separate llama.cpp services when different models need different cache precision, context, or slot geometry.

GPU selection and multi-GPU placement

Ollama aims to choose a sensible placement. It reports whether a model is fully on GPU, fully on CPU, or split, and its scheduler considers available memory when loading models. For a normal single-GPU workstation, automatic placement is often exactly what you want.

llama.cpp exposes the plan. You can select devices, specify GPU layers, choose layer, row, or experimental tensor split modes, set tensor proportions, select the main GPU, and deliberately keep MoE expert weights on CPU. This is substantially better for asymmetric multi-GPU machines and for squeezing an oversized model into a known memory budget.

If your operational notes contain phrases such as "put the KV cache on these devices" or "keep only the experts in system RAM," direct llama.cpp is the natural tool. If the requirement is merely "use the GPU if it fits," Ollama saves time without giving up much.

New features and backend cadence

Direct llama.cpp is where new llama.cpp model architectures, quantization types, GPU kernels, and experimental server options appear first. That is valuable during a model release week, when support may depend on a specific build number rather than the last stable package.

Ollama deliberately pins an upstream revision and applies compatibility patches. This can delay an upstream feature, but it can also shield users from churn and integrate it with Ollama's scheduler, templates, and cross-platform packaging. Faster access is not the same thing as greater reliability.

Ollama 0.30 materially reduced an older gap by expanding GGUF compatibility, improving NVIDIA performance, and enabling Vulkan by default for wider AMD and Intel support. Ollama's release cadence since then has stayed fast — 0.33.3 shipped in early September 2026, roughly three months later, adding cached-prompt-token reporting and another llama.cpp backend bump — so treat any specific version claim in this article, or anywhere else, as something to re-verify against ollama --version rather than a permanent fact. Any comparison that says Ollama cannot run an arbitrary local GGUF, or that Vulkan always requires an experimental opt-in, is now stale.

APIs, tools, vision, and structured output

Both runtimes are credible local API servers in 2026. Both can handle common OpenAI-style chat requests, tools, vision-capable models, embeddings, streaming, and structured output when the model and template support them.

The difference is in the surrounding surface:

Surface	Ollama	llama-server
Native API	`/api/chat`, `/api/generate`, `/api/embed` and model APIs	`/completion` plus server-specific control and inspection APIs
OpenAI API	Compatible with parts of the API, including Chat Completions and Responses	Chat Completions, Responses, embeddings, and other compatible routes
Anthropic-style API	Integrations exist, but check the client path in use	Anthropic Messages-compatible endpoint is documented
Tool calling	Native API, OpenAI-compatible path, and SDK helpers	OpenAI-style tools with Jinja templates and function-call parsing
Structured output	`format: "json"` or a JSON schema	Grammar and JSON-schema constraints plus OpenAI-style response formats
Vision	Simple image messages for supported named models	Multimodal projector control and OpenAI-compatible image input
Observability	Request timings, logs, `ollama ps`, and model APIs	Health, slots, props, and optional Prometheus metrics
Authentication	No API key on the local server by default	Optional API keys and TLS flags are built in

Do not treat "OpenAI-compatible" as a binary certification. Ollama says it supports parts of the OpenAI API, while llama.cpp explicitly avoids making a strong compatibility promise. Before changing runtimes, test streaming frames, tool-call arguments, reasoning fields, usage counters, error bodies, and any endpoint your client actually consumes.

Ollama generally wins when application integration is the job. Its SDKs and documented integrations make the happy path short. llama.cpp wins when the server itself is the object of engineering: its slot view, token timings, metrics, schemas, templates, adapters, and low-level endpoints are unusually useful during diagnosis.

Performance: benchmark the deployment, not the brand

It is tempting to ask whether llama.cpp or Ollama is faster. On a GGUF path, Ollama may be running a pinned, patched llama.cpp underneath, so a universal brand-level answer is not useful. Results change with the build revision, backend, flash attention, context allocation, parallel slots, batch sizes, cache type, model residency, and whether some layers fell back to CPU.

A fair comparison starts with the same GGUF and tests two different questions:

Cold start: include model load time and first response latency.
Warm service: preload the model, then measure prompt processing and generation separately.

Use one request and one slot first. Match context size, K/V cache type, temperature, top-p, seed, maximum output, and chat template; confirm full GPU offload from logs or status output. Only then increase concurrency, because Ollama and llama.cpp allocate and schedule parallel work differently.

For Ollama, the final native API response includes load, prompt-evaluation, and generation durations, and current releases also report cached prompt tokens directly in that response — useful for confirming whether prefix reuse actually happened before you credit a speed win to the runtime. For llama.cpp, enable performance reporting or Prometheus metrics and inspect the startup configuration. A five percent throughput win is meaningless if one run silently used a shorter context, a different cache type, or a different template.

My expectation for the same supported GGUF on one GPU is usually near parity, not a guaranteed llama.cpp victory. Direct llama.cpp can win after deliberate tuning or by adopting a newer optimization; Ollama can be just as fast when its selected engine and defaults line up with the workload. Measure after configuration, not before it.

Failure modes that expose the real difference

The model unexpectedly uses CPU

With Ollama, run ollama ps and inspect PROCESSOR, CONTEXT, and the loaded size. A larger context, another resident model, or an unsupported GPU path may explain the split. Check the service logs rather than assuming the GPU was ignored.

With llama.cpp, start with llama-server --list-devices, then read the startup log for tensor placement and buffer sizes. If you set an exact layer count, device, or split, the command itself is evidence of your intent; this is much easier to reproduce in a bug report.

A longer context causes an out-of-memory error

Ollama chooses default context lengths according to available VRAM, and current documentation recommends at least 64K for agent and coding workloads. That recommendation is not a promise that your model, parallelism, and cache will fit. Reduce num_ctx, reduce parallelism, choose q8_0 KV cache where appropriate, or use a smaller weight quantization. Confirm the actual memory picture with nvidia-smi before and after a long request so you know whether the model, the cache, or both are the constraint.

With llama.cpp, reduce --ctx-size, change --cache-type-k and --cache-type-v, lower --parallel, or adjust offload. Because each choice is explicit, it is easier to build separate long-context and high-concurrency profiles rather than force one compromise onto every model.

The API connects, but the answers are malformed

This is often a template or parser problem, especially with new reasoning and tool-calling models. Verify that the GGUF contains the expected chat template and that the runtime recognizes the architecture. Compare a plain chat request before debugging the agent framework layered above it.

On Ollama, inspect ollama show --modelfile <name> and the reported capabilities. On llama.cpp, inspect the startup template messages, use --jinja, and test /v1/chat/completions directly. Pin the working runtime version before changing another variable.

Requests become slow after switching models

Ollama may need to unload one model and load another, so separate queue time from generation time. Preload the important model with an empty request and set an intentional keep_alive value rather than depending on the five-minute default.

A dedicated llama.cpp process avoids surprise switching because its model stays resident. If you adopt router mode, model loading becomes dynamic again — and every switch between two different models is a full unload-and-reload with no warm pool — so monitor load state and cold-start latency just as you would with Ollama.

Security is not a differentiator unless you configure it

Both servers bind to localhost by default, which is the right workstation behavior. Changing the host to 0.0.0.0 turns a private local inference service into a network service, and neither product should be exposed to the public Internet merely because a firewall rule happened to allow it.

llama.cpp can enforce API keys and can terminate TLS, although a reverse proxy is still useful for policy, rate limits, and logs. Ollama's local API does not require an API key; put it behind an authenticated proxy or private network boundary if remote clients need access. If you do need remote access to Ollama, the Ollama behind a reverse proxy guide covers the Caddy and Nginx setup with streaming and timeout checks. Tool-capable models increase the consequence of exposing the surrounding application, even when the inference server itself does not execute the tools.

When to keep Ollama

Keep Ollama when its automation removes more work than it hides. It is especially strong for shared developer workstations, local desktop applications, demonstrations, coding tools, and small services that rotate among several popular models.

Ollama is also the better default when you want colleagues to reproduce a named configuration without learning llama.cpp flags. A Modelfile, model tag, and two commands are a useful operational contract. The Ollama cheatsheet covers that daily workflow in more detail.

Do not migrate merely because direct llama.cpp looks more technical. If your model fits, the API behaves correctly, latency is stable, and you do not need a missing control, replacing Ollama creates maintenance without creating capability.

One caveat worth tracking over time: Ollama's own product direction has started drifting toward centralized infrastructure. Ollama Turbo is a sign-in-gated cloud acceleration service layered on top of what was originally a local-first, privacy-first tool, and it is not the only recent change that trades local control for a hosted convenience layer. If the reason you chose Ollama in the first place was to avoid sending prompts to someone else's servers, that reasoning deserves a periodic re-check rather than a one-time decision — see Ollama Enshittification: The Early Signs for the specific changes and what to watch for. Direct llama.cpp has no equivalent hosted upsell to drift toward, which is itself a data point when you are weighing long-term control against short-term convenience.

When to move to llama-server

Move to direct llama-server when one or more of these statements are true:

You need a new llama.cpp feature or model fix before it reaches Ollama.
You must pin an exact llama.cpp commit and backend build.
Different models need different K and V cache types or slot layouts.
You need deliberate multi-GPU placement rather than automatic selection.
You are testing speculative decoding, MTP, LoRA scales, prompt caching, or uncommon samplers.
Native metrics, slot state, or server internals are required for diagnosis.
You want the original GGUF files to remain the authoritative model catalog.
A model should remain resident for the lifetime of one supervised process.

The cleanest migration trigger is repeated inspection. If every incident starts with discovering what Ollama chose before you can diagnose the model, make those choices explicit in a llama.cpp service definition.

A low-risk migration from Ollama to llama.cpp

Do not begin by reproducing every Ollama feature. Migrate one model and one client, preserve the current endpoint until the comparison is complete, and keep the same GGUF if possible.

Record ollama --version, ollama show <model>, ollama show --modelfile <model>, and ollama ps.
Locate or download the equivalent GGUF and any multimodal projector.
Start one llama-server with an explicit alias, context, GPU offload, cache types, and slot count.
Send a plain chat request, a structured-output request, and a tool call directly to each API.
Test the real client, including streaming and error handling.
Compare cold-load latency, warm prompt speed, generation speed, VRAM, and answer format.
Only then replace the service URL or add a proxy in front of both backends.

For a simple OpenAI-style smoke test:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-8b",
    "messages": [
      {"role": "user", "content": "Return exactly: runtime-ok"}
    ],
    "temperature": 0,
    "max_tokens": 16
  }'

Then verify the service itself:

# llama.cpp
llama-server --version
llama-server --list-devices
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

# Ollama
ollama --version
ollama ps
curl http://127.0.0.1:11434/api/ps
curl http://127.0.0.1:11434/v1/models

If the application depends on the Ollama-native /api/chat response shape, changing the base URL is not enough. Either migrate the client to an OpenAI-compatible route first or add an adapter. The broader Ollama to vLLM migration guide discusses the same contract-first principle for a larger runtime jump.

Final verdict

Ollama is the better local model appliance. It provides a model catalog, reproducible recipes, sensible automatic placement, convenient APIs, and lifecycle management without demanding that every user become an inference operator.

llama.cpp is the better precision instrument. llama-server exposes enough of the execution plan to make constrained VRAM, long context, unusual hardware, new model support, and controlled experiments understandable rather than mysterious.

For most people, the right sequence is not Ollama or llama.cpp forever. Start with Ollama, learn which constraints actually matter, and move the affected workload to direct llama.cpp when you can name the control you need. That is a much stronger reason than chasing a benchmark measured under someone else's defaults.

References

ROCm vs Vulkan for AMD Local LLM Hosting: 2026 Guide

Rost — Sat, 12 Sep 2026 11:42:17 +0000

ROCm and Vulkan both accelerate AMD GPUs for local LLM hosting, but they are not interchangeable. The right choice depends on the engine, GPU, and workload.

In local LLM hosting the two backends sit at different layers. ROCm is AMD's compute platform under PyTorch, vLLM, and SGLang, while Vulkan is a portable GPU API that llama.cpp-class engines use to run quantized models across a wide range of hardware.

This guide compares the two engine by engine — llama.cpp, Ollama, LM Studio, vLLM, SGLang, TGI, and LocalAI — with build commands, device checks, and failure modes that masquerade as performance problems. If you are new to the hosting landscape, start with the LLM Hosting overview, which maps the tool families this article drills into.

ROCm vs Vulkan: the short answer

Situation	Recommended starting point	Why
llama.cpp with GGUF on Linux	Vulkan	Small install surface, broad GPU coverage, and easy rollback
llama.cpp on a supported RDNA 3 or RDNA 4 GPU	Benchmark both	Kernel performance changes with model shape, quantization, context, and build
Ollama on a listed AMD GPU	ROCm first, verify Vulkan too	Ollama supports both, but backend choice is less explicit than in bare llama.cpp
LM Studio on a desktop AMD GPU	Vulkan first	Runtime switching makes comparison easy and avoids a system-wide compute stack
vLLM or SGLang	ROCm, but verify GPU-family kernel coverage	These are PyTorch/HIP stacks; Vulkan is not an alternative backend, and brand-new architectures can still lack optimized kernels
TGI on supported Instinct hardware	ROCm	The published AMD container path targets MI210, MI250, and MI300 families
Older or unlisted Radeon GPU	Vulkan	Vulkan drivers usually cover more graphics hardware than ROCm libraries do
AMD Instinct server	ROCm	Multi-GPU compute, RCCL, framework kernels, and operational tooling live here
Windows local GGUF serving	Vulkan	It is generally the least restrictive route for llama.cpp-class runtimes
Ryzen AI Max or other large-memory APU	Vulkan first, then ROCm if required	Both can work, but shared memory and kernel support need workload-specific testing

This table is a starting policy, not a benchmark result. A backend that detects the GPU but leaves some operations on the CPU can look healthy while performing badly, so every final decision needs log inspection and an end-to-end prompt test.

What ROCm and Vulkan actually are

ROCm is a compute platform

ROCm includes the HIP runtime, compiler, math libraries, collective communication, profilers, and framework packages needed to run AMD compute workloads. It is the AMD-side foundation beneath PyTorch builds and engines such as vLLM and SGLang, and it can also accelerate llama.cpp through its HIP backend.

That breadth is ROCm's advantage and its cost. The host driver, GPU target, user-space libraries, framework wheel, kernel version, and container image must form a compatible set; when they do, ROCm provides far more than token generation through one local executable.

ROCm 10.0.0, released on August 26, 2026, is built on TheRock (AMD's build and release system since ROCm 7.14), validates PyTorch 2.13, vLLM 0.27, and SGLang 0.5.15, and formally adds RDNA 4 support for gfx1200 (RX 9060/9060 XT/9050) and gfx1201 (RX 9070/9070 XT/9070 GRE, Radeon AI PRO R9700 series). The ROCm compatibility matrix is still the authority for an exact GPU and operating-system combination, not a forum post that happens to use the same marketing family.

Vulkan is a portable GPU interface

Vulkan is a graphics and compute API implemented by a GPU driver. In local LLM hosting, it normally means an inference engine ships or compiles compute shaders that execute through a Vulkan implementation such as Mesa RADV on Linux or the vendor driver on Windows.

Vulkan does not provide a drop-in PyTorch platform comparable to ROCm. Its practical strength is narrower and useful: a llama.cpp-style engine can use the same backend design on AMD, Intel, Nvidia, and other Vulkan-capable hardware without installing a vendor-specific machine-learning stack.

This distinction explains most of the decision. If the application offers only a HIP or PyTorch path, Vulkan cannot rescue it; if the application is already based on llama.cpp and GGUF, installing the whole ROCm stack may solve a problem you did not have.

Engine support matrix in 2026

Engine	ROCm or HIP	Vulkan	Typical model format	Practical note
llama.cpp / llama-server	Yes	Yes	GGUF	Best platform for a controlled A/B backend test
Ollama	Yes	Yes	Managed GGUF-derived models	Convenient, but backend selection and packaging are abstracted
LM Studio	Yes	Yes	GGUF and product-managed formats	Selectable runtimes make desktop testing approachable
vLLM	Yes	No	Safetensors and supported quantizations	Use AMD's matched ROCm image or wheel set; verify GPU-family kernel coverage first
SGLang	Yes	No	Safetensors and supported quantizations	ROCm is part of the deployment architecture
TGI	Yes	No	Safetensors and supported quantizations	Published AMD validation remains Instinct-focused
LocalAI	Yes	Yes	Backend-dependent, commonly GGUF	Uses different ROCm and Vulkan container images

ROCm does not imply Safetensors, and Vulkan does not formally imply GGUF. The useful association comes from engines: llama.cpp can read the same GGUF with either its HIP or Vulkan build, while PyTorch-native servers use ROCm and generally consume Hugging Face model repositories.

That makes model inventory an architectural constraint. A library of carefully selected GGUF quantizations points naturally toward llama-server, Ollama, LM Studio, or LocalAI; a deployment built around tensor parallelism, continuous batching, and framework-native weights points toward ROCm with vLLM or SGLang. For the wider engine landscape beyond AMD backends — API maturity, tool calling, and production readiness across a dozen tools — see our comparison of Ollama, vLLM, LM Studio, LocalAI and other local LLM hosting tools.

llama.cpp: the cleanest ROCm vs Vulkan comparison

llama.cpp exposes both backends without changing the model file or HTTP client. This is the fairest place to compare ROCm and Vulkan because the tokenizer, sampling settings, chat template, quantization, and server behavior can remain fixed.

The current llama.cpp build documentation uses GGML_HIP for ROCm and GGML_VULKAN for Vulkan. Old articles that recommend GGML_ROCM or the removed Makefile flags should not be trusted without checking the project's current CMake options.

Build the Vulkan backend on Ubuntu

Install the Vulkan headers, shader compiler, and SPIR-V headers, then verify that the driver can enumerate the intended GPU:

sudo apt-get update
sudo apt-get install -y libvulkan-dev glslc spirv-headers vulkan-tools

vulkaninfo --summary

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -S . -B build-vulkan \
  -DGGML_VULKAN=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-vulkan --config Release -j

On a mixed iGPU and discrete-GPU system, enumeration order deserves attention. GGML_VK_VISIBLE_DEVICES can restrict llama.cpp to a specific Vulkan device, and the startup log should name the selected card rather than merely report that a Vulkan device exists.

Build the ROCm or HIP backend

First confirm that ROCm identifies the GPU and reports the expected gfx target. The target can be omitted to build for the GPUs in the current system, but pinning it reduces compilation work when you know the deployment hardware.

rocminfo | grep -E 'Name:.*gfx' | head
hipconfig --full

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

HIPCXX="$(hipconfig -l)/clang" \
HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build-rocm \
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1201 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-rocm --config Release -j

Replace gfx1201 with the target reported for the actual card — that value maps to the RX 9070/9070 XT/9070 GRE and Radeon AI PRO R9700 family in RDNA 4, while gfx1200 covers the RX 9060 series and gfx1100/gfx1101/gfx1102 cover RDNA 3's RX 7900/7800/7700/7600 lines. Do not copy HSA_OVERRIDE_GFX_VERSION into a production service merely because it helped someone boot an unsupported GPU; an override can make code load, but it does not turn that hardware into a validated platform.

Benchmark the same workload, not two defaults

Use one GGUF file, the same flash-attention setting, the same layer offload, and repeated runs. Prompt processing (pp) and token generation (tg) exercise the system differently, while a long-context server also adds KV-cache allocation and memory pressure that a short synthetic benchmark will miss.

MODEL=/srv/models/model.gguf

./build-vulkan/bin/llama-bench \
  -m "$MODEL" -ngl 999 -fa 1 -p 512 -n 128 -r 5

./build-rocm/bin/llama-bench \
  -m "$MODEL" -ngl 999 -fa 1 -p 512 -n 128 -r 5

Community results illustrate why a universal winner is misleading, and RDNA 4's gfx1201 target is the clearest recent example. In one same-machine RX 9070 XT submission, Vulkan reached about 143 token/s against ROCm's 128 token/s on the 7B Q4_0 generation test, but later submissions showed smaller gaps as builds changed; the Vulkan and ROCm discussions also contain large differences in prompt-processing results and test conditions. A separate, more detailed OpenBenchmarking.org run on an RX 9070 XT with llama.cpp b6401 found Vulkan ahead on decode across several 8B-class models (Qwen3-8B-Q8_0, Llama-3.1-Tulu-3-8B-Q8_0) but behind HIP on prompt processing at longer prompt lengths — the two backends trade the lead depending on which phase you measure.

The gap can also run the other way, and by a lot, for specific model shapes. An open llama.cpp issue documents Vulkan on gfx1201 becoming 4.7–6.7x slower than HIP on token generation once a model's hidden size reaches 4096 or above (effective decode bandwidth collapsing to roughly 70–100 GB/s on a 640 GB/s card), while a smaller 4B model with hidden size 2560 shows no such regression on either backend. Treat every number here as a snapshot of one build, one driver, and one model shape — not as a rule that generalizes across quant type, model architecture, flash attention, batch sizes, driver version, thermal state, or the llama.cpp commit.

Ollama on AMD: convenient, but verify the backend

Ollama officially supports listed AMD GPUs through ROCm and now documents additional AMD coverage through Vulkan on Windows and Linux. Its current hardware support page says Vulkan is enabled by default when the backend is installed, supports GGML_VK_VISIBLE_DEVICES for device selection, and can disable Vulkan with OLLAMA_VULKAN=0.

This is a meaningful improvement over the period when Vulkan advice depended on experimental builds. It also makes some older tutorials stale: setting an undocumented switch and assuming the service selected Vulkan is weaker evidence than reading the server log.

sudo systemctl edit ollama

For diagnostics, add a drop-in rather than exporting variables only in an interactive shell:

[Service]
Environment="OLLAMA_DEBUG=1"
Environment="GGML_VK_VISIBLE_DEVICES=0"

Then reload, restart, and inspect both process placement and discovery messages:

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -u ollama -b --no-pager | tail -n 200

ollama run qwen3:8b "Return exactly: backend test passed"
ollama ps

Look for the named GPU, selected library, model allocation, and GPU percentage. A log that shows a discovery timeout followed by a successful HTTP response may mean Ollama quietly fell back to the CPU. For the everyday command set around this service, the Ollama CLI cheatsheet is the quicker reference.

Ollama is excellent when model acquisition and a stable local API matter more than backend control. If repeatable ROCm-versus-Vulkan testing is the objective, bare llama-server is the better instrument because the build directory makes the backend explicit.

vLLM and SGLang make ROCm the decision — but check GPU-family kernel coverage first

vLLM and SGLang are not Vulkan applications. Their AMD paths sit on ROCm, PyTorch, and optimized HIP kernels, so choosing one of these engines has already selected the compute platform.

AMD's current vLLM on ROCm guide recommends a prebuilt container and publishes matched images for ROCm, PyTorch, Python, and vLLM. That coupling is useful: it replaces a large dependency-solving exercise with a versioned deployment unit.

At the time of writing, AMD documents this ROCm 10 image for vLLM 0.27:

docker pull \
  rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0

docker run --rm -it \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add video \
  --ipc=host \
  --network=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /srv/models:/app/models \
  -e HF_HOME=/app/models \
  rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 \
  bash

Use the image selected for the exact GPU family in AMD's documentation; RDNA and CDNA images have not always been interchangeable. For a production server, pin the full tag or digest and validate the host driver before blaming vLLM for an initialization failure.

Brand-new GPU generations are the sharpest edge case, and RDNA 4 is a real, documented example rather than a theoretical risk. Independent testing on an RX 9070 XT (gfx1201) in early 2026 found vLLM on ROCm 7.2 silently falling back to FP32 dequantization for FP8 model weights — because gfx1201 was not yet recognized in vLLM's platform detection — which bypassed the GPU's matrix accelerators entirely and produced only 48 tokens/s, versus 62 tokens/s from llama-server on Vulkan running a GGUF quantization of a comparable model on the same card. The lesson generalizes: a ROCm/ PyTorch stack can load successfully on a new architecture and still run an unoptimized fallback path with no error message. Always confirm which kernel path actually executed (via rocprof, vendor profiling notes, or a known-good throughput baseline for the GPU) before trusting a single "it started fine" result on hardware that shipped within the last release cycle or two.

The reason to accept ROCm's larger operational surface, once kernel coverage is confirmed, is throughput architecture — not merely a few more token/s in a single-user test. Continuous batching, framework-native quantization, tensor parallelism, scheduler behavior, and the surrounding PyTorch tooling are the real case for moving to vLLM, and if you are weighing whether that move is justified at all, our Ollama to vLLM migration guide lists the workload signals.

TGI: ROCm support with a narrower target

Hugging Face documents an AMD image for Text Generation Inference, but its published validation is centered on Instinct MI210, MI250, and MI300 hardware. The TGI AMD guide uses the 3.3.5-rocm image and lists unsupported ROCm features, so it should not be generalized into a promise for every Radeon card. Our TGI install guide covers that ROCm image setup in more detail.

There is no Vulkan TGI path to compare. If TGI is a fixed requirement, choose supported ROCm hardware and reproduce the documented container; if the engine is negotiable, current vLLM and SGLang support deserves evaluation before beginning a new AMD deployment.

LM Studio: switch runtimes instead of rebuilding

LM Studio packages multiple inference runtimes and exposes runtime management through the lms command. Its runtime documentation supports listing, downloading, selecting, updating, and removing runtimes, which makes ROCm-versus-Vulkan experiments accessible without maintaining separate source trees.

lms runtime ls
lms runtime get
lms runtime select

Run the same GGUF with the same context length, GPU offload, flash-attention setting, and prompt. Compare time to first token, generation rate, load time, and peak memory rather than judging a backend from one short chat response.

Runtime packaging does not eliminate backend-specific faults. For example, a 2026 LM Studio issue on an R9700 reported a large model hanging near the end of a ROCm load while the Vulkan runtime loaded it, while a separate Vulkan memory-headroom issue described the opposite outcome near full VRAM. These are individual reports, but together they make the right operational point: keep a fallback runtime and leave memory headroom.

LocalAI: choose the image as well as the backend

LocalAI provides separate ROCm or hipblas and Vulkan container variants. Its GPU acceleration guide documents gpu-hipblas images for AMD compute and gpu-vulkan images for the portable path, so a container tag copied from a CUDA guide will not discover the correct backend by magic. The LocalAI quickstart covers the general setup; the backend-specific container choice is what this section adds.

The ROCm container needs /dev/kfd and /dev/dri, while Vulkan normally needs the appropriate render device under /dev/dri. Pin a release tag for a real service; latest and master are useful for diagnosis, but they make rollback and performance comparison unnecessarily vague.

# ROCm or HIP image
docker run --rm -it \
  --device /dev/kfd \
  --device /dev/dri \
  -p 8080:8080 \
  quay.io/go-skynet/local-ai:v4.8.0-gpu-hipblas

# Vulkan image
docker run --rm -it \
  --device /dev/dri \
  -p 8080:8080 \
  localai/localai:v4.8.0-gpu-vulkan

The tag examples reflect the documentation available at publication time; confirm the current registry names before automating a pull. More importantly, do not infer acceleration from the container name alone — inspect LocalAI's debug log and watch GPU utilization during a request.

What changed in ROCm 10 packaging

ROCm 10 is not just another minor package update. AMD's TheRock transition guide says ROCm Core SDK packages now use the amdrocm- prefix, the versioned installation root is /opt/rocm/core-10.0, and several legacy packages have been consolidated.

This is why a command from an older ROCm article may return "package not found" even on a correctly configured repository. For example, HIPCC now comes from amdrocm-llvm, BLAS components are combined in amdrocm-blas, and a full system installation can use an all-architecture or GPU-family-specific Core SDK meta-package — RDNA 4 cards use the gfx120X-all family tag (package suffix -gfx1200-gfx1201), which is worth knowing before you go hunting for a gfx1201-only package name that does not exist.

The amdrocm meta-package configures alternatives and compatibility symlinks under /opt/rocm. A minimal or custom installation may not provide the same paths, so build scripts that hard-code /opt/rocm/bin/hipcc should either use hipconfig or set ROCM_PATH explicitly.

Two diagnostic changes are easy to miss. ROCm SMI has been removed in favor of AMD SMI, and ROCm Bandwidth Test reached end of life; scripts that call rocm-smi or rocm-bandwidth-test need to move to amd-smi and AMD's replacement tools rather than reinstall arbitrary legacy packages.

Containers still depend on the host

A ROCm container carries user-space libraries, not a replacement kernel driver. The host must expose /dev/kfd and /dev/dri, its driver must be compatible with the container stack, and the service user needs permission to open those devices.

Vulkan containers have a similar boundary around the host Vulkan driver and render node. Packaging is lighter, but an incorrect ICD, missing render-group membership, or an accidentally selected iGPU can still turn a working container image into a CPU-bound or unstable service.

Discrete GPUs, APUs, and older Radeon cards

RDNA 3 and RDNA 4 discrete GPUs

Current Radeon RX 7000, RX 9000, and Radeon AI Pro models have the strongest case for testing both llama.cpp backends. ROCm support is now explicit for many gfx110x and gfx120x targets, while Vulkan through a current Mesa RADV or Windows vendor driver is mature enough to be a primary route rather than a desperate fallback. For the hardware side of that decision — VRAM, bandwidth, power, and pricing across vendors — see our GPU comparison for AI workloads in 2026.

Do not convert a 7B benchmark into a rule for a 27B dense model or a mixture-of-experts model. Matrix shapes, active parameters, quantized kernels, prompt length, and memory pressure can change the order, and backend performance has moved substantially between llama.cpp revisions — the gfx1201 hidden-size regression noted above is a concrete case of exactly this kind of shift.

Ryzen APUs and shared memory

Large-memory Ryzen AI Max systems are unusually interesting because the GPU can access a much larger shared-memory pool than a normal discrete consumer card offers. ROCm 10 lists current Ryzen AI families, while Vulkan-capable llama.cpp runtimes can also use the iGPU without building a PyTorch environment.

Capacity is not bandwidth. A model fitting into 64 GB or 96 GB of allocated shared memory does not mean it will decode like a 32 GB discrete card, and aggressive context allocation can starve the operating system even when an application reports ample GPU memory. The same VRAM-budget discipline that applies to discrete NVIDIA and AMD cards applies here too — see KV Cache on 16 GB GPUs for the underlying budget math, which is backend-agnostic.

Mixed iGPU and dGPU machines need explicit device selection. A recent llama.cpp report described excessive system-memory reservation when an unused iGPU remained visible beside an R9700; it is an unconfirmed issue, but it is a good reason to expose only the device the service is intended to use.

Older and unsupported Radeon hardware

Vulkan is usually the first route for an older Radeon because graphics-driver coverage is broader than ROCm's supported compute-target set. ROCm-based projects also note that newer rocBLAS releases removed kernels for some older targets, so forcing a nearby gfx value cannot restore code that is no longer shipped.

An override is acceptable for a laboratory experiment with clear failure expectations. It is a poor foundation for an unattended API, because the next ROCm or application update can replace a tolerated mismatch with a startup failure or incorrect result.

Linux vs Windows for AMD LLM backends

Linux is the natural ROCm host for production inference. It offers the broadest engine support, established container device mapping, current Mesa Vulkan drivers, and the operational tools expected by vLLM and SGLang deployments.

Windows has genuine ROCm support for listed hardware, but the application ecosystem remains narrower. For desktop GGUF inference through llama.cpp, Ollama, or LM Studio, Vulkan is usually the calmer starting point; use ROCm when the application provides a supported Windows path and a concrete feature or benchmark justifies it.

WSL2 should be treated as a third platform, not a synonym for native Linux. Match AMD's documented Windows driver, WSL distribution, ROCm release, and framework package as one supported combination.

Verification checklist before serving traffic

Start below the application. If the driver cannot enumerate the correct device, changing model flags is only rearranging the symptom.

lspci -nnk | grep -A3 -E 'VGA|Display'
ls -l /dev/kfd /dev/dri/renderD* 2>/dev/null
id

# ROCm path
rocminfo | grep -E 'Marketing Name:|Name:.*gfx' | head -n 20
amd-smi list

# Vulkan path
vulkaninfo --summary

Then verify the engine. The startup output must name ROCm or Vulkan, name the intended GPU, and report that model layers or tensors were placed on it; finally, GPU memory and utilization must rise while a request is running.

# Observe an AMD GPU while another terminal sends requests
watch -n1 amd-smi monitor

# Basic OpenAI-compatible API check for llama-server
curl -s http://127.0.0.1:8080/v1/models
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Return exactly: ready"}],
    "max_tokens": 8,
    "temperature": 0
  }'

Record driver, runtime, engine commit or image digest, model file checksum, context, batch settings, and command line with every benchmark. Without that metadata, a token-per-second number is an anecdote that cannot survive the next upgrade — and as the vLLM-on-gfx1201 fallback and the Vulkan hidden-size regression above both show, a plausible-looking number can hide a silently unoptimized code path.

Failure modes that look like backend performance

Silent CPU fallback

The server starts and answers correctly, but generation is unexpectedly slow and GPU utilization remains flat. Check discovery logs, device permissions, model offload, and container devices before tuning threads or sampling parameters.

Silent precision fallback (new hardware, new kernels)

The server starts, GPU utilization looks reasonable, and there is no error — but the framework quietly dropped to an unoptimized numeric path because the GPU's compute capability or architecture string was not yet recognized. This is exactly what happened with vLLM's FP8 kernels on gfx1201; the fix is to check the framework's own platform-detection code or issue tracker for your exact GPU string before trusting a single throughput number on a GPU generation released in the last release cycle or two.

The wrong GPU is selected

A Ryzen desktop may expose an iGPU as Vulkan device 0 and a discrete Radeon as device 1. Restrict visible devices and confirm the full device name in the log; do not assume numbering is stable after a driver or BIOS change.

ROCm target mismatch

rocminfo reports one gfx target while the application image contains kernels for another set. Use a matching image or rebuild for the exact target; reserve HSA_OVERRIDE_GFX_VERSION for explicitly unsupported experiments.

Driver and user-space mismatch

The container has current ROCm libraries but the host driver belongs to an older release stream. Timeouts during discovery, kernel launch errors, or a fall back to CPU are more likely than a clean message explaining the version boundary.

Vulkan ICD confusion

More than one Vulkan implementation is installed, and the loader selects an unexpected ICD. Inspect vulkaninfo, remove accidental duplicates, or select the intended ICD and device explicitly rather than layering another SDK over the problem.

VRAM estimates leave no operating margin

The model appears to fit but fails during warmup, flash-attention setup, or the first long prompt. Leave several gigabytes of headroom on a large model, then reduce context or batch size before concluding that the backend cannot run the quantization.

A practical backend selection procedure

Step 1: choose the serving behavior

If the goal is one or two local users, GGUF files, and a simple OpenAI-compatible endpoint, start with llama-server, Ollama, or LM Studio. If the goal is continuous batching, high concurrency, framework-native models, or tensor parallelism, begin with vLLM or SGLang and accept ROCm as part of the design.

Step 2: check official hardware support

Match the exact GPU target, operating system version, kernel, and driver in the current ROCm matrix. For Vulkan, confirm the intended GPU through vulkaninfo and use a current driver rather than assuming that the presence of libvulkan.so proves useful compute support. If the GPU is from the newest architecture generation, also check the specific framework's platform-detection code or open issues for that exact gfx target — official support and optimized-kernel support are not always released together.

Step 3: establish the simplest working baseline

For GGUF, Vulkan is normally that baseline because it changes fewer system components. For a PyTorch engine, use AMD's pinned ROCm container rather than assembling torch, Triton, AITER, and vLLM from unrelated latest versions.

Step 4: benchmark production-shaped prompts

Measure prompt processing, time to first token, decode rate, peak memory, and concurrent request behavior. Include the context and tool-calling pattern the real service will use; a 128-token microbenchmark does not predict a 100,000-token agent session.

Step 5: keep the fallback deployable

Two llama.cpp build directories cost little compared with a day lost to a driver regression. Keep the last known-good container digest or runtime installed, and roll forward only after the candidate passes the same test set.

The same procedure as a decision flow:

flowchart LR
    A["Choose serving engine"] --> B{"PyTorch/HIP engine?"}
    B -- Yes --> C["ROCm: pinned container<br>+ kernel-coverage check"]
    B -- No --> D{"GGUF on AMD GPU?"}
    D -- Yes --> E["Vulkan baseline"]
    E --> F{"Benchmark: ROCm wins<br>by a measurable margin?"}
    F -- Yes --> G["Switch to ROCm"]
    F -- No --> H["Keep Vulkan,<br>keep ROCm build as fallback"]

Final verdict: ROCm or Vulkan for AMD LLM hosting?

Vulkan is the best default for local GGUF inference when portability, setup speed, Windows support, or older Radeon coverage matters. It is no longer reasonable to describe it as inherently slow; on some recent Radeon and llama.cpp combinations it is the faster backend, and on others it is close enough that lower operational friction wins.

ROCm is the correct choice when the engine is built around PyTorch, when AMD Instinct and multi-GPU compute are central, or when a tested HIP build wins the actual model workload. Its ecosystem is much stronger in 2026, but the new packaging and strict compatibility layers still reward pinned versions and disciplined verification — and on the newest RDNA generation specifically, verifying that the optimized kernel path actually ran is not optional.

For a supported Radeon workstation, my recommendation is deliberately unromantic: install Vulkan first, add ROCm when an engine or benchmark earns the complexity, and keep both llama.cpp builds if the machine regularly serves different model shapes. The best AMD backend is not a permanent property of the card; it is a property of the card, engine, model, driver, and workload together.

References

ROCm 10.0.0 release notes
ROCm compatibility matrix
ROCm TheRock transition and package mapping
llama.cpp HIP and Vulkan build instructions
Ollama AMD and Vulkan hardware support
AMD vLLM inference and serving guide
Hugging Face TGI on AMD GPUs
LM Studio runtime management
LocalAI GPU acceleration
llama.cpp Vulkan performance discussion
llama.cpp ROCm performance discussion
Angelov, I. "Local LLM Inference on AMD RX 9070 XT — Vulkan vs ROCm Benchmarks on RDNA4." digtvbg.com, March 2026. https://digtvbg.com/blog/llama-server-vulkan-rdna4-vllm-rocm-benchmark/
"ROCm Vs. Vulkan Llama.cpp RDNA4 Radeon RX 9070 XT Benchmarks." OpenBenchmarking.org. https://openbenchmarking.org/result/2509078-NE-ROCMVSVUL92
"[Vulkan] Pathological token-generation slowdown on RX 9070 XT (gfx1201) for models with hidden_size >= 4096." llama.cpp GitHub issue.

Choosing free on-prem git server - Gitea is the winner!

Rost — Sat, 12 Sep 2026 11:41:54 +0000

Wanting to move your projects away from open cloud git providers and thinking of self-hosting internal git server locally?

This guide is part of Developer Tools: The Complete Guide to Modern Development Workflows.

Choosing servers

Running your own git server shouldn't be too difficult, right?

So now choosing free git server from a vary short list of options.
Bonobo
Gogs vs
Gitea vs
Gitlab.

Bonobo though is free but for windows, and doesn't have a linux version.

Gitlab is feature-rich and resource-heavy, that's a hands on experience. It's a commertial product but
has a free version
too.

The Gogs is very light-weight, I've tried it and it worked well, but it's missing a container registry

The comparison favors Gitea in my eyes.

Both Gitea and Postgresql dockerized

https://docs.gitea.com/next/installation/install-with-docker

cd ~
mkdir gitea
cd gitea

docker-compose.yml:

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=db:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=gitea
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: postgres:14
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - ./postgres:/var/lib/postgresql/data

then

docker-compose up -d

navigate to http://localhost:3000/

to shutdown

docker-compose down

volumes will stay

Resource usage

Showing containers take 260MB RAM and a bit of CPU.

Docker images size 583MB total.
422MB of which is postgres:14 image.
postgres:14-alpine image is ocupying 239MB, might also work if resources are constrained.

PS. After migrating 10+ repos to gitea, some forking and clonning gitea container alone is using 420MB RAM now.
Need to keep an eye on it.

Sync two repos

https://docs.gitea.com/next/usage/repo-mirror

Can do push and pull.

pull

Select New Migration in the Create... menu on the top right.
Select the remote repository service.
Enter a repository URL.
If the repository needs authentication fill in your authentication information.
Check the box This repository will be a mirror.
Select Migrate repository to save the configuration.

The repository now gets mirrored periodically from the remote repository.
You can force a sync by selecting Synchronize Now in the repository settings.

Can only set up pull mirroring for repos that don't exist yet on your instance.
Once the repo is created, you can't convert it into a pull mirror anymore.

Https config

There is more in ssl config: https://docs.gitea.com/next/administration/https-setup

but we are trying this for now (took from gitea site):

Using the built-in server

Before you enable HTTPS, make sure that you have valid SSL/TLS certificates. You could use self-generated certificates for evaluation and testing.
Please run

gitea cert --host [HOST]

to generate a self signed certificate.

If you are using Apache or nginx on the server, it's recommended to check the reverse proxy guide.

To use Gitea's built-in HTTPS support, you must change your app.ini file:

[server]
PROTOCOL  = https
ROOT_URL  = https://git.example.com:3000/
HTTP_PORT = 3000
CERT_FILE = cert.pem
KEY_FILE  = key.pem

Note that if your certificate is signed by a third party certificate authority (i.e. not self-signed), then cert.pem should contain the certificate chain. The server certificate must be the first entry in cert.pem, followed by the intermediaries in order (if any). The root certificate does not have to be included because the connecting client must already have it in order to establish the trust relationship. To learn more about the config values, please checkout the Config Cheat Sheet.

For the CERT_FILE or KEY_FILE field, the file path is relative to the GITEA_CUSTOM environment variable when it is a relative path. It can be an absolute path as well.

Setting up HTTP redirection
The Gitea server is only able to listen to one port; to redirect HTTP requests to the HTTPS port, you will need to enable the HTTP redirection service:

[server]
REDIRECT_OTHER_PORT = true
; Port the redirection service should listen on
PORT_TO_REDIRECT = 3080

If you are using Docker, make sure that this port is configured in your docker-compose.yml file.

Using reverse proxy

In this post: Gitea-ssl I'm configuring Apache as TLS terminating reverse proxy.

Todo

ssh config: https://docs.gitea.com/next/installation/install-with-docker

backup-restore test. See Backup and Restore Gitea server for detailed backup procedures.

Useful links

KV Cache on 16 GB GPUs: Making Long Context Actually Fit

Rost — Fri, 11 Sep 2026 09:09:53 +0000

A model can advertise a 128K context window and still fail at 40K tokens on a 16 GB GPU. The architecture ceiling never promised that weights, KV cache, compute buffers, and the desktop compositor would fit on your card at the same time.

The KV cache is usually where long-context plans meet that physical limit. It grows with every active token and sequence, so a configuration that looks comfortable at startup can slow sharply, spill into system memory, or fail during a large prefill.

This guide turns the problem into a VRAM budget. It covers the cache formula, reproducible 32K to 128K size tables, and working configurations for llama.cpp --cache-type-k and --cache-type-v, vLLM paged and prefix caching, and Ollama context controls — plus the experimental adaptive-cache forks that deserve interest but not blind trust. For the broader throughput, latency, and benchmark context behind these numbers, start with the LLM Performance hub.

The Short Answer for a 16 GB GPU

Start with one sequence, a realistic maximum context, Flash Attention, and an 8-bit KV cache. Measure that configuration before attempting 4-bit cache, CPU offload, multiple parallel slots, or an experimental fork.

Target	Sensible first attempt on 16 GB	Main risk
32K	Q4 or Q5 weights, Q8 KV, one sequence	Model weights leave too little buffer space
64K	Smaller model or aggressive weight quant, Q8 KV	Prefill latency and cache bandwidth
128K	Small GQA model, Q8 or tested Q4 KV, one sequence	Cache alone may consume most of VRAM
Two concurrent 64K sessions	Treat as roughly a 128K cache budget	Parallel capacity is mistaken for free throughput

My opinion is simple: a stable 64K setup is usually more useful than a nominal 128K setup that runs at the edge of an out-of-memory failure. Context capacity is not a trophy; it is a latency, quality, and concurrency decision.

What the KV Cache Stores

During autoregressive generation, every attention layer produces key and value tensors for each processed token. The runtime retains those tensors so the next token can attend to earlier tokens without recomputing the entire prefix.

The cache saves enormous compute, but it consumes memory in proportion to the retained token count. For a conventional transformer with grouped-query attention, a useful baseline is:

KV bytes = sequences * tokens * layers * 2 * KV heads * head dimension * bytes per value

The factor of two represents keys and values. Multi-head attention uses as many KV heads as query heads, grouped-query attention uses fewer KV heads, and multi-head latent attention or hybrid recurrent architectures need different calculations.

Why Parameter Count Is Not Enough

Two 8B models can have very different KV-cache costs. One may use 32 layers and eight KV heads, while another may use fewer KV heads, shared KV layers, sliding-window attention, or compressed latent states.

Parameter count mostly predicts weight memory. KV geometry comes from attention architecture, so read the model metadata rather than guessing from 8B, 27B, or the GGUF file size. The clearest illustration is how far attention design has moved beyond plain multi-head attention (MHA):

Multi-Query Attention (MQA) shares a single K/V head across all query heads — maximal cache savings, but it is the most aggressive quality compromise and is rarely used alone in current frontier models.
Grouped-Query Attention (GQA) groups query heads into clusters that each share one K/V head — the mainstream compromise used by most open dense models, and the geometry the formula above assumes.
Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2 and carried into DeepSeek-V3 and Kimi K2, takes a different approach entirely: instead of sharing K/V across heads, it projects keys and values into a compressed low-rank latent vector and reconstructs the full-resolution K/V on demand at attention time. DeepSeek reported roughly a 93% KV-cache reduction versus an equivalently sized dense MHA model, while keeping quality competitive with — sometimes ahead of — GQA at the same memory budget.

The practical consequence is that a "27B GQA model" and a "27B MLA model" can have KV-cache footprints that differ by an order of magnitude for the same context length. Do not assume the formula above applies to a model that documents itself as using latent attention, DeltaNet-style state, or sliding-window layers — check the architecture section of the model card first.

With Ollama, ollama show MODEL --verbose exposes model metadata including layer count, attention heads, KV heads, and context length where the format provides them. With llama.cpp, the model-loader output printed at startup usually includes equivalent GGUF metadata and the runtime's actual cache allocation.

Model Limit, Allocated Context, and Used Context

These are three separate numbers. The model limit is the maximum supported by its training and positional encoding, the allocated context is what the runtime reserves or permits, and used context is the tokens currently retained for a sequence.

Raising an engine flag cannot safely extend a model beyond its supported position scheme. RoPE scaling can extend some architectures, but it is a model-quality experiment, not a KV-memory optimization.

KV Cache Size Table: 32K, 64K, and 128K Context Budgets

Consider a representative GQA model with 32 layers, eight KV heads, and a head dimension of 128. These dimensions produce 65,536 key and value elements per token before multiplying by the storage size of each element.

The table uses binary GiB and the physical block sizes commonly associated with llama.cpp f16, q8_0, and q4_0. It is a baseline calculation, not a promise about total process memory; alignment, metadata, hybrid layers, and backend workspaces add overhead.

Cache type	Approx. bytes per stored value	32K context	64K context	128K context
F16	2.0000	4.00 GiB	8.00 GiB	16.00 GiB
Q8_0	1.0625	2.13 GiB	4.25 GiB	8.50 GiB
Q4_0	0.5625	1.13 GiB	2.25 GiB	4.50 GiB
Q8_0 K plus Q4_0 V	Mixed	1.63 GiB	3.25 GiB	6.50 GiB

Now double the layer count to 64 while leaving the other dimensions unchanged. The FP16 cache becomes 8 GiB at 32K, 16 GiB at 64K, and 32 GiB at 128K, which demonstrates why a single context recommendation cannot cover every model.

The Actual 16 GB Equation

A practical budget is broader than the KV formula:

usable VRAM = total VRAM - desktop and driver reserve

KV budget = usable VRAM
          - GPU-resident model weights
          - graph and activation buffers
          - runtime workspace
          - speculative-decoding state
          - safety margin

On a display-attached 16 GB card, do not plan around all 16 GiB being available. Reserve at least several hundred MiB for the desktop and driver, then leave another margin for workload-dependent buffers; 1.0 to 1.5 GiB of total breathing room is a reasonable starting assumption, but your logs are the authority.

Suppose a GGUF model occupies 10.8 GiB on the GPU and runtime overhead peaks near 1.2 GiB. After a 1 GiB safety margin, only about 3 GiB remains for KV, so the representative model fits roughly 45K tokens with Q8_0 or 87K with Q4_0, before engine-specific overhead.

That does not automatically make Q4_0 the right choice. If long-context accuracy drops on your workload, a smaller or more aggressively quantized model with a Q8_0 cache may be better than larger weights paired with a fragile cache. Measured anchors for exactly this arithmetic live in the 16 GB VRAM llama.cpp benchmark tables, where VRAM per model is recorded at 19K, 32K, and 64K context. For a wider survey of which model sizes and quant levels behave well under Ollama on the same class of card, see Comparing LLMs performance on Ollama on 16GB VRAM GPU.

Calculate the KV Cache Budget for Your Model

The following Python snippet estimates a conventional full-attention GQA cache. Replace the geometry with values from the model configuration or GGUF metadata.

def kv_gib(tokens, layers, kv_heads, head_dim, bytes_per_value, sequences=1):
    total = (
        sequences
        * tokens
        * layers
        * 2
        * kv_heads
        * head_dim
        * bytes_per_value
    )
    return total / (1024 ** 3)

model = {
    "layers": 32,
    "kv_heads": 8,
    "head_dim": 128,
}

types = {
    "f16": 2.0,
    "q8_0": 34 / 32,
    "q4_0": 18 / 32,
}

for tokens in (32768, 65536, 131072):
    row = {
        name: round(kv_gib(tokens=tokens, bytes_per_value=size, **model), 2)
        for name, size in types.items()
    }
    print(tokens, row)

The Q8_0 and Q4_0 ratios include simple block metadata, which is why they are slightly larger than exactly one byte and half a byte per value. The runtime's startup report remains more accurate because it knows model-specific cache layouts.

When This Formula Is Wrong: Hybrid and Sliding-Window Architectures

Do not force hybrid architectures into the conventional GQA equation. Sliding-window layers retain only a recent window, shared KV layers reduce duplication, recurrent layers may carry fixed-size state, and multi-head latent attention stores a compressed representation rather than per-head K/V tensors — the MLA case above being the most dramatic example.

Modern engines increasingly manage these mixed layouts explicitly. Use the formula to explain the dominant terms, then confirm the allocation reported by the exact engine build and backend you plan to deploy.

llama.cpp: Direct Control of K and V Precision

llama.cpp exposes separate --cache-type-k and --cache-type-v options in its current argument parser. This is the most useful local-inference interface when you need to trade cache precision against context capacity instead of accepting one global preset. If you need the surrounding install and serving setup first, the llama.cpp guide covers llama-cli, llama-server, and the key VRAM flags.

A conservative 64K single-user configuration looks like this:

./llama-server \
  --model /models/model.gguf \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 1024 \
  --ubatch-size 256

Flag syntax and backend support change quickly, so run llama-server --help for the installed build. More importantly, inspect the startup log: it should show the intended context, cache types, GPU offload, and the allocated K and V buffers.

Which llama.cpp Cache Types to Try

Start with Q8_0 for both K and V. It roughly halves KV memory relative to F16, and independent perplexity tests on 20B-plus models (Qwen3.6-27B, Nemotron-30B) show the aggregate quality delta from F16 is within measurement noise — a much less dramatic gamble than moving directly to Q4_0, which the same tests showed collapsing decode speed and accuracy at long context on smaller models.

If Q8_0 does not fit, test Q8_0 keys with Q4_0 values before quantizing both sides to Q4_0. This ordering has research backing, not just folklore: controlled bit-allocation studies on Llama, Phi-4, Qwen3, and Mistral checkpoints found that key tensors are consistently two to ten times more sensitive to quantization error than value tensors, and that giving keys the larger bit budget (for example 4-bit keys with 2-bit values) recovers up to 94–98% of full-precision accuracy — while the inverted split (2-bit keys, 4-bit values) can lose 30 percentage points on tasks like GSM8K. Keys determine which earlier tokens attention actually matches, so protecting them first is the architecturally sound choice, not just the safer-sounding one.

Configuration	Memory	Quality risk	Recommendation
F16 K and V	Highest	Lowest	Baseline when it fits
Q8_0 K and V	About half of F16	Low but not zero	Default 16 GB starting point
Q8_0 K, Q4_0 V	Between Q8 and Q4	Moderate	Useful second step
Q4_0 K and V	About one quarter of F16	Highest	Validate at target depth

One caveat worth internalising: "low quality risk" on aggregate benchmarks does not mean zero risk at the token level. A controlled test that held Flash Attention constant and only changed KV precision under greedy (deterministic) decoding found that Q8_0 cache changed the exact generated text on the large majority of prompts, and Q4_0 changed it on essentially all of them — once one token flips, the rest of the continuation can diverge. Perplexity and downstream-task scores can look fine on average while individual outputs still differ from the F16 baseline. If your application needs byte-for-byte reproducibility (regression tests, cached responses, deterministic agents), treat any KV quantization as a behavioral change, not just a memory optimization, and validate against your own fixed prompt set.

Quantized V cache may require Flash Attention or a compatible backend path. A server that silently falls back to another type invalidates the experiment, which is why startup logs matter more than copied command lines.

Context, Parallel Slots, and Unified Cache

--ctx-size describes an engine capacity, not a guarantee that every parallel slot receives that many tokens independently. Cache management has evolved in llama.cpp, including unified-cache behavior, so test the exact build rather than relying on an older rule that simply divides context by slot count.

The capacity equation still survives implementation changes: simultaneous unique tokens need storage somewhere. If two agent sessions may each reach 48K, budget for close to 96K live tokens unless the workload shares prefixes or tolerates eviction and recomputation.

Batch Size Does Not Shrink Stored KV

--batch-size and --ubatch-size affect prompt processing and temporary memory. Lowering them can rescue a large prefill from an activation-memory spike, but it does not change the persistent bytes required for each retained token.

This distinction explains a common failure pattern: the model starts and an empty request works, but a 60K prompt fails during ingestion. Reduce the micro-batch to diagnose the transient peak; reduce context, cache precision, parallelism, or weight residency to change persistent capacity.

vLLM: Paged Capacity Is Still Capacity

vLLM approaches the problem as a serving engine. It profiles available memory, reserves a KV-cache pool, and allocates cache in blocks so concurrent sequences do not each require one large contiguous region. If you are deciding whether to move to vLLM in the first place, the Ollama to vLLM migration guide covers the workload signals; here the question is purely how much cache the pool can hold, and the vLLM quickstart covers install and general serving flags beyond the capacity levers below.

PagedAttention reduces fragmentation and waste around variable sequence lengths — paged allocation removes fragmentation, not per-token storage cost, so one unique 128K request still needs enough blocks for its KV state.

The official vLLM memory-conservation guide recommends limiting max_model_len and max_num_seqs when memory is tight, and notes that CUDA graphs consume additional GPU memory. On a 16 GB card, both settings should be intentional rather than inherited from a model's maximum configuration.

A focused single-sequence server might start here:

vllm serve MODEL_ID \
  --max-model-len 65536 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching

Not every 16 GB GPU, model, quantization method, or attention backend supports that exact combination. Treat it as a configuration shape: constrain length and concurrency, reserve headroom, select a supported cache dtype, and validate the initialization report.

FP8 KV Cache in vLLM

The current vLLM quantized KV-cache documentation supports FP8 cache formats on compatible CUDA and ROCm paths. FP8 approximately halves raw cache storage relative to BF16 or FP16 and can therefore increase token capacity or concurrency.

Scaling matters. The documentation distinguishes default scales, warm-up calculation, and dataset calibration, and recommends dataset-based calibration for the highest accuracy; simply setting FP8 with scale 1.0 is convenient but not automatically the most reliable quality choice.

Prefix Caching Is a Reuse Optimization

Automatic prefix caching lets a new request reuse KV blocks for an identical cached prefix. It is excellent for repeated queries over the same long document, shared system prompts, and multi-round conversations because it avoids recomputing the matching prefill.

It does not make a unique long request smaller, and it does not accelerate generation of new tokens. The vLLM prefix-caching documentation explicitly limits the benefit to shared-prefix prefill work.

GPU Memory Utilization Is Not Free Memory

Raising --gpu-memory-utilization gives vLLM a larger reservation target, but it does not create VRAM. Pushing it too close to 1.0 can leave insufficient room for the display, another process, changing activation peaks, or non-PyTorch allocations.

Begin around 0.88 to 0.92 on a dedicated 16 GB GPU, inspect the profile, and increase only if the workload remains stable. If initialization succeeds but real prompts fail, reduce batched tokens, sequence concurrency, CUDA graph capture, or the maximum context before assuming the allocator is broken.

Ollama: Easier Controls, Less Granular Diagnosis

Ollama deliberately provides a smaller operational surface. Its current context-length documentation defaults GPUs below 24 GiB to 4K context, recommends at least 64K for agent and coding workloads, and warns that larger context consumes more memory.

Set the server-wide default and confirm the loaded model like this:

OLLAMA_CONTEXT_LENGTH=65536 ollama serve

ollama ps

You can also set num_ctx per request or model. ollama ps is important because its PROCESSOR and CONTEXT columns reveal whether the model remained fully on the GPU and whether the requested context was actually allocated. Be aware that the scheduling behaviour behind those numbers changed between Ollama versions; my comparison of Ollama v0.12.1 memory allocation shows the new scheduler pushing some models further to CPU on a 16 GB card, so pin the version you measured.

Quantized KV Cache in Ollama

Ollama exposes OLLAMA_KV_CACHE_TYPE with f16, q8_0, and q4_0 choices in its current FAQ. Quantized KV requires Flash Attention, which Ollama uses automatically on supported backends or can be requested with OLLAMA_FLASH_ATTENTION=1.

A 16 GB long-context service can therefore be started as:

OLLAMA_CONTEXT_LENGTH=65536 \
OLLAMA_FLASH_ATTENTION=1 \
OLLAMA_KV_CACHE_TYPE=q8_0 \
OLLAMA_NUM_PARALLEL=1 \
ollama serve

Q8_0 is Ollama's recommended alternative to F16. The FAQ warns that Q4_0 can produce a more noticeable quality loss, especially at higher context, so it should be a measured fallback rather than an automatic 16 GB preset.

Ollama Parallelism Multiplies the Context Budget

Ollama documents a particularly clear rule: required memory scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH. Four parallel requests at a 32K setting can imply a 128K aggregate context allocation for that model.

For a personal agent on 16 GB, keep OLLAMA_NUM_PARALLEL=1 until one long session is stable. Queueing a second request is usually preferable to pushing the first model partly onto the CPU and making both requests slow. The queueing, 503, and model-unloading mechanics behind that choice are documented in how Ollama handles parallel requests.

CPU Offload: A Valid Escape Hatch With a Price

Moving some model layers or KV state into system RAM can turn an allocation failure into a working process. It also places PCIe bandwidth and host-memory latency in the decode path, where each generated token may pay the cost. The lane and generation evidence for when PCIe actually bites is in LLM Performance and PCIe Lanes.

Offload can be sensible for occasional batch work, but it is rarely the best default for an interactive coding agent. First compare a smaller weight quantization, Q8 KV, reduced concurrency, and a realistic context cap; use offload when capacity matters more than latency.

Watch for the cliff rather than the average. A server may decode quickly at 8K, then slow severely after part of the working set spills, so benchmark at 32K, 64K, and the intended maximum rather than reporting only an empty-context token rate.

Sliding-Window and Adaptive KV Caches

Sliding-window attention changes the budget by retaining only a recent window for selected layers. Hybrid models may combine those layers with occasional global attention or recurrent state, making a flat full-context calculation substantially overestimate or misplace memory.

The optimization is part of the model architecture, not a generic switch that can be applied without consequences. An engine must understand the layer pattern, eviction rules, positions, and any global tokens correctly.

What Adaptive KV Tries to Improve

Experimental forks go further by choosing cache precision or layout per layer and context depth. The goal is attractive: preserve higher precision where it matters, compress less sensitive layers, and change the mix before VRAM pressure causes a hard spill — the same key-sensitivity-over-value-sensitivity finding described above is exactly the kind of signal an adaptive allocator would want to exploit automatically instead of leaving it to manual --cache-type-k/--cache-type-v tuning.

One August 2026 downstream project, llama.cpp-adaptive-turboquant, reports an automatic selector for several layer-adaptive modes and publishes long-depth tests on an RTX 5080 16 GB. Those numbers are author-reported results from a specialized fork, not evidence that upstream llama.cpp behaves the same way.

Why It Is Still Experimental

The fork combines custom cache types, CUDA kernels, model-specific paths, and toolchain constraints. That is far more code to trust than switching upstream cache storage from F16 to Q8_0.

Use such a fork only when upstream cannot meet a real requirement and you can reproduce quality, stability, and speed on your model. Record the commit and CUDA version, because a result attached only to a project name is not reproducible.

A Fair Adaptive-Cache Test

Compare the fork against an upstream Q8_0 baseline with the same GGUF, prompt, sampler, context depth, and output length. Measure startup VRAM, peak prefill VRAM, prompt processing speed, decode speed, and a quality task that actually requires evidence from the oldest part of the context.

Do not accept a successful allocation as a complete result. A cache can fit 128K and still lose early facts, corrupt output late in the sequence, or decode too slowly to be useful.

A Worked 16 GB Tuning Procedure: One Variable at a Time

The fastest route to a stable configuration is to change one memory dimension at a time. Randomly altering cache type, batch size, layer offload, parallelism, and context together produces a working command with no explanation.

flowchart TD
    A["Step 1: load at 8K context, one sequence, record warm-up VRAM"] --> B{"Weights + runtime under about 14.5 GiB?"}
    B -- "no" --> C["Pick a smaller quant or model, repeat Step 1"]
    C --> A
    B -- "yes" --> D["Step 2: F16/BF16 KV quality baseline, save task outputs"]
    D --> E["Step 3: switch to Q8_0 / FP8 KV with Flash Attention"]
    E --> F["Step 4: stage context up - 32K, 64K, 96K, 128K"]
    F --> G{"Failure during prefill?"}
    G -- "yes" --> H["Step 5: reduce batch / ubatch size"]
    H --> F
    G -- "no" --> I{"Failure only with concurrent requests?"}
    I -- "yes" --> J["Step 5: reduce parallel slots / max-num-seqs"]
    J --> F
    I -- "no" --> K["Only now: mixed Q8/Q4, full Q4, offload, adaptive fork"]

Step 1: Establish the Weight Floor

Load the model at 8K context, one sequence, and the intended GPU offload. Record process VRAM after warm-up and verify that no layers unexpectedly moved to the CPU.

If the weights and runtime already consume more than about 14.5 to 15 GiB, long context has no healthy margin. Choose a smaller weight quant or model before tuning the cache.

Step 2: Measure F16 or BF16 KV as the Quality Baseline

Run the smallest context that supports your test and retain the default high-precision cache. Save outputs from retrieval, code editing, tool selection, and long-instruction tasks.

This baseline tells you whether later errors come from cache quantization. Without it, a chat-template problem or weak model can easily be blamed on Q4 KV.

Step 3: Move to Q8 or FP8

Enable Flash Attention where required, select Q8_0 in llama.cpp or Ollama, or a supported FP8 mode in vLLM. Repeat the same prompts at the same token depths and confirm the log shows the intended cache type.

For many 16 GB deployments, this is the useful stopping point. It roughly doubles raw KV capacity without making cache compression the most aggressive quantization in the stack.

Step 4: Raise Context in Stages

Test 32K, 64K, 96K, and 128K instead of jumping directly to the advertised maximum. At each stage, record prompt-processing tokens per second, decode tokens per second, peak VRAM, and whether evidence near the start can still be recovered.

Long-context decode often slows even after memory fits because attention reads more cached state. Capacity and performance are separate axes.

Step 5: Tune Transient Memory

If failure occurs during prefill rather than initialization, reduce micro-batch or maximum batched tokens. If failure occurs only with simultaneous requests, reduce sequence concurrency or parallel slots.

Only after those controls are understood should you try mixed Q8/Q4 cache, full Q4 cache, CPU offload, or an adaptive fork. Keep the upstream Q8 run as the comparison baseline. If you later add speculative decoding or MTP, remember its draft buffers are another line in the budget equation, not free speed — the speculative decoding guide covers the mechanics and their VRAM cost, and my Qwen 3.6 27B and 35B MTP vs Standard benchmark shows exactly how much context an MTP head's extra state can cost on a 16 GB card.

What to Record in a Long-Context Benchmark

A single tokens/s figure hides the exact problem this article is trying to solve. Long-context testing should preserve enough detail for another operator to reproduce the memory boundary.

Field	Why it matters
GPU and usable VRAM	Display use and other processes change the budget
Engine version or commit	Cache behavior and flags evolve quickly
Driver, CUDA, ROCm, or Vulkan version	Determines backend and kernel behavior
Exact model and weight quant	Defines weight residency and architecture
K and V cache types	Defines persistent cache size and quality risk
Context capacity and prompt depth	Allocation is not the same as actual depth
Parallel sequences	Multiplies or shares cache demand
Batch and micro-batch	Affects prefill peaks and speed
Prompt processing speed	Exposes long-prefill usability
Decode speed at each depth	Exposes cache-bandwidth slowdown
Peak VRAM and CPU offload	Distinguishes fit from spill
Long-context quality result	Detects compression or position failures

Use nvidia-smi sampling or the equivalent vendor tooling during both prefill and decode. The engine's allocation report is necessary, but peak device memory during a real prompt is the number that decides stability.

Common KV Cache Mistakes on 16 GB GPUs

Treating 128K Support as a Hardware Promise

The context field in a model configuration is an architectural ceiling. It says nothing about the memory left after loading a particular quantization on a particular engine.

Calculate the cache and verify the runtime. Marketing-sized context without a VRAM budget is merely an OOM delayed until the first serious prompt.

Quantizing Weights but Forgetting KV

A 4-bit GGUF reduces model weights, not an F16 KV cache. At long context, the cache can erase the entire saving and eventually exceed the weight footprint.

Report both quantizations. Q4_K_M model, Q8_0 KV is meaningful; 4-bit model is incomplete.

Assuming Paged Attention Compresses Tokens

Paging improves allocation and sharing behavior. It does not change the tensor precision or remove the KV state required by one unique sequence.

Use paged allocation to serve variable workloads efficiently. Use cache precision, model architecture, context caps, and concurrency limits to control capacity.

Assuming Prefix Caching Helps Every Long Prompt

Prefix caching saves repeated prefill computation when requests share an exact prefix. A one-off 100K repository dump receives no magical memory discount simply because prefix caching is enabled.

It is a workload optimization, not a substitute for the budget equation. Measure hit rate and retained cache pressure in multi-user serving.

Using Q4 KV Without a Quality Test

Low-bit cache can fail subtly. The model still writes fluent text, but attention over distant evidence, exact names, tool arguments, or code dependencies may degrade — and as the token-divergence research above shows, even the "safe" Q8_0 setting is not guaranteed to reproduce the exact F16 output under deterministic decoding, only to preserve accuracy on aggregate.

Test the target task at the target depth. Short chat benchmarks are nearly useless for validating a long-context cache.

Leaving Parallelism on Auto

An engine may choose concurrency that is sensible for throughput but impossible for your long-context target. On 16 GB, one deep sequence and several short sequences are fundamentally different workloads.

Set the limit explicitly, then raise it with measured traffic. Otherwise a second request can turn a stable 64K configuration into an allocation or latency surprise.

Recommended 16 GB Profiles

These profiles are starting positions, not universal presets. A model with unusual KV geometry — an MLA or hybrid sliding-window design in particular — can be much cheaper or more expensive than the conventional GQA example.

Interactive Coding Agent

Use one sequence, 48K to 64K context, Q8 cache, Flash Attention, and full GPU weight residency if possible. This profile favors predictable latency and good cache precision over an impressive but rarely useful maximum.

Enable prefix reuse when the engine supports it because coding turns often share a large repository or conversation prefix. Still compact tool output and old transcripts; cache engineering does not make irrelevant tokens valuable.

Long-Document Analysis

Use a smaller model with 64K to 128K capacity, Q8 or calibrated FP8 cache, and repeated-prefix caching when multiple questions target the same document. Measure time to first token because prefill may dominate even when decode remains acceptable.

If only one question will be asked, retrieval or chunked summarization may be faster and more reliable than forcing the entire corpus through a 16 GB card. Long context is a tool, not a replacement for information architecture.

Small Multi-User Server

Cap per-request context and total active sequences rather than advertising the model maximum to every client. vLLM's paged allocation is useful here, while Ollama and llama.cpp also require explicit attention to aggregate live tokens.

Prefer queueing over uncontrolled spill. A slower admission policy is less damaging than every request suddenly crossing PCIe during decode.

Final Recommendation for 16 GB Long Context

For long context on 16 GB, Q8 KV and one active sequence are the right baseline. They expose the real limit without making low-bit cache quality, parallel allocation, and offload latency fail at once.

Calculate from attention geometry, subtract weights and runtime overhead, and then confirm the result in engine logs and peak-memory measurements. If 128K still does not fit, a smaller model is often the cleanest optimization; if it fits but crawls, reducing context is often the honest one.

Paged attention, prefix caching, sliding windows, and adaptive precision all solve useful but different problems. The winning setup is the one that remains on the GPU, retrieves old evidence correctly, and sustains acceptable decode speed at the context depth you actually use.

References

vLLM: conserving GPU memory
vLLM: quantized KV cache (FP8)
vLLM: automatic prefix caching
Ollama: context length documentation
Ollama FAQ: OLLAMA_KV_CACHE_TYPE and Flash Attention
llama.cpp-adaptive-turboquant — experimental layer-adaptive cache fork (author-reported results)
Raschka, S. "Multi-Head Latent Attention (MLA)." LLM Architecture Gallery. https://sebastianraschka.com/llm-architecture-gallery/mla/
"Decoding Multi-Head Latent Attention: The KV Cache Memory Bottleneck, Solved." Vizuara. https://vizuara.substack.com/p/decoding-multi-head-latent-attention
"Quantize What Counts: Bit Allocation Insights Informed by Spectral Gaps in Keys and Values." arXiv:2502.15075. https://ar5iv.labs.arxiv.org/html/2502.15075
v-code01. "kvdivergence — does KV cache quantization change the generated text under greedy decoding?" GitHub. https://github.com/v-code01/kvdivergence
Comparing LLMs performance on Ollama on 16GB VRAM GPU
Qwen 3.6 27B and 35B MTP vs Standard on 16GB GPU