DEV Community: Rost

PARA Method for Engineers: Organize Knowledge by Action

Rost — Sun, 21 Jun 2026 12:18:24 +0000

Organizing notes by topic sounds logical until you have notes on PostgreSQL in five different folders and cannot find the one that matters for today's problem.

The issue is not discipline. The issue is that topic-based organization asks the wrong question. "What is this about?" is useful for libraries. For engineers, the better question is "What am I doing with this?" That is the premise of PARA.

PARA is a simple four-bucket system created by Tiago Forte as the organizational backbone of his Building a Second Brain framework. The idea is that all information can be sorted into four categories: Projects, Areas, Resources, and Archives. Each category represents a different level of actionability, and that distinction drives where every note lives.

This guide applies PARA to engineering work specifically — codebases, documentation, learning material, and the tension between active project work and long-term reference.

The Problem With Topic-Based Organization

Most engineers organize knowledge the way they organize code: by domain.

databases/
  postgresql/
  redis/
api/
  rest/
  graphql/
devops/
  kubernetes/
  terraform/

That structure makes sense when you are browsing. It breaks down when you need something for a specific task. You remember a useful note about database migration safety, but it could be in databases/postgresql/, devops/deployments/, api/versioning/, or nowhere because you saved it somewhere temporary.

Topic folders force you to decide where knowledge belongs before you understand its context. PARA delays that decision — instead of asking what something is about, it asks what you are currently doing with it.

The Four Buckets

Projects

A project is active, time-bound work with a defined outcome.

For engineers, projects are things like:

Migrate billing service to queue v2
Upgrade PostgreSQL from 14 to 16
Write architecture decision record for auth service redesign
Implement rate limiting on public API
Publish article about distributed tracing

Every project has a completion state. When you finish, the project moves to Archives. When you are not actively working on it, it is not a project.

The key constraint: a project note should only contain what you need for that project. Reference material belongs in Resources. Reusable concepts belong in your Zettelkasten or personal notes. Project notes are working documents, not knowledge stores.

Areas

An area is an ongoing responsibility without a deadline.

For engineers, areas include:

System architecture
Infrastructure reliability
Code review quality
Professional development
API design standards
Security posture
On-call responsibilities
Mentoring

Areas do not finish. You are always responsible for infrastructure reliability. You always care about your professional development. The difference between a project and an area is that areas do not have exit criteria — they are things you maintain, not things you complete.

A useful rule: if you can imagine shipping it or closing the ticket, it is a project. If it is just part of what your role means, it is an area.

Resources

Resources are reference material you collected because it might be useful later.

For engineers:

API documentation bookmarks
Cheat sheets
Benchmark results
Architecture diagrams for third-party systems
Conference talks you want to revisit
Library documentation
Research papers
Interesting blog articles

Resources have no active home in your current work. They are collected because you expect to need them eventually. The important discipline here is that resources should not masquerade as projects. A collection of Kubernetes documentation is a resource. A running task to "learn Kubernetes for the platform migration" is a project.

PARA in Practice for Engineers

Here is a concrete example of what an engineer's PARA structure might look like in Obsidian:

Projects/
  billing-queue-migration/
  postgresql-16-upgrade/
  rate-limiting-rfc/
  blog-distributed-tracing/

Areas/
  architecture-standards/
  infrastructure/
  on-call-runbooks/
  career-development/

Resources/
  api-references/
  database-cheatsheets/
  benchmark-results/
  conference-notes/

Archives/
  2025-q4-projects/
  deprecated-services/
  old-runbooks/

The folder structure itself is not sacred. What matters is the discipline of placing notes in the right category based on their relationship to your current work.

Mapping a Typical Engineer's Knowledge

Many engineers start with an undifferentiated pile of notes. Migrating to PARA requires a single audit pass:

Projects — anything with a ticket, a deadline, or a deliverable you are currently working toward.

Areas — recurring responsibilities that define your role.

Resources — reference material you collected without a specific project in mind.

Archives — everything else.

A working rule: when in doubt, Archive it. You can always retrieve it later. An overcrowded Projects folder is more damaging than an underused Archive.

PARA and Zettelkasten: A Practical Hybrid

PARA and Zettelkasten are often compared as competing systems. They are not competing. They solve different problems.

Zettelkasten is for ideas. It captures atomic concepts, links them by meaning, and lets understanding emerge from the connections. Zettelkasten notes are not tied to projects — they belong to no active bucket. A note about idempotency applies to ten different projects, past and future.

PARA is for action. It organizes working context around what you are actively doing, responsible for, or collecting for later use.

A practical hybrid:

Projects/
  billing-queue-migration/
    migration-plan.md
    open-questions.md
    → links to Zettelkasten: [[Idempotency keys turn retries into safe operations]]
    → links to Zettelkasten: [[Outbox pattern separates persistence from delivery]]

Areas/
  architecture-standards/
    current-adr-index.md
    → links to Zettelkasten: [[Database constraints are concurrency control]]

Resources/
  benchmark-results/
    q1-2026-postgres-benchmarks.md

In this model, PARA folders hold working documents and context. Zettelkasten notes hold reusable knowledge. Project notes link to Zettelkasten concepts — the project uses the concept without owning it.

This is more durable than trying to make PARA do the job of Zettelkasten. Projects end. Concepts stay.

Common Failures

Over-Archiving

Some engineers use Archives as a dump for anything they feel guilty discarding. When Archives become large and unsorted, they lose their value. Archives should contain completed work in reasonable shape, not a graveyard of unsorted notes.

A periodic archive sweep — quarterly works well — keeps it manageable. Delete duplicates. Consolidate. Ask whether the old project note still contains anything worth keeping as a Resource or Zettelkasten note before archiving it.

Areas Becoming Dumping Grounds

When Areas grow without pruning, they start to look like a topic-based folder system. An Area called databases/ that contains unsorted notes from three years is not a responsibility — it is a pile.

Keep each Area tight. An area should represent something you are actively accountable for, not a topic you are broadly interested in. Interest goes into Resources. Accountability goes into Areas.

Resources Growing Without Review

Resources are easy to collect and easy to forget. A bookmark dump in Resources/ with 400 unsorted links is harder to use than a bookmark manager. Resources should be curated lightly — remove outdated material, keep the signal.

Skipping the Weekly Review

PARA works best with a weekly ten-minute review of your Projects folder. For each active project:

Is this still active?
What is the next concrete action?
Is there anything to move to Archives?

Without that review, Projects accumulate stale entries and the system loses its value as a current view of your work.

Implementation in Obsidian

Obsidian is a natural fit for PARA because folders map directly to the four buckets and Dataview queries can surface project status automatically.

A basic setup:

vault/
  ├── Projects/
  ├── Areas/
  ├── Resources/
  ├── Archives/
  └── Zettelkasten/     ← concept notes, linked freely

A simple Dataview query to surface active project notes:

LIST FROM "Projects"
WHERE !contains(file.path, "Archives")
SORT file.mtime DESC

Tags can mark status without moving files:

tags: [project, active]
tags: [project, paused]
tags: [project, done]

When a project completes, tag it done, then move the folder to Archives/YEAR-QN/. Simple, auditable, reversible.

Implementation in Plain Files

You do not need Obsidian. PARA works equally well in a Git repository with plain Markdown:

knowledge/
  projects/
    2026-billing-migration/
      README.md
      migration-plan.md
      decisions.md
  areas/
    architecture/
      adr-index.md
  resources/
    databases/
      postgres-16-release-notes.md
  archives/
    2025/
      feature-x-launch/

Git gives you history, diff, search, and portability. That is often more than enough for a personal system.

When PARA Makes Sense

PARA is well suited when:

You juggle multiple active projects at the same time
You need to quickly find what relates to today's work
You want a system that is folder-friendly and tool-agnostic
You combine it with a Zettelkasten or concept-note layer for reusable ideas

PARA is less useful when:

You work on a single long-running project with no clear buckets
You are primarily doing research-oriented work with no active deliverables
You prefer emergent structure over explicit categorization

For engineers doing a mix of active project work and long-term learning, PARA and Zettelkasten together cover most cases: PARA for context, Zettelkasten for thinking.

Decision Framework

When a new note arrives, ask these questions in order:

Is this tied to something I am actively working toward? → Projects
Is this part of an ongoing responsibility I own? → Areas
Is this reference material I might need later? → Resources
Is this finished or inactive? → Archives
Is this a reusable concept or idea not tied to any project? → Zettelkasten

That is the full decision tree. Five options. One rule per option. It takes about ten seconds per note.

Final Thoughts

PARA works because it matches how engineers actually use knowledge — not for browsing, but for acting. You do not open your notes to see what is in databases/. You open them because you are working on a specific problem right now, and you need the relevant material to surface quickly.

The discipline of separating active projects from reference material, and both from finished work, reduces the cognitive overhead of maintaining a personal knowledge base. In combination with a personal knowledge management foundation and a Zettelkasten for concept-level notes, PARA gives you the organizational backbone that keeps everything findable when it matters.

Start with one folder per bucket. Run one audit to sort your existing notes. Review Projects weekly. The rest will follow naturally.

Evergreen Notes: Write Notes That Compound Over Time

Rost — Sun, 21 Jun 2026 12:18:21 +0000

Most engineering notes are written once and forgotten. You capture something during a debugging session, paste it into a doc, and rediscover it two years later with no context for why it mattered.

The problem is not effort. Engineers write constantly — code comments, Slack messages, Confluence pages, Jira descriptions, pull request explanations, architecture diagrams. The problem is that most of those notes are written for a specific moment and age poorly. They do not compound. They accumulate.

Evergreen notes are the alternative. The idea is simple: write each note so that it stays useful indefinitely, improves when you revisit it, and connects to other notes in a way that makes the whole system more valuable over time.

The term was popularized by researcher Andy Matuschak, whose own public notes demonstrate the idea at scale. For engineers, the principle has direct applications in technical writing, documentation, architecture decisions, and the long-term capture of hard-won lessons.

What Makes a Note Evergreen

Atomic

An evergreen note contains one idea. Not one topic — one idea.

A note called "PostgreSQL" is not evergreen. It is a container waiting to be filled. A note called "Partial indexes reduce write overhead when queries target a small subset" is evergreen. It states a specific, portable claim.

The atomic constraint is important because it controls reuse. A container note can only be linked as a vague topic. An atomic note can be linked wherever that specific idea applies — in a discussion of query optimization, in a comparison of indexing strategies, in a project note about a specific performance problem.

Standalone

An evergreen note should be understandable without its original source.

That means writing in your own words. A note that says "See the linked article — good stuff on caching" is not evergreen. A note that says "Write-through caching updates the cache synchronously with the database on every write, improving read consistency at the cost of higher write latency" is evergreen. You can read it a year later without chasing the original source.

This is harder than it sounds. Writing a standalone note requires actually understanding what you read, not just tagging it. That processing step is where most of the learning happens.

Evolving

Evergreen notes improve over time rather than going stale.

A fleeting note has a lifecycle: you write it, it serves a moment, it becomes irrelevant. An evergreen note should be worth revisiting and refining six months or two years later. You might add a counterexample, update it with a production experience, link it to a new pattern, or simply rewrite it more precisely.

The word "evergreen" is intentional: these notes do not die after harvest. They persist and improve.

Linked

Evergreen notes connect to other notes rather than sitting in isolation.

A standalone note about write-through caching connects naturally to notes about read-heavy workloads, cache invalidation, eventual consistency, and database write performance. Each link makes both notes more useful — the connection surfaces context that neither note contains alone.

The linking habit is what turns a collection of individual insights into a network of connected understanding.

Note Types and When to Use Each

Understanding evergreen notes requires understanding what they are not.

Fleeting notes are temporary captures. A line scribbled during a debugging session, a bookmark to revisit, a question to follow up on. Fleeting notes serve a moment. They should be processed quickly and either discarded or promoted into something more durable. Most fleeting notes never become evergreen notes, and that is fine.

Literature notes are summaries of external sources — a documentation page, a postmortem, a book chapter, a conference talk. Literature notes preserve what a source said. They are a step toward understanding, not understanding itself. A literature note says "this source claims X." An evergreen note says "I believe X for these reasons."

Evergreen notes synthesize what you have come to understand. They live at the output of the learning process, not the input.

Note type	Purpose	Lifespan	Example
Fleeting	Quick capture	Hours to days	"Look into why Postgres vacuum missed this row"
Literature	Source summary	Medium term	"Redis docs say AOF fsync default is 1s"
Evergreen	Portable idea	Years	"Fsync-on-write durability trades throughput for crash safety"

Writing Evergreen Technical Notes

The structure of a good evergreen technical note follows a simple logic: claim, evidence, implication.

# Write-through caching improves read consistency at the cost of write latency

Write-through caching updates the cache at the same time as the underlying store
on every write. Every read hits fresh data because the write path ensures
consistency before the write is acknowledged.

The tradeoff is write latency — every write now requires two operations (store
and cache) to complete before the caller receives a confirmation.

This pattern suits read-heavy workloads where cache staleness has real
business impact, such as product inventory counts or user settings.

Links:
- [[Read-through caching shifts cache population to read time]]
- [[Cache invalidation is a coordination problem]]
- [[Write-behind caching trades consistency for write throughput]]

That note is useful without the source. It states the claim, explains the tradeoff, gives a context where it applies, and links to related ideas.

What to Avoid

Time-sensitive references age badly. "As of Postgres 14, this behavior works this way" is a literature note, not an evergreen note. Write the principle instead: "The planner skips index scans when estimated row count exceeds a threshold relative to table size." That claim survives version changes even if the threshold changes.

Tool-specific commands without context are snippets, not notes. A note that is just a kubectl command copied from a StackOverflow answer is not evergreen. A note about why that command works — what Kubernetes resource it affects and what problem it solves — has a chance.

Assumptions about reader knowledge degrade fast. Write as if explaining to a competent colleague who is not inside your current context.

Good Candidates for Evergreen Notes in Engineering

Almost any hard-won lesson with broad applicability is a good candidate:

Architecture tradeoffs and the reasoning behind decisions
Debugging patterns that apply across systems
API design rules and their edge cases
Performance characteristics with real-world numbers attached
Security assumptions that turned out to be wrong
Test strategy lessons from projects where the approach failed
Deployment constraints that changed how the team worked

The common thread: specific enough to be actionable, general enough to apply more than once.

The Evergreen Workflow

Step 1: Capture Fleeting Notes

Capture quickly without overthinking. The goal is not to produce an evergreen note in the moment — it is to preserve the raw material for one.

During a debugging session:

Found that the cache was returning stale user permissions after role changes.
The TTL was 5 minutes but the role update was immediate.
Need to think through how to handle this — invalidation on write?
Or shorter TTL? Or event-driven update?

That is a fleeting note. It is not an evergreen note, but it contains the seeds of several.

Step 2: Process Into Evergreen Notes Within 48 Hours

Processing is where the value appears. Take the raw capture and extract the ideas that are worth preserving.

From that debugging note, you might write:

# Role-based cache entries require invalidation on write, not just TTL expiry

When cached data encodes permissions or roles, TTL-based expiry is not safe.
A user whose role is downgraded keeps elevated permissions until the TTL expires.
Write-time invalidation — or event-driven cache updates on role change — is required
for correctness in permission-sensitive caches.

Links:
- [[Cache invalidation is a coordination problem]]
- [[Authorization decisions should not be cached at rest without validation]]

The debugging context is gone. The portable idea remains.

Step 3: Connect to Existing Notes

After writing the note, spend two minutes asking:

What existing note does this relate to?
What concept does this depend on?
What does this extend or contradict?

Add links in both directions. The new note links to existing notes. Existing notes that are now richer for the connection link back.

Step 4: Revisit and Improve

Evergreen notes do not have a single correct state. Every time you encounter the idea again — in a production incident, a design review, a code review comment — consider returning to the note and making it better.

You might:

Add a more concrete example
Update the claim based on new evidence
Remove a caveat that turned out not to matter
Add a link to a new related note
Rewrite the opening sentence for clarity

That cycle of refinement is what makes notes compound rather than decay.

Evergreen Notes and Documentation

There is a useful distinction between personal evergreen notes and team documentation.

Personal evergreen notes are your understanding, written for future you. They can be rough, opinionated, and incomplete. Their value is in being reusable for your thinking.

Team documentation is for shared understanding. It needs accuracy, accessibility, and maintenance ownership.

The two layers complement each other. Your evergreen notes about why a system was designed a certain way can become the raw material for the architecture decision record. Your debugging notes can feed the runbook. Your API design notes can inform the style guide.

The direction of flow is usually: evergreen notes → polished documentation, not the reverse.

Evergreen Notes and RAG Systems

As AI-augmented knowledge tools become more practical, well-written evergreen notes become increasingly valuable as retrieval source material. The retrieval versus representation problem in knowledge management is essentially about quality of source material — and evergreen notes, being atomic, standalone, and written for comprehension, chunk well for vector search.

A Zettelkasten of atomic evergreen notes is a natural foundation for a personal RAG system. The atomic structure aligns with retrieval chunk size. The standalone property means retrieved notes need no additional context to be useful. The linking structure provides graph traversal opportunities beyond keyword search.

This is increasingly relevant for engineers who want to query their own knowledge base with an LLM rather than starting from scratch each time.

Common Pitfalls

Writing Too Broadly

A note that covers an entire topic is not an evergreen note — it is a draft article. If your note is longer than a single screen and covers more than one claim, break it into smaller notes and link them.

Writing Too Narrowly

A note that is too specific to one context has no reuse value. "Fixed the billing service cache bug on 2024-03-14" is a log entry, not an evergreen note. Raise the abstraction level until the idea applies in at least three different contexts.

Confusing "Evergreen" With "Never Changes"

Evergreen does not mean immutable. It means the note remains worth returning to. A note about Go generics written in 2022 is still evergreen if you update it to reflect how patterns evolved in 2024. A note that you never touch because you believe it is permanently correct is a note that will eventually become wrong in silence.

Skipping the Processing Step

The most common failure is treating evergreen notes as a collection target rather than a writing practice. You cannot grow a collection of high-quality atomic notes by saving bookmarks. The evergreen note is not the article you read — it is what you extracted from it in your own words.

Tools

Obsidian

Obsidian is the most popular tool for evergreen notes. Its local Markdown files, bidirectional links, and graph view align well with the practice. A simple structure:

vault/
  fleeting/
    daily/
  literature/
  evergreen/
  maps/       ← index notes for clusters of evergreen notes

The graph view in Obsidian makes link clusters visible — useful for discovering which concepts form natural groups that might become index notes or published articles.

Plain Markdown With Git

A Git repository of Markdown files works well and has no dependency on any specific tool. Standard Markdown links connect notes. Search is handled by your editor or grep. Version history comes from Git.

knowledge/
  evergreen/
    caching/
    api-design/
    performance/
  literature/
  fleeting/

The discipline is the same regardless of tool — one idea per note, written in your own words, linked to related notes.

Starting From Zero

The most useful way to start is not to migrate your existing notes. It is to write one evergreen note today.

Take something you learned in the last week. Write it as a claim. Explain it in your own words in one paragraph. Add links to zero or one related ideas.

That is a complete evergreen note. Repeat once per week for six months and you have a working system.

The compounding effect takes time to become visible. Engineers who maintain evergreen notes for a year often report that their notes start answering questions before they finish asking them — because they have already written the answer in a previous context.

Final Thoughts

The reason evergreen notes work is not that they are better at storage. They are better at thinking. The discipline of writing one portable idea per note, in your own words, with links to related ideas, forces understanding that passive collection does not.

For engineers, this has practical consequences. The notes from a production incident that you process into evergreen format are more useful than the incident log. The design tradeoff you distill into an atomic note is more useful than the architecture diagram. The debugging pattern you generalize from a specific bug is more reusable than the ticket.

Used alongside the PARA method for organizing active work, evergreen notes give you the conceptual layer that PARA does not provide — a growing network of reusable understanding that persists across projects, across roles, and across years.

Cost Optimization for LLM Systems: Where the Money Actually Goes

Rost — Fri, 19 Jun 2026 09:52:51 +0000

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000.

Cost optimization isn't about cutting corners. It's about spending tokens where they matter.

Every token you waste is a token you could have spent on a better answer.

Token budgeting

The simplest way to control costs is to set limits. Per session, per task, or per day.

Strategy 1: Per-Session Budgets

Per-session budgets are straightforward:

class SessionBudget:
    def __init__(self, budget_tokens: int = 10000):
        self.budget = budget_tokens
        self.used = 0

    def allocate(self, tokens: int) -> bool:
        if self.used + tokens <= self.budget:
            self.used += tokens
            return True
        return False

    def remaining(self) -> int:
        return self.budget - self.used

Strategy 2: Per-Task Budgets

Per-task budgets are more useful. Different tasks need different amounts of context:

task_budgets:
  classify:
    max_tokens: 100
    model: qwen2.5-1.5b
  summarize:
    max_tokens: 500
    model: qwen2.5-7b
  code_review:
    max_tokens: 2000
    model: qwen2.5-coder-7b
  reason:
    max_tokens: 4000
    model: qwen2.5-32b

Strategy 3: Adaptive Budgets

Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100:

class AdaptiveBudget:
    def __init__(self):
        self.task_history = {}

    def allocate(self, task_type: str) -> int:
        if task_type in self.task_history:
            return int(self.task_history[task_type] * 1.5)
        return 1000

    def record(self, task_type: str, tokens_used: int):
        if task_type not in self.task_history:
            self.task_history[task_type] = tokens_used
        else:
            self.task_history[task_type] = (
                0.9 * self.task_history[task_type] + 0.1 * tokens_used
            )

The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are.

API vs local inference

Local inference is cheaper at scale. The break-even depends on your hardware and API rates.

Model	API ($/M tokens)	Local cost/hour	Break-even
GPT-4o	$2.50 / $10.00	—	N/A
Claude Sonnet 4	$3.00 / $15.00	—	N/A
Qwen2.5-72B	$0.50 / $2.00	~$0.50	~4 hours/day
Qwen2.5-32B	$0.30 / $1.20	~$0.20	~2 hours/day
Qwen2.5-7B	$0.10 / $0.40	~$0.05	~1 hour/day

The hardware math:

Hardware	Upfront	Monthly electricity	Break-even vs API
RTX 3090 (used)	$600	$15	~4 months
RTX 4090	$1,500	$20	~6 months
RTX 5080	$1,000	$18	~5 months
DGX Spark	$2,000	$30	~8 months

At moderate usage — an hour or more per day — local inference pays for itself. At high usage, the savings are dramatic. The catch is upfront capital. A RTX 5080 is $1,000. An API bill you can pause. Hardware you can't.

Fallback strategies

When your preferred model is too expensive or too slow, fall back to something cheaper. The key is knowing when quality is "good enough."

Strategy 1: Quality-Based Fallback

Quality-based fallback tries models until the output meets a threshold:

class QualityFallback:
    def __init__(self, quality_threshold: float = 0.8):
        self.threshold = quality_threshold
        self.models = [
            {"model": "claude-sonnet-4", "cost": 0.015},
            {"model": "qwen2.5-72b", "cost": 0.002},
            {"model": "qwen2.5-32b", "cost": 0.001},
            {"model": "qwen2.5-7b", "cost": 0.0004},
        ]

    def route(self, prompt: str) -> str:
        for model_config in self.models:
            result = self.call_model(model_config["model"], prompt)
            if self.evaluate_quality(result) >= self.threshold:
                return result
        return self.call_model(self.models[0]["model"], prompt)

The problem is evaluation itself. How do you measure quality without calling another model? Some systems use a small classifier. Others use heuristic checks — length, structure, keyword presence. None of these are perfect.

Strategy 2: Latency-Based Fallback

Latency-based fallback is simpler. Route to the fastest model that meets your time budget:

class LatencyFallback:
    def __init__(self, max_latency: float = 5.0):
        self.max_latency = max_latency
        self.models = [
            {"model": "qwen2.5-1.5b", "latency": 0.5},
            {"model": "qwen2.5-7b", "latency": 2.0},
            {"model": "qwen2.5-32b", "latency": 10.0},
            {"model": "claude-sonnet-4", "latency": 5.0},
        ]

    def route(self, prompt: str) -> str:
        for model_config in sorted(self.models, key=lambda x: x["latency"]):
            if model_config["latency"] <= self.max_latency:
                return self.call_model(model_config["model"], prompt)
        return self.call_model(self.models[0]["model"], prompt)

Caching

Caching is the most underrated cost optimization. Identical prompts happen more often than you think — classification requests, FAQ-style queries, repeated tool calls.

Strategy 1: Prompt Caching

Exact prompt caching is simple:

import hashlib

class PromptCache:
    def __init__(self, max_size: int = 1000):
        self.cache = {}
        self.max_size = max_size

    def get(self, prompt: str) -> str | None:
        key = hashlib.sha256(prompt.encode()).hexdigest()
        return self.cache.get(key)

    def set(self, prompt: str, response: str):
        key = hashlib.sha256(prompt.encode()).hexdigest()
        if len(self.cache) >= self.max_size:
            self.cache.pop(next(iter(self.cache)))
        self.cache[key] = response

Strategy 2: Semantic Caching

Semantic caching is more useful. It catches prompts that are different but mean the same thing:

from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.95):
        self.model = SentenceTransformer('all-MiniLM-L6-v2')
        self.cache = {}
        self.threshold = similarity_threshold

    def get(self, prompt: str) -> str | None:
        prompt_embedding = self.model.encode([prompt])[0]
        for cached_prompt, cached_response in self.cache.items():
            cached_embedding = self.model.encode([cached_prompt])[0]
            similarity = self.cosine_similarity(
                prompt_embedding, cached_embedding
            )
            if similarity >= self.threshold:
                return cached_response
        return None

    def set(self, prompt: str, response: str):
        self.cache[prompt] = response

The threshold matters. 0.95 is aggressive — only very similar prompts match. 0.85 is more forgiving but risks returning wrong answers. Measure your miss rate and adjust.

Response caching for common queries is worth it too. If users ask "what's the weather" or "what time is it" repeatedly, cache the pattern, not just the exact prompt:

class ResponseCache:
    def __init__(self):
        self.common_queries = {
            "what is the weather": "Check weather API",
            "what is the time": "Check system time",
            "who is the president": "Check current president",
        }

    def get(self, query: str) -> str | None:
        query_lower = query.lower()
        for common_query, response in self.common_queries.items():
            if common_query in query_lower:
                return response
        return None

This isn't sophisticated, but it works. Common queries are common for a reason.

When optimization helps

Optimization matters when you're processing high volumes, running mixed workloads, or paying API costs that add up.

It doesn't matter when you're prototyping, using a single model, or processing low volumes. The complexity of budgeting, fallback, and caching isn't worth it for a system that makes 100 requests a day.

Get the basic flow working first. Add optimization when the bill comes in.

Tradeoffs

Strategy	Cost	Quality	Complexity
No optimization	Highest	Consistent	Lowest
Token budgeting	Moderate	Variable	Medium
Fallback models	Low-Medium	Variable	Medium
Caching	Lowest	High (for cache hits)	Medium
Hybrid	Optimized	Optimized	Highest

Production systems usually run hybrid. Budget per session, fall back on quality or latency, cache what you can. The complexity is real, but so are the savings.

Model Routing Strategies — capability-based, cost-aware, latency-aware routing
LLM Guardrails in Practice — input validation, output filtering, safety
Multi-Model System Design — architecture for multiple models
LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration

LLM Guardrails in Practice: What Actually Works

Rost — Fri, 19 Jun 2026 09:52:41 +0000

LLMs are unpredictable. They hallucinate, leak data, generate harmful content, or refuse legitimate requests. Guardrails constrain model behavior without sacrificing capability.

The key is knowing which guardrails matter and which are just noise.

Guardrails aren't about controlling the model. They're about controlling the risk.

Input validation

The most important guardrail. Bad input gets bad output, and bad input can also prompt-inject your system.

Strategy 1: Prompt Sanitization

Sanitize dangerous patterns early:

import re

class PromptSanitizer:
    def __init__(self):
        self.dangerous_patterns = [
            r"ignore\s+previous\s+instructions",
            r"system\s+prompt",
            r"you\s+are\s+now\s+free",
            r"break\s+out\s+of",
        ]

    def sanitize(self, prompt: str) -> str:
        for pattern in self.dangerous_patterns:
            prompt = re.sub(pattern, "[REDACTED]", prompt, flags=re.IGNORECASE)
        return prompt

This isn't bulletproof. Adversarial inputs are creative. But it catches the obvious ones, and the obvious ones are the most common.

Strategy 2: Input Length Limits

Length limits prevent token waste and timeouts:

class InputValidator:
    def __init__(self, max_length: int = 10000):
        self.max_length = max_length

    def validate(self, prompt: str) -> tuple[bool, str]:
        if len(prompt) > self.max_length:
            return False, f"Input too long: {len(prompt)} > {self.max_length}"
        return True, "OK"

Strategy 3: Content Filtering

Content filtering blocks policy violations. The patterns here depend on your domain:

class ContentFilter:
    def __init__(self):
        self.blocked_topics = [
            "violence", "hate speech", "self-harm",
            "sexual content", "illegal activities",
        ]

    def filter(self, prompt: str) -> tuple[bool, str]:
        prompt_lower = prompt.lower()
        for topic in self.blocked_topics:
            if topic in prompt_lower:
                return False, f"Blocked: {topic}"
        return True, "OK"

Simple string matching is fast but imprecise. For production, use a classifier model — even a small one like Qwen2.5-1.5B — to detect policy violations. It's more accurate and harder to evade.

Output filtering

The model's output needs checking too. Structure, content, and facts.

Strategy 1: Response Validation

Validate structure first. If you expect JSON, check for JSON:

class ResponseValidator:
    def __init__(self):
        self.required_fields = ["answer", "confidence"]

    def validate(self, response: dict) -> tuple[bool, str]:
        for field in self.required_fields:
            if field not in response:
                return False, f"Missing field: {field}"
        return True, "OK"

Strategy 2: Content Filtering

Filter harmful content:

class OutputFilter:
    def __init__(self):
        self.blocked_patterns = [
            r"kill\s+someone",
            r"bomb\s+recipe",
            r"hate\s+speech",
            r"self-harm",
        ]

    def filter(self, response: str) -> tuple[bool, str]:
        for pattern in self.blocked_patterns:
            if re.search(pattern, response, re.IGNORECASE):
                return False, f"Blocked: {pattern}"
        return True, "OK"

Strategy 3: Fact-Checking

Fact-checking is harder. You can't validate every claim, so pick the ones that matter:

class FactChecker:
    def __init__(self):
        self.known_facts = {
            "capital of france": "Paris",
            "population of usa": "330 million",
            "speed of light": "299,792,458 m/s",
        }

    def check(self, claim: str) -> tuple[bool, str]:
        claim_lower = claim.lower()
        for fact, truth in self.known_facts.items():
            if fact in claim_lower and truth not in claim_lower:
                return False, f"Fact check failed: {fact}"
        return True, "OK"

For real fact-checking, you need a retrieval pipeline. Check claims against a knowledge base, not a hardcoded dictionary.

Safety mechanisms

Strategy 1: Rate Limiting

Rate limiting prevents abuse:

import time
from collections import deque

class RateLimiter:
    def __init__(self, max_requests: int = 10, window: int = 60):
        self.max_requests = max_requests
        self.window = window
        self.requests = deque()

    def allow(self) -> bool:
        now = time.time()
        while self.requests and self.requests[0] < now - self.window:
            self.requests.popleft()

        if len(self.requests) >= self.max_requests:
            return False

        self.requests.append(now)
        return True

Strategy 2: Token Budgeting

Token budgeting caps per-request costs:

class TokenBudget:
    def __init__(self, max_tokens: int = 1000):
        self.max_tokens = max_tokens

    def validate(self, response: str) -> tuple[bool, str]:
        token_count = len(response.split())
        if token_count > self.max_tokens:
            return False, f"Token limit exceeded: {token_count} > {self.max_tokens}"
        return True, "OK"

Strategy 3: Context Window Management

Context window management prevents overflow:

class ContextManager:
    def __init__(self, max_context: int = 4096):
        self.max_context = max_context
        self.context = []

    def add(self, message: str):
        self.context.append(message)
        self.trim()

    def trim(self):
        while len(" ".join(self.context)) > self.max_context:
            self.context.pop(0)

Sliding window trimming is simple but loses early context. Better approaches use summarization or attention-based compression, but those add latency.

Compliance

Enterprise systems need compliance guardrails. Two that matter most:

Pattern 1: Data Residency

Data residency — ensure data stays within required geographic boundaries:

class DataResidency:
    def __init__(self, allowed_regions: list[str]):
        self.allowed_regions = allowed_regions

    def validate(self, region: str) -> tuple[bool, str]:
        if region not in self.allowed_regions:
            return False, f"Region not allowed: {region}"
        return True, "OK"

Pattern 2: Audit Logging

Audit logging — log all model interactions:

import json
from datetime import datetime

class AuditLogger:
    def __init__(self, log_file: str = "audit.log"):
        self.log_file = log_file

    def log(self, request: dict, response: dict):
        entry = {
            "timestamp": datetime.now().isoformat(),
            "request": request,
            "response": response,
        }
        with open(self.log_file, "a") as f:
            f.write(json.dumps(entry) + "\n")

Audit logs are critical for debugging and compliance. Make them structured, append-only, and stored securely.

Putting it together

Pattern 1: Simple Guardrails

A simple guardrail pipeline:

class SimpleGuardrails:
    def __init__(self):
        self.input_validator = InputValidator(max_length=10000)
        self.output_filter = OutputFilter()

    def process(self, prompt: str) -> str:
        valid, message = self.input_validator.validate(prompt)
        if not valid:
            return f"Error: {message}"

        response = self.call_model(prompt)

        valid, message = self.output_filter.filter(response)
        if not valid:
            return f"Error: {message}"

        return response

Pattern 2: Advanced Guardrails

Advanced guardrails add sanitization, rate limiting, and token budgets:

class AdvancedGuardrails:
    def __init__(self):
        self.sanitizer = PromptSanitizer()
        self.input_validator = InputValidator(max_length=10000)
        self.content_filter = ContentFilter()
        self.output_filter = OutputFilter()
        self.rate_limiter = RateLimiter(max_requests=10)
        self.token_budget = TokenBudget(max_tokens=1000)

    def process(self, prompt: str) -> str:
        prompt = self.sanitizer.sanitize(prompt)

        valid, message = self.input_validator.validate(prompt)
        if not valid:
            return f"Error: {message}"

        valid, message = self.content_filter.filter(prompt)
        if not valid:
            return f"Error: {message}"

        if not self.rate_limiter.allow():
            return "Error: Rate limit exceeded"

        response = self.call_model(prompt)

        valid, message = self.output_filter.filter(response)
        if not valid:
            return f"Error: {message}"

        valid, message = self.token_budget.validate(response)
        if not valid:
            return f"Error: {message}"

        return response

When guardrails matter

Guardrails matter when you're building user-facing systems, handling sensitive data, or running in production. They also matter when you have compliance requirements — GDPR, HIPAA, SOC 2.

They don't matter when you're prototyping, using models for internal tools only, or not handling sensitive data. Skip them until you need them.

The tradeoff is always capability versus safety. More guardrails mean fewer failures but also fewer capabilities. Find the balance that works for your system.

Tradeoffs

Strategy	Safety	Capability	Latency
No guardrails	Lowest	Highest	Lowest
Input validation	High	Medium	Low
Output filtering	High	Medium	Low
Safety mechanisms	Highest	Lowest	Highest
Compliance	Highest	Lowest	Highest

Model Routing Strategies — capability-based, cost-aware, latency-aware routing
Cost Optimization for LLM Systems — token budgeting, fallback models, caching
Multi-Model System Design — architecture for multiple models
LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration

Model Routing: Stop Using One Model for Everything

Rost — Fri, 19 Jun 2026 09:51:51 +0000

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in.

It matches task complexity to model capability. The tradeoffs are real, but the savings are too.

The routing problem

People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request.

Four strategies work in practice:

Capability-based — route by what the model can do
Cost-aware — route by what you're willing to spend
Latency-aware — route by how fast you need it
Hybrid — combine them

Each optimizes something different. Picking one is usually a decision about what hurts most.

Capability-based routing

The simplest approach. Classify the task, send it to the model that handles it.

Task	Model size	Examples
Classification, tagging	1-3B	Qwen2.5-1.5B, Gemma-2-2B
Summarization, extraction	3-7B	Qwen2.5-7B, Llama-3.1-8B
Code generation	7-14B	Qwen2.5-Coder-7B, DeepSeek-Coder-V2
Complex reasoning	14-32B	Qwen2.5-32B, Llama-3.1-70B
Creative writing, analysis	32B+	Qwen2.5-72B, Claude, GPT-4

If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay.

Implementation is straightforward:

ROUTING_RULES = {
    "classify": {"model": "qwen2.5-1.5b", "max_tokens": 100},
    "summarize": {"model": "qwen2.5-7b", "max_tokens": 500},
    "code_review": {"model": "qwen2.5-coder-7b", "max_tokens": 2000},
    "reason": {"model": "qwen2.5-32b", "max_tokens": 4000},
    "creative": {"model": "claude-sonnet-4", "max_tokens": 8000},
}

def route_request(task_type: str) -> dict:
    return ROUTING_RULES.get(task_type, ROUTING_RULES["reason"])

The catch is classification itself. If you get the task type wrong, you route to the wrong model. I've seen systems classify code review as "summarization" and lose quality silently.

Cost-aware routing

Local inference shines here. Local models are effectively free after hardware amortization. A RTX 5080 pays for itself in about six months at moderate API usage.

Model	Input ($/M tokens)	Output ($/M tokens)	Local cost/hour
GPT-4o	$2.50	$10.00	—
Claude Sonnet 4	$3.00	$15.00	—
Qwen2.5-72B (API)	$0.50	$2.00	—
Qwen2.5-32B (local)	$0.00	$0.00	~$0.10
Qwen2.5-7B (local)	$0.00	$0.00	~$0.05

If you're processing thousands of requests per session, even $0.05 in electricity beats $15/M tokens.

Budget-based routing falls back as you spend:

class CostAwareRouter:
    def __init__(self, budget_per_session: float = 0.10):
        self.budget = budget_per_session
        self.spent = 0.0
        self.models = {
            "cheap": {"model": "qwen2.5-7b", "cost": 0.0},
            "medium": {"model": "qwen2.5-32b", "cost": 0.0},
            "expensive": {"model": "claude-sonnet-4", "cost": 0.000015},
        }

    def route(self, task: str) -> str:
        ratio = self.spent / self.budget
        if ratio < 0.5:
            return self.models["expensive"]["model"]
        elif ratio < 0.8:
            return self.models["medium"]["model"]
        return self.models["cheap"]["model"]

Quality degrades as you fall back. You start with Claude, move to Qwen-32B, then to Qwen-7B. By the end of a long session, the output is noticeably worse. Whether that matters depends on what you're building.

Latency-aware routing

Interactive tools need fast first tokens. Batch jobs can wait. The difference is usually a factor of five in model size.

Use case	First token	Complete	Max model size
Real-time chat	< 200ms	< 2s	< 7B
Interactive tools	< 500ms	< 5s	< 14B
Batch processing	< 1s	< 30s	Any
Research/analysis	< 2s	< 60s	Any

When you're streaming tokens to a user, first token latency is what they feel. A 32B model taking half a second to start feels sluggish compared to a 1.5B model that fires instantly.

class LatencyAwareRouter:
    def __init__(self):
        self.model_latencies = {
            "qwen2.5-1.5b": {"first_token": 0.05, "complete": 0.5},
            "qwen2.5-7b": {"first_token": 0.15, "complete": 2.0},
            "qwen2.5-32b": {"first_token": 0.5, "complete": 10.0},
            "claude-sonnet-4": {"first_token": 0.3, "complete": 5.0},
        }

    def route(self, target_latency: float) -> str:
        for model, latencies in sorted(
            self.model_latencies.items(),
            key=lambda x: x[1]["complete"]
        ):
            if latencies["complete"] <= target_latency:
                return model
        return "qwen2.5-1.5b"

The latency numbers are rough — they depend on your hardware, quantization, and batch size. Measure on your own setup.

Fallback strategies

Models fail. APIs rate-limit. Timeouts happen. The pattern that works is a fallback chain, ordered from best to most reliable:

class FallbackRouter:
    def __init__(self):
        self.fallback_chain = [
            {"model": "claude-sonnet-4", "timeout": 30},
            {"model": "qwen2.5-72b", "timeout": 60},
            {"model": "qwen2.5-32b", "timeout": 120},
            {"model": "qwen2.5-7b", "timeout": 300},
        ]

    def route_with_fallback(self, prompt: str) -> str:
        for config in self.fallback_chain:
            try:
                return self.call_model(
                    config["model"], prompt,
                    timeout=config["timeout"]
                )
            except (TimeoutError, APIError) as e:
                log.warning(f"Model {config['model']} failed: {e}")
                continue
        raise RuntimeError("All fallback models failed")

The last model in the chain should be local. It's slower, but it won't fail because of a network issue or an API key.

When routing helps

Routing makes sense when your workload is mixed. If you're doing classification, summarization, and reasoning in the same system, a router saves money and latency.

It doesn't make sense when everything you do is the same complexity. Just use the model that's good at that task. The router adds complexity you don't need.

Early prototyping is another reason to skip it. Get the task working with one model, then add routing when cost or latency actually becomes a problem.

Tradeoffs

Every routing strategy optimizes something and sacrifices something else:

Single model — simplest, most expensive, consistent quality
Capability-based — better cost, higher quality per task, moderate complexity
Cost-aware — cheapest, quality varies, moderate complexity
Latency-aware — fastest, may sacrifice quality, moderate complexity
Hybrid — best of all, most complex to implement

Production systems usually converge on hybrid. Start with capability-based routing, add cost awareness when the bill comes in, add latency awareness when users complain about slowness.

Cost Optimization for LLM Systems — token budgeting, caching, fallback models
LLM Guardrails in Practice — input validation, output filtering, safety
Multi-Model System Design — architecture for multiple models
LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration

Multi-Model System Design: When One Model Isn't Enough

Rost — Fri, 19 Jun 2026 09:51:46 +0000

Single-model systems are simple. Multi-model systems are powerful. The challenge isn't choosing models — it's designing the architecture that orchestrates them.

A multi-model system isn't about having more models. It's about having the right model for the right task at the right time.

Architecture patterns

Five patterns cover most use cases:

Pattern	Complexity	When to use	Tradeoff
Single Model	Lowest	Prototyping, simple tasks	Limited capability
Sequential	Low	Multi-step workflows	Higher latency
Parallel	Medium	Independent tasks	Higher cost
Hierarchical	High	Complex reasoning	Complex orchestration
Ensemble	Highest	Critical decisions	Highest cost

Pick the simplest one that works. Complexity is real, and it compounds.

Sequential architecture

Process tasks through a chain of models, each specializing in a step.

Pattern 1: Pipeline

Pipeline pattern — each model's output feeds the next:

class ModelPipeline:
    def __init__(self):
        self.models = [
            {"model": "qwen2.5-1.5b", "task": "classify"},
            {"model": "qwen2.5-7b", "task": "extract"},
            {"model": "qwen2.5-32b", "task": "reason"},
        ]

    def process(self, input: str) -> str:
        current = input
        for model_config in self.models:
            current = self.call_model(
                model_config["model"],
                self.create_prompt(model_config["task"], current)
            )
        return current

Latency adds up. Three models in sequence means three times the latency. Only use this when each step actually needs a different model.

Pattern 2: Router

Router pattern — classify the task, route to the specialist:

class ModelRouter:
    def __init__(self):
        self.classifier = "qwen2.5-1.5b"
        self.specialists = {
            "code": "qwen2.5-coder-7b",
            "math": "qwen2.5-32b",
            "creative": "claude-sonnet-4",
            "general": "qwen2.5-7b",
        }

    def route(self, prompt: str) -> str:
        task_type = self.classify(prompt)
        model = self.specialists.get(task_type, self.specialists["general"])
        return self.call_model(model, prompt)

The classifier is the weak link. If it misclassifies, you route to the wrong model and lose quality. Use a classifier that's good enough — even a small one works if the categories are clear.

Parallel architecture

Process independent tasks simultaneously.

Pattern 1: Fan-Out

Fan-out — run the same prompt through multiple models:

import asyncio

class ModelFanOut:
    def __init__(self):
        self.models = [
            "qwen2.5-7b",
            "qwen2.5-32b",
            "claude-sonnet-4",
        ]

    async def process(self, prompt: str) -> list[str]:
        tasks = [self.call_model(model, prompt) for model in self.models]
        return await asyncio.gather(*tasks)

Useful for comparison, A/B testing, or when you want to pick the best output. Expensive, but the quality gain is worth it for critical decisions.

Pattern 2: Voting

Voting — combine outputs through consensus:

class ModelVoting:
    def __init__(self):
        self.models = [
            "qwen2.5-7b",
            "qwen2.5-32b",
            "claude-sonnet-4",
        ]

    def vote(self, prompt: str) -> str:
        responses = [self.call_model(model, prompt) for model in self.models]
        from collections import Counter
        votes = Counter(responses)
        return votes.most_common(1)[0][0]

Majority voting works for classification. For generation tasks, it's harder — you need semantic similarity, not exact matches.

Hierarchical architecture

Use models at different levels of abstraction.

Pattern 1: Planner-Executor

Planner-executor — a strong model plans, smaller models execute:

class PlannerExecutor:
    def __init__(self):
        self.planner = "qwen2.5-32b"
        self.executors = {
            "code": "qwen2.5-coder-7b",
            "search": "qwen2.5-7b",
            "math": "qwen2.5-7b",
        }

    def process(self, task: str) -> str:
        plan = self.call_model(self.planner, f"Plan: {task}")
        results = []
        for step in self.parse_plan(plan):
            executor = self.executors.get(step["type"], "qwen2.5-7b")
            result = self.call_model(executor, step["prompt"])
            results.append(result)
        return self.call_model(self.planner, f"Synthesize: {results}")

The planner does the heavy lifting. The executors handle specific tasks. This pattern works well when the planning step is expensive but the execution steps are cheap.

Pattern 2: Supervisor-Worker

Supervisor-worker — a supervisor delegates and reviews:

class SupervisorWorker:
    def __init__(self):
        self.supervisor = "qwen2.5-32b"
        self.workers = ["qwen2.5-7b", "qwen2.5-coder-7b"]

    def process(self, task: str) -> str:
        assignments = self.call_model(self.supervisor, f"Assign: {task}")
        results = []
        for assignment in self.parse_assignments(assignments):
            result = self.call_model(
                assignment["worker"], assignment["task"]
            )
            results.append(result)
        return self.call_model(self.supervisor, f"Review: {results}")

The supervisor is the bottleneck. It plans, delegates, and reviews. Make sure it's fast enough, or the whole system slows down.

Ensemble architecture

Combine multiple models for critical decisions.

Pattern 1: Weighted Ensemble

Weighted ensemble — score each model's output, pick the highest:

class WeightedEnsemble:
    def __init__(self):
        self.models = {
            "qwen2.5-32b": 0.5,
            "claude-sonnet-4": 0.3,
            "qwen2.5-7b": 0.2,
        }

    def decide(self, prompt: str) -> str:
        responses = {
            model: self.call_model(model, prompt)
            for model in self.models
        }
        scores = {}
        for model, response in responses.items():
            score = self.evaluate(response) * self.models[model]
            scores[response] = scores.get(response, 0) + score
        return max(scores, key=scores.get)

Weights reflect your confidence in each model. Adjust them based on actual performance, not benchmarks.

Pattern 2: Consensus Ensemble

Consensus ensemble — require agreement, escalate if there isn't any:

class ConsensusEnsemble:
    def __init__(self, threshold: float = 0.7):
        self.threshold = threshold
        self.models = [
            "qwen2.5-32b",
            "claude-sonnet-4",
            "qwen2.5-7b",
        ]

    def decide(self, prompt: str) -> str:
        responses = [
            self.call_model(model, prompt)
            for model in self.models
        ]
        from collections import Counter
        votes = Counter(responses)
        max_votes = max(votes.values())

        if max_votes / len(self.models) >= self.threshold:
            return votes.most_common(1)[0][0]

        return self.call_model("qwen2.5-32b", prompt)

The threshold controls how strict consensus is. 0.7 means two-thirds agreement. Lower it for faster decisions, raise it for higher confidence.

When multi-model systems make sense

Multi-model systems make sense when you have mixed workloads, need high quality for critical decisions, or are optimizing for cost or latency.

They don't make sense when all tasks are similar complexity, you're prototyping, or simplicity matters more than optimization.

The rule of thumb: start with one model. Add more when you hit a real constraint — cost, latency, or quality. Don't architect complexity before you need it.

Tradeoffs

Pattern	Cost	Latency	Quality	Complexity
Single Model	Lowest	Lowest	Variable	Lowest
Sequential	Medium	High	High	Medium
Parallel	High	Low	High	Medium
Hierarchical	High	High	Highest	High
Ensemble	Highest	Medium	Highest	Highest

Every pattern trades something. Pick the one that matches your constraints.

Model Routing Strategies — capability-based, cost-aware, latency-aware routing
Cost Optimization for LLM Systems — token budgeting, fallback models, caching
LLM Guardrails in Practice — input validation, output filtering, safety
LLM Architecture — system design pillar: routing, cost, guardrails, and orchestration

AI Assistant Architecture: LLM, Memory, Tools, Routing, Observability

Rost — Tue, 16 Jun 2026 12:20:21 +0000

A production AI assistant is not "an LLM with a prompt". It is a system that accepts intent, keeps state, decides when to retrieve or act, and exposes enough runtime detail to debug failures.

That systems-level view is what the AI Systems cluster explores when assistants move beyond a single model invocation.

OpenAI describes agents as applications that plan, call tools, collaborate, and keep enough state for multi-step work, while Anthropic frames the same problem as a managed harness that can run files, commands, web access, and code securely.

The cleanest architecture splits responsibilities into five layers: LLM, Memory, Tooling, Routing, and Observability. That split matches the capabilities exposed by major provider APIs, by MCP, by self-hosted runtimes such as vLLM and llama.cpp, and by real assistant systems such as OpenClaw and Hermes.

Memory should be treated as more than "longer context". Retrieval systems turn external knowledge into explicit non-parametric memory — the same design space covered in depth by Retrieval-Augmented Generation (RAG) — and both Anthropic's context guidance and the "Lost in the Middle" paper warn that merely cramming more tokens into context does not guarantee reliable recall.

Tool use is a contract boundary, not magic. OpenAI function calling, Anthropic tool use, and MCP all rely on the same pattern: the model emits a structured request, some runtime executes it, and the result flows back into the conversation. If that boundary is sloppy, the assistant becomes sloppy.

My bias is simple: start boring. One orchestrator, one durable memory path, one trace per request, and one explicit policy for tool execution. Multi-agent graphs are useful, but only after you can explain your single-agent failure cases without guessing.

What an AI assistant system is

A practical definition is this: an AI assistant system is a runtime that turns user intent into a response or action by combining a model interface, context assembly, tool execution, state management, and telemetry. That is why the useful docs are not just model cards. The useful docs are API references, tool contracts, retrieval guides, routing docs, and tracing docs. OpenAI's Responses API exposes stateful interactions, built-in tools, and function calling. Anthropic's Claude API exposes direct Messages access as well as Managed Agents. OpenClaw and Hermes go one step further and show what happens when you put those capabilities behind persistent gateways, channels, sessions, and memory.

In other words, an assistant system has a broader contract than a chat completion. A good internal contract looks something like this:

AssistantRequest  = user intent + identity + session + attachments + policy
AssistantResponse = answer + actions + citations + state changes + trace id

That contract matters because every production disagreement eventually reduces to one of these questions: what context was visible, which tool executed, which model answered, which memory was read or written, and where the trace says the system spent time. OpenTelemetry defines traces as the path of a request through an application, which is exactly the abstraction serious assistants need. LangSmith and OpenLIT then specialise that idea for LLMs, tools, vector stores, and agent workflows.

Core components and interfaces

The component split below is the one I find most durable. It is also the split that lines up best with the official APIs and the open-source runtimes people actually operate.

Layer	Main responsibility	Typical interface	Example technologies
LLM layer	Reason, generate, decide, emit structured calls	Responses API, Messages API, OpenAI-compatible or Anthropic-compatible endpoints	OpenAI, Anthropic, vLLM, llama.cpp, Ollama
Memory layer	Hold session state, durable notes, and searchable knowledge	embeddings, vector search, memory read/write tools, retrieval APIs	OpenAI embeddings and vector stores, Pinecone, Weaviate, pgvector, Milvus, Hermes memory, OpenClaw memory
Tooling layer	Read data and perform actions outside the model	JSON-schema tools, MCP tools, file and web search, native runtime tools	OpenAI function calling, Anthropic tool use, MCP, LangChain tools, LlamaIndex query tools
Routing layer	Choose model, backend, policy, and tenant path	model aliases, failover groups, health checks, budgets, channel bindings	LiteLLM, OpenClaw multi-agent routing, Hermes provider runtime resolution
Observability	Explain what happened and why	traces, spans, logs, metrics, eval runs	OpenTelemetry, LangSmith, OpenLIT

The table above is derived from the official provider interfaces, MCP, vector database docs, and the runtime docs for vLLM, llama.cpp, OpenClaw, and Hermes.

The LLM layer should do three things well: consume a current working context, emit either a final answer or a structured action request, and return enough metadata to support retries and tracing. OpenAI's Responses API is explicitly designed for stateful interactions plus built-in tools and function calling. Anthropic's Messages API exposes the same core loop through tool_use blocks and tool_result returns, while Managed Agents gives you a hosted harness if you do not want to build the loop yourself. Self-hosted runtimes such as vLLM and llama.cpp matter because they preserve familiar provider-style interfaces while letting you place inference inside your own environment.

The Memory layer should be split mentally into three buckets: working memory, durable symbolic memory, and searchable semantic memory. OpenAI embeddings return vectors that can be indexed and searched; OpenAI Retrieval and File Search then layer semantic and keyword search on top of vector stores. Pinecone, Weaviate, pgvector, and Milvus represent four common storage shapes: fully managed, open-source vector-native, Postgres-native, and distributed vector database. Hermes and OpenClaw add a useful reminder that not all memory belongs in a vector store: file-backed notes, reviewed promotions, and session-scoped snapshots are often the more honest design — patterns unpacked in Hermes Agent Memory System for bounded core memory and frozen session snapshots.

The Tooling layer is where an assistant stops being a summariser and starts being software. OpenAI function calling treats tools as schema-defined functionality the model may decide to invoke. Anthropic says the same thing more explicitly: tool use is a contract between your application and the model, and the model never executes anything on its own. MCP generalises that contract into a client-server protocol where hosts connect to one or more servers that expose tools, prompts, and resources — the same boundary described step by step in MCP Server in Go. LangChain and LlamaIndex sit comfortably here as orchestration libraries: LangChain focuses on a prebuilt agent architecture and integrations, while LlamaIndex focuses on context-augmented data access, query engines, and workflows.

The Routing layer exists because "which model?" is never the only question. You also need "which provider path, which tenant, which budget, which latency class, and which fallback?". LiteLLM is useful because its official docs are refreshingly concrete: weighted pick, least-busy, latency-based, cost-based routing, and bounded failovers are all first-class patterns. OpenClaw extends routing upward into channel and agent isolation, while Hermes extends it downward into model slots for main and auxiliary work such as summarisation, context compression, and MCP tool routing. That is the right mental model: the router chooses more than a model, it chooses an execution lane.

The Observability layer is what prevents architecture from turning into folklore. OpenTelemetry gives you the trace abstraction. LangSmith gives you end-to-end visibility over LLM application steps and supports cloud, hybrid, and self-hosted deployment shapes. OpenLIT gives you OpenTelemetry-native AI observability with zero-code and manual instrumentation options, including support for LLMs, agent frameworks, vector databases, and GPUs. For production metrics, traces, and SLO patterns across inference and agent workflows, see Observability for LLM Systems. If your assistant has no trace per request, no span per model call, and no event history for tool execution, you do not really have an architecture yet. You have vibes.

Capture, enrich, respond

The sequence that keeps showing up in real systems is capture -> enrich -> respond -> record. Different frameworks wrap it differently, but the flow is stable enough to treat as the backbone.

sequenceDiagram
    participant U as User or Channel
    participant G as Gateway or UI
    participant R as Router
    participant M as Memory and Retrieval
    participant L as LLM
    participant T as Tools or MCP
    participant O as Observability

    U->>G: message, file, or command
    G->>O: start root trace
    G->>R: request + identity + session + policy
    R->>M: load session state and retrieve context
    M-->>R: notes, chunks, metadata
    R->>L: prompt + context + tool schemas
    L-->>R: answer or tool call
    alt tool call
        R->>T: execute tool or MCP action
        T-->>R: tool result
        R->>L: tool result + updated context
        L-->>R: final answer
    end
    R->>M: persist session changes and memory candidates
    R->>O: spans, metrics, eval events
    G-->>U: response

The capture step is usually more important than it looks. OpenClaw and Hermes both put a persistent gateway in front of the assistant because ingress is not just text entry. It includes channel metadata, identities, authorisation, session boundaries, direct messages, groups, cron ticks, and delivery semantics. If you skip that layer and rely on a raw chat widget abstraction, you will eventually bolt it back on as ad hoc middleware anyway.

The enrich step is where mature systems diverge from toy demos. OpenAI Retrieval and File Search make retrieval explicit through vector stores and search calls. LlamaIndex formalises the same pattern through data connectors, indexes, query engines, and workflows. Hermes goes further by splitting the model estate into main and auxiliary slots, offloading work such as compression, summarisation, and routing to smaller or more specialised models. That is a design pattern worth stealing: do not spend your most expensive model tokens on chores.

The respond step is not "generate text". It is "close the current loop". If the model can answer directly, it does. If it needs a tool, it emits a structured request. Anthropic's tool-use contract and OpenAI's function-calling guide both make this explicit. The reason this matters architecturally is that outputs now include both language and control flow. Your response object is partly prose and partly runtime plan.

The record step is where consistency semantics show up. Pinecone separates write and read paths and processes writes after durable acknowledgement. Hermes memory is injected as a frozen snapshot per session so it can preserve prefix-cache performance, which means new writes do not automatically appear in the current session prompt. OpenClaw's Dreaming system only promotes reviewed, grounded candidates into MEMORY.md, and it is opt-in rather than always-on. The practical lesson is that memory is rarely truly read-after-write across every layer. You need to design for staged visibility.

OpenClaw and Hermes as reference systems

OpenClaw and Hermes are useful reference cases because they are not just wrappers around one provider API. Both present an assistant as a long-running system with gateways, sessions, tools, memory, and multiple model backends.

Architectural concern	OpenClaw mapping	Hermes mapping
Ingress and surfaces	Self-hosted gateway connecting chat apps and channel surfaces	Single background messaging gateway connecting many external platforms
Orchestration	Gateway-centric control plane for channels and AI interactions	`AIAgent` loop handling prompt assembly, provider selection, tool dispatch, retries, and failover
Routing	Multi-agent routing binds inbound traffic to isolated agents with separate workspaces and sessions	Main and auxiliary model slots split core reasoning from compression, summarisation, approvals, and MCP routing
Memory	File-backed memory plus optional active memory and background Dreaming promotion	`MEMORY.md` and `USER.md` injected as a frozen session snapshot, plus external memory providers
Tooling and extension	Built-in tools, session tools, provider plugins, custom and self-hosted endpoints	40+ tools, built-in MCP client, toolsets, skills, and memory-provider plugins

This mapping is grounded in the official OpenClaw and Hermes docs and repos. OpenClaw documents a gateway architecture, multi-agent routing, custom and self-hosted provider support including vLLM and Ollama, optional active memory, and Dreaming-based promotion. Hermes documents a messaging gateway, a central AIAgent loop, main and auxiliary model slots, built-in memory, and native MCP integration.

My slightly opinionated read is that both systems make the same architectural argument in different accents. OpenClaw is strongly gateway-first. Hermes is strongly agent-loop-first. But both reject the shallow idea that an assistant is just "prompt plus model". They model channels, identities, memory semantics, tool surfaces, and backend heterogeneity as first-class concerns. That is exactly what a production architecture should do.

A practical hybrid stack inspired by both systems looks like this:

edge:
  gateway: hermes or openclaw

routing:
  proxy: litellm
  policy: latency and budget aware
  tenancy: session and channel scoped

llm:
  primary: openai responses or anthropic messages
  local_fallback: vllm
  local_dev: ollama or llama.cpp

memory:
  session: sqlite or postgres
  semantic: pgvector or weaviate
  embeddings: openai embeddings or ollama embeddings

tools:
  contract: json schema tools plus mcp
  examples: filesystem, browser, web search, internal APIs

observability:
  traces: opentelemetry
  ai_dashboards: openlit or langsmith
  evals: openai evals plus app-specific regression sets

That stack is a reasoned deployment pattern rather than a vendor-prescribed blueprint. It works because the official interfaces line up: OpenAI and Anthropic expose tool-oriented APIs, vLLM and llama.cpp emulate provider-style endpoints, Ollama handles local models and embeddings, MCP standardises external tools, LiteLLM handles routing and failover, and OpenTelemetry-compatible platforms can trace the whole path.

Patterns, tables, and tradeoffs

There are a few repeatable assistant patterns worth naming. A managed assistant keeps most of the runtime inside provider APIs. A retrieval-first assistant treats memory and search as the main differentiator. A tool-first assistant behaves more like an operator than a chatbot. A gateway assistant prioritises always-on access through messaging surfaces. A specialist mesh decomposes work into multiple agents or routes. Official docs across OpenAI, Anthropic, LlamaIndex, LiteLLM, OpenClaw, and Hermes all support versions of these patterns, even if they name them differently.

Pattern	What it optimises for	Best use case	Hidden cost
Managed assistant	Speed of delivery	Internal copilots and support bots	Provider lock-in and less control over runtime details
Retrieval-first assistant	Grounded answers over owned data	Docs, support, knowledge work	Retrieval quality becomes the real product
Tool-first assistant	Action over conversation	Ops workflows, data pulls, automations	Side effects, retries, and approvals become core concerns
Gateway assistant	Ubiquitous access	Personal and team assistants across chat surfaces	Identity, session, and security complexity
Specialist mesh	Division of labour	Complex workflows with real ownership boundaries	Harder debugging, orchestration, and eval design

This pattern table is a synthesis from the provider docs, framework docs, and reference systems rather than a claim from any one vendor.

Option shape	Typical components	Strength	Weakness
Managed	OpenAI Responses or Anthropic Managed Agents, hosted file search or vector stores	Fastest path, fewer moving parts, hosted tools	Lowest control over data path and runtime semantics
Hybrid	Provider API plus self-hosted router and vector store	Good balance of speed and control	More contracts to maintain
Self-hosted	vLLM or llama.cpp or Ollama, MCP, self-hosted vector DB, OTel	Strong privacy and deployment control	Highest ops burden, hardware and tuning overhead

Table notes: OpenAI hosted File Search is a managed tool, Anthropic offers a managed harness, Pinecone is a managed vector service, while vLLM, llama.cpp, Ollama, pgvector, Weaviate, Milvus, LangSmith self-hosted, and OpenLIT all support self-managed or hybrid operation to varying degrees.

Vector store	Shape	Why teams choose it	Watchout
Pinecone	Managed vector service	Strong operational simplicity and scalable managed architecture	External dependency and managed-service economics
Weaviate	Open-source vector database	Vector plus inverted indexes and flexible index choices	More cluster tuning than a hosted-only path
pgvector	Postgres extension	Keep vectors with relational data and existing SQL stack	Not the best fit for every high-scale ANN workload
Milvus	Distributed vector database	Purpose-built scale and ecosystem around managed Zilliz Cloud	Another specialist datastore to operate

Table notes: Pinecone documents a managed control plane and regional data planes. Weaviate documents vector and inverted indexes with multiple vector index types. pgvector adds exact and approximate nearest-neighbour search to Postgres. Milvus positions itself as an open-source high-performance, scalable vector database, with Zilliz Cloud as the managed option.

LLM option	Interface style	Best at	Watchout
OpenAI Responses	Stateful responses plus built-in tools	Fast start, hosted tools, structured loops	You inherit platform-specific abstractions
Anthropic Messages	Direct model access with explicit tool-use contract	Clear tool boundaries and good control in custom loops	More runtime is your responsibility unless you use Managed Agents
vLLM	OpenAI-compatible and Anthropic-compatible self-hosted serving	High-throughput self-hosted inference	Real infrastructure and model-serving work
Ollama	Simple local model and embedding runtime	Local development and small self-hosted stacks	Not the same class of serving system as a tuned distributed runtime
llama.cpp	Lightweight local server with provider-compatible routes	Edge, CPU-first, constrained environments	You do more manual tuning and capability matching

Table notes: OpenAI documents Responses as its advanced interface for stateful responses and built-in tools. Anthropic documents the Messages API and the tool-use contract separately from Managed Agents. vLLM exposes an OpenAI-compatible server plus Anthropic Messages API support. Ollama documents local embedding and model workflows. llama.cpp documents OpenAI-compatible chat, responses, and embeddings routes, plus Anthropic-compatible chat completions.

Constraint or tradeoff	Bias toward managed	Bias toward self-hosted	Practical mitigation
Latency	Often better first iteration and fewer local tuning tasks	Can win when model and data are colocated and kept warm	Use routing tiers, hot caches, and smaller auxiliary models
Cost	Easy to start, variable at token scale	Better amortisation at steady utilisation	Measure real traffic before optimising by instinct
Privacy and residency	Simpler for non-sensitive data	Stronger control for sensitive and regulated flows	Use hybrid boundaries and keep only what must move
Consistency	Hosted tools still have staged visibility semantics	Self-hosted memory pipelines also stage and promote data	Define read-after-write rules explicitly by layer
Scaling	Less control-plane pain	Better tailoring for steady, specialised workloads	Use batching, queueing, and isolated tenants
Debuggability	Easy to miss opaque provider internals	Easy to drown in self-made complexity	Trace every request and evaluate every route

This tradeoff matrix is an architectural inference from the official docs, not a vendor benchmark. The consistency row matters more than many blog posts admit: Pinecone separates write and read paths, Hermes freezes memory into session-start prompts, and OpenClaw promotes durable memory through staged review. That means "memory updated" and "memory visible to the current answer" are often different truths.

Failure modes and mitigations

Most assistants do not fail because the base model is "bad". They fail because the surrounding system lies to the model, starves it of the right context, lets tools drift, or makes debugging impossible.

Where it breaks	Typical symptom	Usual cause	Mitigation
Prompt assembly	Confident but off-target answer	Too much irrelevant context, poor ordering	Budget context, rerank, keep key facts near the top
Retrieval	Correct tone, wrong facts	Bad chunking, stale index, weak filters	Evaluate retrieval separately, add metadata filters and hybrid search
Tool boundary	Wrong action or duplicate action	Loose schemas, retries without idempotency	Tight schemas, idempotency keys, approval gates
Routing	Wildly inconsistent behaviour by request	Cost or latency routing without quality controls	Add sticky sessions and per-route evals
Memory	Stale or poisoned recall	Over-eager writes, weak review, cross-session leakage	Separate working and durable memory, review promotions
Observability	No idea what happened	Missing traces or no span granularity	Emit root and subspans for retrieval, model, and tool calls
Hallucination control	Plausible but unsupported claims	Weak grounding or no validation pass	Reference-doc validation, self-consistency checks, eval gates

The evidence base for this table is broad but consistent. Anthropic's tool docs make clear that tool use is a contract boundary. OpenAI Guardrails includes hallucination detection against a reference knowledge base via File Search. SelfCheckGPT shows that self-consistency across samples can help detect unsupported claims. The "Lost in the Middle" results and Anthropic's context guidance both reinforce the same operational lesson: more tokens do not remove the need for context curation.

Preferred mitigation stack could be boring and repetitive: trace every request, version prompts, evaluate retrieval independently, keep tools idempotent, and run regression evals before you change routes or memory policy. OpenAI's Evals docs and repo are blunt about why: without evals, it is hard and time-consuming to understand how model or prompt changes affect your use case. That applies just as much to routers and retrieval as it does to prompts.

Memory Systems in AI Assistants

Rost — Tue, 16 Jun 2026 12:20:17 +0000

Memory turns assistants from reactive to persistent, but it is also where many systems quietly rot. Surveys argue the short-term versus long-term split is no longer enough for modern agent memory; OpenAI and LangGraph SDKs point to a simpler stack — working memory, durable state, and retrieval.

Assistants need working memory for the current run, durable state for stable facts and preferences, and retrieval memory for relevant supporting context. My slightly opinionated view is that structured state is underused, vector retrieval is overused, and most memory failures come from promotion and injection policy rather than storage choice.

The other important point is that memory does not automatically fix long context. LoCoMo shows that very long-term conversational recall remains hard, and "Lost in the Middle" shows that simply throwing more tokens at the model can degrade performance when relevant information lands in the middle of the prompt. Good memory systems are selective, layered, and explicit about precedence.

This guide sits in the AI Systems Memory hub as the cross-framework map for the memory layer inside AI Assistant Architecture.

How to think about assistant memory

Assistant memory is not the same problem as PKM, wikis, or standalone RAG pipelines — PKM vs RAG vs Wiki vs Memory Systems maps those paradigms at the knowledge-architecture level. This guide stays one layer down, in the runtime contracts assistants actually implement.

The cleanest way to think about memory is not as "chat history", but as a set of storage contracts with different jobs. One store preserves the active thread. Another store keeps durable user state. Another supports semantic lookup over documents or past interactions. OpenAI's memory guidance for personalisation makes this explicit by separating global and session memory, while LangGraph separates thread-level persistence from long-term stores across conversations.

Memory matters because production assistants repeat work, revisit goals, and operate across days or weeks. Generative Agents popularised the pattern of storing experiences, reflecting on them, and retrieving them dynamically for future planning. MemGPT pushed that further by modelling memory as tiers and movement between fast and slow stores. More recent systems such as A-MEM and Mem0 focus on linking, consolidation, and deployment efficiency rather than just recall volume.

Types of memory

Production assistants typically need three cooperating layers. The FAQ above names them; the sections below explain how each behaves in real systems.

Short-term memory

Short-term memory is the working context of the current conversation or run. OpenAI Sessions automatically prepend conversation history before each run and append new items after each run. LangGraph implements the same idea as thread-level persistence through a checkpointer. This layer keeps local coherence, but it is also the first thing that explodes when tool results, file reads, or long chats pile up.

Long-term retrieval memory

Long-term retrieval memory stores items that are looked up when relevant rather than replayed every turn. That overlaps with RAG as a retrieval technique, but it is not the whole assistant memory story — wikis and PKM corpora often feed the index while structured state and session memory live elsewhere, as the PKM/RAG/wiki/memory comparison above makes clear. In classical RAG, the model combines parametric memory with non-parametric memory such as a dense vector index. Self-RAG improves on naive retrieval by making retrieval on-demand rather than fixed for every request. In practical assistant systems, this is usually the vector store or searchable transcript layer.

Structured memory

Structured memory stores durable facts, preferences, or constraints in explicit fields with precedence rules. OpenAI's personalisation cookbook is unusually clear here. Global and session memory have different roles, the latest user instruction wins, session memory can override global memory for the current task, and memory that conflicts with current user intent should trigger clarification rather than silent obedience. This is why structured state is often better than retrieval for stable preferences, policies, or standing constraints.

Retrieval mechanics

A typical retrieval flow has five steps: capture, encode, search, rerank or filter, then inject. Pinecone, Weaviate, Qdrant, Redis, and Milvus all document variants of this pattern. Some support dense vectors only, others support hybrid retrieval that combines semantic and lexical search, and some expose metadata filters or namespaces for tenancy and scope control. The engineering point is straightforward. Retrieval quality depends as much on filtering, chunking, and ranking strategy as on the embedding model itself.

Hybrid retrieval is usually the sensible default when queries mix meaning and exact terms. Weaviate documents hybrid search with an alpha parameter balancing vector and keyword components, Qdrant supports hybrid and multi-stage queries through its Query API and score-fusion methods, and Milvus describes dense, sparse, and hybrid retrieval in the same system. That matters for assistants because users often ask for both approximate meaning and exact identifiers, file names, revision numbers, or product codes. When the lexical side lives in Postgres or Elasticsearch rather than inside the vector database, PostgreSQL full text search vs Elasticsearch helps you choose where keyword search should run in production.

One more opinionated point: retrieval should not decide policy. It should supply candidates. The assistant still needs structured rules for precedence, privacy, recency, and conflict resolution. OpenAI's state-based memory example makes this explicit, and it is a much healthier pattern than pretending similarity search alone can resolve contradictory user state.

Common issues

The most common failure is stale or contradictory memory. OpenAI's long-term memory cookbook calls memory consolidation the most sensitive and error-prone stage, listing context poisoning, memory loss, duplicate memories, and contradiction handling as core concerns. That is correct, and it is where many assistants fail quietly. They remember too much, too early, and without a rule for forgetting.

The second failure is context overload. LangGraph warns that long conversations can exceed the LLM context window and recommends trimming, deletion, summarisation, or checkpoint management. OpenClaw similarly prunes old tool outputs from in-memory context while preserving the full on-disk transcript. These are not optional optimisations. They are required if your assistant reads, searches, or executes anything non-trivial.

The third failure is assuming long context equals reliable recall. LoCoMo shows that long-term conversational memory is still difficult, and "Lost in the Middle" shows position sensitivity inside long prompts. If memory is important, do not rely on brute-force prompt stuffing. Use compaction, retrieval, and explicit state.

Tradeoffs

The vector database layer is where many assistant teams make early platform bets. The comparison below focuses on documented product characteristics that matter for assistant memory design.

System	What stands out	Best fit
Pinecone	Managed vector database with integrated embedding, reranking, metadata filters, namespaces, and support for dense, sparse, and BM25-style full-text in one schema	Teams that want managed retrieval with minimal infra
Weaviate	Open-source vector database storing objects and vectors, with semantic and hybrid search and strong RAG positioning	Teams that want open-source flexibility with hybrid retrieval
Qdrant	AI-native vector search with filtering, hybrid and multi-stage queries, plus an embedded offline-capable Edge mode	Teams that want search control, edge deployment, or strong filtering
pgvector	Vector similarity search inside Postgres, with exact and approximate search plus ACID, JOINs, and recovery features	Teams already standardised on Postgres and relational data
Milvus	Cloud-native vector database with disaggregated storage and compute, plus dense, sparse, and hybrid retrieval	Large-scale retrieval workloads and distributed deployments

Once you pick a backend, operating it is a data infrastructure problem — Postgres with pgvector for session metadata and vectors on one stack, or Neo4j when retrieval memory is graph-shaped rather than flat chunks.

The latency and cost pattern below is a design synthesis based on the operational models described in OpenAI Sessions and compaction guidance, LangGraph memory management, OpenAI state-based memory, and the documented retrieval behaviour of Redis and vector stores. It is intentionally qualitative, because real numbers depend on corpus size, embedding model, network placement, and caching.

Memory tactic	Read latency	Write latency	Token cost pressure	Infra cost	When it is worth it
Raw session history	Lowest	Lowest	Highest	Lowest	Simple multi-turn chat and short runs
Summary or compaction memory	Low to medium	Medium, because summarisation itself is a model step	Medium to low	Low to medium	Long-running work where the active run must continue
Structured profile and state	Low	Medium	Low	Low	Durable preferences, rules, and standing constraints
Vector or hybrid retrieval	Medium	Medium	Low to medium	Medium	Large corpora, searchable history, document grounding
Full replay of everything	High and increasingly unstable	Low	Highest	Low infra, high model spend	Almost never, except tiny corpora and debugging

Implementation examples

OpenAI's current stack gives two useful reference patterns. The first is Sessions for short-term continuity across runs. The second is state-based long-term memory, where structured profile fields and global memory notes are injected at session start, session notes are distilled during the run, and a consolidation step promotes only durable items into global memory. That inject → reason → distill → consolidate loop is one of the clearest public memory patterns available right now.

LangGraph provides a similar but framework-agnostic split. Checkpointers handle short-term thread memory and stores handle long-term search across conversations. The store can be searched inside nodes at runtime, which makes it a good reference design for assistants that need explicit orchestration rather than hidden framework magic.

Hermes is a useful public example of layered memory in the wild. Its built-in memory uses MEMORY.md, USER.md, and SQLite FTS5 session search, while external provider plugins add graph memory, semantic retrieval, automatic fact extraction, and user modelling. The full mechanics are documented in Hermes Agent Memory System, and the eight pluggable backends are compared in Agent memory providers compared.

OpenClaw offers a different take, with session pruning, optional active memory that runs before the main reply, and an opt-in Dreaming system for background memory consolidation. Those examples are worth paying attention to because they treat memory as an operational subsystem, not just a retrieval trick. For how OpenClaw maps onto the wider five-layer assistant stack, see the OpenClaw system overview.

Research prototypes point in the same direction. MemGPT uses hierarchical memory tiers and control flow for context management, A-MEM uses dynamic indexing and linking inspired by Zettelkasten, and Mem0 reports better accuracy with much lower p95 latency and token cost than full-context baselines on LoCoMo. You do not need to copy these systems wholesale, but their shared lesson is clear. Memory quality comes from selection and organisation, not from storing everything forever.

When memory helps versus hurts

Memory helps when the assistant repeatedly encounters stable preferences, durable constraints, reusable workflow lessons, or large external corpora that cannot fit in a prompt. OpenAI's reliable agents guide makes the distinction well. Compaction helps the current long-running run continue, while memory helps future runs reuse workflow lessons. That is the right mental model for most business assistants.

Memory hurts when the task is one-shot, the user state changes often, the retrieval index is noisy, or the system cannot reconcile conflicts. OpenAI's travel-memory example warns that session memory should not automatically become global memory, and it explicitly states that memory is not a security boundary. If your assistant treats every recalled string as truth, you have built a confusion engine, not a memory system.

A selective memory loop

The simplest robust memory loop is selective and staged. Load durable state, retrieve supporting context, answer, capture only candidate memories, then consolidate later. Both OpenAI's state-based pattern and recent memory papers move in this direction.

Without tracing and evals, memory changes are hard to debug. When you promote new facts or change retrieval policy, pair those changes with the observability patterns in Observability for LLM Systems so you can see which layer injected what.

Take-Away

The practical memory stack for assistants is not "just use a vector DB". It is working memory for the live run, structured state for durable truth, retrieval memory for supporting evidence, and a conservative consolidation policy that forgets as deliberately as it remembers. Recent research and current SDK guidance both point in that direction.

For the full assistant stack around this layer, start with AI Assistant Architecture. For Hermes-specific bounded memory and provider plugins, follow Hermes Agent Memory System and Agent memory providers compared.

AI for Knowledge Management: Real Workflows That Hold Up

Rost — Sun, 31 May 2026 13:51:08 +0000

AI is not replacing knowledge management; it is changing the shape of it for both individuals and teams.

Microsoft's Work Trend Index describes a move toward hybrid teams of humans and agents, and NIST's AI RMF argues that trustworthy AI systems need explicit roles, evaluation, and oversight rather than vague automation.
Those ideas fit neatly beside the human-centred practices in the site's Knowledge Management in 2026 pillar, which focuses on tools and methods long before any model is involved.

That is exactly the right frame for knowledge work: AI is best treated as an enrichment layer over notes, docs, runbooks, and research, not as a magical second brain that works without structure. A useful mental model is the one developed in PKM vs RAG vs Wiki vs Memory Systems, where human note systems, shared wikis, retrieval pipelines, and agent memory each play a distinct role instead of collapsing into a single tool.

The slightly opinionated version is this: if your notes are chaotic, AI will not rescue them. It will often make the chaos more fluent. Good knowledge management still starts with capture, naming, ownership, and source discipline. What AI changes is what you can do after capture: compress, extract, link, retrieve, and repackage information at useful speed. That view fits both modern prompting guidance, which recommends small, well-scoped tasks, and chunking guidance that preserves semantic units for retrieval instead of flattening everything into one blob.

Why AI changes knowledge management

The core shift is from static archives to active memory. Embeddings convert text into vectors that reflect relatedness and are commonly used for search, clustering, and recommendations. Retrieval systems can then surface semantically similar material even when the query shares few or no keywords with the source text. In practical terms, that means a note about "incident review" can still find a runbook chunk titled "post-deployment outage steps" without brittle exact-match rules.

This is why AI-augmented knowledge management is worth doing now. The enabling pieces are no longer exotic: embedding APIs are mainstream, vector stores are standard, local embedding models are easy to run, and production databases such as Postgres can do both exact and approximate nearest-neighbour search with pgvector. The result is not artificial knowledge in the philosophical sense. It is a much more practical thing: better recall, better compression, and better context at the moment someone needs to think, especially when paired with solid representation choices from work such as Retrieval vs Representation in Knowledge Systems. If your next step is implementation detail, the RAG cluster covers chunking, retrieval, reranking, and production patterns in depth.

Workflow patterns that actually work

The patterns that hold up in production are boring in the best way. They use AI for bounded transformations, not vague autonomy. In practice, three patterns show up again and again: summarisation, extraction, and linking suggestions. Those map neatly to what current tools do well: summarise within a clear scope, extract structured data with schemas, and compute semantic relatedness through embeddings and retrieval. They also map cleanly onto the layered view of knowledge systems behind concepts such as second brain workflows and LLM Wiki style compiled knowledge.

Summaries that preserve decisions

Summarisation works best when it stays close to the source and preserves the parts humans actually need later: decisions, unresolved questions, owners, dates, and links back to the original material. OpenAI's enterprise prompting guidance explicitly recommends "one prompt, one deliverable", simple headings, and clear success criteria. That is a good discipline for knowledge work too: summarise one meeting, one document, or one research item at a time, then store the summary beside the source. Do not ask a model to "summarise my knowledge base" and expect anything trustworthy.

A real workflow looks like this: capture meeting notes or a PDF, run a scoped summary prompt, store the summary with source references, then add a human check before it becomes canonical. If the source is a rich PDF, multimodal parsing can matter because slide decks and exported web pages often contain layout cues that plain text extraction misses. OpenAI's PDF parsing cookbook shows a practical split between text extraction and page-image analysis for turning rich PDFs into retrievable content.

# Context
You are assisting with team knowledge capture.

# Instructions
Summarise this meeting note in:
- 5 key points
- decisions made
- open questions
- actions with owners
- terms that should link to existing notes

# Constraints
- Do not invent details
- If something is unclear, mark it as uncertain
- Include the source note ID

Extraction that creates reusable fields

Extraction is where AI starts to feel genuinely infrastructural. Instead of storing only prose, you ask the model to populate reusable fields such as entities, systems, APIs, owners, action items, products, dates, claims, or risk tags. OpenAI's Structured Outputs feature is designed to keep responses aligned to a JSON Schema, and Ollama offers the same pattern locally with schema-based JSON output. That matters because useful knowledge systems are made of fields you can sort, filter, compare, and validate, not just paragraphs that sound clever.

OpenAI's long-document entity extraction example follows the right operational pattern: chunk the document, extract the relevant facts from each chunk, and then combine results. That same workflow works for postmortems, research papers, product docs, customer interviews, and support transcripts. In practice, I would extract more than named entities: I would also pull "needs follow-up", "contradicts existing note", and "candidate for evergreen note" because those fields create action, not just metadata.

{
  "source_id": "note-2026-05-22-incident-review",
  "summary": "Short summary here.",
  "entities": ["service-a", "postgres", "oauth"],
  "actions": [
    {"owner": "ops", "task": "rotate keys", "due": "2026-05-24"}
  ],
  "related_terms": ["token refresh", "deployment checklist"],
  "confidence": "medium"
}

Linking that turns notes into a graph

Link suggestions are the quiet workhorse of AI for knowledge management. Embeddings are explicitly used for search, clustering, and recommendations, which makes them a natural fit for related notes, similar incidents, see also, and you may want to merge these two docs features. Semantic retrieval is especially good at surfacing conceptually related content even when wording differs. That makes it far better than folder hierarchies alone for large note sets and technical documentation.

Dense semantic search should not be your only retrieval signal, though. Exact identifiers still matter: function names, package names, issue IDs, error codes, SKUs, regulation numbers. Google Research has shown that hybrid retrieval, which combines semantic and lexical signals, improves recall because each method finds relevant material the other misses. In a technical knowledge base, that is not an academic detail. It is the difference between finding the conceptually related design note and also finding the exact migration command someone needs at 2 a.m.

If you are already on Postgres, pgvector is the pragmatic option. It stores vectors with the rest of your data, supports exact search by default, and offers approximate indexing through HNSW and IVFFlat when you need more speed and can tolerate some recall trade-off. That is enough to build related-content suggestions, semantic search, and note deduplication without adding a separate vector database on day one.

The human plus AI loop

The model that actually works is not human or AI. It is capture -> AI enrich -> human refine. Microsoft describes the broader shift as humans working with assistants and then agent teams, while NIST's AI RMF and Playbook stress clearly defined human roles, responsibilities, and oversight in human-AI configurations. For knowledge management, that means humans remain accountable for the canonical note, the source of truth, and the final merge or publication decision. AI does the first-pass compression and cross-linking; humans do the judgement.

capture -> parse -> chunk -> embed -> enrich -> review -> publish
             |         |        |
             |         |        +-> related notes
             |         +-> retrieval index
             +-> structure-aware extraction

This division of labour is more than cautious process design. It matches how risk accumulates. NIST notes that understanding the limitations of human-AI interaction improves AI risk management, and that roles in oversight and use should be clearly differentiated. In practice, that means the model can draft titles, tags, summaries, and candidate links, but a person should approve anything that changes taxonomy, publishes external content, or overwrites an existing note. If you let the model silently rewrite your knowledge base, you are not building memory. You are outsourcing editorial control to a probabilistic system.

The tool choices that matter

The base layer is embeddings plus retrieval. OpenAI's embeddings guide frames embeddings as a way to measure relatedness between text strings, while the Retrieval API handles semantic search over your data through vector stores. For many teams, that is the minimum viable stack for AI-augmented knowledge management: parse content, chunk it well, embed it, and retrieve the right fragments before synthesis. If you only do one serious thing this quarter, make it retrieval-backed recall instead of a chat wrapper over raw documents.

Local models are the right answer when privacy, offline use, or cost control dominate. Ollama documents both local embeddings and structured outputs, and its product pages emphasise that data stays yours and that workloads can run entirely offline. That makes local-first pipelines sensible for internal notes, engineering runbooks, and sensitive research archives. My bias is simple: use local models for indexing, classification, and routine enrichment; reach for hosted APIs when you need stronger reasoning, multimodal extraction, or the best available model quality.

Do not ignore parsing and chunking. Unstructured's chunking docs recommend building chunks from semantic document elements rather than raw character boundaries when possible, and OpenAI's PDF cookbook shows why rich-document parsing matters for RAG. Structure-aware PDF work goes further: naive parsing can destroy tables, scramble reading order, and strip hierarchical headings, while structure-aware parsing preserves paragraphs, tables, and document hierarchy. In knowledge management, that is the difference between an index that understands your corpus and one that merely tokenises it.

Limitations worth respecting

Hallucination is still the obvious risk, but the more useful framing is insufficient context. RAG exists because large language models can hallucinate, use stale knowledge, and produce answers with weak traceability; retrieval helps by grounding generation in external knowledge. Even so, Google Research found that models often answer incorrectly instead of abstaining when the provided context is not sufficient. That matters for knowledge management because "I found something similar" is not the same as "I found enough to answer". Your system should preserve source references, expose uncertainty, and prefer abstention over confident fabrication.

Long context does not remove the need for retrieval discipline. The 2023 "Lost in the Middle" paper showed that model performance could degrade when relevant information sat in the middle of long inputs, and newer Google results show that at least some newer models have improved substantially on simple needle-in-a-haystack retrieval near context limits. The sober lesson is not "long context solves it" or "long context is useless". It is that you should test your actual workflows and corpus, because position effects, task type, and document structure still matter.

Loss of structure is the quieter failure mode, and in technical documentation it can be worse than hallucination because it poisons retrieval before the model even starts reasoning. Structure-aware PDF research shows that naive parsing can split tables, destroy their internal meaning, and break reading order, while semantic chunking systems try to preserve coherent document elements. If your source material includes tables, diagrams, code examples, or multi-column layouts, your parser is part of your knowledge system, not a boring preprocessing detail.

So the practical rule is this: keep the human editorial loop, preserve source links, use schemas for extraction, and treat retrieval quality as a product feature. AI does not replace PKM, team docs, or knowledge architecture. It changes the leverage. Used well, it turns raw notes into searchable, linkable, structured memory. Used badly, it turns your documentation into high-speed drift.

Multi-Tenancy Database Patterns with examples in Go

Rost — Thu, 28 May 2026 13:13:31 +0000

Multi-tenancy is a fundamental architectural pattern for SaaS applications, allowing multiple customers (tenants) to share the same application infrastructure while maintaining data isolation.

Choosing the right database pattern is crucial for scalability, security, and operational efficiency.

Overview of Multi-Tenancy Patterns

When designing a multi-tenant application, you have three primary database architecture patterns to choose from:

Shared Database, Shared Schema (most common)
Shared Database, Separate Schema
Separate Database per Tenant

Each pattern has distinct characteristics, trade-offs, and use cases. Let's explore each in detail.

Pattern 1: Shared Database, Shared Schema

This is the most common multi-tenancy pattern, where all tenants share the same database and schema, with a tenant_id column used to distinguish tenant data.

Architecture

┌─────────────────────────────────────┐
│     Single Database                 │
│  ┌───────────────────────────────┐  │
│  │  Shared Schema                │  │
│  │  - users (tenant_id, ...)     │  │
│  │  - orders (tenant_id, ...)    │  │
│  │  - products (tenant_id, ...)  │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘

Implementation Example

When implementing multi-tenant patterns, understanding SQL fundamentals is crucial. For a comprehensive reference on SQL commands and syntax, check out our SQL Cheatsheet. Here's how to set up the shared schema pattern:

-- Users table with tenant_id
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    tenant_id INTEGER NOT NULL,
    email VARCHAR(255) NOT NULL,
    name VARCHAR(255),
    created_at TIMESTAMP DEFAULT NOW(),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);

-- Index on tenant_id for performance
CREATE INDEX idx_users_tenant_id ON users(tenant_id);

-- Row-Level Security (PostgreSQL example)
ALTER TABLE users ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON users
    FOR ALL
    USING (tenant_id = current_setting('app.current_tenant')::INTEGER);

For more PostgreSQL-specific features and commands, including RLS policies, schema management, and performance tuning, refer to our PostgreSQL Cheatsheet.

Application-Level Filtering

When working with Go applications, choosing the right ORM can significantly impact your multi-tenant implementation. The examples below use GORM, but there are several excellent options available. For a detailed comparison of Go ORMs including GORM, Ent, Bun, and sqlc, see our comprehensive guide to Go ORMs for PostgreSQL.

// Example in Go with GORM
func GetUserByEmail(db *gorm.DB, tenantID uint, email string) (*User, error) {
    var user User
    err := db.Where("tenant_id = ? AND email = ?", tenantID, email).First(&user).Error
    return &user, err
}

// Middleware to set tenant context
func TenantMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        tenantID := extractTenantID(r) // From subdomain, header, or JWT
        ctx := context.WithValue(r.Context(), "tenant_id", tenantID)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

Shared Schema Pros

Lowest cost: Single database instance, minimal infrastructure
Easiest operations: One database to backup, monitor, and maintain
Simple schema changes: Migrations apply to all tenants at once
Best for high tenant count: Efficient resource utilization
Cross-tenant analytics: Easy to aggregate data across tenants

Shared Schema Cons

Weaker isolation: Data leakage risk if queries forget tenant_id filter
Noisy neighbor: One tenant's heavy workload can affect others
Limited customization: All tenants share the same schema
Compliance challenges: Harder to meet strict data isolation requirements
Backup complexity: Can't restore individual tenant data easily

Shared Schema Best For

SaaS applications with many small-to-medium tenants
Applications where tenants don't need custom schemas
Cost-sensitive startups
When tenant count is high (thousands+)

Pattern 2: Shared Database, Separate Schema

Each tenant gets their own schema within the same database, providing better isolation while sharing infrastructure.

Separate Schema Architecture

┌─────────────────────────────────────┐
│     Single Database                 │
│  ┌──────────┐  ┌──────────┐         │
│  │ Schema A │  │ Schema B │  ...    │
│  │ (Tenant1)│  │ (Tenant2)│         │
│  └──────────┘  └──────────┘         │
└─────────────────────────────────────┘

Separate Schema Implementation

PostgreSQL schemas are a powerful feature for multi-tenancy. For detailed information on PostgreSQL schema management, connection strings, and database administration commands, consult our PostgreSQL Cheatsheet.

-- Create schema for tenant
CREATE SCHEMA tenant_123;

-- Set search path for tenant operations
SET search_path TO tenant_123, public;

-- Create tables in tenant schema
CREATE TABLE tenant_123.users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    name VARCHAR(255),
    created_at TIMESTAMP DEFAULT NOW()
);

Application Connection Management

Managing database connections efficiently is critical for multi-tenant applications. The connection management code below uses GORM, but you might want to explore other ORM options. For a thorough comparison of Go ORMs including connection pooling, performance characteristics, and use cases, refer to our Go ORMs comparison guide.

// Connection string with schema search path
func GetTenantDB(tenantID uint) *gorm.DB {
    db := initializeDB()
    db.Exec(fmt.Sprintf("SET search_path TO tenant_%d, public", tenantID))
    return db
}

// Or use PostgreSQL connection string
// postgresql://user:pass@host/db?search_path=tenant_123

Separate Schema Pros

Better isolation: Schema-level separation reduces data leakage risk
Customization: Each tenant can have different table structures
Moderate cost: Still single database instance
Easier per-tenant backups: Can backup individual schemas
Better for compliance: Stronger than shared schema pattern

Separate Schema Cons

Schema management complexity: Migrations must run per tenant
Connection overhead: Need to set search_path per connection
Limited scalability: Schema count limits (PostgreSQL ~10k schemas)
Cross-tenant queries: More complex, requires dynamic schema references
Resource limits: Still shared database resources

Separate Schema Best For

Medium-scale SaaS (dozens to hundreds of tenants)
When tenants need schema customization
Applications needing better isolation than shared schema
When compliance requirements are moderate

Pattern 3: Separate Database per Tenant

Each tenant gets their own complete database instance, providing maximum isolation.

Separate Database Architecture

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Database 1  │  │  Database 2  │  │  Database 3  │
│  (Tenant A)  │  │  (Tenant B)  │  │  (Tenant C)  │
└──────────────┘  └──────────────┘  └──────────────┘

Separate Database Implementation

-- Create database for tenant
CREATE DATABASE tenant_enterprise_corp;

-- Connect to tenant database
\c tenant_enterprise_corp

-- Create tables (no tenant_id needed!)
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    name VARCHAR(255),
    created_at TIMESTAMP DEFAULT NOW()
);

Dynamic Connection Management

// Connection pool manager
type TenantDBManager struct {
    pools map[uint]*gorm.DB
    mu    sync.RWMutex
}

func (m *TenantDBManager) GetDB(tenantID uint) (*gorm.DB, error) {
    m.mu.RLock()
    if db, exists := m.pools[tenantID]; exists {
        m.mu.RUnlock()
        return db, nil
    }
    m.mu.RUnlock()

    m.mu.Lock()
    defer m.mu.Unlock()

    // Double-check after acquiring write lock
    if db, exists := m.pools[tenantID]; exists {
        return db, nil
    }

    // Create new connection
    db, err := gorm.Open(postgres.Open(fmt.Sprintf(
        "host=localhost user=dbuser password=dbpass dbname=tenant_%d sslmode=disable",
        tenantID,
    )), &gorm.Config{})

    if err != nil {
        return nil, err
    }

    m.pools[tenantID] = db
    return db, nil
}

Separate Database Pros

Maximum isolation: Complete data separation
Best security: No risk of cross-tenant data access
Full customization: Each tenant can have completely different schemas
Independent scaling: Scale tenant databases individually
Easy compliance: Meets strictest data isolation requirements
Per-tenant backups: Simple, independent backup/restore
No noisy neighbors: Tenant workloads don't affect each other

Separate Database Cons

Highest cost: Multiple database instances require more resources
Operational complexity: Managing many databases (backups, monitoring, migrations)
Connection limits: Each database instance has connection limits
Cross-tenant analytics: Requires data federation or ETL
Migration complexity: Must run migrations across all databases
Resource overhead: More memory, CPU, and storage needed

Separate Database Best For

Enterprise SaaS with high-value customers
Strict compliance requirements (HIPAA, GDPR, SOC 2)
When tenants need significant customization
Low to medium tenant count (dozens to low hundreds)
When tenants have very different data models

Security Considerations

Regardless of the pattern chosen, security is paramount:

1. Row-Level Security (RLS)

PostgreSQL RLS automatically filters queries by tenant, providing a database-level security layer. This feature is particularly powerful for multi-tenant applications. For more details on PostgreSQL RLS, security policies, and other advanced PostgreSQL features, see our PostgreSQL Cheatsheet.

-- Enable RLS
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;

-- Policy to isolate by tenant
CREATE POLICY tenant_isolation ON orders
    FOR ALL
    USING (tenant_id = current_setting('app.current_tenant')::INTEGER);

-- Application sets tenant context
SET app.current_tenant = '123';

2. Application-Level Filtering

Always filter by tenant_id in application code. The examples below use GORM, but different ORMs have their own approaches to query building. For guidance on choosing the right ORM for your multi-tenant application, check our comparison of Go ORMs.

// ❌ BAD - Missing tenant filter
db.Where("email = ?", email).First(&user)

// ✅ GOOD - Always include tenant filter
db.Where("tenant_id = ? AND email = ?", tenantID, email).First(&user)

// ✅ BETTER - Use scopes or middleware
db.Scopes(TenantScope(tenantID)).Where("email = ?", email).First(&user)

3. Connection Pooling

Use connection poolers that support tenant context:

// PgBouncer with transaction pooling
// Or use application-level connection routing

4. Audit Logging

Track all tenant data access:

type AuditLog struct {
    ID        uint
    TenantID  uint
    UserID    uint
    Action    string
    Table     string
    RecordID  uint
    Timestamp time.Time
    IPAddress string
}

Performance Optimization

Indexing Strategy

Proper indexing is crucial for multi-tenant database performance. Understanding SQL indexing strategies, including composite indexes and partial indexes, is essential. For a comprehensive reference on SQL commands including CREATE INDEX and query optimization, see our SQL Cheatsheet. For PostgreSQL-specific indexing features and performance tuning, refer to our PostgreSQL Cheatsheet.

-- Composite indexes for tenant queries
CREATE INDEX idx_orders_tenant_created ON orders(tenant_id, created_at DESC);
CREATE INDEX idx_orders_tenant_status ON orders(tenant_id, status);

-- Partial indexes for common tenant-specific queries
CREATE INDEX idx_orders_active_tenant ON orders(tenant_id, created_at)
WHERE status = 'active';

Query Optimization

// Use prepared statements for tenant queries
stmt := db.Prepare("SELECT * FROM users WHERE tenant_id = $1 AND email = $2")

// Batch operations per tenant
db.Where("tenant_id = ?", tenantID).Find(&users)

// Use connection pooling per tenant (for separate database pattern)

Monitoring

Effective database management tools are essential for monitoring multi-tenant applications. You'll need to track query performance, resource usage, and database health across all tenants. For comparing database management tools that can help with this, check out our DBeaver vs Beekeeper comparison. Both tools offer excellent features for managing and monitoring PostgreSQL databases in multi-tenant environments.

Monitor per-tenant metrics:

Query performance per tenant
Resource usage per tenant
Connection counts per tenant
Database size per tenant

Migration Strategy

Shared Schema Pattern

When implementing database migrations, your choice of ORM affects how you handle schema changes. The examples below use GORM's AutoMigrate feature, but different ORMs have different migration strategies. For detailed information on how various Go ORMs handle migrations and schema management, see our Go ORMs comparison.

// Migrations apply to all tenants automatically
func Migrate(db *gorm.DB) error {
    return db.AutoMigrate(&User{}, &Order{}, &Product{})
}

Separate Schema/Database Pattern

// Migrations must run per tenant
func MigrateAllTenants(tenantIDs []uint) error {
    for _, tenantID := range tenantIDs {
        db := GetTenantDB(tenantID)
        if err := db.AutoMigrate(&User{}, &Order{}); err != nil {
            return fmt.Errorf("tenant %d: %w", tenantID, err)
        }
    }
    return nil
}

Decision Matrix

Factor	Shared Schema	Separate Schema	Separate DB
Isolation	Low	Medium	High
Cost	Low	Medium	High
Scalability	High	Medium	Low-Medium
Customization	None	Medium	High
Operational Complexity	Low	Medium	High
Compliance	Limited	Good	Excellent
Best Tenant Count	1000+	10-1000	1-100

Hybrid Approach

You can combine patterns for different tenant tiers:

// Small tenants: Shared schema
if tenant.Tier == "standard" {
    return GetSharedDB(tenant.ID)
}

// Enterprise tenants: Separate database
if tenant.Tier == "enterprise" {
    return GetTenantDB(tenant.ID)
}

Best Practices

Always filter by tenant: Never trust application code alone; use RLS when possible. Understanding SQL fundamentals helps ensure proper query construction—refer to our SQL Cheatsheet for query best practices.
Monitor tenant resource usage: Identify and throttle noisy neighbors. Use database management tools like those compared in our DBeaver vs Beekeeper guide to track performance metrics.
Implement tenant context middleware: Centralize tenant extraction and validation. Your ORM choice affects how you implement this—see our Go ORMs comparison for different approaches.
Use connection pooling: Efficiently manage database connections. PostgreSQL-specific connection pooling strategies are covered in our PostgreSQL Cheatsheet.
Plan for tenant migration: Ability to move tenants between patterns
Implement soft delete: Use deleted_at instead of hard deletes for tenant data
Audit everything: Log all tenant data access for compliance
Test isolation: Regular security audits to prevent cross-tenant data leakage

Conclusion

Choosing the right multi-tenancy database pattern depends on your specific requirements for isolation, cost, scalability, and operational complexity. The Shared Database, Shared Schema pattern works well for most SaaS applications, while Separate Database per Tenant is necessary for enterprise customers with strict compliance needs.

Start with the simplest pattern that meets your requirements, and plan for migration to a more isolated pattern as your needs evolve. Always prioritize security and data isolation, regardless of the pattern chosen.

Useful Links

Zettelkasten for Developers: A Practical Method That Works

Rost — Mon, 25 May 2026 13:09:20 +0000

Developers do not usually suffer from a lack of information. We suffer from too much of it.

There are API docs, pull requests, production incidents, design discussions, meeting notes, architecture diagrams, code comments, Slack threads, research papers, experiments, bookmarks, and half-finished ideas sitting in five different tools. The hard part is not saving information. The hard part is turning it into reusable thinking.

That is where Zettelkasten becomes useful.

A Zettelkasten is often described as a note-taking system, but that undersells it. Used well, it is a personal knowledge system for developing ideas over time. For developers, it can become a practical bridge between code, architecture, debugging, learning, and writing.

The opinionated part is this: most developers should not use Zettelkasten as a romantic productivity hobby. Do not build a beautiful note museum. Build a working system that helps you solve problems, explain systems, and make better engineering decisions.

What Is Zettelkasten?

Zettelkasten means "slip box". The method is associated with sociologist Niklas Luhmann, who used a large collection of linked notes to develop ideas and write extensively.

The important lesson is not that he used paper cards. The important lesson is that his notes were not isolated files. Each note had a clear idea, a place in the system, and links to other notes. Over time, the system became more valuable because connections accumulated.

For developers, the modern version is simple:

Write one useful idea per note.
Link it to related notes.
Use those links to grow explanations, decisions, patterns, and articles.

That is it. The rest is implementation detail.

Why Developers Struggle With Knowledge Overload

Software development creates knowledge that is both detailed and temporary.

You learn why a cache invalidation bug happened. You discover a weird edge case in a framework. You compare two queueing strategies. You debug a production outage. You understand why a legacy service behaves strangely. You read a great article about distributed tracing.

Then, two months later, you vaguely remember that you once knew the answer.

The usual developer knowledge stack makes this worse:

Bookmarks store sources, not understanding.
Folders force early categorization.
Wikis become stale when nobody owns them.
TODO lists mix tasks with ideas.
Code comments explain local details, not broader concepts.
Chat messages disappear into history.

A Zettelkasten helps because it treats knowledge as a network, not a warehouse. If that framing sounds familiar from reading about building a second brain, that is not a coincidence — both methods attack the same gap between capture and reuse, but Zettelkasten's discipline of atomic notes and explicit links gives developers a more granular handle on technical ideas.

Core Principles of Zettelkasten

Atomic Notes

An atomic note contains one idea.

Not one topic. Not one article summary. Not one giant page called "PostgreSQL". One idea.

For example, these are too broad:

PostgreSQL notes
Kubernetes
Caching
System design

These are closer to atomic:

text Partial indexes reduce write overhead when queries target a small subset Kubernetes readiness probes protect traffic routing, not container startup Write-through caching improves consistency but increases write latency Idempotency keys turn retries into safe operations

Atomic notes are powerful because they are easier to link. A huge page can only be linked as a vague topic. A focused note can be connected to an exact concept, decision, bug, or system.

A good developer note should usually answer one of these questions:

What is the idea?
When does it matter?
What tradeoff does it expose?
Where have I seen it in real code?
What other concept does it connect to?

Linking

Links are the heart of the system.

The point is not to create a pretty graph. The point is to make ideas reusable.

When you write a note about idempotency keys, link it to notes about retries, distributed systems, payment processing, message queues, API design, and incident prevention. When you write a note about database migrations, link it to deploy safety, rollback strategy, backward compatibility, and feature flags.

A link should usually mean one of these things:

"This explains the same concept from another angle."
"This is a practical example of the idea."
"This is a tradeoff or counterpoint."
"This concept depends on that concept."
"This note belongs in a larger argument."

Avoid lazy links. Linking every note to every other note creates noise. The best links are intentional.

Emergence

Emergence is the part of Zettelkasten that sounds mystical, but it is practical.

You do not need to design the perfect structure upfront. You add useful notes, connect them honestly, and let clusters appear over time.

After a few months, you may notice that many notes connect around topics like:

API reliability
Observability
Developer experience
Event-driven architecture
Database performance
Technical debt
Documentation
Security reviews

Those clusters become future articles, internal docs, design principles, conference talks, onboarding material, or better engineering decisions.

This is why Zettelkasten is different from a folder hierarchy. Folders ask you to decide where knowledge belongs before you fully understand it. Links let knowledge belong to multiple contexts.

A Developer Adaptation Of Zettelkasten

Classic Zettelkasten advice often comes from academic writing — the personal knowledge management literature covers that tradition well. Developers need a slightly different version.

A developer Zettelkasten should connect three things:

Concepts
Code
Systems

Concepts

Concept notes explain reusable ideas.

Examples:

text Backpressure prevents fast producers from overwhelming slow consumers Optimistic locking detects conflicting writes without blocking readers Circuit breakers protect dependencies from repeated failing calls

These notes should be written in your own words. Copying documentation is not enough. The value comes from forcing yourself to explain the concept clearly.

A useful concept note can include:

A short explanation
A concrete example
A tradeoff
A link to a related pattern
A link to a real system where you used it

Code

Code notes capture practical implementation knowledge.

They are not random snippet dumps. A snippet is useful only when it explains a decision or pattern.

For example:

`markdown

Idempotent request handling with a database constraint

The safest implementation is often a unique constraint on the idempotency key.
The application can retry safely because duplicate requests resolve to the same
stored result instead of creating a second side effect.

[[Retries need idempotent operations]]
[[Database constraints are concurrency control]]
[[Payment APIs should treat network failure as unknown outcome]] `

Good code notes explain why the code works, when to use it, and what can go wrong.

Systems

System notes connect abstract ideas to your actual architecture.

For example:

text The billing service uses idempotency keys because payment provider calls may succeed even when our HTTP client times out.

This note can link to:

text Idempotency keys turn retries into safe operations Timeouts do not prove failure Payment APIs should model unknown outcomes Outbox pattern separates database writes from external side effects

This is where Zettelkasten becomes valuable for senior engineering work. It helps you build a memory of why systems are shaped the way they are.

A Practical Workflow

Step 1: Capture Fleeting Notes

A fleeting note is a rough capture. It does not need to be polished.

Examples:

text Look into why readiness probe failed during deploy. Maybe retries made the duplicate invoice bug worse. Good quote from incident review: timeout is not failure. Research: Postgres partial index for active rows only.

Use whatever is fastest: Obsidian daily note, Logseq journal, a text file, mobile notes, or a scratch buffer.

The rule is simple: capture quickly, process later.

Step 2: Process Notes Into Permanent Notes

Processing is where the value appears.

Turn rough notes into clear, reusable notes. Rewrite in your own words. Give each note a title that states the idea.

Bad title:

text Retries

Better title:

text Retries are safe only when the operation is idempotent

Bad note:

text Need idempotency for retries.

Better note:

text Retries can turn a temporary network problem into duplicate side effects. A retry is safe only when the operation can run more than once and still produce the same business result. For APIs, this often requires an idempotency key, a unique constraint, or a stored request result.

Step 3: Add Links While The Context Is Fresh

After writing the note, ask:

What does this explain?
What does this depend on?
Where have I seen this in code?
What is the opposite view?
What system would benefit from this?

Add only the links that help future you think.

Step 4: Create Index Notes Or Maps Of Content

Once a cluster grows, create an index note.

For example:

`markdown

API Reliability

Core ideas

[[Retries are safe only when the operation is idempotent]]
[[Timeouts do not prove failure]]
[[Circuit breakers reduce pressure on failing dependencies]]
[[Rate limits protect shared resources]]

Implementation patterns

[[Idempotency keys turn retries into safe operations]]
[[Outbox pattern separates persistence from delivery]]
[[Dead letter queues preserve failed messages for inspection]]

System examples

[[Billing service payment retry design]]
[[Webhook delivery failure handling]] `

This gives you navigation without forcing everything into folders.

Step 5: Use Notes To Produce Output

A Zettelkasten should produce something.

For developers, output can be:

Architecture decision records
Design documents
Blog posts
Debugging guides
Onboarding docs
Pull request explanations
Internal talks
Refactoring plans
Incident review insights

If your notes never influence your work, the system is too decorative.

Recommended Note Types For Developers

Fleeting Notes

Temporary notes for quick capture.

Use them for:

Ideas during coding
Debugging observations
Meeting fragments
Questions
Bookmarks to process later

Delete or convert them quickly. Do not let them become a swamp.

Literature Notes

Notes about external sources.

For developers, a source can be:

Documentation
Blog article
RFC
Source code
Conference talk
GitHub issue
Postmortem
Book chapter

Keep source notes separate from your own permanent notes. A source note says, "This source said this." A permanent note says, "I understand this idea this way."

Permanent Notes

These are the core of the Zettelkasten.

A permanent note should be:

Atomic
Written in your own words
Linked to related notes
Useful without needing the original source
Stable enough to revisit later

Project Notes

Project notes are allowed, but do not confuse them with permanent notes.

A project note might be:

text Migrate billing worker to queue v2

It can link to permanent notes like:

text Backpressure prevents queue consumers from collapsing Outbox pattern separates persistence from delivery Feature flags reduce deployment risk

Projects end. Concepts stay.

Tool Examples

Obsidian

Obsidian works well for developer Zettelkasten because it uses local Markdown files and supports internal links.

A simple Obsidian structure:

text notes/ fleeting/ sources/ permanent/ maps/ projects/

Example note:

`markdown

Timeouts do not prove failure

A timeout means the client stopped waiting. It does not prove the server failed.
The operation may have succeeded, failed, or still be running.

This matters for payment APIs, job queues, and any external side effect.

[[Retries are safe only when the operation is idempotent]]
[[Idempotency keys turn retries into safe operations]]
[[External side effects need reconciliation]] `

Obsidian is a good fit if you like file ownership, plain text, and editor-like workflows.

Logseq

Logseq is useful if you prefer outlining, daily journals, and block-level references.

Its block model works well for capturing small units of thought. You can write rough notes in the journal, then promote useful blocks into permanent notes.

Example Logseq-style workflow:

`text

Timeout during payment request does not prove payment failure.
- This should become a permanent note about unknown outcomes.
- Related: [[Idempotency]], [[Retries]], [[Payment APIs]] `

Logseq is a good fit if your thinking starts as outlines and you like block references. For a side-by-side comparison of both tools across workflow style, sync options, and plugin ecosystems, Obsidian vs Logseq maps the trade-offs clearly.

Plain Markdown And Git

You do not need a special app.

A Git repository of Markdown files can be enough:

text knowledge/ permanent/ sources/ maps/

Use normal Markdown links:

markdown [Retries are safe only when operations are idempotent](../permanent/retries-safe-only-with-idempotency.md)

This approach is boring, durable, and developer-friendly. That is a compliment.

Naming Notes

Prefer titles that make claims.

Weak titles:

text Caching Queues OAuth PostgreSQL indexes

Strong titles:

text Cache invalidation is a coordination problem Queues hide latency but do not remove work OAuth access tokens should be short lived Partial indexes are useful when queries target a subset

A claim-based title makes the note easier to understand and easier to link.

What To Put In A Developer Zettelkasten

Good candidates:

Architecture principles
Debugging lessons
Production incident insights
API design rules
Database patterns
Security assumptions
Performance tradeoffs
Framework edge cases
Refactoring heuristics
Testing strategies
Deployment lessons
Code review patterns

Poor candidates:

Raw meeting transcripts
Unprocessed bookmarks
Huge copied documentation pages
Random snippets with no explanation
Task lists
Secrets
Credentials
Anything that belongs in official company documentation only

A personal Zettelkasten can reference work, but it should not become an unsafe shadow copy of private systems.

Common Mistakes

Mistake 1: Over-Structuring Too Early

Developers love structure. That is sometimes a problem.

Do not spend the first week designing folders, tags, templates, naming conventions, dashboards, and automation. You do not yet know what structure your notes need.

Start with a small number of note types:

text fleeting sources permanent maps projects

Let complexity earn its place.

Mistake 2: Treating It Like Folders

A Zettelkasten is not a better folder tree.

If every note belongs to exactly one folder and has no meaningful links, you have built a filing cabinet. That may still be useful, but it is not Zettelkasten.

The value comes from connections:

text API retries -> idempotency -> database constraints -> payment safety -> incident prevention

That chain is more useful than a folder called "Backend".

Mistake 3: Saving Instead Of Thinking

Copying is not learning.

A saved paragraph from documentation may help later, but a rewritten explanation helps now. The act of restating an idea in your own words is where understanding improves.

A good rule:

text Do not create a permanent note until you can explain the idea without copying.

Mistake 4: Linking Everything

Too many links are as bad as too few.

Do not link words just because they exist. Link ideas because the relationship matters.

A useful link should help future you answer:

text Why is this connected?

Mistake 5: Confusing Tags With Structure

Tags are useful for status and broad grouping:

`text

todo

source

security

draft

But tags should not carry the whole system. If you rely only on tags, you lose the richer meaning of direct links.

A link says:

text This idea relates to that idea in a specific way.

A tag usually says:

text This belongs to a broad bucket.

Both are useful. They are not the same.

Mistake 6: Never Producing Output

A Zettelkasten that never produces output becomes a private archive.

Output does not have to mean public writing. It can be a design doc, an incident review, a better pull request, or a clear explanation to a teammate.

The system should make your thinking easier to reuse.

A Minimal Template

Use a small template. Resist the urge to create a form with fifteen fields.

`markdown

Title as a claim

Idea

Explain the idea in your own words.

Why it matters

Describe the practical impact.

Example

Show a code, system, or debugging example.

Tradeoffs

Mention limits, risks, or counterpoints.

Example: From Bug To Zettelkasten Notes

Imagine you fixed a bug where users were charged twice after a timeout.

A weak note would be:

text Payment bug - retries caused duplicate charge.

A stronger set of notes might be:

text Timeouts do not prove failure Retries are safe only when the operation is idempotent Idempotency keys turn retries into safe operations Payment APIs should model unknown outcomes Database constraints are concurrency control

Now the bug has become reusable engineering knowledge.

Later, those notes can support:

A postmortem
A design doc for payment retries
A blog post about idempotency
A checklist for external API integrations
A code review comment
A safer implementation

That is the practical value of Zettelkasten.

A Weekly Maintenance Routine

You do not need a complicated review process.

Once a week:

Process rough notes.
Delete notes that no longer matter.
Convert useful ideas into permanent notes.
Add missing links.
Promote clusters into map notes.
Pick one note and turn it into output.

Keep it lightweight. The system should support development, not compete with it.

Practical Rules

Use these rules to keep the system healthy:

One idea per note.
Write titles as claims.
Prefer links over folders.
Keep source notes separate from your own ideas.
Connect notes to real code and real systems.
Create map notes only when a cluster exists.
Delete low-value notes.
Do not automate before you understand your workflow.
Use the system to produce something.

When Zettelkasten Is Not Worth It

Zettelkasten is not the answer to every problem.

It may be overkill if:

You only need a task manager.
You rarely revisit technical ideas.
You do not write, teach, design, or document.
Your notes are mostly short-lived project details.
You are using it to avoid doing the actual work.

It is most useful when your work depends on compounding understanding.

That includes senior engineering, architecture, technical leadership, debugging complex systems, writing, consulting, research, and learning deeply over many years.

Final Thoughts

For developers, Zettelkasten is not about collecting notes. It is about building a thinking environment.

The method works best when it stays practical: atomic notes, meaningful links, real examples, and regular output. Connect concepts to code. Connect code to systems. Connect systems to decisions.

Do not try to build the perfect second brain. Build a useful one.

A good developer Zettelkasten should help you answer better questions:

text Where have I seen this problem before? What concept explains this bug? What tradeoff are we making? What pattern applies here? What should I write down so I do not relearn this again?

That is enough.

OpenClaw vs Hermes Agent: Stars, Downloads & Usage 2026

Rost — Mon, 25 May 2026 13:09:06 +0000

Open-source AI agent frameworks are exploding in popularity on GitHub.
Two projects at the core of the self-hosted AI systems ecosystem — OpenClaw and Hermes Agent — have pulled so far ahead that the rest of the field is fighting for a distant third place.

Here is the full picture as of May 2026.

The Leaderboard

Star counts are live data fetched from the GitHub API on May 21, 2026. Repos are sorted by current stars, descending.

Rank	Project	GitHub repo	Language	Stars	Releases last 30 days
1	OpenClaw	`openclaw/openclaw`	TypeScript	373,616	62
2	Hermes Agent	`NousResearch/hermes-agent`	Python	160,175	5
3	Nanobot	`HKUDS/nanobot`	Python	42,873	2
4	AstrBot	`AstrBotDevs/AstrBot`	Python	32,709	11
5	ZeroClaw	`zeroclaw-labs/zeroclaw`	Rust	31,500	≥1
6	NanoClaw	`nanocoai/nanoclaw`	TypeScript	29,143	≥1
7	PicoClaw	`sipeed/picoclaw`	Go	29,121	3
8	AionUi	`iOfficeAI/AionUi`	TypeScript	26,025	≥3
9	NemoClaw	`NVIDIA/NemoClaw`	TypeScript	20,571	0
10	OpenFang	`RightNow-AI/openfang`	Rust	17,599	≥5
11	LangBot	`langbot-app/LangBot`	Python	16,084	1
12	memU	`NevaMind-AI/memU`	Python	13,672	0
13	IronClaw	`nearai/ironclaw`	Rust	12,305	4
14	Moltworker	`cloudflare/moltworker`	TypeScript	9,899	0
15	MemOS	`MemTensor/MemOS`	Python	9,246	≥2
16	ClawWork	`HKUDS/ClawWork`	Python	8,111	0
17	NullClaw	`nullclaw/nullclaw`	Zig	7,603	2
18	MimicLaw	`memovai/mimiclaw`	C	5,422	0
19	Moltis	`moltis-org/moltis`	Rust	2,697	≥3
20	Clawra	`SumeLabs/clawra`	TypeScript	2,298	0

OpenClaw: 373k Stars and Still Growing

OpenClaw is a personal AI assistant framework built in TypeScript. It runs entirely on the user's own device and connects to over 50 messaging platforms — WhatsApp, Telegram, Slack, Discord, and more — through a single unified interface.

The project launched in November 2025 but truly ignited on January 30, 2026, reaching 100,000 stars within 48 hours of its relaunch. By April 2026 it had overtaken React to become the most-starred software repository in GitHub's history. At the time of this writing it sits at 373,616 stars, 72,000+ forks, and 360 contributors.

The release cadence is extraordinary: 62 tagged releases in the last 30 days puts it in a category of its own in terms of iteration speed. The full arc of how OpenClaw grew from a weekend prototype to GitHub's most-starred repository — including the economics behind the viral spike and the April 2026 subscription cutoff that reshaped weekly growth — is detailed in the OpenClaw rise and fall timeline.

Hermes Agent: The Challenger

Nous Research's Hermes Agent markets itself as "the agent that grows with you." It is a self-improving AI agent built in Python with a built-in learning loop — it creates new skills from experience, searches past conversations for relevant context, and can run on a range of infrastructure options from local hardware to cloud.

Created in July 2025 and now at 160,175 stars, Hermes Agent recently surpassed OpenClaw as the world's most-used open-source AI agent by daily token processing on OpenRouter — though OpenClaw still leads in cumulative all-time usage. The gap between the two in raw GitHub stars remains large (over 200k), but Hermes Agent's trajectory is notably steeper.

Mid-field: The 20k–45k Band

The third through eighth positions are all clustered between 26k and 43k stars, making ranking changes here frequent:

Nanobot (HKUDS, 42,873 ⭐) — Python, lightweight graph-based task orchestration from the HKU Data Science lab.
AstrBot (AstrBotDevs, 32,709 ⭐) — Python, multi-platform chatbot framework with active release history (11 releases in the last 30 days).
ZeroClaw (zeroclaw-labs, 31,500 ⭐) — Rust, systems-level agent runtime targeting low-latency deployments.
NanoClaw (nanocoai, 29,143 ⭐) — TypeScript, recently migrated from qwibitai/nanoclaw to nanocoai/nanoclaw; the rename caused a brief star-count gap in trackers.
PicoClaw (Sipeed, 29,121 ⭐) — Go, embedded-friendly agent framework. Only 22 stars separate it from NanoClaw.
AionUi (iOfficeAI, 26,025 ⭐) — TypeScript, focuses on agentic UI generation with a visual workflow editor.

Language Breakdown

Language	Repos in top 20	Total stars
Python	8	294,897
TypeScript	7	470,155
Rust	3	51,502
Go	1	29,121
Zig	1	7,603
C	1	5,422

TypeScript leads in total star weight — largely because of OpenClaw itself — while Python holds the most individual projects. Rust is carving out a niche in the performance-sensitive tier (ZeroClaw, OpenFang, IronClaw).

Release Velocity vs Star Count

High star counts do not always mean high release velocity. Several top-starred repos (NemoClaw, memU, ClawWork, Clawra, MimicLaw) show zero releases in the last 30 days — they may be in maintenance mode or experiencing slower development cycles.

AstrBot stands out in the mid-field with 11 releases in 30 days, suggesting active feature development. OpenFang (≥5) and Moltis (≥3) are also moving quickly relative to their star counts, which may signal emerging momentum.

Notable Moves Since Last Snapshot

NanoClaw renamed org from qwibitai to nanocoai; updated link in the table above.
NemoClaw language corrected to TypeScript (previously listed as JavaScript in older data).
AionUi gained ~800 stars, moving from 8th to a stronger 8th position.
MemOS crossed 9,000 stars.

OpenRouter Usage Rankings

GitHub stars measure mindshare; OpenRouter token volume measures actual runtime usage. The two charts tell different stories.

The table below shows the global daily ranking on OpenRouter as of May 21, 2026, filtered to apps and agents that have opted into usage attribution. Counts are daily tokens processed through the platform.

Rank	App / Agent	Category	Daily tokens
1	Hermes Agent	Personal / CLI Agents	458 B
2	OpenClaw	Personal / CLI Agents	173 B
3	Kilo Code	CLI / IDE Agents	163 B
4	Descript	Video Generation	68.1 B
5	Claude Code	CLI Agents	64.1 B
6	pi	CLI Agents	58 B
8	Janitor AI	Roleplay	28.4 B
9	ISEKAI ZERO	Game	26.8 B
10	CSS AI Pro	—	25.4 B
11	Cline	IDE / CLI Agents	23.5 B
12	Roo Code	IDE / Cloud Agents	20.1 B
13	Lemonade	Programming App	20 B
14	Mira	Personal Agents	15.2 B
15	VidMuse	Video Generation	13.3 B
16	AA-LCR Benchmark	Research	9.42 B
18	SillyTavern	Roleplay	7.84 B
19	OpenHands	CLI Agents	7.21 B
20	Nous Research API	General Chat	6.63 B

Gaps in rank numbers (e.g., no 7 or 17) reflect apps without public attribution at the time of retrieval; OpenRouter lists 60 apps in total.

All-time cumulative tokens

The daily leader and the all-time leader have swapped since earlier in the year. As of today Hermes Agent has also overtaken OpenClaw on the all-time chart — a milestone that crossed some time after the May 10 daily flip.

App / Agent	All-time tokens
Hermes Agent	8.14 T
OpenClaw	7.18 T
Kilo Code	5.21 T
Claude Code	2.6 T

What the numbers mean

The gap between Hermes Agent (458 B daily) and OpenClaw (173 B) is now wider than it was on May 10, when the flip first happened at 224 B vs 186 B. Hermes has more than doubled its daily volume in 11 days; OpenClaw's daily volume has declined.

The architecture difference explains a lot of this. OpenClaw is session-native — it resets between runs, which means every session re-pays the full context-stuffing cost. Hermes is a persistent runtime with a three-layer memory system (identity snapshot, SQLite FTS5 session database, self-written procedural skill files). Once a skill is written, repeat tasks cost a fraction of the tokens.

For the coding-agent sub-category specifically, the top five are Hermes Agent, OpenClaw, Kilo Code, Claude Code, and pi. Cline (#11) and Roo Code (#12) round out the open-source coding-agent tier, both crossing 20 B daily tokens.

The driver of Hermes's May acceleration was the v0.13.0 "Tenacity" release (May 7, 2026): 864 commits, 588 merged PRs, 295 contributors. That release shipped a Kanban-style durable multi-agent task board with heartbeat monitoring and hallucination recovery, plus eight P0 security fixes and Google Chat as the 20th messaging integration.

Community Health

GitHub repository metrics reveal a sharp contrast in project maturity and maintenance style between the two leaders.

Metric	OpenClaw	Hermes Agent
Issue close rate	89.9 %	37.2 %
Contributors	360	400
Forks	72,696	26,000
Releases shipped (total)	82+	14+
Disclosed CVEs (2026 YTD)	9 in 4 days (March 2026)	0
Worst CVE severity	CVSS 9.9	—
Exposed public instances	135,000+ across 82 countries	Not separately tracked
Security response (v0.13.0)	—	8 P0 fixes, default redaction on

OpenClaw's 89.9 % issue close rate reflects a well-staffed, responsive maintainer team — the highest of any project in this space. Its release cadence (62 in the last 30 days alone) is exceptional, but that velocity has a cost: roughly a quarter of updates reportedly break response delivery on at least one channel, and the March 2026 CVE cluster (nine issues in four days, the worst at CVSS 9.9) forced emergency patching at scale. Shadowserver confirmed over 135,000 exposed Gateway instances across 82 countries in the same window. The OpenClaw team does publish fixes fast; the problem is that a community of this size patches slowly.

Hermes Agent's 37.2 % issue close rate is the expected profile for a three-month-old project with a backlog accumulating faster than it can be triaged. The security record so far is clean — zero disclosed agent-specific CVEs as of May 2026 — though that partly reflects fewer eyes on the codebase. The v0.13.0 "Tenacity" release shipped eight P0 fixes proactively, before any public disclosure, which is a good signal of security culture.

Ecosystem Size

Package downloads

Package	Registry	Weekly downloads
`openclaw` (main)	npm	5,344,931
`@tencent-weixin/openclaw-weixin`	npm	230,903
`@ollama/openclaw-web-search`	npm	160,221
`@paperclipai/adapter-openclaw-gateway`	npm	159,310
`@larksuite/openclaw-lark`	npm	115,964
`hermes-agent` (main)	PyPI	53,134

The raw numbers are not directly comparable — npm counts installs on every npm install (including CI runs), while PyPI counts pip installs. OpenClaw also has a larger ecosystem of third-party adapter packages that each pull in the core. Even so, the order-of-magnitude difference reflects OpenClaw's deeper penetration of automated pipelines and developer toolchains.

Hermes Agent at 53,000 PyPI downloads per week is not a small number for a three-month-old Python tool. Its install rate has grown roughly linearly with the GitHub star count.

Skills and integrations

Dimension	OpenClaw	Hermes Agent
Third-party skill marketplace	ClawHub — 44,000+ skills	None yet (self-generated only)
Messaging integrations (official)	50+ channels	20 channels
Community repositories (GitHub)	Large (untracked by maintainers)	80+ quality-filtered
Skill libraries (community)	Embedded in ClawHub	17 curated
Multi-agent orchestration frameworks	Built-in ACP swarm	9 third-party
External memory providers	Via skills	8 native

ClawHub is OpenClaw's most durable moat: 44,000 community-maintained skills covering integrations, automations, and workflows that would take months to replicate. Hermes's answer is to generate skills from its own task completions rather than pull them from a marketplace — a fundamentally different philosophy that pays off on deep, repeated tasks but leaves gaps on long-tail integrations. The eight external memory backends Hermes ships natively — Honcho, OpenViking, Mem0, Hindsight, and four more — are compared in detail in Agent Memory Providers Compared.

One security note on ClawHub: in Q1 2026 Koi Security identified 341 malicious entries in the registry, prompting OpenClaw to add a verification layer to the skill submission pipeline. A detailed guide to vetting skills, understanding which are safe to install, and navigating ClawHub's quality tiers is in OpenClaw Skills Ecosystem and Practical Production Picks.

Community Sentiment

A synthesis of Reddit threads across r/homeautomation, r/selfhosted, and r/MachineLearning (compiled by kilo.ai) breaks down operator preferences as follows:

Stance	Share
Stay on OpenClaw	35 %
Switched fully to Hermes	30 %
Run both side by side	20 %
Withholding judgment on Hermes	15 %

The 15 % holding off on Hermes are primarily concerned about what some users characterise as coordinated promotion activity from newly created accounts in Hermes-related threads — a pattern common to fast-growing projects but notable enough that veteran community members flag it.

Top OpenClaw complaints (by upvote volume)

Release breakage — most-upvoted complaint has 305 votes: "Every single update ships more bugs and problems than before." An estimated 25 % of releases break response delivery on at least one channel.
Memory drift — agents forget prior instructions across sessions, requiring users to re-establish context manually.
Self-host friction — disproportionate time spent on Docker configuration, SSH setup, and YAML tuning relative to actual agent work.

Top Hermes Agent complaints (by frequency)

Unreliable self-evaluation — the agent occasionally reports task success when the outcome was a partial failure.
Skill file overwriting — auto-improvement rewrites manually tuned skill files, discarding intentional customisation.
Integration gaps — ClawHub has a skill for almost everything; Hermes does not, and self-generation takes time to catch up.

The "run both" pattern (20 % of operators) is the most architecturally interesting: OpenClaw as the channel-and-routing layer up front, with Hermes as the deep-specialist backend. Messages arrive via Telegram or Slack, OpenClaw routes them, and the tasks where compounding matters are dispatched to a Hermes instance that has been improving on exactly those workflows for weeks.

Search Interest Trend

Tracking the project growth leaderboard (weekly new GitHub stars, a cleaner signal than raw star count) shows a clear momentum reversal as of May 2026.

Project	Weekly star growth	Leaderboard position
Claw-code	+7,000	#1
Hermes Agent	+3,800	#3
OpenClaw	+1,700	#11

OpenClaw had a +40,000/week peak in early February 2026 during the post-relaunch explosion. At +1,700/week in May, it is still growing in absolute terms — 373k stars does not happen without weekly adds — but it has settled into a mature project cadence, not a growth sprint.

Hermes Agent at +3,800/week is the fastest-growing agent runtime on the leaderboard right now, despite having less than half of OpenClaw's cumulative stars. Its growth curve is steeper than OpenClaw's was at the same age (week 12 post-launch).

The broader search interest trend corroborates the star-growth pattern. Queries for "Hermes Agent" and "hermes-agent install" have been rising consistently since the February launch; "OpenClaw" search volume peaked in late January and has been flat-to-declining since. The intersection point — where Hermes search volume equals OpenClaw's — has not yet been reached, but the trajectories suggest it will cross sometime in Q3 2026 if current rates hold.

The HN community has also shifted: threads about OpenClaw now centre on security hardening, transport trust (Telegram's lack of default end-to-end encryption), and maintenance overhead. Threads about Hermes Agent are still mostly "how do I set this up for X" — an earlier-stage energy that reflects a project still in its adoption phase.

DEV Community: Rost

PARA Method for Engineers: Organize Knowledge by Action

The Problem With Topic-Based Organization

The Four Buckets

Projects

Areas

Resources

Archives

PARA in Practice for Engineers

Mapping a Typical Engineer's Knowledge

PARA and Zettelkasten: A Practical Hybrid

Common Failures

Over-Archiving

Areas Becoming Dumping Grounds

Resources Growing Without Review

Skipping the Weekly Review

Implementation in Obsidian

Implementation in Plain Files

When PARA Makes Sense

Decision Framework

Final Thoughts

Evergreen Notes: Write Notes That Compound Over Time

What Makes a Note Evergreen

Atomic

Standalone

Evolving

Linked

Note Types and When to Use Each

Writing Evergreen Technical Notes

What to Avoid

Good Candidates for Evergreen Notes in Engineering

The Evergreen Workflow

Step 1: Capture Fleeting Notes

Step 2: Process Into Evergreen Notes Within 48 Hours

Step 3: Connect to Existing Notes

Step 4: Revisit and Improve

Evergreen Notes and Documentation

Evergreen Notes and RAG Systems

Common Pitfalls

Writing Too Broadly

Writing Too Narrowly

Confusing "Evergreen" With "Never Changes"

Skipping the Processing Step

Tools

Obsidian

Plain Markdown With Git

Starting From Zero

Final Thoughts

Cost Optimization for LLM Systems: Where the Money Actually Goes

Token budgeting

Strategy 1: Per-Session Budgets

Strategy 2: Per-Task Budgets

Strategy 3: Adaptive Budgets

API vs local inference

Fallback strategies

Strategy 1: Quality-Based Fallback

Strategy 2: Latency-Based Fallback

Caching

Strategy 1: Prompt Caching

Strategy 2: Semantic Caching

When optimization helps

Tradeoffs

Related

LLM Guardrails in Practice: What Actually Works

Input validation

Strategy 1: Prompt Sanitization

Strategy 2: Input Length Limits

Strategy 3: Content Filtering

Output filtering

Strategy 1: Response Validation

Strategy 2: Content Filtering

Strategy 3: Fact-Checking

Safety mechanisms

Strategy 1: Rate Limiting

Strategy 2: Token Budgeting

Strategy 3: Context Window Management

Compliance

Pattern 1: Data Residency

Pattern 2: Audit Logging

Putting it together