DEV Community: Zoricic

Three Things Had to Align: The Real Story Behind the LLM Revolution

Zoricic — Tue, 31 Mar 2026 20:39:35 +0000

ChatGPT didn't come out of nowhere. It's the result of 60 years of dead ends, one accidental breakthrough, and three completely separate technologies all maturing at exactly the same moment.

Before the Revolution: Chatbots That Didn't Learn

AI is older than most people realize. But early AI looked nothing like what we have today.

ELIZA (1966) was the first chatbot. It simulated conversation using pattern matching — if you typed "I feel sad," it replied "Why do you feel sad?" It didn't learn anything. It followed hand-written rules. Impressive for 1966. Useless for anything complex.

RNNs emerged in the 1990s as the dominant approach for language processing. LSTMs (Long Short-Term Memory networks), invented by Hochreiter & Schmidhuber in 1997, improved on RNNs specifically to address memory limitations — but both shared the same fundamental design: reading text sequentially, word by word, like looking through a keyhole. By the time either reached the end of a long sentence, the beginning was largely forgotten.

"The dog, which had been chasing the cat down the long street, was tired."
                                                               ↑
                               RNN had forgotten "dog" here ──┘

Meanwhile, Google Was Running on Rules

While researchers struggled with language models, Google was building search using a series of named algorithms — each solving a specific problem:

Algorithm	Year	What It Did
PageRank	1998	Ranked pages by how many other pages linked to them
Panda	2011	Penalized low-quality and duplicate content
Penguin	2012	Penalized sites that bought fake links
Hummingbird	2013	First steps toward understanding concepts, not just keywords
RankBrain	2015	First use of machine learning — handled never-seen-before queries

RankBrain was Google's first real AI in search. It used mathematical vectors to connect unknown words to known ones. But it still couldn't write a single sentence — and it still saw words as isolated units, not as parts of connected thought.

The forgetting problem and the keyword problem were the same problem in different clothes. Both systems failed to capture relationships between words across distance. The fix came from a single paper.

The Breakthrough: "Attention Is All You Need" (2017)

In 2017, a team of Google researchers published a paper with a confident title: Attention Is All You Need. The paper addressed sequence transduction broadly — mapping one sequence to another — and used machine translation as the benchmark task, showing the Transformer outperforming existing approaches on English-to-German and English-to-French. What they invented was the Transformer architecture, and it accidentally became the foundation for every major AI system since. Google Translate's adoption of Transformers was gradual and varied by language pair; the architecture didn't replace the previous approach across the board overnight.

The Core Idea: Self-Attention

Instead of reading text sequentially, the Transformer looks at the entire input at once and assigns a weight — an importance score — to each word relative to every other word.

When processing "tired" in our earlier example:

"The dog, which had been chasing the cat down the long street, was tired."

  Word      Attention Weight
  ───────────────────────────
  dog        HIGH  ← subject; dogs can be tired
  chasing    MED   ← action dog performed
  cat        LOW   ← object of chasing; not tired
  street     LOW   ← location; not tired

The model builds a relationship map of the entire sentence simultaneously. Nothing is read sequentially. Nothing is forgotten mid-sentence.

Why This Required New Hardware

Self-attention is parallelizable — every word's relationship to every other word can be computed at the same time, not one after another.

RNN:         [word1] → [word2] → [word3] → result
             Sequential. Each step waits for the previous.

Transformer: [word1] ↔ [word2] ↔ [word3]
               ↕          ↕         ↕
             All relationships computed simultaneously.

GPUs were designed to render thousands of pixels in parallel for video games. That same parallel architecture maps directly onto self-attention calculations. The match was accidental — and decisive.

Training GPT-3 without modern GPU hardware wouldn't take months. It would take hundreds of years.

Three Things Had to Align

Here's the part most explanations skip: if the Transformer paper had been published in 2005, nothing would have happened.

Modern LLMs exist because three things converged at the same moment:

┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
│  THE ALGORITHM  │   │    THE DATA     │   │  THE HARDWARE   │
│                 │   │                 │   │                 │
│  Transformer    │ + │  Billions of    │ + │  GPU clusters   │
│  architecture   │   │  internet pages,│   │  powerful       │
│  (the recipe)   │   │  books, Wikipedia│  │  enough to run  │
│                 │   │  (ingredients)  │   │  the math       │
└─────────────────┘   └─────────────────┘   └─────────────────┘
                              ↓
                  All three align ~2017
                              ↓
                  LLM revolution begins

The data requirement is the most underestimated factor. LLMs need billions of sentences to develop statistical understanding of language. The internet, Wikipedia, and digitized books only reached the necessary scale in the early 2010s. The algorithm existing in 2005 would have had nothing meaningful to train on.

Miss any one of the three — and ChatGPT doesn't exist.

Two Directions from One Architecture

Once the Transformer existed, two companies used it to solve two completely different problems.

BERT — Google (2018)

Google needed to understand language, not generate it. Search requires answering: what does this query actually mean?

BERT (Bidirectional Encoder Representations from Transformers) trained using Masked Language Modeling: researchers masked 15% of words in a sentence and trained the model to predict them from surrounding context.

"Paris is the [MASK] of France."  →  BERT learns: "capital"

The key innovation: bidirectionality. Previous models read left-to-right. BERT reads the full sentence in all directions at once.

Real-world impact: Google Search circa 2019. Before BERT, searching "can I pick up a prescription for someone else at the pharmacy" matched keywords. After BERT, Google understood that "for someone else" was the semantically critical phrase.

GPT Series — OpenAI (2018–2020)

OpenAI needed to generate language. The decoder half of the Transformer — predicting the next word given all previous words — became the basis for GPT.

Model	Year	Milestone
GPT-1	2018	Proved large-scale pre-training transfers to other tasks
GPT-2	2019	Outputs convincing enough to trigger public misuse debates
GPT-3	2020	175 billion parameters — first truly massive, general-purpose model

BERT vs. GPT compared:

Aspect	BERT (Google)	GPT (OpenAI)
Reading direction	Bidirectional	Left to right
Primary goal	Understanding	Generation
Training signal	Masked word prediction	Next word prediction
Used for	Search, Q&A, classification	Chatbots, writing, code

Neither is strictly better. They're optimized for different jobs.

The Missing Piece: Why GPT-3 Wasn't ChatGPT

GPT-3 could generate fluent text. But left to its own training objective — predict the next word — it was unreliable as an assistant. Ask it a question and it might answer it, continue the question, or list five more similar questions. It had no concept of being helpful.

The gap between GPT-3 (2020) and ChatGPT (2022) is two techniques:

Instruction tuning fine-tunes a pre-trained model on examples of instructions paired with good responses. Instead of "predict the next token in internet text," the training signal becomes "given this instruction, produce a useful response." The model learns to follow directions.

RLHF (Reinforcement Learning from Human Feedback) goes further. Human raters compare pairs of model outputs and rank which is better. Those rankings train a reward model — a separate model that scores outputs. The language model is then fine-tuned using reinforcement learning to maximize that score.

Pre-trained GPT-3
        │
        ▼
Instruction tuning (supervised, labeled examples)
        │
        ▼
RLHF — human raters rank outputs → reward model → RL fine-tuning
        │
        ▼
InstructGPT / ChatGPT — helpful, harmless, honest

This is why ChatGPT felt qualitatively different from GPT-3 API users' experience. The underlying model was similar in architecture. What changed was the training objective: from "predict internet text" to "be useful to a human."

OpenAI published the InstructGPT paper in early 2022 describing this approach. ChatGPT launched in November 2022 using the same method.

Beyond Text: Multimodal Models

LLMs started as text-only systems. By 2024, the leading models — GPT-4o, Gemini 1.5, Claude 3 — process text, images, audio, and video within the same model.

The core insight: the Transformer's attention mechanism is not specific to text. An image can be divided into patches and treated as a sequence of tokens. Audio can be converted to spectrograms and similarly tokenized. The same self-attention machinery that relates words to each other can relate image patches to words to audio frames.

┌─────────────────────────────────────────────────────┐
│              Multimodal Input                       │
│                                                     │
│  "What is in this image?"  +  [image file]         │
│         ↓                          ↓               │
│   Text tokens               Image patches           │
│         └──────────┬───────────────┘               │
│                    ▼                                │
│           Shared Transformer                        │
│           (attention over all tokens)               │
│                    ↓                                │
│            "A dog chasing a cat"                    │
└─────────────────────────────────────────────────────┘

Practical implications:

A model can answer questions about a screenshot, diagram, or photograph
Audio can be transcribed, translated, and responded to in a single pass
Video frames become a sequence of image tokens the model can reason over

The boundary between "language model" and "AI system" has effectively dissolved. What were separate specialized models (vision model, speech model, text model) are converging into single multimodal architectures.

Hardware: The Unsung Hero

Algorithm and data get most of the coverage. Hardware rarely does — despite being the actual constraint.

Microsoft built a dedicated supercomputer with 10,000 NVIDIA V100s specifically for OpenAI's research. OpenAI never disclosed exact training figures for GPT-3, but estimates put large training runs in the thousands of GPUs running for weeks. Today, systems running Gemini and GPT use clusters of H100 cards at similar scale. This is why NVIDIA became one of the most valuable companies in the world.

Google took a different path, designing custom silicon specifically for Transformer math:

Hardware	Used by	Strength
NVIDIA H100	OpenAI, Anthropic, Meta	General-purpose, widely available
Google TPUs	Google (Gemini)	Purpose-built for Transformer math, more efficient per watt

Gemini has been trained and served across multiple TPU generations (v4, v5e, v5p, and the newer Trillium/v6 for more recent models). TPUs aren't available to external developers for training — they're Google's internal advantage.

Practical Recommendations

If you're building on top of LLMs today, the historical distinctions above translate into concrete choices.

Choosing between understanding and generation models:

Task	Model type	Examples
Classifying text, ranking documents, Q&A over a corpus	Encoder (BERT-style)	BERT, RoBERTa, sentence-transformers
Generating text, summarizing, conversing, coding	Decoder (GPT-style)	GPT-4, Claude, Gemini
Most new product work in 2024+	Multimodal API	GPT-4o, Gemini 1.5 Pro, Claude 3

The Full Timeline

1966        ELIZA — first chatbot (rules only, no learning)
1990s–1997  RNNs (1990s) + LSTMs (1997) — sequential processing, forgetting problem
1998        PageRank — Google's original ranking algorithm
2013        Hummingbird — first concept-based search
2015        RankBrain — first ML in Google Search
2017        "Attention Is All You Need" — Transformer invented
2018        BERT (Google) + GPT-1 (OpenAI) — the split path begins
2019        BERT deployed in Google Search
2020        GPT-3 — 175B parameters, first truly massive LLM
2022        InstructGPT — instruction tuning + RLHF; ChatGPT launches November 2022
2024–2025   Multimodal models — GPT-4o, Gemini 1.5, Claude 3 unify text, image, audio, video

The Bottom Line

The Transformer's attention mechanism solved the forgetting problem that had bottlenecked language AI for decades — by computing relationships across the entire input simultaneously rather than word by word. From that single architectural change, both understanding models (BERT) and generation models (GPT) were built. But the algorithm alone wasn't the revolution. It was the algorithm converging with enough training data and hardware powerful enough to run the math — and none of that aligned until ~2017. Even then, GPT-3 in 2020 wasn't ChatGPT: the final piece was RLHF, which replaced "predict internet text" with "be useful to a human" as the training objective. "AI" and "LLM" are not synonyms. Neither are "pre-trained model" and "assistant." The distinctions matter when evaluating what any given system can actually do.

Resources

Attention Is All You Need — the original 2017 Transformer paper
BERT: Pre-training of Deep Bidirectional Transformers — Google's 2018 paper
Language Models are Few-Shot Learners — the GPT-3 paper
Training language models to follow instructions with human feedback — the InstructGPT/RLHF paper
The Illustrated Transformer — best visual walkthrough of attention mechanisms, no math required

Sub-Agent Architectures: Patterns, Trade-offs, and a Kotlin Implementation

Zoricic — Sun, 22 Mar 2026 20:04:28 +0000

The Problem

Single-agent systems hit a wall fast. Give one agent too many tools and a complex task, and you get:

Context bloat — every intermediate result accumulates in the conversation history
Tool confusion — a model with 20 tools tends to make worse decisions than one with 5 (tool-use accuracy degrades with scale; see the Gorilla LLM benchmark for empirical evidence)
No isolation — a failed step can corrupt the entire conversation state
No parallelism — sequential execution even when tasks are independent

Larger context windows reduce but don't eliminate these problems. A 1M-token context still degrades with irrelevant history, and tool confusion is a model behavior issue independent of window size. Architecture is the more scalable solution — but it's a trade-off, not a free lunch. Sub-agents add latency, coordination complexity, and cost.

What Is a Sub-Agent?

A sub-agent is a separate LLM call with its own isolated conversation context, invoked by a parent agent to handle a specific, self-contained task. "Separate instance" is a common shorthand but can mislead — it's typically the same model, same API key, same underlying compute. The isolation is logical, not physical. Every sub-agent call is another round of LLM inference costs.

What a sub-agent gets depends on how you design it. At minimum:

A system prompt tailored to its job
A lifecycle — it runs, returns a result, and is discarded

Beyond that, context isolation is a design variable, not a defining property of the pattern. In AIgent, sub-agents start with a fresh conversation history and never see the parent's history. In AutoGen, agents share a group chat history by default. In LangGraph, nodes can read and write shared state. Whether a sub-agent has isolated context or shared context depends entirely on the framework and implementation choices.

This article describes AIgent's approach — isolated context, restricted tool sets — as one point in that design space, not the only valid one.

┌──────────────────────────────────────────────────┐
│                  Parent Agent                    │
│  • Full tool set       • Persistent history      │
│  • User-facing         • Skill injection         │
└────────────────────┬─────────────────────────────┘
                     │
              [LLM model decides]
              to call spawn_agent
              tool with task + tools
                     │
                     ▼
          ┌──────────────────────┐
          │     Sub-Agent        │
          │  • Limited tools     │
          │  • Isolated context* │
          │  • Focused task      │
          └──────────┬───────────┘
                     │ returns result string
                     ▼
┌──────────────────────────────────────────────────┐
│           Parent Agent (continues)               │
└──────────────────────────────────────────────────┘

* Context isolation is a design choice — see note below.

When Did Sub-Agents Emerge?

Sub-agents didn't appear overnight. They evolved through distinct phases:

2022 — ReAct and Tool Use

Tool-augmented LMs existed before 2022. What the ReAct paper (Yao et al., 2022) contributed was empirical evidence that interleaving reasoning traces with actions outperforms either alone — it beat action-only approaches (like WebGPT) and reasoning-only approaches (like chain-of-thought) on question-answering and fact-verification tasks. The interleaving structure — think, act, observe, think again — became the blueprint for agent loops that followed. Still single-agent, but foundational.

2023 — Autonomous Agent Experiments

Projects like AutoGPT and BabyAGI showed what happens when you give a model the ability to spawn tasks and run them recursively. They were chaotic and unreliable, but they proved the concept: agents could delegate to themselves. The LangChain and LlamaIndex ecosystems formalized tool-using agents and chains, making sub-agent patterns accessible to developers.

2024 — Production-Grade Orchestration

CrewAI, AutoGen, and LangGraph moved the conversation from "can this work?" to "how do you build it reliably?" Role-based agent teams, graph-based execution, and structured handoffs between agents became first-class patterns. The focus shifted from raw autonomy to controlled delegation.

By 2025 — Native SDK Support

API providers began shipping sub-agent patterns as first-class features. Anthropic's Claude Agent SDK and Google's Gemini multi-agent APIs made spawning isolated sub-agents a few lines of code rather than a framework to build yourself.

Orchestration Patterns

These are structural decisions about how agents are created, what work they do, and how results flow back. They're distinct from capability mechanisms (covered in the next section), which are about what information an agent has access to.

1. ReAct Loop (Single Agent)

The building block. The model reasons, acts, observes, and repeats until done.

User input → [Think] → [Tool Call] → [Observe result] → [Think] → ... → Final answer

Every agent loop in every framework is a variation of this. In production, it's typically bounded by a max-iterations guard to prevent infinite loops:

User input → [Think] → [Tool Call] → [Observe] → ... (max N iterations) → Final answer
                                                              ↓
                                                    [Forced termination if N exceeded]

Pros:

Simple to implement
Easy to trace and debug
No coordination overhead

Cons:

Context grows linearly with each tool call
All tools visible at once — decision quality degrades with scale
No parallelism

2. Hierarchical Delegation (Orchestrator + Sub-Agents)

A parent agent breaks work into tasks and delegates each to a specialized sub-agent with a restricted tool set and fresh context.

┌─────────────────────────────────────────────────────┐
│                   Orchestrator                      │
│   Tools: [spawn_agent, search_web, get_time, ...]   │
└───────────┬──────────────────┬──────────────────────┘
            │                  │
            ▼                  ▼
   ┌──────────────┐   ┌──────────────────┐
   │  Sub-Agent A │   │   Sub-Agent B    │
   │  [search_web]│   │ [push_to_github] │
   └──────────────┘   └──────────────────┘

The key insight is tool isolation: sub-agents only get the tools relevant to their task. This improves decision quality and limits blast radius.

Context isolation — whether sub-agents see the parent's history or share state — is a design variable. Parallelism is structurally possible (sub-agents with isolated state have no ordering dependency), but requires explicit async wiring; a blocking implementation runs them sequentially.

Pros:

Tool scoping improves decision quality per sub-task
Parallelism is possible when sub-tasks are independent and isolated
Limits damage from a misbehaving sub-agent (if contexts are isolated)

Cons:

Each sub-agent is a full LLM call — latency and cost multiply with every delegation
Silent failures: a sub-agent that returns an empty or nonsensical result is indistinguishable from a valid response at the string boundary. Single-agent loops fail loudly (the model keeps trying); hierarchical delegation fails quietly (the parent receives garbage and reasons from it)
No mechanism for sub-agents to ask for clarification; ambiguous tasks produce hallucinated results
Coordination logic lives in the parent prompt — hard to test explicitly

3. Plan & Execute

The agent first produces a full plan (a list of steps), then executes each step — potentially spawning sub-agents per step. Planning and execution are separated.

[User input] → [Planning Agent] → [Step 1, Step 2, Step 3]
                                         │
                          ┌──────────────┼──────────────┐
                          ▼              ▼              ▼
                     [Executor 1]  [Executor 2]  [Executor 3]
                          └──────────────┼──────────────┘
                                         ▼
                                   [Synthesizer]

Pros:

Predictable execution path
Easy to inspect and approve before running

Cons:

Rigid — the plan can't adapt to what it discovers mid-execution
Upfront planning requires good foresight from the model

4. Role-Based Multi-Agent Teams

Multiple agents with distinct roles coordinate via shared memory or message passing. Common roles: Researcher, Writer, Critic, Executor.

┌──────────┐    findings    ┌──────────┐    draft    ┌──────────┐
│Researcher│ ─────────────▶ │  Writer  │ ──────────▶ │  Critic  │
└──────────┘                └──────────┘             └──────────┘
                                                           │
                                                    feedback│
                                                           ▼
                                                    ┌──────────┐
                                                    │  Writer  │
                                                    └──────────┘

Used heavily in CrewAI and AutoGen. Great for creative or review-heavy pipelines; complex to coordinate reliably.

Capability Mechanisms

These are distinct from orchestration patterns. They don't change how agents are spawned or how work is divided — they change what knowledge or behavior an agent has access to within a single context.

Mechanism	How It Works	When to Use
Skill injection	Relevant instructions prepended to system prompt	Conditional behavior without spawning overhead
RAG	Retrieved document chunks injected into the message	Domain knowledge too large for static prompts
Memory	Previous conversation facts loaded into context	Personalization, continuity across sessions

Skill-Based Injection

The parent agent dynamically loads context ("skills") into its system prompt based on the user's request. No new LLM instance — just different instructions per request.

User: "Remind me to call John tomorrow at 9am"
        │
        ▼
┌──────────────────────────────────────────────────┐
│  selectRelevant("remind", allSkills)             │
│  → matches "reminders" skill                     │
│  → injects into system prompt                    │
└──────────────────────────────────────────────────┘

In AIgent, selectRelevant is word intersection — no embeddings, no LLM call:

// src/main/kotlin/SkillLoader.kt
fun selectRelevant(input: String, all: List<Skill>): List<Skill> {
    val words = input.lowercase().split(Regex("\\W+")).filter { it.length > 2 }.toSet()
    val matched = all.filter { skill ->
        val skillWords = (skill.name + " " + skill.description).lowercase().split(Regex("\\W+")).toSet()
        words.intersect(skillWords).isNotEmpty()
    }
    return matched.ifEmpty { all }  // fallback: inject ALL skills if nothing matches
}

It splits both the user input and each skill's name+description into lowercase words (≥3 chars) and checks for any overlap. Fast and cheap — no extra API calls. The fallback on the last line is significant: if nothing matches, every skill gets injected. With two skills that's fine; with twenty it's a prompt full of unrelated instructions.

This is what "lightweight" means here. Alternative approaches — embedding similarity or a classifier LLM call — are more accurate but add latency and cost before the main response even starts.

Pros:

Zero extra API calls — no latency overhead
Easy to add new behaviors: drop a .md file in skills/
Skills can reference specific tools, guiding the model without hard-coding logic

Cons:

Word intersection is brittle — "Set an alert" won't match a skill named "reminders" unless the words overlap
Wrong skill injection is hard to detect: the model follows injected guidance even when irrelevant
Fallback injects all skills when nothing matches — prompt can balloon unexpectedly
No isolation — a buggy skill affects the whole agent, not just a sub-task

Architecture Comparison

Pattern	Type	Context Isolation	Tool Scoping	Complexity	Best For
Single ReAct loop	Orchestration	None	None	Low	Simple, focused tasks
Hierarchical delegation	Orchestration	Design variable²	Per sub-agent	Medium	Multi-step workflows
Plan & Execute	Orchestration	Partial	Per step	Medium	Predictable pipelines
Role-based teams	Orchestration	Shared state	Per role	High	Creative, review workflows
Skill injection	Capability mechanism	Static (prompt-level)¹	None	Low	Conditional behavior
RAG	Capability mechanism	Static (prompt-level)	None	Low	Large knowledge bases

¹ Skill injection changes what context is active at prompt-construction time. The agent still runs in a single LLM context — there is no new instance or conversation boundary.

² Context isolation in hierarchical delegation is a design variable. AIgent uses isolated contexts per sub-agent. AutoGen shares a group chat history by default. LangGraph nodes can read and write shared state.

Practical Recommendations

Use a single ReAct loop when:

The task needs fewer than ~5 tool calls
All tools are clearly relevant to the task

Use skill injection when:

The agent needs different behavioral modes
You want guidance without spinning up new LLM instances
Capabilities are well-defined but not always relevant

Use sub-agents when:

A sub-task is self-contained
You want a model to operate with a focused tool set
Tasks are independent and you're willing to implement async dispatch for parallelism

Prevent recursive spawning explicitly. Don't rely on the model to know not to spawn sub-agents of sub-agents. Enforce it structurally — pass a baseTools list (without the spawn tool) to sub-agents.

Guard your ReAct loops. Add a max-iterations limit. Without it, a model that gets stuck will keep calling tools until you run out of tokens or budget.

Plan for sub-agent failures. The string "(no response)" is indistinguishable from a valid empty result. Use a structured return type that separates success from error, and make the parent handle failure explicitly.

Case Study: AIgent — A Kotlin Implementation

AIgent implements the hierarchical delegation pattern on top of Gemini's function calling API, with a skills system for capability injection. It's a working project — the code below is from the actual repo, not a simplified demo.

The Tool Interface

Every capability — from web search to spawning agents — implements the same contract:

// src/main/kotlin/tools/AgentTool.kt
interface AgentTool {
    val declaration: FunctionDeclaration
    fun handle(args: Map<String, Any>, chatId: Long?): Map<String, Any>
}

FunctionDeclaration is what gets passed to Gemini's function calling API. handle() is what runs when the model calls the tool. Adding a new capability means implementing two things and nothing else.

The Spawn Agent Tool

The parent agent delegates by calling spawn_agent:

// src/main/kotlin/tools/SpawnAgentTool.kt
override fun handle(args: Map<String, Any>, chatId: Long?): Map<String, Any> {
    val task = args["task"]?.toString() ?: return mapOf("error" to "Missing task")
    val toolNames = args["tools"] as? List<String> ?: return mapOf("error" to "Missing tools")

    val selectedTools = availableTools.filter {
        it.declaration.name().orElse("") in toolNames
    }

    val result = SubAgentService(client, selectedTools, task).run()
    return mapOf("result" to result)
}

One scoping detail: availableTools is the parent-defined pool. But toolNames comes from args["tools"] — meaning the model picks which tools to request from that pool. The parent sets hard limits; the sub-agent selects within them. A hallucinating model can't exceed the pool, but it could request the wrong subset. For security-sensitive tools, validate toolNames against an explicit allowlist.

This run() call is synchronous and blocking — sub-agents execute sequentially in the current implementation. The architecture supports parallelism (sub-agents share no state), but it requires explicit async wiring. With Kotlin coroutines:

// Parallel dispatch — not in current implementation, but straightforward to add
suspend fun spawnParallel(
    tasks: List<Pair<String, List<AgentTool>>>,
    client: Client
): List<String> = coroutineScope {
    tasks.map { (task, tools) ->
        async(Dispatchers.IO) { SubAgentService(client, tools, task).run() }
    }.awaitAll()
}

Each async block runs on the IO dispatcher — network-bound LLM calls benefit immediately. awaitAll() collects results once all sub-agents finish.

The Sub-Agent Service

Each sub-agent runs its own ReAct loop in complete isolation:

// src/main/kotlin/SubAgentService.kt
fun run(): String {
    val config = GenerateContentConfig.builder()
        .systemInstruction(Content.fromParts(Part.fromText(
            "You are a focused sub-agent. Complete the given task using the available tools. Be concise and return only the result."
        )))
        .tools(listOf(Tool.builder().functionDeclarations(tools.map { it.declaration }).build()))
        .build()

    val history = mutableListOf<Content>()
    history.add(Content.builder().role("user").parts(listOf(Part.fromText(task))).build())
    var response = client.models.generateContent("gemini-2.5-flash", history, config)

    while (response.functionCalls()?.isNotEmpty() == true) {
        val responseParts = mutableListOf<Part>()
        for (fc in response.functionCalls()!!) {
            val tool = tools.find { it.declaration.name().orElse("") == fc.name().orElse("") }
            val result = tool?.handle(fc.args().orElse(emptyMap()), null)
                ?: mapOf("error" to "Unknown function: ${fc.name().orElse("")}")
            responseParts.add(Part.fromFunctionResponse(fc.name().orElse(""), result))
        }
        history.add(Content.builder().role("model").parts(response.parts()).build())
        history.add(Content.builder().role("user").parts(responseParts).build())
        response = client.models.generateContent("gemini-2.5-flash", history, config)
    }

    return response.text() ?: "(no response)"
}

The system prompt — "Be concise and return only the result" — works well for self-contained tasks where the expected output is clear. The trade-off: the sub-agent has no way to ask for clarification. If the task is ambiguous, it will either hallucinate a plausible answer or return an empty result. The narrow prompt is a deliberate choice to keep sub-agents focused, but it means the parent must write unambiguous task descriptions. A task like "research the company" will fail silently; "find the founding year and current CEO of Stripe" will not.

Notice what's structurally absent in AIgent's implementation: no parent history passed in, no access to tools outside the assigned set, no ability to spawn further sub-agents. This is a design choice — other frameworks (AutoGen, LangGraph) make different ones.

Preventing Recursive Spawning

This design decision prevents one specific class of runaway agent bug — sub-agents spawning further sub-agents:

// src/main/kotlin/AgentService.kt

// baseTools does NOT include SpawnAgentTool
private val baseTools: List<AgentTool> = scheduler.tools + githubDocs.tools +
    listOf(CurrentTimeTool(), GoogleSearchTool(client))

// Only the parent agent gets spawn_agent
private val tools: List<AgentTool> = baseTools + listOf(SpawnAgentTool(client, baseTools))

Sub-agents receive baseTools. SpawnAgentTool is only in tools — the parent's list. This is enforced structurally, not by trusting the model.

What it does not prevent: a parent that misinterprets a sub-agent's result and spawns it again in a loop, or a sub-agent whose own ReAct while loop runs indefinitely. SubAgentService.run() has no max-iteration guard — a model that keeps generating function calls will keep looping until it hits the token limit or the API times out. A production implementation should add an iteration counter and a hard cutoff.

Dynamic Skill Injection

Before each response, the agent injects relevant skills into its system prompt based on keyword matching:

// src/main/kotlin/AgentService.kt
private fun buildConfig(input: String): GenerateContentConfig {
    val relevantSkills = skillLoader.selectRelevant(input, allSkills)
    val systemPrompt = buildString {
        append(soul)
        if (relevantSkills.isNotEmpty()) {
            append("\n\n# Active Skills\n")
            relevantSkills.forEach { skill -> append("\n## ${skill.name}\n${skill.content}\n") }
        }
    }
    return GenerateContentConfig.builder()
        .systemInstruction(Content.fromParts(Part.fromText(systemPrompt)))
        .tools(listOf(Tool.builder().functionDeclarations(tools.map { it.declaration }).build()))
        .build()
}

Skills are markdown files with YAML frontmatter:

---
name: reminders
description: Schedule and manage reminders, alerts, and time-based notifications
tools: [schedule_reminder, list_reminders, get_current_time]
---

Always call `get_current_time` first before scheduling a reminder...

Zero extra API calls, zero latency overhead. The brittle point is selectRelevant: it matches on overlapping words between the input and the skill name/description. "Set an alert for Monday" won't match a skill named "reminders" unless "alert" or "Monday" appears in the skill's metadata.

The Bottom Line

Sub-agents solve three real problems: context that grows without bound, tools that dilute decision quality, and tasks that could run in parallel but don't. The trade-offs are real too: every sub-agent is an additional LLM call with its own latency and cost, coordination logic is hard to test, and a sub-agent that fails silently is worse than one that fails loudly.

The patterns that matter most in practice: isolate tools per sub-task, prevent recursive spawning structurally, write unambiguous task descriptions, and handle errors explicitly at the boundary where sub-agents return results. The architecture is sound; the failure modes live in the implementation details.

Resources

AIgent on GitHub — The Kotlin agent that powers the case study in this article
ReAct: Synergizing Reasoning and Acting in Language Models — The original 2022 paper
Gorilla LLM Benchmark — Empirical data on tool-use accuracy at scale
LangGraph Documentation — Graph-based agent orchestration
AutoGen Multi-Agent Conversation Framework — Microsoft's role-based agent framework
Anthropic Agent SDK — Native sub-agent support in Claude

You Built a Kotlin AI Agent. Should You Migrate to Koog?

Zoricic — Thu, 05 Mar 2026 16:18:13 +0000

JetBrains Koog vs Google GenAI SDK — Should You Switch?

You've built a working AI agent with Google GenAI. Now JetBrains releases Koog, a Kotlin-first AI framework. Should you migrate? Probably not — here's why.

See also: SQLite Embeddings for Local Agents for the RAG implementation referenced in this article.

The Problem

When building AI agents in Kotlin, you have two main options:

Google GenAI SDK — Direct API wrapper, full control, manual tool loop
JetBrains Koog — Higher-level framework, Kotlin DSL, built-in agent patterns

Both work. The question is: what do you gain and lose by switching?

What Koog Offers

Koog is JetBrains' attempt to bring Kotlin-native ergonomics to AI agent development. It wraps multiple LLM providers (Gemini, OpenAI, Anthropic) behind a unified DSL.

Cleaner Tool Definitions

With Google GenAI, tool definitions are verbose:

val currentTimeDeclaration = FunctionDeclaration.builder()
    .name("get_current_time")
    .description("Get the current date and time")
    .parameters(
        Schema.builder()
            .type(Type.Known.OBJECT)
            .properties(emptyMap<String, Schema>())
            .build()
    )
    .build()

Koog simplifies this with a DSL approach:

val getCurrentTime = tool(
    name = "get_current_time",
    description = "Get the current date and time"
) {
    TimeResult(
        datetime = LocalDateTime.now().toString(),
        formatted = LocalDateTime.now().format(pattern)
    )
}

Automatic Tool Execution Loop

With Google GenAI, you write the loop yourself:

while (response.functionCalls()?.isNotEmpty() == true) {
    val functionCalls = response.functionCalls()!!
    val responseParts = mutableListOf<Part>()

    for (fc in functionCalls) {
        val result = handleFunction(fc.name(), fc.args())
        responseParts.add(Part.fromFunctionResponse(name, result))
    }

    history.add(functionResponseContent)
    response = client.models.generateContent(model, history, config)
}

Koog handles this internally — you just define tools and let the framework orchestrate.

Multi-Provider Support

Koog abstracts the LLM provider:

// Conceptual example — verify against current Koog docs
val agent = AIAgent(
    executor = simpleGeminiExecutor(apiKey),
    toolRegistry = buildToolRegistry { tool(getCurrentTime) }
)

Switching from Gemini to OpenAI means swapping the executor, not rewriting tools.

What You Lose

Fine-Grained Control

Your manual loop lets you inject logic between iterations. For example, adding RAG context before each user message:

val relevantChunks = documents.search(input)
val message = if (relevantChunks.isNotEmpty()) {
    val context = relevantChunks.joinToString("\n\n")
    "Context:\n$context\n\nUser: $input"
} else {
    input
}

With Koog, you'd need to work within its abstraction — possible, but less direct.

Maturity

Google GenAI SDK is battle-tested. Koog shipped in 2024 and is still evolving. Documentation gaps and edge cases are expected.

Simplicity

Your current stack is lean. Adding Koog introduces another layer:

Current:                With Koog:
┌──────────────────┐    ┌──────────────────┐
│    Your Code     │    │    Your Code     │
├──────────────────┤    ├──────────────────┤
│  Google GenAI    │    │      Koog        │
└──────────────────┘    ├──────────────────┤
                        │  Provider SDK    │
                        └──────────────────┘

More layers = more things that can break.

A Related Decision: Structured Output

While evaluating frameworks, another question often comes up: should you use structured output?

Structured output forces the LLM to respond in a specific JSON schema:

val config = GenerateContentConfig.builder()
    .responseMimeType("application/json")
    .responseSchema(Schema.builder()
        .type(Type.Known.OBJECT)
        .properties(mapOf(
            "intent" to Schema.builder().type(Type.Known.STRING).build(),
            "confidence" to Schema.builder().type(Type.Known.NUMBER).build()
        ))
        .build())
    .build()

Use it when:

Extracting specific fields from user input
Building APIs that need predictable JSON responses
Downstream processing requires consistent format

Skip it when:

Building conversational bots (like Telegram)
Free-form responses are the goal
You want the model to format naturally

For a chat agent, free-form text is usually what you want.

When to Switch to Koog

Consider Koog if:

Scenario	Koog Value
Supporting multiple LLM providers	High
Adding 10+ tools with complex schemas	High
Using advanced patterns (subagents, planning)	Medium
Current code is becoming hard to maintain	Medium
Just want cleaner syntax	Low

Don't switch just for syntactic sugar. Your 30-line tool loop isn't the bottleneck.

Practical Recommendations

Stay with Google GenAI if:

Your agent works and you have full control
You're only using Gemini
RAG or custom logic is tightly integrated into your flow

Consider Koog if:

You're starting a new project with multi-provider requirements
You need complex agent patterns out of the box
You value Kotlin DSL ergonomics over explicit control

Hybrid approach:

Keep Google GenAI as the backend
Extract your tool definitions into a more DSL-like pattern
Get some ergonomic wins without a full framework migration

The Bottom Line

JetBrains Koog trades control for convenience. If you've already built a working agent with Google GenAI and manual tool orchestration, migration isn't worth it unless you need multi-provider support or advanced agent patterns. Your explicit loop gives you flexibility that abstractions hide. For new projects with complex requirements, Koog is worth evaluating — but for a functional Telegram bot with RAG, keep what works.

Resources

SQLite as a Vector Database — Yes, Really

Zoricic — Wed, 04 Mar 2026 19:10:49 +0000

Do you really need a vector database? For local AI agents, SQLite handles embeddings just fine — and it's already on your machine.

This article explores a practical approach for local-first AI tools. For context on MCP transports and how agents communicate, see Understanding MCP Server Transports.

The Problem with "Production-Grade" Vector Databases

When building AI agents that use embeddings (for RAG, semantic search, or memory), the default advice is: "Use a vector database like Pinecone, Weaviate, or Chroma."

But for local agents — tools running on your machine, personal assistants, or development prototypes — this advice creates unnecessary complexity:

What you need:                    What you're told to set up:
┌─────────────────────┐          ┌─────────────────────────────┐
│ Store ~10K vectors  │          │ Docker container            │
│ Query occasionally  │    →     │ Separate database server    │
│ Works offline       │          │ API keys and auth           │
│ Easy to backup      │          │ Cloud dependencies          │
└─────────────────────┘          └─────────────────────────────┘

There's a mismatch between the problem and the solution.

SQLite as an Embedding Store

SQLite is a serverless, file-based database that ships with most operating systems. It turns out it's perfectly capable of storing and querying embeddings for local use cases.

Basic Approach: Store Embeddings as BLOBs

The simplest approach stores embeddings as binary data:

CREATE TABLE embeddings (
    id INTEGER PRIMARY KEY,
    content TEXT NOT NULL,
    embedding BLOB NOT NULL,
    metadata JSON,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.sql.Connection
import kotlinx.serialization.json.Json
import kotlinx.serialization.encodeToString

fun storeEmbedding(
    conn: Connection,
    content: String,
    embedding: FloatArray,
    metadata: Map<String, Any>? = null
) {
    val embeddingBlob = ByteBuffer
        .allocate(embedding.size * 4)
        .order(ByteOrder.LITTLE_ENDIAN)
        .apply { embedding.forEach { putFloat(it) } }
        .array()

    conn.prepareStatement(
        "INSERT INTO embeddings (content, embedding, metadata) VALUES (?, ?, ?)"
    ).use { stmt ->
        stmt.setString(1, content)
        stmt.setBytes(2, embeddingBlob)
        stmt.setString(3, metadata?.let { Json.encodeToString(it) })
        stmt.executeUpdate()
    }
}

fun getEmbedding(blob: ByteArray): FloatArray {
    val buffer = ByteBuffer.wrap(blob).order(ByteOrder.LITTLE_ENDIAN)
    return FloatArray(blob.size / 4) { buffer.getFloat() }
}

Similarity Search: Brute Force Works

For local agents with thousands (not millions) of vectors, brute-force cosine similarity is fast enough:

data class SearchResult(
    val id: Int,
    val content: String,
    val metadata: String?,
    val similarity: Double
)

fun searchSimilar(
    conn: Connection,
    queryEmbedding: FloatArray,
    topK: Int = 5
): List<SearchResult> {
    val results = mutableListOf<SearchResult>()

    conn.prepareStatement(
        "SELECT id, content, embedding, metadata FROM embeddings"
    ).use { stmt ->
        stmt.executeQuery().use { rs ->
            while (rs.next()) {
                val storedVec = getEmbedding(rs.getBytes("embedding"))
                val similarity = cosineSimilarity(queryEmbedding, storedVec)
                results.add(
                    SearchResult(
                        id = rs.getInt("id"),
                        content = rs.getString("content"),
                        metadata = rs.getString("metadata"),
                        similarity = similarity
                    )
                )
            }
        }
    }

    return results.sortedByDescending { it.similarity }.take(topK)
}

fun cosineSimilarity(a: FloatArray, b: FloatArray): Double {
    val dot = a.zip(b.toList()).sumOf { (x, y) -> (x * y).toDouble() }
    val normA = kotlin.math.sqrt(a.sumOf { (it * it).toDouble() })
    val normB = kotlin.math.sqrt(b.sumOf { (it * it).toDouble() })
    return dot / (normA * normB)
}

Performance Reality Check

How fast is brute-force search? Faster than you'd expect:

Vectors	Dimensions	Search Time	Memory
1,000	1536	~5ms	~6 MB
10,000	1536	~50ms	~60 MB
100,000	1536	~500ms	~600 MB

For a local agent querying a personal knowledge base, 50ms is imperceptible. You don't need approximate nearest neighbors (ANN) algorithms until you're well past 100K vectors.

SQLite-VSS: When You Need More Speed

For larger datasets, sqlite-vss adds vector search capabilities to SQLite using Facebook's FAISS library:

-- Load the extension
.load ./vector0
.load ./vss0

-- Create a virtual table for vector search
CREATE VIRTUAL TABLE vss_embeddings USING vss0(
    embedding(1536)  -- OpenAI embedding dimensions
);

-- Insert vectors
INSERT INTO vss_embeddings (rowid, embedding)
SELECT id, embedding FROM embeddings;

-- Search with approximate nearest neighbors
SELECT rowid, distance
FROM vss_embeddings
WHERE vss_search(embedding, ?)
LIMIT 10;

This gives you sub-millisecond searches even at 1M+ vectors, while keeping everything in a single SQLite file.

Real-World Example: Local-First AI Tools

Many local-first AI tools use SQLite for codebase indexing and embeddings storage. The pattern is common — a single database file holds everything the agent needs:

┌───────────────────────────────────────────────────────┐
│                    Local AI Agent                     │
├───────────────────────────────────────────────────────┤
│                                                       │
│  ┌─────────────┐   ┌─────────────┐   ┌────────────┐  │
│  │  Embeddings │   │    Chat     │   │   Config   │  │
│  │  (vectors)  │   │   History   │   │  & State   │  │
│  └──────┬──────┘   └──────┬──────┘   └─────┬──────┘  │
│         │                 │                │         │
│         └─────────────────┼────────────────┘         │
│                           │                          │
│                  ┌────────▼────────┐                 │
│                  │    SQLite DB    │                 │
│                  │  (single file)  │                 │
│                  └─────────────────┘                 │
│                           │                          │
│                  ~/.agent/data.db                    │
└───────────────────────────────────────────────────────┘

Everything lives in one file. Backup? Copy the file. Reset? Delete it. Move to another machine? Take it with you.

Advantages of SQLite for Embeddings

1. Zero Infrastructure

# Pinecone setup:
pip install pinecone-client
# Create account, get API key, configure environment...

# SQLite setup:
# Nothing. It's already there.

No servers, no Docker, no accounts, no API keys. Your agent works offline, on airplanes, and in restricted networks.

2. Single-File Simplicity

Your entire knowledge base is one .db file:

~/.my-agent/
├── config.json
└── knowledge.db    ← Everything: embeddings, chat history, metadata

This makes debugging trivial. You can open the database with any SQLite client and inspect exactly what's stored.

3. Atomic Transactions

SQLite gives you ACID transactions for free:

fun updateDocument(
    conn: Connection,
    docId: Int,
    newContent: String,
    newEmbedding: FloatArray
) {
    conn.autoCommit = false
    try {
        conn.prepareStatement("DELETE FROM embeddings WHERE doc_id = ?").use { stmt ->
            stmt.setInt(1, docId)
            stmt.executeUpdate()
        }
        storeEmbedding(conn, newContent, newEmbedding, mapOf("doc_id" to docId))
        conn.commit()
    } catch (e: Exception) {
        conn.rollback()
        throw e
    } finally {
        conn.autoCommit = true
    }
    // Either both succeed or both fail
}

No partial updates, no orphaned embeddings, no consistency issues.

4. Rich Querying

Combine vector search with SQL's full power. Pre-filter with SQL, then compute similarity in your application:

fun searchRecentNonArchived(
    conn: Connection,
    queryEmbedding: FloatArray,
    topK: Int = 10
): List<SearchResult> {
    val candidates = mutableListOf<SearchResult>()

    // SQL handles filtering — only fetch what matches
    conn.prepareStatement("""
        SELECT id, content, embedding, metadata
        FROM embeddings
        WHERE created_at > datetime('now', '-7 days')
          AND json_extract(metadata, '$.archived') != true
    """).use { stmt ->
        stmt.executeQuery().use { rs ->
            while (rs.next()) {
                val storedVec = getEmbedding(rs.getBytes("embedding"))
                candidates.add(
                    SearchResult(
                        id = rs.getInt("id"),
                        content = rs.getString("content"),
                        metadata = rs.getString("metadata"),
                        similarity = cosineSimilarity(queryEmbedding, storedVec)
                    )
                )
            }
        }
    }

    return candidates.sortedByDescending { it.similarity }.take(topK)
}

Try doing that with a pure vector database — most don't support SQL-level filtering before vector search.

5. Portable and Debuggable

# Inspect your embeddings
sqlite3 knowledge.db "SELECT COUNT(*) FROM embeddings"

# Backup before experimenting
cp knowledge.db knowledge.db.backup

# Share with a colleague
scp knowledge.db user@other-machine:~/

Disadvantages and Limitations

1. No Native Vector Operations

SQLite doesn't understand vectors natively. You're either:

Computing similarity in application code (fine for <100K vectors)
Using sqlite-vss extension (adds complexity)

// This works, but it's not "native"
val similarities = rows.map { row ->
    row to cosineSimilarity(query, getEmbedding(row.embedding))
}

2. Memory Constraints

Brute-force search loads all vectors into memory. At 1M vectors with 1536 dimensions:

1,000,000 × 1,536 × 4 bytes = ~6 GB

That's too much for most laptops. You'll need sqlite-vss or a different approach.

3. No Built-in Sharding

SQLite is single-file by design. If you need to scale across machines, you're on your own:

SQLite:
┌─────────┐
│ One DB  │  ← Can't split across servers
└─────────┘

Distributed vector DB:
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │  ← Built-in distribution
└─────────┘ └─────────┘ └─────────┘

4. Write Concurrency

SQLite uses file-level locking. Multiple processes writing simultaneously will block:

// Process A: Writing embeddings...
// Process B: Waiting for lock...
// Process C: Waiting for lock...

For single-user local agents, this is rarely a problem. For multi-user services, it's a showstopper.

5. No Managed Updates or Filtering Indexes

Dedicated vector databases offer features like:

Automatic re-indexing when embeddings change
Pre-filtering indexes for metadata queries
Quantization for memory efficiency

With SQLite, you implement these yourself or go without.

When to Use SQLite for Embeddings

Use Case	SQLite?	Why
Personal knowledge base	Yes	Thousands of docs, single user
Local coding assistant	Yes	Codebase fits in memory
Development prototype	Yes	Fast iteration, no setup
Chat memory for agent	Yes	Small dataset, needs persistence
Production RAG service	No	Need scale, concurrency, managed ops
Multi-tenant application	No	Need isolation, distribution
Real-time search at scale	No	Need millisecond latency at 10M+ vectors

When NOT to Use SQLite

Don't use SQLite when:

You have more than ~500K embeddings
Multiple services need to write concurrently
You need sub-10ms search latency at scale
Your application is multi-tenant
You're building a service that needs to scale horizontally

In these cases, use a proper vector database: Pinecone for managed simplicity, Weaviate for open-source flexibility, or pgvector if you're already on PostgreSQL.

Practical Implementation Tips

1. Index Your Metadata

CREATE INDEX idx_metadata_type ON embeddings(json_extract(metadata, '$.type'));
CREATE INDEX idx_created_at ON embeddings(created_at);

Pre-filter with SQL before computing similarities.

2. Batch Your Inserts

fun storeEmbeddingsBatch(
    conn: Connection,
    items: List<Triple<String, FloatArray, Map<String, Any>?>>
) {
    conn.prepareStatement(
        "INSERT INTO embeddings (content, embedding, metadata) VALUES (?, ?, ?)"
    ).use { stmt ->
        items.forEach { (content, embedding, metadata) ->
            val blob = ByteBuffer
                .allocate(embedding.size * 4)
                .order(ByteOrder.LITTLE_ENDIAN)
                .apply { embedding.forEach { putFloat(it) } }
                .array()
            stmt.setString(1, content)
            stmt.setBytes(2, blob)
            stmt.setString(3, metadata?.let { Json.encodeToString(it) })
            stmt.addBatch()
        }
        stmt.executeBatch()
    }
}

3. Use WAL Mode for Better Concurrency

val conn = DriverManager.getConnection("jdbc:sqlite:knowledge.db").apply {
    createStatement().use { stmt ->
        stmt.execute("PRAGMA journal_mode=WAL")    // Write-ahead logging
        stmt.execute("PRAGMA synchronous=NORMAL")  // Faster writes, still safe
    }
}

4. Consider Dimensionality Reduction

If memory is tight, reduce embedding dimensions:

// Using Smile library for dimensionality reduction
import smile.projection.PCA

// Reduce 1536 → 512 dimensions (loses some accuracy)
val pca = PCA.fit(originalEmbeddings).getProjection(512)
val reducedEmbeddings = originalEmbeddings.map { pca.project(it) }

The Bottom Line

SQLite is a legitimate choice for embedding storage in local AI agents. It's not a hack or a compromise — it's the right tool for single-user, offline-capable, moderate-scale use cases.

The key insight: most local agents don't need distributed vector databases. They need simple, reliable, file-based storage that works without infrastructure. SQLite delivers exactly that.

Start with plain SQLite and brute-force search. If you outgrow it (you probably won't for local use), add sqlite-vss. Only reach for Pinecone or Weaviate when you're building a service, not a tool.

Resources

SQLite Documentation
sqlite-vss — Vector similarity search for SQLite
OpenAI Embeddings Guide
FAISS — The library behind sqlite-vss
pgvector — PostgreSQL alternative if you need more power
Chroma — Embedded vector database (SQLite under the hood!)

Understanding MCP Server Transports: STDIO, SSE, and HTTP Streamable

Zoricic — Tue, 03 Mar 2026 21:10:12 +0000

If you've been exploring Claude Code or building AI tools, you've probably heard about MCP (Model Context Protocol). But when it comes to setting up your own MCP server, one question often comes up: which transport should I use?

In this article, I'll break down the three main MCP transports—STDIO, SSE, and HTTP Streamable—explain when each was introduced, what problems they solve, and help you choose the right one for your project.

What is MCP?

MCP (Model Context Protocol) is an open protocol that lets AI assistants like Claude connect to external tools and data sources. Think of it as a standardized way for AI to "talk" to your code.

An MCP server exposes tools (functions the AI can call) and resources (data the AI can access). The transport is simply how the client (like Claude Code) communicates with your server.

The Three Transports at a Glance

Transport	Use Case	Network	Complexity
STDIO	Local tools, scripts	None (subprocess)	Simple
SSE	Web environments, legacy	HTTP	Medium
HTTP Streamable	Production, remote servers	HTTP	Medium

Let's dive into each one.

Transport 1: STDIO (Standard Input/Output)

What is it?

STDIO is the simplest transport. Your MCP server runs as a subprocess, and communication happens through stdin (input) and stdout (output)—the same streams you use when piping commands in a terminal.

Claude Code  --stdin-->  Your Server
             <-stdout--

When was it introduced?

STDIO has been part of MCP since the protocol's initial release in November 2024. It's the original transport for local tools and remains the most common choice for Claude Code integrations.

What problems does it solve?

No network setup: No ports, no HTTP, no CORS issues
Simple deployment: Just run a script
Security: Runs locally with your permissions

When to use STDIO

Building local tools for Claude Code
Quick prototypes and experiments
Tools that access local files or system resources
When you don't need network access

Code Example

Here's a minimal STDIO MCP server:

#!/usr/bin/env node
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";

const server = new Server(
  { name: "my-mcp-server", version: "1.0.0" },
  { capabilities: { tools: {} } }
);

// Define tools
const tools = [
  {
    name: "greet",
    description: "Greet someone by name",
    inputSchema: {
      type: "object",
      properties: {
        name: { type: "string", description: "Name to greet" }
      },
      required: ["name"]
    }
  }
];

// Handle tool listing
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools }));

// Handle tool calls
server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const { name, arguments: args } = request.params;
  if (name === "greet") {
    return {
      content: [{ type: "text", text: `Hello, ${args.name}!` }]
    };
  }
  return { content: [{ type: "text", text: "Unknown tool" }], isError: true };
});

// Start server
const transport = new StdioServerTransport();
await server.connect(transport);

Configuration for Claude Code

{
  "mcpServers": {
    "my-server": {
      "command": "node",
      "args": ["/path/to/server.js"]
    }
  }
}

Pros and Cons

Pros:

Zero network configuration
Simple to implement and debug
Secure by default (local only)

Cons:

Local only—can't share with others
One client per server instance
Requires the server script on the local machine

Transport 2: SSE (Server-Sent Events)

What is it?

SSE transport uses HTTP with Server-Sent Events for real-time communication:

Client → Server: HTTP POST requests
Server → Client: SSE event stream (long-lived connection)

Client  --POST /messages-->  Server
        <----SSE stream----

When was it introduced?

SSE transport was introduced in the MCP specification 2024-11-05 as the first HTTP-based transport, enabling web-based integrations and remote access. However, it has since been deprecated in favor of HTTP Streamable as of specification version 2025-03-26.

What problems does it solve?

Real-time updates: Server can push messages without polling
HTTP-friendly: Works through firewalls and proxies
Web compatibility: Works in browsers

When to use SSE

Legacy systems that already use SSE
When you specifically need the SSE pattern
Web applications with existing SSE infrastructure

Note: For new projects, consider HTTP Streamable instead—it's the modern recommended approach.

Code Example

#!/usr/bin/env node
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { SSEServerTransport } from "@modelcontextprotocol/sdk/server/sse.js";
import express from "express";

const app = express();
app.use(express.json());

const transports = new Map();

function createServer() {
  const server = new McpServer({
    name: "my-mcp-server-sse",
    version: "1.0.0"
  });

  server.tool("greet", "Greet someone", {
    name: { type: "string", description: "Name to greet" }
  }, async ({ name }) => ({
    content: [{ type: "text", text: `Hello, ${name}!` }]
  }));

  return server;
}

// SSE endpoint - client connects here
app.get("/sse", async (req, res) => {
  const transport = new SSEServerTransport("/messages", res);
  const server = createServer();

  transports.set(transport.sessionId, { transport, server });

  res.on("close", () => transports.delete(transport.sessionId));

  await server.connect(transport);
});

// Messages endpoint - client sends requests here
app.post("/messages", async (req, res) => {
  const sessionId = req.query.sessionId;
  const session = transports.get(sessionId);

  if (session) {
    await session.transport.handlePostMessage(req, res);
  } else {
    res.status(400).json({ error: "No session" });
  }
});

app.listen(3002, () => console.log("SSE server on port 3002"));

Pros and Cons

Pros:

Real-time server push
Works over HTTP (firewall-friendly)
Good browser support

Cons:

More complex than STDIO
Requires maintaining long-lived connections
Deprecated in favor of HTTP Streamable

Transport 3: HTTP Streamable (Recommended)

What is it?

HTTP Streamable is the modern, recommended transport for remote MCP servers. It uses standard HTTP requests with optional streaming:

Client → Server: HTTP POST requests
Server → Client: HTTP responses (can be streamed)

Client  --POST /mcp-->  Server
        <--Response---

When was it introduced?

HTTP Streamable was introduced in the MCP specification version 2025-03-26 as the recommended approach for remote servers. It replaces SSE as the primary HTTP transport and represents the evolution of MCP's HTTP transport capabilities.

What problems does it solve?

Scalability: Stateless architecture for horizontal scaling

Unlike SSE which requires maintaining persistent connections, HTTP Streamable can operate in a fully stateless manner. This means you can deploy multiple instances of your MCP server behind a load balancer, and any instance can handle any request. When traffic spikes, simply spin up more server instances—no need to worry about connection affinity or sticky sessions.

In practice, this looks like:

                    ┌─ MCP Server Instance 1
Client ─► Load ─────┼─ MCP Server Instance 2
          Balancer  └─ MCP Server Instance 3

Each request is independent, making auto-scaling straightforward with Kubernetes, AWS ECS, or any container orchestration platform.

Authentication: Native HTTP auth support

HTTP Streamable uses standard HTTP, which means you get the entire HTTP authentication ecosystem for free:

Bearer tokens: Pass JWT tokens in the Authorization: Bearer <token> header
API keys: Use custom headers like X-API-Key for simple integrations
OAuth 2.0: Implement full OAuth flows for enterprise applications
mTLS: Use client certificates for machine-to-machine authentication

This is a significant advantage over STDIO (which has no authentication) and SSE (where auth can be awkward to implement). Your existing auth middleware—whether it's Passport.js, Auth0, or a custom solution—works out of the box.

Multi-client: One server, many users

With HTTP Streamable, a single server deployment can handle requests from hundreds or thousands of different clients simultaneously. Each request carries its own session context, so there's no confusion about which client is which.

Consider a team scenario:

Developer A uses the MCP server from their laptop
Developer B uses it from their desktop
A CI/CD pipeline calls it for automated tasks
A web dashboard makes calls for analytics

All these clients hit the same server endpoint, each with their own authentication credentials and session state. The server doesn't need to maintain separate long-lived connections for each client.

Infrastructure: Works with your existing HTTP stack

HTTP Streamable doesn't require special infrastructure—it's just HTTP. This means you can leverage all the battle-tested tools you already know:

Load balancers: AWS ALB, nginx, HAProxy all work without special configuration
API gateways: Kong, AWS API Gateway, or Cloudflare can add rate limiting, caching, and monitoring
Reverse proxies: Standard nginx or Traefik configs apply
CDNs: Edge caching for read-heavy operations
Monitoring: Prometheus, Datadog, New Relic—any HTTP metrics tool works
Logging: Standard access logs capture every request

You don't need to learn new tools or convince your ops team to adopt unfamiliar technology. HTTP Streamable fits into existing infrastructure patterns.

When to use HTTP Streamable

Production deployments

When your MCP server moves from your laptop to serving real users, HTTP Streamable is the right choice. It provides the reliability, observability, and operational patterns that production systems require. You can deploy with confidence knowing that standard deployment practices—blue-green deployments, canary releases, health checks—all work as expected.

Remote servers (not on the same machine as the client)

If your MCP server needs to run somewhere other than the user's local machine, HTTP Streamable is designed for this. Whether your server lives in:

A cloud VM (AWS EC2, Google Compute, DigitalOcean)
A container platform (Kubernetes, ECS, Cloud Run)
A serverless function (with some adaptations)
A different machine on your local network

HTTP provides the network transport you need. Unlike STDIO which requires the server to be a local subprocess, HTTP Streamable works across any network boundary.

When you need authentication

If you need to answer "who is making this request?", use HTTP Streamable. Common scenarios include:

Paid APIs: Verify users have valid subscriptions before processing requests
Enterprise deployments: Integrate with corporate SSO (SAML, OIDC)
Rate limiting per user: Track and limit usage by authenticated identity
Audit logging: Record who did what for compliance requirements
Feature flags: Enable different capabilities for different user tiers

Multi-user applications

Building an MCP server that multiple people will use? HTTP Streamable handles this naturally. Each user authenticates independently, and the server can:

Maintain per-user quotas and limits
Store user-specific preferences or context
Isolate data between users for security
Scale to handle concurrent users without connection management headaches

Cloud-hosted MCP servers

If you're deploying to any cloud platform, HTTP Streamable is the natural fit. Cloud providers are built around HTTP:

Serverless: AWS Lambda, Google Cloud Functions, and Azure Functions all trigger on HTTP requests
Container services: Cloud Run, App Runner, and Fargate expect HTTP workloads
PaaS platforms: Heroku, Railway, and Render deploy HTTP apps seamlessly

You'll also benefit from cloud-native features like automatic HTTPS, DDoS protection, and global distribution—all without extra configuration.

Code Example

#!/usr/bin/env node
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import express from "express";
import crypto from "crypto";

const app = express();
app.use(express.json());

const sessions = new Map();

function createServer() {
  const server = new McpServer({
    name: "my-mcp-server-http",
    version: "1.0.0"
  });

  server.tool("greet", "Greet someone", {
    name: { type: "string", description: "Name to greet" }
  }, async ({ name }) => ({
    content: [{ type: "text", text: `Hello, ${name}!` }]
  }));

  return server;
}

async function handleMcp(req, res) {
  const sessionId = req.headers["mcp-session-id"];
  let session = sessions.get(sessionId);

  if (!session && req.method === "POST") {
    const transport = new StreamableHTTPServerTransport({
      sessionIdGenerator: () => crypto.randomUUID(),
      onsessioninitialized: (id) => {
        sessions.set(id, { transport, server });
      }
    });

    const server = createServer();
    session = { transport, server };
    await server.connect(transport);
  }

  if (session) {
    await session.transport.handleRequest(req, res, req.body);
  } else {
    res.status(400).json({ error: "No session" });
  }
}

app.post("/mcp", handleMcp);
app.get("/mcp", handleMcp);
app.delete("/mcp", (req, res) => {
  sessions.delete(req.headers["mcp-session-id"]);
  res.status(200).end();
});

app.listen(3001, () => console.log("HTTP server on port 3001"));

Configuration for Claude Code

{
  "mcpServers": {
    "my-server-http": {
      "url": "http://localhost:3001/mcp"
    }
  }
}

Pros and Cons

Pros:

Production-ready
Supports authentication
Scalable (can be stateless)
Works with standard HTTP infrastructure

Cons:

More setup than STDIO
Requires network access
Need to handle session management

Want to understand the technical differences between SSE and HTTP Streamable? Read the companion article: Deep Dive: SSE vs HTTP Streamable — What's the Difference?

Comparison: Which Transport Should You Choose?

Question	STDIO	SSE	HTTP Streamable
Building a local tool?	Yes	No	No
Need remote access?	No	Yes	Yes
Need authentication?	No	Possible	Yes
Multiple clients?	No	Yes	Yes
Production deployment?	No	Possible	Yes
Simplest setup?	Yes	No	No

Decision Flowchart

Is this for local use only?
- Yes → Use STDIO
- No → Continue...
Do you need production features (auth, scaling)?
- Yes → Use HTTP Streamable
- No → Continue...
Do you have existing SSE infrastructure you must integrate with?
- Yes → Use SSE (but plan to migrate)
- No → Use HTTP Streamable (it's the modern default)

Getting Started with Templates

I've created a template repository with all three transports: my-mcp-server

my-mcp-server/
├── stdio/     # STDIO transport (local tools)
├── sse/       # SSE transport (Server-Sent Events)
└── http/      # HTTP Streamable transport (recommended for remote)

Quick Start

# Clone the repo
git clone https://github.com/fzoricic/my-mcp-server.git
cd my-mcp-server

# Try STDIO (local)
cd stdio && npm install && npm start

# Or try HTTP Streamable (remote)
cd http && npm install && npm start

Each template includes:

Working server with a greet tool
Clear comments explaining each part
README with setup instructions
Instructions for adding your own tools

Conclusion

MCP transports might seem confusing at first, but the choice is usually straightforward:

Local tools? → STDIO
Remote/production? → HTTP Streamable
Legacy SSE system? → SSE (but plan migration)

Start with STDIO for experimentation—it's the simplest. When you're ready to deploy remotely or need authentication, upgrade to HTTP Streamable.

Happy building!

Resources

MCP Official Documentation
MCP Architecture Guide
MCP Specification Changelog — Track transport changes across versions
Building MCP Servers
MCP SDK on npm

Deep Dive: SSE vs HTTP Streamable — What's the Difference?

Zoricic — Tue, 03 Mar 2026 21:08:41 +0000

You might be wondering: "Both SSE and HTTP Streamable work over HTTP, so why can't SSE do everything HTTP Streamable does?" Great question. The answer lies in how they manage connections and state.

This is a companion article to Understanding MCP Server Transports: STDIO, SSE, and HTTP Streamable. If you're new to MCP transports, start there first.

How SSE Works Under the Hood

SSE uses a persistent, long-lived connection:

1. Client opens connection    GET /sse HTTP/1.1
                              Accept: text/event-stream

2. Server keeps it open       HTTP/1.1 200 OK
                              Content-Type: text/event-stream

                              data: {"event": "connected"}

                              data: {"event": "update"}

                              ... connection stays open ...

3. Client sends requests      POST /messages?sessionId=abc123
   on separate channel        {"method": "callTool", ...}

The key characteristics:

Connection-based: Server maintains an open connection for each client
Stateful by nature: Server must track each connection in memory
Two channels: SSE stream for server→client, POST requests for client→server
Session affinity required: The client's POST requests must reach the same server instance holding their SSE connection

How HTTP Streamable Works Under the Hood

HTTP Streamable uses standard request/response with optional streaming:

1. Client sends request       POST /mcp HTTP/1.1
                              Content-Type: application/json
                              Mcp-Session-Id: abc123

                              {"method": "callTool", ...}

2. Server responds            HTTP/1.1 200 OK
   (can stream if needed)     Content-Type: application/json

                              {"result": ...}

3. Connection closes          (or streams multiple responses)

The key characteristics:

Request/response based: Each interaction is a complete HTTP request
Stateless capable: Server doesn't need to hold connections open
Single channel: Everything goes through the same endpoint
No affinity required: Any server instance can handle any request

Why SSE Can't Match HTTP Streamable's Capabilities

1. The Persistent Connection Problem

SSE requires the server to maintain an open connection for every client. This creates real problems at scale:

SSE with 1,000 clients:
┌─────────────────────────────────────┐
│           SSE Server                │
│  ┌─────┐ ┌─────┐ ┌─────┐           │
│  │Conn1│ │Conn2│ │...  │ │Conn1000││
│  └─────┘ └─────┘ └─────┘           │
│  Memory: ~1KB × 1000 = 1MB+        │
│  File descriptors: 1000            │
│  Cannot simply restart server      │
└─────────────────────────────────────┘

HTTP Streamable with 1,000 clients:
┌─────────────────────────────────────┐
│      HTTP Streamable Server         │
│                                     │
│  No persistent connections          │
│  Memory: only during requests       │
│  Can restart/deploy anytime         │
└─────────────────────────────────────┘

With SSE, if your server restarts (for a deployment, crash, or scaling event), all 1,000 clients lose their connections and must reconnect. With HTTP Streamable, clients just send their next request — they don't even notice.

2. The Load Balancing Problem

With SSE, you need "sticky sessions" — the load balancer must route each client to the same server:

SSE Load Balancing (Complex):
                         ┌─ Server A (holds Client 1's connection)
Client 1 ─► Load ────────┤
            Balancer     │  Must remember: Client 1 → Server A
            (sticky)     │                 Client 2 → Server B
Client 2 ─►─────────────┼─ Server B (holds Client 2's connection)

If Server A dies, Client 1 loses connection and must reconnect!

HTTP Streamable Load Balancing (Simple):
                         ┌─ Server A
Client 1 ─► Load ────────┤
            Balancer     │  Any request → Any server
            (stateless)  │
Client 2 ─►─────────────┼─ Server B

If Server A dies, next request just goes to Server B!

3. The Authentication Problem

SSE authentication is awkward because the EventSource API (used in browsers) doesn't support custom headers:

// SSE: Can't set Authorization header easily
const eventSource = new EventSource('/sse'); // No header support!

// Workarounds are hacky:
// - Pass token in URL: /sse?token=xyz (visible in logs, insecure)
// - Use cookies (but what about non-browser clients?)

HTTP Streamable uses standard HTTP requests where auth is straightforward:

// HTTP Streamable: Standard auth headers
fetch('/mcp', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer xyz',  // Clean and standard
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({...})
});

4. The Infrastructure Problem

Many infrastructure tools assume short-lived HTTP connections:

Infrastructure	SSE Support	HTTP Streamable Support
AWS Lambda	❌ Times out	✅ Perfect fit
Cloud Run	⚠️ Needs config	✅ Works by default
Cloudflare	⚠️ Connection limits	✅ Standard HTTP
API Gateways	⚠️ Often problematic	✅ Native support
Corporate proxies	⚠️ May kill long connections	✅ No issues

When SSE Still Makes Sense

Despite these limitations, SSE isn't obsolete. Use it when:

You need true push notifications: SSE excels when the server needs to push updates unprompted (though HTTP Streamable can also stream)
Browser-native simplicity: EventSource API is dead simple for basic use cases
Existing SSE infrastructure: If you've already built around SSE, migration may not be worth it
Single-server deployments: The scaling issues don't matter if you only have one server

The Bottom Line

SSE and HTTP Streamable both use HTTP, but they represent fundamentally different architectural patterns:

Aspect	SSE	HTTP Streamable
Connection model	Persistent	Request/Response
Server state	Must maintain connections	Can be stateless
Scaling model	Vertical (bigger servers)	Horizontal (more servers)
Load balancing	Sticky sessions required	Stateless routing
Auth support	Awkward	Native HTTP auth
Infrastructure fit	Specialized	Universal

SSE = Keep a phone line open
HTTP Streamable = Send text messages

Both communicate, but one requires maintaining an open line while the other sends discrete messages that can be routed anywhere.

What Should You Choose?

For most new MCP projects, HTTP Streamable is the better choice. It's more flexible, scales better, and works with modern cloud infrastructure out of the box.

Use SSE only if you have a specific reason—like existing infrastructure or a genuine need for server-initiated push without polling.

Resources

Understanding MCP Server Transports — Overview of all three transports
MCP Official Documentation
MCP Template Repository — Working examples of all transports