DEV Community: Vahid Aghajani

Why Your LLM Keeps Returning Garbage JSON (And How to Stop It)

Vahid Aghajani — Sat, 04 Jul 2026 19:22:01 +0000

Prefer audio? Spotify episode · Telegram

You wire up an LLM call. The demo is magical. You ship it.

The next morning Sentry is on fire. json.JSONDecodeError: Expecting value: line 1 column 1 (char 0). You open the failed payload and the model has politely returned:

Sure! Here's the JSON you asked for:

```json
{
  "name": "Acme Corp",
  "founded": 1998,
}
```

Let me know if you need anything else!

Three things broke at once: a chatty preamble, a markdown code fence, and a trailing comma. Every team building on LLMs hits this within the first week of going to production. The fix isn't one trick — it's three layers, each catching what the previous one misses.

This post is the layered playbook: native structured-output APIs first, typed validation second, repair-and-retry third. It's the same pattern shipping in our own production code today.

Why LLMs Fail at JSON in the First Place

A language model doesn't output JSON. It outputs the most likely next token, repeatedly. JSON is just a particular sequence of tokens it has seen a lot during training. So whenever the prompt is even slightly ambiguous about format — or the model is fine-tuned to be helpful and conversational — those token probabilities drift toward natural language.

Common failure modes, ranked by frequency:

Wrapped in markdown — json ... because the training data is full of code blocks.
Chatty preamble or trailing chatter — "Here's your JSON:", "Hope this helps!".
Trailing commas — JSON forbids them, JavaScript and Python don't.
Single quotes — looks like JSON, isn't.
Unescaped quotes inside strings — "He said "hi"".
Truncation mid-object — token limit hit, last brace missing.
Hallucinated fields — extra keys you didn't ask for; missing required ones.
Wrong types — "founded": "1998" (string) when you asked for an integer.

You can't prompt your way out of all of these. You need the model to be constrained, the output to be typed, and a fallback for when both still fail.

Layer 1: Use the Native Structured-Output API

Every major provider now ships a way to constrain decoding to a schema. Use it. This single change kills 80–90% of the failure modes above.

OpenAI — `response_format={"type": "json_schema", ...}`

The model is forced to produce tokens that satisfy the schema. No prose, no fences.

from openai import OpenAI
from pydantic import BaseModel

class Company(BaseModel):
    name: str
    founded: int
    industry: str

client = OpenAI()
resp = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[{"role": "user", "content": "Extract from: 'Acme Corp, founded 1998, makes industrial widgets.'"}],
    response_format=Company,
)
company: Company = resp.choices[0].message.parsed

parsed is already a typed Pydantic object. No json.loads. No regex. No prayer.

Anthropic — Tool use as a schema enforcer

Claude doesn't have a response_format field, but tool definitions act the same way: declare a tool with an input_schema, force the model to call it, and the input you get back is schema-conformant JSON.

import anthropic

client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=[{
        "name": "extract_company",
        "description": "Extract structured company data.",
        "input_schema": Company.model_json_schema(),
    }],
    tool_choice={"type": "tool", "name": "extract_company"},
    messages=[{"role": "user", "content": "Acme Corp, founded 1998, makes industrial widgets."}],
)
company = Company.model_validate(resp.content[0].input)

tool_choice forces the model to call the tool, which forces schema compliance.

Gemini — `response_mime_type` + `response_schema`

Gemini is especially strong at enforcing enums during decoding. If a field has a fixed set of valid values, declare it as a Literal — Gemini will physically refuse to emit a token outside that set, killing a whole class of Layer-2 validation failures before they happen.

from typing import Literal
from google import genai
from google.genai import types
from pydantic import BaseModel

class CompanyStatus(BaseModel):
    name: str
    founded: int
    industry: Literal["software", "manufacturing", "finance", "other"]
    status: Literal["active", "acquired", "defunct"]

client = genai.Client()
resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Acme Corp, founded 1998, makes industrial widgets. Still trading.",
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=CompanyStatus,
        temperature=0,
    ),
)
company = CompanyStatus.model_validate_json(resp.text)

Rule of thumb: if the API has a structured-output mode, use it. Don't hand-roll prompts that say "return only valid JSON, no other text" — that worked in 2023 and it still kind of works in 2026, but it's strictly worse than constrained decoding.

Layer 2: Validate With Pydantic (Even If Layer 1 Worked)

Constrained decoding gives you syntactic JSON. It does not give you semantic correctness. The model can still:

Return founded: 1 when you wanted a 4-digit year.
Return an empty name.
Return a plausible-but-wrong industry.
Return all-nulls because the source text didn't actually contain the fields.

Pydantic catches these at the boundary, before bad data enters your domain logic:

from pydantic import BaseModel, Field, field_validator
from typing import Literal

class Company(BaseModel):
    name: str = Field(min_length=1, max_length=200)
    founded: int = Field(ge=1800, le=2030)
    industry: Literal["software", "manufacturing", "finance", "other"]

    @field_validator("name")
    @classmethod
    def name_not_placeholder(cls, v: str) -> str:
        if v.strip().lower() in {"unknown", "n/a", "none", ""}:
            raise ValueError("name looks like a placeholder")
        return v

Validation failures here are useful signal, not just errors. If founded=1 keeps tripping the validator, your prompt is ambiguous — fix the prompt. If industry="other" appears too often, your enum is too narrow — fix the schema.

The pattern: every LLM call returns into a Pydantic model, and the rest of your codebase only ever sees validated objects. Treat the LLM like a third-party API you don't trust.

Layer 3: Repair and Retry When Both Layers Above Fail

For ~95% of calls, layers 1 and 2 are enough. The remaining 5% — long inputs, weird edge cases, model degradation, rate-limit retries that hit a different model version — still fails. You need a fallback.

`json_repair` for almost-JSON

json-repair is a small library that fixes the common malformations: trailing commas, single quotes, missing closing braces, markdown fences, prose-around-JSON.

from json_repair import repair_json
import json

raw = response.text  # the model's raw string

try:
    data = json.loads(raw)
except json.JSONDecodeError:
    data = json.loads(repair_json(raw))

It's not magic — it's a forgiving parser. It will succeed on inputs that strict JSON refuses, and it has saved more production calls than any single prompt change.

Retry with the validation error fed back to the model

If repair_json fails and Pydantic validation fails, retry with the error message in the next prompt. The model is genuinely good at fixing its own mistakes when you tell it what broke:

def call_with_repair(messages, schema_cls, max_retries=2):
    for attempt in range(max_retries + 1):
        resp = call_llm(messages)  # native structured-output call
        try:
            return schema_cls.model_validate_json(resp.text)
        except ValidationError as e:
            if attempt == max_retries:
                raise
            messages.append({"role": "assistant", "content": resp.text})
            messages.append({
                "role": "user",
                "content": f"Your previous response failed validation: {e}\nFix it and return only valid JSON matching the schema.",
            })

Two retries is the sweet spot. One isn't enough for sticky failures; three burns tokens for almost no extra success rate.

Make failure structured, too

When all retries are exhausted, don't raise Exception("LLM failed"). Raise a typed exception your caller can branch on:

class InsufficientDataError(Exception):
    """The source material genuinely didn't contain the requested fields."""

class SchemaViolation(Exception):
    """The model couldn't conform to the schema after retries."""

Different failures deserve different handling. A SchemaViolation is a model/prompt problem — log it, alert. An InsufficientDataError is a data problem — surface it to the user as "we couldn't extract X from this document", not as a 500.

Putting It All Together

Here's the full pattern, condensed:

def extract_company(text: str) -> Company:
    messages = [
        {"role": "system", "content": "Extract structured company data from the text."},
        {"role": "user", "content": text},
    ]
    return call_with_repair(messages, schema_cls=Company, max_retries=2)

Three layers, one entry point. The caller never sees a JSONDecodeError. They get a Company or a typed exception they can handle.

Decision Matrix

You're calling…	Use
OpenAI GPT-4o or newer	`response_format=PydanticModel` (`.parse()` API)
Anthropic Claude	Tool with forced `tool_choice` + `model_json_schema()`
Gemini 2.5+	`response_mime_type="application/json"` + `response_schema=PydanticModel`
OpenRouter wrapper	Whatever the underlying model supports — check, don't assume
Local Llama/Mistral via vLLM	Outlines or LM Format Enforcer for grammar-constrained decoding
Anything older or weirder	Plain prompt + `json_repair` + Pydantic + retry loop

And one more axis: model size

Model class	Strategy
Frontier (GPT-4o, Claude Opus 4.x, Gemini 2.5 Pro)	Layers 1 + 2 are usually enough. Layer 3 catches the long tail.
Small / edge (Gemini Flash, Llama 3.x 8B, Phi-4, Mistral 7B)	Layer 3 is mandatory. Small models trip on nested schemas, Optional fields, and long enums far more often. Budget for 2–5% retry rate even with structured outputs on.

Gotchas Nobody Tells You

Schemas with Optional[T] get filled with None aggressively. The model treats "I don't know" as a valid answer when nullability is allowed. If you need extraction to be honest about missing data, use a typed exception path instead.
Enums with too many values regress to "other". Keep Literal[...] lists short. If you need 50 categories, use a two-step pipeline: free-text → embedding → nearest enum.
additionalProperties: false matters. Without it, the model invents fields. Pydantic v2 emits this by default; double-check if you write the JSON Schema by hand.
Streaming + structured outputs is half-broken everywhere. You can stream the JSON, but you can't typed-parse it until the stream finishes. Don't promise users a typewriter effect on extracted data.
JSON mode is not free. Constrained decoding adds 10–30% latency on most providers. Worth it for correctness, but budget for it.
Retries cost money quadratically when prompts are long. A 50K-token prompt that retries twice is 150K tokens. Cache the system prompt aggressively.

Closing

The "LLM-as-software-component" problem isn't solved by a smarter model. It's solved by treating the model like every other unreliable upstream service: constrain what it can return, validate what it does return, repair what's almost right, retry with feedback, fail loudly with typed errors when all else breaks.

Three layers. Each one catches what the previous one missed. Ship that and your Sentry inbox stops lighting up at 6 a.m.

Vision Language Models — When AI Learns to See and Talk (Part 3 of 3)

Vahid Aghajani — Sat, 04 Jul 2026 18:17:21 +0000

This is Part 3 of a 3-part series on the transformer revolution in vision and language:

Part 1: Transformers — The Architecture That Changed AI
Part 2: Vision Transformers — How Transformers Learned to See
Part 3: Vision Language Models — When AI Learns to See and Talk (this post)

In Part 1, we covered how the transformer architecture replaced RNNs and CNNs as the backbone of modern AI. In Part 2, we saw how Vision Transformers (ViTs) brought that same architecture to image understanding — splitting images into patches and treating them like tokens.

Now comes the question that drives this entire field forward: what happens when you combine both?

Why Combine Vision and Language?

Humans don't process the world in isolated channels. When you look at a photo of a dog catching a frisbee in a park, you don't separately "see" the image and then "think" in language. Your understanding is multimodal from the start — you perceive the scene, recognize objects, understand spatial relationships, and can describe it all in natural language without effort.

Traditional AI couldn't do this. Computer vision models could classify images or detect objects, but they couldn't explain what they saw. Language models could write eloquently, but they were blind. These were separate systems with separate training pipelines, separate datasets, and no shared understanding.

Vision Language Models (VLMs) change this. They bridge the gap between pixels and words, creating systems that can look at an image and answer questions about it, generate descriptions, follow visual instructions, or reason about what they see.

The applications are enormous: a doctor uploads a medical scan and asks "What do you see?"; a warehouse robot reads labels and navigates shelves; a visually impaired user points their phone at a restaurant menu and gets it read aloud. All of these require a model that sees and speaks.

The Evolution: From Captioning to True Multimodal Understanding

The journey from "AI that describes pictures" to "AI that understands images and reasons about them" happened in distinct phases.

Phase 1: Image Captioning (2015-2019)

Early systems used a CNN encoder (like ResNet) to extract image features, then fed those features into an RNN or LSTM decoder to generate a caption. The architecture was straightforward: image -> CNN -> feature vector -> RNN -> "A dog catches a frisbee". These systems worked but were brittle — they could generate grammatically correct captions, but didn't truly understand the scene. Ask a follow-up question and they'd fall apart.

Phase 2: Contrastive Pre-training (2021)

CLIP changed the game by learning to align images and text in a shared embedding space, without generating anything. This allowed zero-shot classification, image search, and open-vocabulary recognition. It was the first time a single model could handle visual concepts it had never been explicitly trained on.

Phase 3: Generative Multimodal Models (2022-2023)

Models like Flamingo, BLIP-2, and LLaVA took things further — they could not just align images and text, but generate free-form text responses about images. You could have a conversation about a photo.

Phase 4: Natively Multimodal Systems (2023-present)

GPT-4V, Gemini, and Claude represent the current frontier: models trained from the ground up to handle text, images, video, and audio as first-class inputs. These aren't vision modules bolted onto a language model — they are unified systems.

Four Architectural Approaches to VLMs

Not all VLMs are built the same way. There are four fundamental design patterns, each with distinct trade-offs.

1. Contrastive Learning (The CLIP Approach)

CLIP (Contrastive Language-Image Pre-training, OpenAI 2021) uses a dual-encoder architecture. One encoder processes images (a ViT or ResNet), and a separate encoder processes text (a transformer). Both encoders map their inputs into the same embedding space.

During training, CLIP sees 400 million image-text pairs scraped from the internet. For each batch, it computes the cosine similarity between every image embedding and every text embedding. The training objective is simple: maximize the similarity between matching image-text pairs and minimize it for non-matching ones. This is contrastive learning — the model learns by contrasting positives against negatives.

Image Encoder (ViT)  ──→  Image Embedding  ──┐
                                               ├──→  Cosine Similarity
Text Encoder (Transformer) → Text Embedding ──┘

Why was CLIP revolutionary? Three reasons:

Zero-shot transfer. To classify an image, you don't fine-tune. You just compute similarity between the image embedding and text embeddings like "a photo of a dog" or "a photo of a cat". The highest similarity wins. This means CLIP can classify images into any categories you define at inference time — no retraining needed.
Open vocabulary. Traditional classifiers are limited to their fixed label set. CLIP understands free-form language, so you can classify images using any text description you can think of.
Web-scale training. By using image-text pairs from the internet instead of hand-labeled datasets, CLIP trained on far more diverse data than any supervised model.

The limitation: CLIP is an alignment model, not a generative model. It can tell you which text best matches an image, but it can't generate a detailed description or answer complex questions.

2. Cross-Attention Fusion (The Flamingo Approach)

Flamingo (DeepMind, 2022) takes a different strategy. Instead of aligning two separate encoders, it injects visual information directly into a frozen large language model using cross-attention layers.

The architecture works like this: a vision encoder (a frozen NFNet or ViT) extracts visual features from the input image. These features are then compressed by a Perceiver Resampler — a small transformer module that reduces the variable number of visual tokens into a fixed set (typically 64). These compressed visual tokens are fed into newly added cross-attention layers that are interleaved between the existing self-attention layers of the frozen LLM.

Image ──→ Vision Encoder ──→ Perceiver Resampler ──→ Visual Tokens
                                                         │
Text ──→ [Frozen LLM with interleaved cross-attention] ←─┘
                         │
                    Generated Text

The key insight: the LLM itself stays frozen. Only the Perceiver Resampler and the cross-attention layers are trained. This preserves the language model's capabilities while teaching it to attend to visual information.

Flamingo excelled at few-shot learning. You could show it a few image-text examples as context, and it would generalize to new tasks — much like how GPT-3 demonstrated few-shot language capabilities.

3. Visual Tokens into LLM (The LLaVA Approach)

LLaVA (Large Language and Vision Assistant, 2023) takes the simplest possible approach: use a linear projection (or MLP) to map visual features into the token embedding space of an LLM, then just prepend them to the text tokens.

The architecture is refreshingly minimal:

Pass the image through a pre-trained CLIP ViT to get visual features.
Project those features through a trained MLP to match the LLM's embedding dimension.
Concatenate the projected visual tokens with the text tokens.
Feed everything into the LLM as a single sequence.

Image ──→ CLIP ViT ──→ MLP Projection ──→ Visual Tokens
                                              │
                              [v1, v2, ..., vN, text tokens] ──→ LLM ──→ Response

From the LLM's perspective, visual tokens look just like any other tokens. The model processes them using its standard self-attention mechanism. No architectural changes to the LLM are needed.

LLaVA's training happens in two stages:

Pre-training the projection. The vision encoder and LLM are frozen; only the MLP is trained on image-caption pairs to align the visual feature space with the language embedding space.
Visual instruction tuning. The MLP and LLM are fine-tuned together on instruction-following data — conversations about images, visual question answering, and complex reasoning tasks.

LLaVA proved that you don't need exotic architectural innovations. A well-trained projection layer and good instruction-tuning data can produce remarkably capable multimodal models. Its open-source nature made it enormously influential — LLaVA-NeXT, LLaVA-OneVision, and many derivative models followed.

4. Natively Multimodal (The Gemini Approach)

Gemini (Google, 2023) takes the most ambitious approach: train a single transformer from scratch on interleaved text, images, audio, and video. There is no separate vision encoder bolted on — the model natively processes all modalities through a unified architecture.

Images are tokenized using SentencePiece-style visual tokenization or learned patch embeddings, and these visual tokens are interleaved with text tokens during both training and inference. The model processes everything through the same transformer layers.

[text tokens, image tokens, text tokens, audio tokens, ...] ──→ Unified Transformer ──→ Output

The advantage is deep fusion: visual and textual understanding develop together during training, rather than being stitched together after the fact. The model can reason across modalities in a way that adapter-based approaches struggle with.

The disadvantage is cost. Training a natively multimodal model from scratch requires enormous compute budgets and carefully curated multimodal training data. This is why only a handful of labs (Google, OpenAI, Anthropic) have built models in this category.

Major VLM Models: A Landscape Overview

The VLM space has exploded. Here's a tour of the most important models and what makes each one notable.

CLIP (OpenAI, 2021)

The model that started the modern VLM era. Trained on 400M image-text pairs using contrastive learning. CLIP is not generative — it aligns images and text in a shared space — but it became the backbone vision encoder for dozens of subsequent models (LLaVA, BLIP-2, and more all use CLIP's ViT).

BLIP and BLIP-2 (Salesforce, 2022-2023)

BLIP introduced "bootstrapping" — using a model to generate and then filter its own training captions, creating higher-quality data. BLIP-2 took this further with the Q-Former (Querying Transformer), a lightweight module that bridges a frozen image encoder and a frozen LLM. The Q-Former uses a set of learnable query tokens that interact with visual features through cross-attention, then pass the result to the LLM. This made it possible to combine powerful pre-trained components with minimal training.

Flamingo (DeepMind, 2022)

The few-shot champion. By interleaving cross-attention layers into a frozen Chinchilla LLM, Flamingo showed that you could give a language model vision capabilities without retraining it. Its few-shot performance on visual QA benchmarks was remarkable — you could show it 4-8 example image-text pairs and it would generalize effectively.

LLaVA / LLaVA-NeXT (University of Wisconsin, 2023-2024)

The open-source workhorse. LLaVA proved that a simple projection MLP between a CLIP ViT and a Vicuna/LLaMA LLM, combined with high-quality visual instruction-tuning data, could match or exceed far more complex architectures. LLaVA-NeXT improved resolution handling with dynamic image partitioning — splitting high-resolution images into tiles, encoding each tile, and concatenating the features.

GPT-4V / GPT-4o (OpenAI, 2023-2024)

GPT-4V brought multimodal capabilities to the most capable commercial LLM. Architectural details are not published, but it handles complex visual reasoning, OCR, chart understanding, and multi-image comparisons. GPT-4o ("omni") extended this to audio and real-time interaction, processing text, images, and audio natively rather than through separate pipelines.

Gemini (Google, 2023-2024)

Google's natively multimodal family. Available in Ultra, Pro, and Nano sizes. Gemini processes text, images, audio, and video through a unified transformer trained from scratch on multimodal data. Gemini 1.5 Pro introduced a 1M+ token context window, enabling processing of hour-long videos or hundreds of pages of documents alongside text queries.

Claude (Anthropic, 2024-present)

Anthropic's Claude models support image understanding with strong performance on document analysis, chart reading, and visual reasoning. Like GPT-4V and Gemini, the exact architecture is proprietary, but Claude demonstrates particularly strong performance on tasks requiring careful analysis and reduced hallucination.

PaliGemma (Google, 2024)

An open-weight, lightweight VLM combining a SigLIP vision encoder with a Gemma language model. Designed for fine-tuning on specific tasks — OCR, visual QA, object detection, image segmentation. At 3B parameters, PaliGemma showed that you don't need massive models for practical VLM applications.

Qwen-VL (Alibaba, 2023-2024)

Alibaba's open-source multimodal model. Supports image, video, and text inputs. Qwen2-VL introduced Naive Dynamic Resolution — handling images at their native resolution by dynamically adjusting the number of visual tokens rather than resizing all images to a fixed size. This was a meaningful advance for tasks requiring fine-grained detail.

InternVL (Shanghai AI Lab, 2023-2024)

A strong open-source contender, combining an InternViT vision encoder with an InternLM language model. InternVL 2.0 scaled to 108B parameters and achieved competitive performance with commercial models on benchmarks. Notable for its progressive training strategy — scaling both the vision encoder and LLM together.

VLM Comparison

Model	Architecture Type	Open/Closed	Key Strength	Parameters
CLIP	Dual encoder (contrastive)	Open	Zero-shot classification, backbone encoder	~400M
BLIP-2	Q-Former bridge	Open	Efficient frozen model connection	~3-12B
Flamingo	Cross-attention fusion	Closed	Few-shot multimodal learning	80B
LLaVA / LLaVA-NeXT	Projection into LLM	Open	Simple, effective, easy to reproduce	7-34B
GPT-4V / GPT-4o	Natively multimodal	Closed	Strongest general reasoning	Undisclosed
Gemini	Natively multimodal	Closed (API)	Long context, video understanding	Undisclosed
Claude	Natively multimodal	Closed (API)	Document analysis, reduced hallucination	Undisclosed
PaliGemma	SigLIP + Gemma projection	Open	Lightweight, fine-tunable	3B
Qwen2-VL	Dynamic resolution + LLM	Open	Native resolution, multilingual	2-72B
InternVL 2.0	Progressive scaling ViT + LLM	Open	Competitive with closed models	1-108B

Real-World Applications

VLMs have moved well beyond research benchmarks into production systems.

Visual Question Answering and Conversational AI

The most visible application: upload an image and ask questions about it. This powers customer support (photograph a broken product and describe the issue), education (point at a math problem and get step-by-step solutions), and accessibility (describe scenes for visually impaired users).

Document Understanding and OCR

VLMs excel at understanding the structure of documents, not just the text. They can read invoices, parse tables, understand forms, and extract information from complex layouts that traditional OCR systems struggle with. Financial services, legal, and healthcare all benefit here.

Autonomous Driving and Robotics

Self-driving systems need to understand scenes in context: "Is that person about to cross the street?" requires combining visual perception with semantic reasoning. VLMs can provide this contextual understanding as part of the driving stack. In robotics, VLMs enable robots to follow natural language instructions in the physical world — "pick up the red cup next to the keyboard."

Medical Imaging

Radiologists can use VLMs as a second-opinion tool — upload a chest X-ray and ask about potential findings. Models like Med-PaLM M (Google) were specifically fine-tuned for medical multimodal tasks. The combination of visual understanding and natural language output makes findings more accessible to non-specialists.

Creative and Design Work

VLMs can critique designs ("What's wrong with this UI layout?"), provide alt-text for images at scale, analyze competitors' visual branding, and help with content moderation by understanding both images and their context.

Challenges and Open Problems

Despite rapid progress, VLMs still have significant limitations.

Hallucination

This is the biggest problem. VLMs confidently describe objects that don't exist in the image, misread text, or invent details. A model might claim there are three people in an image that shows two, or describe a red car as blue. The language model's tendency to generate plausible-sounding text sometimes overrides what it actually "sees." Reducing multimodal hallucination is an active research area, with approaches like RLHF on visual tasks and grounding mechanisms showing promise.

Spatial Reasoning

VLMs struggle with precise spatial relationships. "Is the cup to the left or right of the book?" or "How many windows are on the second floor?" often produce wrong answers. The image-to-tokens pipeline loses fine-grained spatial information, and current training data doesn't emphasize spatial reasoning enough.

Fine-Grained Understanding

Counting objects accurately, reading small text in images, distinguishing between visually similar items (different bird species, similar product models) — these remain difficult. Higher image resolutions help, but processing 4K images means thousands of visual tokens, which strains context windows and compute budgets.

Temporal Reasoning in Video

While models like Gemini can process video, true temporal reasoning ("What happened right before the person fell?") remains limited. Most video VLMs sample frames rather than processing continuous video, losing temporal dynamics.

Safety and Bias

VLMs inherit biases from both their visual and textual training data. They may generate stereotypical descriptions, fail to recognize people from underrepresented groups, or be manipulated through adversarial images. Multimodal safety is harder than text-only safety because the attack surface is larger — adversarial patterns can be embedded in images in ways that are invisible to humans but manipulate model outputs.

Where Multimodal AI Is Headed

The trajectory is clear: modality barriers are dissolving.

Unified models are winning. The trend is away from adapter-based approaches (bolt a vision encoder onto an LLM) and toward natively multimodal training. Future models will likely process text, images, video, audio, 3D data, and sensor readings through a single architecture, trained together from the start.

Reasoning over visual information is improving fast. Chain-of-thought prompting is being extended to visual reasoning — models that can "think step by step" about what they see, breaking complex visual scenes into sequential reasoning steps. This addresses spatial reasoning and counting weaknesses.

Smaller, specialized models are becoming practical. Not every application needs GPT-4V. Models like PaliGemma and Qwen2-VL show that focused, open-weight models in the 2-7B parameter range can handle specific visual tasks effectively. Expect more task-specific VLMs that can run on edge devices.

The agentic future is multimodal. AI agents that can browse the web, interact with GUIs, and operate in physical environments need vision-language understanding as a core capability. VLMs are the perceptual backbone of autonomous AI systems — from computer-use agents that navigate screens to robots that manipulate objects based on verbal instructions.

Real-time multimodal interaction is arriving. GPT-4o's ability to process audio, video, and text simultaneously in real-time conversation points toward a future where AI assistants see, hear, and respond as naturally as a human conversation partner.

Conclusion

The three-part journey from transformers to VLMs tells a coherent story about how a single architectural idea — self-attention over sequences — scaled from text to images to true multimodal understanding.

Transformers gave us attention. Vision Transformers showed that images are just sequences of patches. And Vision Language Models proved that you can unify perception and language in a single model.

We are still early. Current VLMs hallucinate, struggle with spatial reasoning, and require enormous compute. But the rate of improvement is remarkable — the gap between the best VLMs in early 2024 and late 2025 is larger than the gap between 2018 and 2023. The models are getting smaller, faster, more capable, and more accessible.

The endgame isn't just a model that can look at a picture and answer a question. It's an AI that perceives the world as richly as humans do — that can watch a video, read a document, listen to a conversation, and reason across all of it simultaneously. VLMs are the foundation that makes this possible.

Vision Transformers — How Transformers Learned to See (Part 2 of 3)

Vahid Aghajani — Sat, 04 Jul 2026 18:11:58 +0000

Recap: The Transformer Revolution (Part 1)

In Part 1 of this series, we explored how the Transformer architecture — introduced in Google's 2017 paper "Attention Is All You Need" — upended natural language processing. The key ideas were self-attention (letting every token attend to every other token), positional encodings (injecting sequence order without recurrence), and multi-head attention (learning multiple relationship patterns in parallel). Transformers replaced RNNs and LSTMs as the backbone of language models, eventually powering GPT, BERT, and everything that followed.

But Transformers were designed for sequences of tokens — words, subwords, characters. Images are not sequences. They are 2D grids of pixels with spatial structure, local patterns, and hierarchical features. For decades, a completely different family of architectures dominated vision: convolutional neural networks.

So how did Transformers learn to see? That is the story of this post.

The CNN Era: What Worked and What Did Not

Convolutional Neural Networks (CNNs) have been the workhorse of computer vision since AlexNet won ImageNet in 2012. The architecture is elegant: small learnable filters slide across the image, detecting local patterns like edges, textures, and shapes. Stacking convolutional layers builds a hierarchy — early layers detect edges, middle layers detect parts (eyes, wheels), and deep layers detect entire objects.

CNNs come with strong inductive biases baked into their design:

Locality: Each filter looks at a small patch of the image. A 3x3 convolution only sees 9 pixels at a time.
Translation equivariance: The same filter is applied everywhere, so a cat detected in the top-left corner uses the same weights as a cat in the bottom-right.
Hierarchical feature extraction: Pooling layers progressively reduce spatial resolution, forcing the network to build abstract representations.

These biases are a gift when data is limited. They tell the network how to look at images before it sees a single training example. Models like ResNet, EfficientNet, and ConvNeXt achieved remarkable accuracy and efficiency.

But CNNs have limitations:

Limited receptive field: Even deep CNNs struggle to capture long-range dependencies. A pixel in the top-left corner has no direct connection to a pixel in the bottom-right until very late in the network. This matters for understanding scene-level context — knowing that a person is holding a surfboard requires relating distant image regions.
Fixed geometric structure: Convolutions are rigid. They process fixed-size local neighborhoods regardless of content. They cannot dynamically decide which parts of the image are most relevant to each other.
Scaling bottlenecks: While CNNs scale reasonably, the relationship between model size, data, and performance plateaus compared to what Transformers achieve in NLP.

Researchers asked: could Transformers, with their ability to let any part of the input attend to any other part, do better?

The Key Insight: Images as Sequences of Patches

The breakthrough idea behind Vision Transformers is disarmingly simple: treat an image as a sequence of patches, just like a sentence is a sequence of words.

Take a 224x224 pixel image. Divide it into a grid of non-overlapping 16x16 patches. You get 14 x 14 = 196 patches. Each patch is a small image region containing 16 x 16 x 3 = 768 pixel values (for RGB images). Flatten each patch into a vector, project it through a linear layer, and you have a sequence of 196 "tokens" — each one representing a patch of the image.

Now you can feed this sequence into a standard Transformer encoder. Self-attention lets every patch attend to every other patch, regardless of spatial distance. The patch in the top-left corner can directly interact with the patch in the bottom-right corner in a single layer. No need to stack dozens of layers to build a large receptive field.

This is the core idea of the Vision Transformer (ViT), published by Google Research in late 2020.

ViT Architecture: A Step-by-Step Walkthrough

Let us trace how ViT processes a single image from input to classification.

Step 1: Patch Embedding

The input image (e.g., 224x224x3) is divided into a grid of P x P patches (typically P=16). Each patch is flattened into a vector of length P^2 x C (where C is the number of channels), giving a 768-dimensional vector for 16x16 RGB patches. A learned linear projection maps each flattened patch to the model's hidden dimension D (e.g., 768). The result: a sequence of N patch embeddings, where N = (224/16)^2 = 196.

Image (224x224x3)
  → Split into 196 patches of 16x16x3
  → Flatten each patch to 768-dim vector
  → Linear projection to D-dim embedding
  → Sequence of 196 token embeddings

Step 2: The [CLS] Token

ViT prepends a special learnable [CLS] token to the patch sequence, borrowed directly from BERT. This token does not correspond to any image patch. Instead, it serves as an aggregation point: through self-attention across all layers, it collects information from every patch. After the final Transformer layer, the [CLS] token's representation is used for classification.

The sequence is now 197 tokens long: 1 [CLS] token + 196 patch tokens.

Step 3: Position Embeddings

Unlike convolutions, the Transformer has no built-in notion of spatial position. If you shuffle the patch order, the self-attention output is identical (it is permutation-equivariant). To encode spatial information, ViT adds learnable 1D position embeddings to each token. Position 0 is the [CLS] token, positions 1-196 correspond to patches in raster order (left-to-right, top-to-bottom).

Interestingly, the original ViT paper found that 1D positional embeddings work just as well as explicit 2D positional encodings. The model learns the 2D structure from data — nearby position embeddings end up with similar values, effectively reconstructing a 2D grid.

Step 4: Transformer Encoder

The sequence of 197 position-encoded embeddings is fed into a standard Transformer encoder — the exact same architecture from the NLP world. Each layer consists of:

Layer Normalization (applied before attention, following the Pre-Norm convention)
Multi-Head Self-Attention (MHSA): Every token attends to every other token. For 197 tokens, this means a 197x197 attention matrix per head. Each head can learn different relationships — some might focus on nearby patches, others on semantically related distant patches.
Residual Connection: Add the attention output back to the input
Layer Normalization
MLP (Feed-Forward Network): Two linear layers with GELU activation, expanding the dimension by 4x then projecting back
Residual Connection

ViT-Base uses 12 such layers, ViT-Large uses 24, and ViT-Huge uses 32.

Step 5: Classification Head

After the final encoder layer, the [CLS] token's output representation is extracted and passed through a simple MLP head (one hidden layer during pre-training, a single linear layer during fine-tuning) to produce class logits.

Input Image
  → Patch Embedding (196 patches)
  → Prepend [CLS] token (197 tokens)
  → Add Position Embeddings
  → Transformer Encoder (L layers of MHSA + MLP)
  → Extract [CLS] token output
  → MLP Classification Head
  → Class Prediction

The elegance is striking: no pooling layers, no convolutions, no hand-crafted feature extractors. Just patches, linear projections, and attention.

The Data Hunger Problem

Here is the catch. When ViT was trained on ImageNet alone (1.3 million images), it performed worse than comparable CNNs like ResNet. The Transformer's lack of inductive bias is both its strength and its weakness.

CNNs "know" to look locally and share weights spatially. ViT knows nothing — it must learn everything from data, including the fact that nearby pixels are related and that patterns can appear anywhere in the image. Learning these priors from scratch requires enormous amounts of data.

The original ViT paper showed that the picture flips dramatically with scale: when pre-trained on JFT-300M (Google's internal dataset of 300 million images), ViT-Huge outperformed every CNN on ImageNet, CIFAR-100, and other benchmarks. The takeaway was clear: Transformers for vision work, but they are data-hungry.

This raised a practical question: most researchers and companies do not have 300 million labeled images. Can Vision Transformers work without massive private datasets?

The Ecosystem: Vision Transformer Variants

The original ViT paper sparked an explosion of follow-up work addressing its limitations. Here are the most significant models and what they contribute.

DeiT — Data-Efficient Image Transformers (Facebook, 2021)

DeiT proved that ViT can be trained effectively on ImageNet alone (1.3M images) — no JFT needed. The key innovations:

Strong data augmentation (RandAugment, Mixup, CutMix, random erasing) to compensate for the lack of inductive bias
Regularization techniques (stochastic depth, repeated augmentation)
Knowledge distillation from a CNN teacher (RegNet): a special distillation token learns to mimic the CNN's predictions, effectively transferring the CNN's inductive bias to the Transformer

DeiT-Base matched ViT-Base performance while training only on ImageNet with 4 GPUs — a massive reduction in compute requirements.

Swin Transformer — Shifted Windows (Microsoft, 2021)

Swin Transformer addressed ViT's two biggest architectural issues: quadratic attention cost and single-scale representation.

Hierarchical feature maps: Like a CNN, Swin produces feature maps at multiple resolutions (1/4, 1/8, 1/16, 1/32 of input size) by merging patches between stages. This makes it a drop-in backbone replacement for CNNs in detection and segmentation frameworks like FPN and UPerNet.
Window-based attention: Instead of global self-attention over all patches (O(N^2) cost), Swin computes attention within local windows of fixed size (e.g., 7x7 patches). This reduces complexity to O(N).
Shifted windows: In alternating layers, the window partition is shifted by half the window size, allowing cross-window information flow without the cost of global attention.

Swin became the de facto backbone for dense prediction tasks (object detection, semantic segmentation, instance segmentation) and won the "best paper" at ICCV 2021.

BEiT — BERT Pre-Training for Images (Microsoft, 2021)

BEiT brought BERT-style masked pre-training to vision. During pre-training, random image patches are masked, and the model must predict the visual tokens of the masked patches (using a discrete visual codebook from a tokenizer called dVAE). This self-supervised objective lets ViT learn powerful representations without any labels.

BEiT showed that self-supervised pre-training dramatically improves ViT's performance when fine-tuned on smaller labeled datasets, partially solving the data hunger problem.

CvT and CoAtNet — Hybrid CNN + Transformer

These models combine CNN and Transformer strengths:

CvT (Convolutional Vision Transformer): Replaces the linear patch embedding with convolutional token embeddings and uses depthwise convolutions inside the attention projection. This injects locality bias into the Transformer while keeping the global attention mechanism.
CoAtNet (Google, 2021): Stacks depthwise convolution layers (for local patterns in early stages) with Transformer layers (for global attention in later stages). By systematically combining convolutions and attention, CoAtNet achieves state-of-the-art ImageNet accuracy (90.88% top-1) with strong efficiency.

The hybrid approach is pragmatic: use convolutions where locality matters most (early layers processing raw pixels) and attention where global reasoning matters (later layers composing high-level features).

DINO and DINOv2 — Self-Supervised Learning (Meta)

DINO (Self-DIstillation with NO Labels, 2021) showed that ViT trained with self-supervised distillation learns remarkably structured features. The attention maps of self-supervised ViTs spontaneously learn to segment objects — without ever seeing a segmentation label. The model learns by having a student network match the output of a momentum-updated teacher network on different augmented views of the same image.

DINOv2 (2023) scaled this approach with curated data, larger models, and improved training recipes. DINOv2 features are so general that they work as frozen feature extractors for depth estimation, segmentation, classification, and retrieval — often matching or beating supervised models without any fine-tuning.

EVA and InternImage — Pushing Scale

EVA (Exploring the Limits of Masked Visual Representation Learning, 2022): Combined masked image modeling with CLIP-style vision-language alignment at billion-parameter scale. EVA-02 demonstrated that scaling ViT with the right pre-training recipe achieves new state-of-the-art results across many benchmarks.
InternImage (2023): Took a different path — a large-scale model based on deformable convolutions (not Transformers) that matched or exceeded ViT performance, proving that the CNN vs. Transformer debate is not settled. InternImage uses dynamic sparse attention patterns through deformable convolutions, getting some of the benefits of attention without the architecture.

SAM — Segment Anything (Meta, 2023)

SAM is a foundation model for image segmentation. Its image encoder is a ViT-Huge pre-trained with MAE (Masked Autoencoder). Given an image and a prompt (point, box, or text), SAM produces high-quality segmentation masks for any object — including objects it has never seen before.

SAM demonstrated that ViT backbones, trained at scale with the right objectives, can power general-purpose visual understanding that transfers to virtually any segmentation task. SAM 2 extended this to video.

Comparison: CNN vs. ViT vs. Hybrid

Property	CNN (e.g., ResNet, ConvNeXt)	ViT (Pure Transformer)	Hybrid (e.g., CoAtNet, CvT)
Inductive bias	Strong (locality, translation equivariance)	Minimal (learns from data)	Moderate (conv early, attention late)
Data efficiency	Good with small datasets	Poor without large-scale pre-training	Good — best of both worlds
Scalability	Moderate — diminishing returns at scale	Excellent — performance scales with data and compute	Excellent
Global context	Limited — requires deep stacking	Full — every patch sees every patch	Progressive — local to global
Compute cost	Efficient (linear in image size)	Expensive (quadratic in patch count)	Moderate
Dense prediction	Natural multi-scale features	Single-scale (ViT) — needs adaptation	Multi-scale (Swin, CvT)
Transfer learning	Strong	Exceptional at scale	Strong

Comparison of Major Vision Transformer Models

Model	Year	Key Innovation	Pre-training	Best For
ViT	2020	Pure Transformer for vision	Supervised (JFT-300M)	Classification at scale
DeiT	2021	Data-efficient training + distillation	Supervised (ImageNet-1K)	Classification without massive data
Swin	2021	Hierarchical + shifted windows	Supervised (ImageNet)	Detection, segmentation
BEiT	2021	Masked image modeling	Self-supervised	Low-data fine-tuning
CoAtNet	2021	Conv + Attention hybrid staging	Supervised (JFT)	Top accuracy on ImageNet
DINO/DINOv2	2021/2023	Self-supervised distillation	Self-supervised	General-purpose features
EVA	2022	Scaled masked modeling + CLIP alignment	Self-supervised + CLIP	Vision-language tasks
InternImage	2023	Large-scale deformable convolutions	Supervised	Dense prediction at scale
SAM	2023	Promptable segmentation foundation model	MAE + SA-1B dataset	Zero-shot segmentation

When to Use CNN vs. ViT in Practice

Choosing between a CNN and a Vision Transformer is not about which is "better" — it depends on your constraints.

Use a CNN (ResNet, EfficientNet, ConvNeXt) when:

You have a small to medium dataset (under 100K images) and no pre-trained ViT is available for your domain
You need real-time inference on edge devices or mobile — CNNs are still more efficient at small model sizes
Your task is straightforward classification or detection with well-established CNN pipelines
You want a battle-tested, well-understood architecture with extensive tooling

Use a ViT (or Swin/DeiT) when:

You can leverage a strong pre-trained checkpoint (ImageNet-21K, CLIP, DINOv2, etc.)
Your task benefits from global context (scene understanding, medical imaging with long-range dependencies, satellite imagery)
You are working at scale — more data and compute reliably improve ViT performance
You need a backbone that connects to modern multimodal systems (CLIP, LLaVA, GPT-4V)

Use a hybrid (CoAtNet, CvT, or ConvNeXt + attention) when:

You want the best accuracy-efficiency tradeoff
You need multi-scale features for dense prediction (detection, segmentation) without Swin's complexity
You are building a production system where you need both the efficiency of convolutions and the expressiveness of attention

A practical note: in 2025-2026, the default starting point for most vision tasks is a pre-trained ViT or Swin backbone, fine-tuned on your data. The pre-training handles the data hunger problem. If you are training from scratch on a small dataset with no relevant pre-trained model, CNNs remain the safer choice.

The Bigger Picture

Vision Transformers did more than improve accuracy numbers. They unified the architecture across modalities. The same Transformer that processes text can now process images — and critically, the same architecture can process both at the same time.

This unification is what enables the next wave: models that see and read simultaneously. CLIP learns to align images and text in a shared embedding space. Flamingo, LLaVA, and GPT-4V combine a ViT image encoder with a language model decoder to answer questions about images, describe scenes, and reason visually.

What is Next: Part 3 — Vision Language Models

In Part 3 of this series, we will explore Vision Language Models (VLMs) — the architectures that combine Vision Transformers with Large Language Models. We will cover how models like CLIP, LLaVA, and GPT-4V bridge the gap between seeing and understanding, enabling AI systems that can describe images, answer visual questions, and reason about the visual world in natural language.

The Transformer learned to read in 2017. It learned to see in 2020. Now it is learning to do both at once.

Read Part 3: Vision-Language Models — When AI Learns to See and Talk

Semantic vs Keyword vs Hybrid Search: What Every RAG Demo Skips

Vahid Aghajani — Sat, 04 Jul 2026 18:02:34 +0000

Prefer audio? Spotify episode · Telegram

Every RAG tutorial starts the same way: chunk your docs, embed them, throw them in a vector store, query with cosine similarity. Done.

It's a great demo. It's also not how serious search systems work.

The moment a user types error code E_1042 or Llama-3.1-70B or a product SKU, pure semantic search starts quietly failing — because an embedding of E_1042 is a vector of noise. Meanwhile, keyword search has the opposite problem: type "how do I cancel my subscription" and it misses the document titled "ending your membership".

So real systems use both. This post is about what each one is actually doing, why hybrid beats either alone, and how to build it in Postgres in ~40 lines.

Keyword Search: BM25 in One Paragraph

Keyword search finds documents containing your query terms. The question is how to rank them.

BM25 (the de-facto ranking function since the 90s) is basically three ideas stacked:

Term frequency — the more a word appears in a doc, the more relevant, but with diminishing returns (the 20th occurrence of "database" doesn't help much).
Inverse document frequency — rare words count more. "the" is useless, "pgvector" is a strong signal.
Length normalization — longer documents would otherwise win by accident, so scores are normalized by length.

That's it. No machine learning, no GPU, no training. Under the hood it's an inverted index (word → list of documents containing it), which makes it blindingly fast even at billions of documents.

What it's good at: exact matches, IDs, rare tokens, acronyms, product names, filenames, version numbers, anything literal.

What it fails at: synonyms ("car" ≠ "automobile"), paraphrase, conceptual queries, cross-language.

Semantic Search: Embeddings in One Paragraph

An embedding model turns text into a vector — say, 768 floats — such that texts with similar meaning end up close in that vector space. "How do I cancel?" and "ending your subscription" become near-neighbors.

To search, you embed the query and find the nearest vectors. Naively that's O(N) — check distance to every doc — which doesn't scale past a few million. So we use approximate nearest neighbor (ANN) indexes:

HNSW (Hierarchical Navigable Small World) — a graph where each node links to its near neighbors at multiple resolutions. Fast, accurate, memory-hungry.
IVF (Inverted File) — cluster all vectors first, only search the nearest clusters at query time. Lighter, slightly less accurate.

What it's good at: paraphrase, synonyms, conceptual similarity, cross-language, fuzzy intent.

What it fails at: rare tokens (the embedding has never seen them), acronyms, identifiers, numbers, exact-match requirements. Also expensive — every query means an embedding model call plus a vector search.

Why Hybrid Beats Both

Run the same query through both systems and you get two ranked lists. How do you merge them?

The surprisingly simple answer that keeps winning benchmarks: Reciprocal Rank Fusion (RRF). For each document d that appears in any ranker r, sum 1 / (k + rank_r(d)) across all the rankers:

           ┌── keyword ranker (BM25) ──┐
query ────┤                            ├──► RRF merge ──► final ranking
           └── semantic ranker (vec) ──┘

           score(d) =  Σ   1 / (k + rank_r(d))
                      r∈R

k is a smoothing constant — typically 60. It dampens the gap between rank 1 and rank 2 so the top result doesn't utterly dominate the fused score. Raise k and lower ranks contribute more; lower k and the top ranks dominate. The default works for almost everyone.

def rrf(ranked_lists, k=60):
    scores = {}
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

No calibration between score scales, no tuning, no training. A document that ranks well in either list bubbles up; a document that ranks in both lists bubbles up strongly. That's why it works — the two systems have different failure modes, and RRF exploits that.

Build It In Postgres

Postgres does both keyword (via tsvector + GIN index) and semantic (via pgvector + HNSW index) natively. One table, two indexes, one hybrid query.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE docs (
  id SERIAL PRIMARY KEY,
  content TEXT,
  content_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
  embedding VECTOR(768)
);

CREATE INDEX docs_tsv_idx ON docs USING GIN (content_tsv);
CREATE INDEX docs_vec_idx ON docs USING HNSW (embedding vector_cosine_ops);

Now a hybrid query with RRF, all in one SQL statement:

WITH kw AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(content_tsv, query) DESC) AS rnk
  FROM docs, plainto_tsquery('english', 'how to cancel subscription') query
  WHERE content_tsv @@ query
  LIMIT 50
),
vec AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1::vector) AS rnk
  FROM docs
  ORDER BY embedding <=> $1::vector
  LIMIT 50
)
SELECT d.id, d.content,
       COALESCE(1.0/(60 + kw.rnk), 0) + COALESCE(1.0/(60 + vec.rnk), 0) AS score
FROM docs d
LEFT JOIN kw  ON kw.id  = d.id
LEFT JOIN vec ON vec.id = d.id
WHERE kw.id IS NOT NULL OR vec.id IS NOT NULL
ORDER BY score DESC
LIMIT 10;

$1 is the query embedding — a VECTOR(768) your application layer (Python, Node, Go) has already computed by calling the embedding model. Postgres doesn't embed on its own; it just stores and searches the result. So your app does: (1) embed the query, (2) send the vector as a parameter to this SQL. That's your hybrid retriever — one database, one query, no extra infrastructure.

What About Elasticsearch, Qdrant, Weaviate?

Postgres is a fine choice for most teams. The dedicated tools matter when scale, flexibility, or specific features push you past it.

Tool	Best at	Weak at
Postgres (pgvector + tsvector)	Teams already on Postgres, moderate scale (< ~10M vectors), transactional data next to embeddings	Billion-scale vector search, complex BM25 tuning, multi-tenant reranking
Elasticsearch / OpenSearch	Mature BM25, aggregations, faceting, geo, log search; native hybrid via RRF since 8.x	Embeddings are a second-class citizen; heavier ops
Qdrant	Pure vector workloads, clean Rust implementation, fast filters on payload, simple to run	Keyword search is basic (no BM25) — you'll pair it with something else
Weaviate	Built-in hybrid (BM25 + vectors) as a first-class feature, strong schema + modules for embedding pipelines	Opinionated architecture; lock-in on their query language
Vespa	The only one that plays seriously in all three axes — scale, keyword quality, vector quality — at FAANG scale	Steep learning curve; overkill for most teams

Rule of thumb: start with Postgres. Move to Qdrant or Weaviate when vector count crosses ~10M or you need low-latency ANN at high QPS. Use Elasticsearch/OpenSearch when keyword quality and faceting are the main product. Reach for Vespa when all three dimensions matter and nothing else scales.

Decision Matrix

Your query looks like…	Reach for	Why
Product SKUs, error codes, version numbers, filenames	Keyword	Embeddings have never seen these exact tokens; BM25 treats them as high-IDF signals
Conceptual, paraphrased, cross-language	Semantic	Different words, same meaning — keyword can't see the connection, vectors can
Real user queries in a product — mixed and messy	Hybrid	Users type both kinds in the same session; hybrid has no downside when one side is weak
"Most similar to this other document"	Semantic	Pure vector problem — rank docs by distance in embedding space
Log search, structured fields, faceting	Keyword / Elasticsearch	Exact matches + aggregations matter more than meaning

If you're not sure — and in production you're usually not — just use hybrid. RRF has no downside: if one side is useless for a particular query, it simply contributes near-zero and the other side wins the ranking.

Gotchas Nobody Tells You

Chunking matters more than the embedding model. A great model on badly chunked docs loses to a mediocre model on well-chunked docs. Start with ~300-token chunks with overlap.
Multilingual content breaks keyword search. to_tsvector('english', ...) silently butchers non-English text. Either detect language and use the right dictionary, or lean more on semantic.
Stop words cut both ways. Removing "the" helps keyword search. Removing "not" changes the meaning completely for semantic.
Rerankers beat tuning. After your hybrid retrieves 50 candidates, a cross-encoder reranker (e.g., bge-reranker) re-scores them pairwise against the query. It's the single biggest quality lift you can add in an afternoon.
Latency budget is a product decision. Pure keyword: ~5ms. Pure semantic with HNSW: ~20ms. Hybrid with a reranker: ~200ms. That last one can't live in a typeahead; it can live in RAG.

Closing

The "semantic vs keyword" framing is a false choice. They're complementary — each covers where the other is blind. Hybrid retrieval with RRF is almost free to add, needs no training, and works across every vector store that also supports BM25.

If you take one thing away: before you pick a new vector database, check whether Postgres already does what you need. For most teams, it does.

Next time someone shows you a RAG demo with just embeddings, you'll know what's missing.

Transformers — The Architecture That Changed AI (Part 1 of 3)

Vahid Aghajani — Sat, 04 Jul 2026 17:53:35 +0000

In June 2017, a team at Google published a paper with a deceptively simple title: "Attention Is All You Need." Eight authors, fourteen pages, and one architecture that would go on to power GPT-4, Claude, Gemini, DALL-E, Stable Diffusion, AlphaFold, and virtually every breakthrough in AI since.

The Transformer didn't just improve on existing models. It replaced the entire paradigm. Recurrent neural networks, LSTMs, sequence-to-sequence models with attention — all of them became legacy architectures almost overnight.

This is Part 1 of a 3-part series. Here we cover the Transformer itself — the core architecture, the intuition behind each component, and why it scales so remarkably well. Part 2 will cover Vision Transformers (how this architecture learned to see), and Part 3 will cover Vision-Language Models (when AI learned to see and talk).

The Problem: Why RNNs Hit a Wall

To understand why Transformers matter, you need to understand what came before.

Recurrent Neural Networks (RNNs) process sequences one token at a time, left to right. Each step takes the previous hidden state and the current input, produces a new hidden state, and passes it forward. This is elegant in theory: the hidden state is a compressed summary of everything the model has seen so far.

In practice, it has three devastating problems:

The bottleneck problem. By the time an RNN reaches the 500th word in a paragraph, the information from the 1st word has been compressed through 499 sequential transformations. Important early context gets diluted or lost entirely. Imagine trying to remember the first sentence of a book after reading 500 pages, where each page partially overwrites your memory of the previous one.
No parallelization. Because each step depends on the previous step's output, you cannot process tokens in parallel. Training is inherently sequential. On modern GPUs with thousands of cores designed for parallel computation, this is a catastrophic bottleneck.
Vanishing and exploding gradients. During backpropagation through time, gradients must flow backwards through every sequential step. Over long sequences, they either shrink to near-zero (vanishing) or blow up to infinity (exploding), making it extremely hard to learn long-range dependencies.

LSTMs and GRUs partially addressed problem 3 by adding gating mechanisms — explicit "remember" and "forget" controls. They helped, but they didn't solve the fundamental sequential nature of the computation (problem 2) or the information bottleneck (problem 1).

The sequence-to-sequence model with attention (Bahdanau et al., 2014) made a crucial step forward. Instead of forcing the decoder to work from a single compressed context vector, it allowed the decoder to "look back" at all encoder hidden states and attend to the most relevant ones at each decoding step. This was the birth of attention as a mechanism.

But even seq2seq with attention still relied on an RNN backbone. The encoder still processed tokens sequentially. The Transformer's radical insight was: what if we throw away the recurrence entirely and use only attention?

The Core Idea: Attention as the Only Mechanism

The Transformer computes relationships between all tokens in a sequence simultaneously. Instead of passing information through a chain of hidden states, every token can directly attend to every other token in a single operation.

Think of it this way. An RNN is like a game of telephone — each person whispers the message to the next, and by the end of the line, the message is garbled. A Transformer is like a round table where everyone can hear everyone else directly. No information loss from sequential passing. No bottleneck.

This has a profound consequence: the entire sequence can be processed in parallel. During training, all tokens are known in advance, so every attention computation can happen simultaneously across the GPU. This is why Transformers train orders of magnitude faster than RNNs on the same hardware.

The Architecture: Component by Component

The original Transformer uses an encoder-decoder structure, designed for sequence-to-sequence tasks like machine translation (English to German, for example). Let's walk through each component.

Encoder-Decoder Overview

The encoder takes the input sequence (e.g., an English sentence) and produces a rich representation of it — a set of vectors that capture meaning and context. The decoder takes that representation and generates the output sequence (e.g., the German translation) one token at a time.

The encoder is a stack of 6 identical layers. Each layer has two sub-components: a multi-head self-attention mechanism and a position-wise feed-forward network. The decoder is also 6 layers, but each layer has three sub-components: masked multi-head self-attention, multi-head cross-attention (attending to the encoder output), and a feed-forward network.

Every sub-component is wrapped with a residual connection and layer normalization. We'll cover each piece.

Input Embeddings and Positional Encoding

Before anything else, input tokens are converted to dense vectors via a learned embedding table. If the model dimension is d_model = 512, each token becomes a 512-dimensional vector.

But here's a problem the RNN never had: since the Transformer processes all tokens simultaneously, it has no inherent notion of order. The sentence "the cat sat on the mat" and "mat the on sat cat the" would produce identical attention patterns without some way to encode position.

The solution is positional encoding — adding a position-dependent signal to each token embedding. The original paper uses sinusoidal functions:

PE(pos, 2i)     = sin(pos / 10000^(2i/d_model))
PE(pos, 2i + 1) = cos(pos / 10000^(2i/d_model))

Each position gets a unique pattern of sine and cosine values across the embedding dimensions. The key properties: (1) each position has a unique encoding, (2) the encoding is deterministic (no learned parameters), and (3) the model can generalize to sequence lengths longer than those seen during training because the functions are continuous.

The analogy: think of positional encoding as a unique "address" stamped onto each word. The model learns to read these addresses and factor position into its attention decisions.

Modern Transformers often use learned positional embeddings (just another embedding table indexed by position) or Rotary Position Embeddings (RoPE), which encode relative position directly into the attention computation. But the core insight remains the same: you must inject position information explicitly.

Scaled Dot-Product Attention: The Q/K/V Framework

This is the heart of the Transformer. Every attention mechanism in the architecture is built on this single operation.

For each token, we compute three vectors from its embedding:

Query (Q): "What am I looking for?"
Key (K): "What do I contain?"
Value (V): "What information do I provide if you attend to me?"

These are produced by multiplying the input by three learned weight matrices: W_Q, W_K, and W_V.

The attention computation works in three steps:

Step 1: Compute compatibility scores. Multiply each query by all keys (dot product). This produces a score matrix: how relevant is each key to each query. High dot product = the query and key are aligned = this token is relevant to that token.

Step 2: Scale and normalize. Divide scores by the square root of the key dimension (sqrt(d_k)). This scaling prevents the dot products from growing too large in magnitude, which would push the softmax into regions with tiny gradients. Then apply softmax row-wise to get attention weights that sum to 1.

Step 3: Weighted sum of values. Multiply the attention weights by the value vectors. Each token's output is a weighted combination of all value vectors, with weights determined by how relevant each key was to that token's query.

In matrix form:

Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V

The analogy: imagine you're at a library (the sequence). Your query is the question you're researching. Each book has a title (key) and content (value). You scan all the titles, figure out which books are most relevant to your question, and then read those books more carefully — weighting your reading time based on relevance.

Multi-Head Attention: Attending to Different Things Simultaneously

A single attention head learns one kind of relationship. But language has many simultaneous relationships: syntactic, semantic, coreference, positional, topical.

Multi-head attention runs multiple attention operations in parallel, each with its own learned Q/K/V projections. The original Transformer uses 8 heads with d_k = d_v = 64 each (total: 8 * 64 = 512 = d_model).

Each head can specialize. Research has shown that different heads learn to capture different linguistic phenomena:

One head might track subject-verb agreement
Another might focus on adjacent word relationships
Another might capture long-range coreference ("she" referring to "Dr. Smith" from three sentences ago)

The outputs of all heads are concatenated and linearly projected back to d_model:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) * W_O
where head_i = Attention(Q * W_Qi, K * W_Ki, V * W_Vi)

This is one of the Transformer's most powerful design choices — it gets multiple "perspectives" on the same data for the cost of one full-dimensional attention computation.

Three Types of Attention in the Transformer

The encoder-decoder Transformer uses attention in three distinct ways:

Encoder self-attention. Each token in the input attends to all other tokens in the input. This builds a contextual representation where each word's embedding is enriched by the full context of the sentence.
Masked decoder self-attention. Each token in the output sequence attends to all previous output tokens (but not future ones). The masking prevents the model from "cheating" by looking ahead during training. Future positions are set to negative infinity before the softmax, zeroing out their attention weights.
Encoder-decoder cross-attention. Each decoder token attends to all encoder outputs. The queries come from the decoder; the keys and values come from the encoder. This is how the decoder "reads" the input sentence to produce the translation.

Feed-Forward Network

After each attention sub-layer, every token passes independently through the same two-layer feed-forward network:

FFN(x) = ReLU(x * W_1 + b_1) * W_2 + b_2

The inner dimension is expanded (typically 4x, from 512 to 2048 in the original), then projected back down. This gives the model per-token nonlinear processing capacity — attention handles inter-token relationships, while the FFN handles intra-token transformation.

Recent research suggests the FFN layers serve as the model's "memory," storing factual knowledge, while the attention layers handle relational reasoning.

Residual Connections and Layer Normalization

Every sub-layer (attention and FFN) is wrapped with:

output = LayerNorm(x + Sublayer(x))

The residual connection (x + Sublayer(x)) allows gradients to flow directly through the network without degradation, enabling very deep stacks (6, 12, 24, 96+ layers). Without residuals, training deep Transformers would be nearly impossible.

Layer normalization stabilizes the hidden state magnitudes, preventing the distribution of activations from drifting as the signal passes through many layers. It normalizes across the feature dimension for each token independently.

These aren't glamorous components, but they're essential. The Transformer's depth — and therefore its capacity — depends on them.

Why This Design Scales So Well

The Transformer has a property that no previous architecture achieved to the same degree: predictable, smooth scaling.

In 2020, Kaplan et al. (OpenAI) published the scaling laws paper, showing that Transformer performance improves as a smooth power law with respect to three factors:

Model size (number of parameters)
Dataset size (number of training tokens)
Compute budget (FLOPs spent on training)

Double the parameters, and you get a predictable improvement. Double the data, same thing. This is remarkably different from previous architectures where scaling often hit diminishing returns or instabilities.

Why do Transformers scale so well?

Full parallelism means more compute directly translates to faster training and larger batch sizes. No sequential bottleneck limits how much hardware you can throw at the problem.
Attention across all positions means the model's capacity to capture relationships grows with sequence length and model width, without architectural changes.
Depth composability — stacking more identical layers adds representational power smoothly, thanks to residual connections and layer normalization.
No information bottleneck — unlike RNNs, there's no fixed-size hidden state that all information must pass through.

The Chinchilla paper (Hoffmann et al., 2022) later refined these laws, showing that models should be trained on roughly 20 tokens per parameter for optimal compute efficiency. This led to a shift from the "bigger model" paradigm to the "more data" paradigm.

Transformer Variants: A Family of Architectures

The original Transformer is encoder-decoder. But researchers quickly discovered that using parts of the architecture for specific tasks yielded remarkable results. Three major paradigms emerged.

Encoder-Only: BERT and Its Descendants

BERT (Bidirectional Encoder Representations from Transformers, 2018) uses only the encoder stack. During pre-training, it masks random tokens in the input and trains the model to predict them — this is Masked Language Modeling (MLM). Because there's no autoregressive generation, every token can attend to every other token bidirectionally.

BERT excels at understanding tasks: classification, named entity recognition, question answering, semantic similarity. It produces rich contextual embeddings where the same word gets different representations depending on context ("bank" in "river bank" vs. "bank account").

Key descendants:

RoBERTa — Same architecture, better training: more data, longer training, no next-sentence prediction objective. Showed BERT was significantly undertrained.
ALBERT — Parameter-efficient BERT with cross-layer parameter sharing and factorized embeddings. Smaller model, competitive performance.
DeBERTa — Disentangles content and position into separate attention streams, then combines them late. Achieved human-level performance on the SuperGLUE benchmark.
XLNet — Uses a permutation-based training objective to capture bidirectional context while maintaining autoregressive formulation. Avoids the pretrain-finetune discrepancy of BERT's masking.

Decoder-Only: GPT and the Autoregressive Revolution

GPT (Generative Pre-trained Transformer, 2018) uses only the decoder stack. It's trained to predict the next token given all previous tokens — pure autoregressive language modeling. The masked self-attention ensures each position can only attend to earlier positions.

This paradigm turned out to be the one that scales the furthest. The progression:

GPT-1 (2018): 117M parameters. Showed that unsupervised pre-training + supervised fine-tuning works.
GPT-2 (2019): 1.5B parameters. Showed that scale enables zero-shot task performance. "Too dangerous to release" (it wasn't, but the PR worked).
GPT-3 (2020): 175B parameters. Showed that in-context learning emerges at scale — the model can perform tasks from a few examples in the prompt, no fine-tuning needed.
GPT-4 (2023): Architecture undisclosed, rumored mixture-of-experts. Multimodal (text + vision). State-of-the-art across dozens of benchmarks.

Other major decoder-only models:

PaLM (Google, 540B) — Showed breakthrough performance on reasoning tasks with chain-of-thought prompting.
LLaMA (Meta) — Open-weight models proving that smaller, well-trained models can match much larger ones. LLaMA 2 (7B-70B) and LLaMA 3 catalyzed the open-source AI ecosystem.
Mistral — Efficient open models using Grouped-Query Attention and Sliding Window Attention. Mistral 7B outperformed LLaMA 2 13B.
Claude (Anthropic) — Constitutional AI approach with RLHF. Strong reasoning and instruction-following with emphasis on safety and helpfulness.

Encoder-Decoder: T5 and Unified Frameworks

T5 (Text-to-Text Transfer Transformer, 2019) keeps the full encoder-decoder architecture but reframes every NLP task as a text-to-text problem. Classification? Input: "classify: this movie was great", output: "positive". Translation? Input: "translate English to German: Hello", output: "Hallo".

This unified framing is elegant — one architecture, one training procedure, one format for everything. T5 also systematically studied every architectural choice (model size, pre-training objective, dataset), making it one of the most thorough papers in the field.

Model Comparison

Model	Type	Year	Parameters	Key Innovation
Original Transformer	Encoder-Decoder	2017	65M	Self-attention replacing recurrence entirely
BERT	Encoder-only	2018	110M / 340M	Bidirectional pre-training with masked language modeling
GPT-1	Decoder-only	2018	117M	Unsupervised pre-training + fine-tuning paradigm
GPT-2	Decoder-only	2019	1.5B	Zero-shot task transfer via scale
T5	Encoder-Decoder	2019	220M - 11B	Unified text-to-text framing for all NLP tasks
XLNet	Autoregressive	2019	340M	Permutation-based training for bidirectional context
RoBERTa	Encoder-only	2019	355M	Optimized BERT training procedure
ALBERT	Encoder-only	2019	12M - 235M	Cross-layer parameter sharing, factorized embeddings
DeBERTa	Encoder-only	2020	134M - 1.5B	Disentangled attention for content and position
GPT-3	Decoder-only	2020	175B	In-context learning, few-shot capabilities
PaLM	Decoder-only	2022	540B	Pathways system, breakthrough reasoning
LLaMA 2	Decoder-only	2023	7B - 70B	Open-weight, efficient training, GQA
Mistral 7B	Decoder-only	2023	7B	Sliding window attention, grouped-query attention
GPT-4	Decoder-only (MoE?)	2023	Undisclosed	Multimodal, state-of-the-art reasoning
Claude 3.5	Decoder-only	2024	Undisclosed	Constitutional AI, strong reasoning + safety
LLaMA 3	Decoder-only	2024	8B - 405B	15T tokens training data, extended context

Why Decoder-Only Won (For Now)

A natural question: if the original Transformer is encoder-decoder, why are the largest and most capable models decoder-only?

Several factors converged:

Simplicity. One stack is easier to scale than two. Fewer architectural decisions, fewer hyperparameters, simpler training pipelines.
Unification of understanding and generation. Encoder-only models are great at understanding but cannot generate. Decoder-only models can do both — they understand context through the process of predicting what comes next.
Emergent capabilities. As decoder-only models scaled, unexpected abilities appeared: chain-of-thought reasoning, in-context learning, instruction following. These emergent behaviors were less pronounced in encoder-only or encoder-decoder models at similar scales.
Training efficiency. Next-token prediction is a dense supervision signal — every token in the training data provides a training signal. Masked language modeling only trains on the ~15% of tokens that are masked.

That said, encoder-decoder architectures aren't dead. They excel at tasks where you have a clear input-output mapping (translation, summarization), and models like T5 and its successors remain competitive in many benchmarks.

The Lasting Impact

The Transformer didn't just change NLP. It became the universal architecture for deep learning:

Computer Vision: Vision Transformers (ViT) treat image patches as tokens and achieve state-of-the-art classification, detection, and segmentation.
Audio: Whisper uses Transformers for speech recognition. MusicLM generates music.
Protein Structure: AlphaFold 2 uses a modified attention mechanism to predict 3D protein structures, solving a 50-year-old biology problem.
Robotics: RT-2 uses Transformers to translate language instructions into robot actions.
Code: Codex, CodeLlama, and StarCoder generate and understand programming languages.

The architecture is so general that the main research question shifted from "what architecture should we use?" to "how much data and compute should we invest?"

What Comes Next: From Text to Vision

The Transformer started with language, but it didn't stay there. In Part 2 of this series, we'll explore Vision Transformers (ViT) — how researchers adapted the attention mechanism to work with images, why it works so well, and how it dethroned CNNs as the dominant architecture in computer vision.

From pixels to patches to attention maps — the next chapter of the Transformer story is just as transformative.

Next up: Part 2 — Vision Transformers: How Transformers Learned to See

How LLM Function Calling Actually Works — From Tokens to Tool Orchestration

Vahid Aghajani — Sat, 04 Jul 2026 17:42:25 +0000

When you ask an LLM "Compare the weather in Tokyo and Berlin," what actually happens? The model can't browse the internet — but it can decide to call a weather API. Twice. In the same turn.

This article covers how function calling works, how the LLM returns structured data despite generating tokens one by one, and what happens when the model needs to orchestrate multiple tool calls to answer a single question.

Part 1: What Is Function Calling?

"Function calling" is one of several ways to get output from an LLM API. Here are the three main methods:

Method A: Plain Text Completion (Simplest)

response = client.chat.completions.create(
    model="gemini-2.5-flash-lite",
    messages=[{"role": "user", "content": "Classify this email: ..."}],
)
text = response.choices[0].message.content
# "This is a job_search email because..."

You get back free-form text. Then you'd have to parse it yourself — maybe with regex, or hoping the LLM follows instructions like "respond with JSON only". This is fragile because the LLM might say:

...and now your regex breaks.

Method B: JSON Mode

response = client.chat.completions.create(
    model="...",
    messages=[...],
    response_format={"type": "json_object"},  # force JSON output
)
data = json.loads(response.choices[0].message.content)

The LLM is forced to output valid JSON, but you still have no guarantee of the schema — it might return {"cat": "job"} instead of {"category": "job_search"}.

Method C: Function Calling (What We Use)

response = client.chat.completions.create(
    model="...",
    messages=[...],
    tools=[{
        "type": "function",
        "function": {
            "name": "classify_email",
            "parameters": {
                "properties": {
                    "category": {
                        "type": "string",
                        "enum": ["job_search", "spam", "newsletter", ...]
                    },
                    "confidence": {"type": "number"},
                    "summary": {"type": "string"},
                    "reasoning": {"type": "string"}
                },
                "required": ["category", "confidence"]
            }
        }
    }],
    tool_choice={
        "type": "function",
        "function": {"name": "classify_email"}
    },
)

You define the exact schema you want — field names, types, enums, required fields. The API forces the LLM to fill in that schema. The result comes back as:

{
    "category": "job_search",
    "confidence": 0.95,
    "summary": "LinkedIn job alert for Senior Python Developer",
    "reasoning": "Sender is LinkedIn, contains job listings..."
}

This is the most reliable way to get structured output. The LLM cannot deviate from the schema.

Comparison

Method	Output	Schema Guarantee	Reliability
Plain text	Free-form string	None	Low — requires manual parsing
JSON mode	Valid JSON	No schema enforcement	Medium — valid JSON but unpredictable keys
Function calling	Schema-constrained JSON	Full schema + types + enums	High — enforced at token generation level

Part 2: How Does an LLM Return a Dict If It Generates Tokens?

The LLM still generates tokens one by one. It doesn't "natively" return a Python dictionary. Here's what actually happens under the hood.

What the LLM Actually Generates

Tokens:  {  "  category  "  :  "  job  _  search  "  ,  "  confidence  "  :  0  .  95  }
         ↑    ↑       ↑   ↑  ↑    ↑   ↑   ↑      ↑   ↑       ↑       ↑  ↑   ↑   ↑
       token token  token ... (still just text tokens)

The LLM is still producing text — it's just text that happens to be valid JSON. The API layer does the magic.

The Constrained Decoding Process

When you use function calling, the API applies constrained decoding (also called "guided generation"):

The API receives your tools schema definition
It constrains the LLM's token generation — at each step, only tokens that would produce valid JSON matching your schema are allowed
For "enum": ["job_search", "spam", ...], the LLM can literally only pick from those values
For "type": "number", only numeric tokens are valid at that position

This is fundamentally different from just asking "please reply in JSON" in the prompt. The constraints are enforced at the token-generation level, not via prompt instructions.

The Full Flow

LLM brain
  │
  ▼ (generates tokens, constrained by schema)
'{"category":"job_search","confidence":0.95,"summary":"..."}'
  │
  ▼ (API parses & validates)
response.choices[0].message.tool_calls[0].function.arguments
  │  (this is still a STRING)
  ▼
json.loads(arguments)
  │
  ▼ (now it's a Python dict)
{"category": "job_search", "confidence": 0.95, "summary": "..."}

In Code

# The API returns tool_calls as part of the response
tool_call = response.choices[0].message.tool_calls[0]

# .arguments is a STRING containing JSON
raw = tool_call.function.arguments
# '{"category":"job_search","confidence":0.95,...}'

# We parse it into a Python dict
data = json.loads(raw)
# {"category": "job_search", "confidence": 0.95, ...}

TL;DR: The LLM is still generating text/tokens. Function calling constrains which tokens it can generate (must match your schema), and the API wraps the result in a structured format. We then json.loads() that string into a Python dict.

Part 3: Multiple Tools — "Compare the Weather in Tokyo and Berlin"

So far we've seen one function called once. But what happens when the user's question requires multiple tool calls?

The Setup: Defining a Weather Tool

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"},
                "units": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "Temperature units"
                }
            },
            "required": ["city"]
        }
    }
}]

We have one tool definition — get_weather. Now watch what happens when the user asks a question that requires it twice.

The Request

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "Compare the weather in Tokyo and Berlin right now"
    }],
    tools=tools,
)

What the LLM Returns: Two Tool Calls

The LLM doesn't return a text answer. Instead, it returns two tool calls in a single response:

message = response.choices[0].message

# message.content is None — no text response
# message.tool_calls has TWO entries:

for tc in message.tool_calls:
    print(f"ID: {tc.id}")
    print(f"Function: {tc.function.name}")
    print(f"Args: {tc.function.arguments}")
    print()

Output:

ID: call_abc123
Function: get_weather
Args: {"city": "Tokyo", "units": "celsius"}

ID: call_def456
Function: get_weather
Args: {"city": "Berlin", "units": "celsius"}

The LLM decided on its own to:

Call the same function twice with different arguments
Pick "celsius" as the unit (reasonable default for these cities)
Return both calls in the same turn (parallel tool calls)

Your Code Executes the Tools

Now you run both calls and feed the results back:

import json

# Execute each tool call
tool_results = []
for tc in message.tool_calls:
    args = json.loads(tc.function.arguments)
    # Call your actual weather API
    weather = get_weather_from_api(args["city"], args.get("units", "celsius"))
    tool_results.append({
        "role": "tool",
        "tool_call_id": tc.id,     # must match the ID from the LLM
        "content": json.dumps(weather)
    })

# Send results back to the LLM
messages = [
    {"role": "user", "content": "Compare the weather in Tokyo and Berlin right now"},
    message,           # the assistant's tool_calls response
    *tool_results,     # both tool results
]

final = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
)

print(final.choices[0].message.content)

The Final Answer

The LLM now has both weather results and generates a natural comparison:

The Complete Flow

User: "Compare weather in Tokyo and Berlin"
  │
  ▼
LLM (Turn 1): I need weather for both cities
  │
  ├─→ tool_call: get_weather(city="Tokyo")    ──→ Your code calls API ──→ {"temp": 22, ...}
  │
  └─→ tool_call: get_weather(city="Berlin")   ──→ Your code calls API ──→ {"temp": 8, ...}
  │
  ▼
LLM (Turn 2): Now I have both results
  │
  └─→ "Tokyo is 22°C, Berlin is 8°C. Tokyo is 14 degrees warmer..."

Parallel vs Sequential Tool Calls

Parallel (what happened above): The LLM returns multiple tool_calls in a single response. Both calls are independent — your code can execute them concurrently:

import asyncio

async def execute_tools_parallel(tool_calls):
    tasks = [execute_single_tool(tc) for tc in tool_calls]
    return await asyncio.gather(*tasks)

Sequential: Sometimes the LLM needs the result of one call before making the next. For example: "What's the weather in the capital of France?"

Turn 1: LLM calls get_capital(country="France")
  → Your code returns "Paris"
Turn 2: LLM calls get_weather(city="Paris")
  → Your code returns weather data
Turn 3: LLM generates final answer

The LLM decides which pattern to use based on whether the calls depend on each other.

Part 4: Different Tools in One Turn

The LLM can also call different tools in the same turn. Suppose you define two tools:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_exchange_rate",
            "description": "Get currency exchange rate",
            "parameters": {
                "type": "object",
                "properties": {
                    "from_currency": {"type": "string"},
                    "to_currency": {"type": "string"}
                },
                "required": ["from_currency", "to_currency"]
            }
        }
    }
]

If the user asks: "I'm traveling from NYC to Tokyo next week. What's the weather like and how much is 1 USD in Yen?"

The LLM returns two different tool calls in one turn:

# tool_calls[0]
{"name": "get_weather", "arguments": '{"city": "Tokyo"}'}

# tool_calls[1]
{"name": "get_exchange_rate", "arguments": '{"from_currency": "USD", "to_currency": "JPY"}'}

Your code routes each call to the right function:

tool_handlers = {
    "get_weather": handle_weather,
    "get_exchange_rate": handle_exchange_rate,
}

for tc in message.tool_calls:
    handler = tool_handlers[tc.function.name]
    args = json.loads(tc.function.arguments)
    result = handler(**args)
    # ... send result back

This is essentially the registry pattern — a dictionary maps function names to handlers. No if/else chains needed.

What Controls This Behavior?

The tool_choice parameter controls whether and how the LLM uses tools:

`tool_choice`	Behavior	Use Case
`"auto"`	LLM decides whether to call tools or respond with text	General-purpose agents
`"required"`	LLM must call at least one tool	When you always need structured output
`{"type": "function", "function": {"name": "..."}}`	LLM must call this specific function	Email classification (always classify)
`"none"`	LLM cannot call any tools	Force a text-only response

For the weather comparison, we use "auto" — the LLM decides on its own that it needs to call get_weather twice.

Key Takeaways

Function calling > JSON mode > plain text for getting structured data from LLMs. Function calling enforces your schema at the token generation level, not just via prompt instructions.
LLMs still generate tokens — they don't natively return dicts. The API layer applies constrained decoding to ensure the token output matches your schema, then you json.loads() the resulting string.
One question can trigger multiple tool calls. The LLM decides whether to call the same tool with different arguments (Tokyo + Berlin) or different tools entirely (weather + exchange rate) — all in a single turn.
Parallel vs sequential is decided by the LLM. Independent calls (two cities) come back in one turn. Dependent calls (get capital → get weather) happen across multiple turns.
Route tool calls with a registry, not if/else. A dictionary mapping function names to handlers keeps your code clean and extensible.

DEV Community: Vahid Aghajani

Why Your LLM Keeps Returning Garbage JSON (And How to Stop It)

Why LLMs Fail at JSON in the First Place

Layer 1: Use the Native Structured-Output API

OpenAI — response_format={"type": "json_schema", ...}

Anthropic — Tool use as a schema enforcer

Gemini — response_mime_type + response_schema

Layer 2: Validate With Pydantic (Even If Layer 1 Worked)

Layer 3: Repair and Retry When Both Layers Above Fail

json_repair for almost-JSON

Retry with the validation error fed back to the model

Make failure structured, too

Putting It All Together

Decision Matrix

And one more axis: model size

Gotchas Nobody Tells You

Closing

Vision Language Models — When AI Learns to See and Talk (Part 3 of 3)

Why Combine Vision and Language?

The Evolution: From Captioning to True Multimodal Understanding

Phase 1: Image Captioning (2015-2019)

Phase 2: Contrastive Pre-training (2021)

Phase 3: Generative Multimodal Models (2022-2023)

Phase 4: Natively Multimodal Systems (2023-present)

Four Architectural Approaches to VLMs

1. Contrastive Learning (The CLIP Approach)

2. Cross-Attention Fusion (The Flamingo Approach)

3. Visual Tokens into LLM (The LLaVA Approach)

4. Natively Multimodal (The Gemini Approach)

Major VLM Models: A Landscape Overview

CLIP (OpenAI, 2021)

BLIP and BLIP-2 (Salesforce, 2022-2023)

Flamingo (DeepMind, 2022)

LLaVA / LLaVA-NeXT (University of Wisconsin, 2023-2024)

GPT-4V / GPT-4o (OpenAI, 2023-2024)

Gemini (Google, 2023-2024)

Claude (Anthropic, 2024-present)

PaliGemma (Google, 2024)

Qwen-VL (Alibaba, 2023-2024)

InternVL (Shanghai AI Lab, 2023-2024)

VLM Comparison

Real-World Applications

Visual Question Answering and Conversational AI

Document Understanding and OCR

Autonomous Driving and Robotics

Medical Imaging

Creative and Design Work

Challenges and Open Problems

Hallucination

Spatial Reasoning

Fine-Grained Understanding

Temporal Reasoning in Video

Safety and Bias

Where Multimodal AI Is Headed

Conclusion

Vision Transformers — How Transformers Learned to See (Part 2 of 3)

Recap: The Transformer Revolution (Part 1)

The CNN Era: What Worked and What Did Not

The Key Insight: Images as Sequences of Patches

ViT Architecture: A Step-by-Step Walkthrough

Step 1: Patch Embedding

Step 2: The [CLS] Token

Step 3: Position Embeddings

Step 4: Transformer Encoder

Step 5: Classification Head

The Data Hunger Problem

The Ecosystem: Vision Transformer Variants

DeiT — Data-Efficient Image Transformers (Facebook, 2021)

Swin Transformer — Shifted Windows (Microsoft, 2021)

BEiT — BERT Pre-Training for Images (Microsoft, 2021)

CvT and CoAtNet — Hybrid CNN + Transformer

DINO and DINOv2 — Self-Supervised Learning (Meta)

EVA and InternImage — Pushing Scale

SAM — Segment Anything (Meta, 2023)

Comparison: CNN vs. ViT vs. Hybrid

Comparison of Major Vision Transformer Models

When to Use CNN vs. ViT in Practice

The Bigger Picture

What is Next: Part 3 — Vision Language Models

Semantic vs Keyword vs Hybrid Search: What Every RAG Demo Skips

OpenAI — `response_format={"type": "json_schema", ...}`

Gemini — `response_mime_type` + `response_schema`

`json_repair` for almost-JSON