DEV Community: Shakti Wadekar

[Boost]

Shakti Wadekar — Mon, 29 Jun 2026 14:00:03 +0000

Shakti Wadekar

Jun 27

1M Context Tokens Is Not Memory: The Beginner’s Guide to Long Context

#ai #beginners #learning #llm

6 min read

1M Context Tokens Is Not Memory: The Beginner’s Guide to Long Context

Shakti Wadekar — Sat, 27 Jun 2026 12:23:04 +0000

So your favorite LLM now supports a 1 million token context window. Marketing slides everywhere: “Fits the entire Harry Potter series! Twice! With footnotes!”

A model with a 1 million token context window sounds powerful. And it is powerful.

But here are the key points:

A model having 1M context means it can receive a lot of input. Whether it remembers, finds, connects, or uses all of it correctly is a completely separate problem.

Long Context Is Capacity, Not Capability

Context length = how much the model can receive. Capability = how well the model can use it.

Access is not the same as intelligence.

“fits in context” does not mean “understood perfectly”

“Okay but it can read a lot, so it understands a lot, right?”
Not necessarily.

Reading ≠ remembering accurately. Reading ≠ using everything you read correctly when it matters.

So, Is Long Context Bad?

No.

Long context is extremely useful.

It reduces the need for aggressive chunking (RAG).
It helps with large documents and big codebases.
It makes many workflows easier.

The problem is NOT long context.

The problem is expecting long context to behave like perfect memory, perfect search, perfect reasoning, and perfect summarization all at once.

That is NOT how production AI works.

A good AI system usually combines:

Long context
+ retrieval
+ memory
+ summarization
+ structured context
+ evaluation

Three known problems with long context models:

1. Forgetting (a.k.a. “Lost in the Middle”)

Studies on long-context models found something very interesting:

Models are great at remembering stuff at the start and end of a long input, and surprisingly bad at the middle.

If you bury the one important text of your 80-page document in the middle, the model might just… not notice it. Even though it “read” it.

2. Missing (needle-in-a-haystack failures)

Hide one specific sentence (“The secret code is 4471”) inside a huge pile of text, then ask the model to find it. Sometimes it nails it. Sometimes gives you a confident wrong answer.

More tokens means more haystack, and more haystack means more places for the needle to hide.

3. Failing multi-hop reasoning

Multi-hop reasoning means the model must connect multiple facts from different places.

Multi-hop reasoning = needing to connect Fact A (page 3) with Fact B (page 250) with Fact C (page 800) to answer one question.

The longer and more scattered the chain of facts or critical information, the more likely the model is to drop a link.

Rather than say “I don’t know,” it’ll often just invent a plausible-sounding connection (Hallucination).

So… is there anything you can actually do about this?

Yes. And that’s the more useful half of this article, so let’s get into it.

The Solution: Stop Trusting, Start Evaluating

Okay, enough doom-scrolling through failure modes.

Here’s the actual fix, and it’s less glamorous than “buy a bigger context window”:

Evaluate the model thoroughly on your long-context use case before you let it anywhere near your application or business workflow.

A long-context model should not be judged only by how much text it can receive. It should be judged by how well it can:

1. Find the right information
2. Remember the important constraints during the task
3. Connect facts across distant sections
4. Ignore irrelevant noise
5. Avoid hallucination
6. Produce a faithful final answer

There are two ways to do this, and you need both:

Academic benchmarks: LongBench and LongGenBench. Good for understanding a model’s general long-context behavior before you even pick which model to use.
Your own domain-specific evaluation pipeline: Because no academic benchmark knows what “correct” means for your 200-page insurance policy or your codebase’s internal logic.

Let’s take both in turn.

LongBench and LongGenBench

LongBench and LongGenBench exist precisely to measure the gap between “received the text” and “remembers, finds, connects, or uses it correctly”

LongBench: A benchmark suite that tests models on real long-document tasks: long Q&A, summarization, code understanding, few-shot learning, all stretched across long inputs in multiple languages.

The point: see how performance holds up as documents get longer and more complex, not just whether the model can technically accept the tokens.

LongGenBench: Focuses on something sneakier: long-form generation, not just long-form reading.

It checks whether a model can produce a long, coherent piece of output (think: a long structured document with consistent constraints throughout) without contradicting itself, drifting off-topic, or quietly forgetting an instruction it agreed to 3,000 words ago.

Use these two benchmarks the way you’d use a car’s official mileage rating: useful for comparing models before you buy, but not a guarantee of what will happen on your specific roads, in your specific traffic. For that, you need your own test.

There are various other benchmarks, but mentioning here 2 which covers long context understanding and long context generation.

Design your own domain-specific evaluation pipeline

This is the part most people skip, and it’s the part that actually saves you when production breaks at 2 AM. A solid pipeline looks like this:

Build a test set out of real examples from your domain, not generic Wikipedia paragraphs. If you’re building a legal-contract assistant, your test documents should be actual long contracts, with real clauses buried in real places.
Plant “needles” deliberately, at different positions. Put your critical facts at the start, middle, and end of test documents on purpose. Remember the “lost in the middle” problem? This is how you measure whether your model, on your documents, suffers from it, and how badly.
Include multi-hop questions, not just single-fact lookups. A question that requires connecting a clause on page 4 with an exception on page 60 will expose reasoning failures that simple needle-in-haystack tests won’t.
Score for correctness, not just “an answer was produced.” A confident, fluent, completely wrong answer should fail your eval just as hard as a refusal.
Automate the grading where you can. Exact-match for factual lookups, a separate LLM-as-judge step or rubric for open-ended answers, human spot-checks for anything high-stakes.
Set a minimum acceptable threshold before shipping (Example: “95%+ accuracy on critical-fact retrieval across all document positions”) and treat dropping below it as a blocking bug.
Re-run the eval whenever you change anything: model version, prompt, chunking strategy, retrieval logic. Long-context behavior is surprisingly sensitive to small changes, and “it worked last month” is not a test result.

The academic benchmarks tell you whether a model is generally trustworthy with long context.

Your own pipeline tells you whether it’s trustworthy with your documents, your questions, and your definition of “correct.” Skip the second one, and you’re deploying on hope.

What Metrics Should You Track?

1. Answer accuracy
2. Faithfulness to the provided context
3. Evidence citation quality
4. Multi-hop reasoning correctness
5. Instruction following
6. Long-output consistency
7. Hallucination rate
8. Latency
9. Cost

Accuracy tells you whether the answer is correct. Faithfulness tells you whether the answer is grounded in the provided context.

Citation quality tells you whether the model can point to the right evidence.

Latency and cost tell you whether the solution is actually usable, or whether every user question requires a small financial ceremony :)

Compare Different Context Strategies

Do not evaluate only one setup.

Compare multiple approaches:

1. Full long context
2. RAG-based retrieval
3. Summarized context
4. Hybrid approach: retrieval + summaries + long context

Sometimes full long context works well. Sometimes retrieval works better.

Sometimes a structured summary beats dumping raw text. Sometimes the best solution is a hybrid system.

Concluding remarks:

The solution to long-context risk is not avoiding long-context models. They are powerful and useful. The solution is to evaluate them properly.

So when you see:

Supports 1M tokens

the better question is:

What can it reliably do with 1M tokens?

Because context length is a specification. Performance is an evaluation result.

Marketing loves the first one. Engineers should care about the second one.

Editing credit goes to an AI (ChatGPT and Claude). It suggested better phrasing, cleaner diagrams, and only hallucinated few facts, which I caught using the multi-hop reasoning skills it taught me two sections ago. Synergy :)

Stop Letting LLMs Hallucinate Your Codebase: A Graph-First Way to Summarize Repos

Shakti Wadekar — Fri, 26 Jun 2026 04:34:36 +0000

1. The problem we’re actually trying to solve

Ask any LLM to “summarize this repository” and it will happily oblige, and it will also happily make things up. It will mention a test suite that doesn’t exist. It will describe an API endpoint it inferred from the folder name api/. It will confidently tell you about a “data flow” it never actually traced :(

Reason:

LLMs are pattern completers, not code analyzers. When you dump a pile of files into a context window and ask for a summary, the model is guessing based on naming conventions and statistical priors from millions of other repos it has seen, not from understanding this code.

Solution:

code-graph-ai-summarizer is a small, Python project that takes a different approach: don't let the LLM look at raw code at all.

Instead, build a precise, structured, graph-derived set of facts about the repo first, using real static analysis, and only then hand the LLM a curated fact-sheet and ask it to write summary.

The LLM's job shrinks from “understand this codebase” to “narrate these facts I already verified,” which is a job LLMs are very good at.

That one design decision is the whole story of this repo, and it’s a pattern worth learning regardless of whether you ever run this exact tool.

2. The core idea, in one picture

Five stages. Each stage only passes forward what the next stage needs and never the raw source code itself.

Let’s walk through each one, building up from the basics.

3. Stage 1 : Turning code into a graph (Joern + CPG)

Before we can reason about a codebase, we need a representation of it that a program can query.

Reading characters in a .py file tells you nothing about which function calls which other function. You need structure. This is where Joern comes in.

Joern is an open-source static analysis platform that parses source code into a Code Property Graph (CPG): a single graph data structure that fuses together several representations:

The Abstract Syntax Tree (what the code literally says)
The Control Flow Graph (what order things execute in)
The Data Dependence Graph (which values flow into which)
Call graph edges (what calls what)

Once your repo is imported into Joern, all of that becomes queryable through CPGQL : a Scala-based query language that treats the whole codebase as one big graph you can filter, map, and traverse.

In code-graph-ai-summarizer repo, that connection lives in joern/client.py:

class JoernRunner:
    def __init__(self, server: str) -> None:
        self.client = CPGQLSClient(server)

    def import_repo(self, repo_path: Path, project_name: str) -> None:
            result = self.client.execute(import_code_query(str(repo_path), project_name))
            ...

JoernRunner is just a thin wrapper around cpgqls_client, which talks to a Joern server running locally (joern --server, listening on localhost:8080 by default).

You point it at a local folder, it imports the repo, and from then on you can fire CPGQL queries at it. (This is done by code-graph-ai-summarizer so you don’t have to)

Why a graph and not just an AST per file?

Because the interesting questions about a codebase are inherently cross-file: “what calls this function,” “where does this value end up,” “which file is the most central.”

Those are graph-traversal questions, not single-file parsing questions.

The CPG gives you one graph spanning the entire repo, so those questions become tractable.

4. Stage 2 : Asking the graph the right questions

A CPG by itself is just a big graph sitting in memory.

The value comes from the specific queries you run against it.

joern/queries.py defines six of them, and reading through them is basically a mini-lesson in “what does a useful static analysis tool actually need to know about a codebase.”

Query and What it extracts:

files : Every source file in the repo

methods: Every function/method, with file, full name, signature, line

types: Every class/type declared

call_edges: For each method, which other methods it calls (internal + external)

calls: Every individual call site, with its code text

entry_candidates: Methods that look like entry points

source_sink_calls: Calls that look like data sources or data sinks

entry_candidates

The entry_candidates query is critical. There’s no universal CPGQL way to say “find the main function” across Python, JS, Go, etc.

So the repo uses a name/filename heuristic instead:

val entryRe = "(?i).*(main|run|start|serve|handler|handle|route|controller|command|execute|process|consume|worker|app).*"

cpg.method
  .filterNot(_.isExternal)
  .filter(m => m.name.matches(entryRe) || m.filename.matches(entryRe))
  .take(maxItems)

One more detail worth noticing:

joern/client.py wraps every single query in a try/except:

for name, query in joern_queries(max_items).items():
    try:
        facts[name] = self.run_json_query(name, query)
    except Exception as exc:
        print(f"[warn] Joern query failed: {name}: {exc}")
        facts[name] = []

If one query fails (say, the data-flow query isn’t supported for a given language overlay), the pipeline doesn’t crash, it just records an empty result and moves on.

5. Stage 3 : From raw facts to ranked signal (the Python analysis layer)

Joern hands back raw lists: every file, every method, every call. That’s hundreds or thousands of items, too much, too unstructured, and too noisy to throw straight at an LLM.

This is where the repo’s analysis/ package comes in.

Its whole job is compression with judgment: turning a flood of graph facts into a small set of ranked, labeled signals.

5.1 Classifying what code “is”, without a single import statement

analysis/patterns.py defines keyword buckets for common categories: api_web, cli, storage_db, filesystem, llm, network, auth, queue_worker.

A snippet:

CATEGORY_PATTERNS = {
    "storage_db": [
        "sqlite", "postgres", "mysql", "mongodb", "redis", "sqlalchemy",
        "save", "insert", "update", "delete", "select", "execute", "commit", "query",
    ],
    "llm": [
        "openai", "ollama", "anthropic", "gemini", "groq", "cerebras",
        "completion", "chat.completions", "llm", "model", "generate",
    ],
    ...
}

analysis/classify.py then just checks whether any of these substrings show up in a call's name/code/target text.

This is deliberately simple: no embeddings, no ML model, just substring matching. Reason: it's fast, debuggable, language-agnostic, and “good enough” because its output isn't the final answer, it's a signal that downstream ranking and the LLM will further interpret.

Don't reach for a heavyweight model when a keyword list solves 90% of the problem at near-zero cost.

5.2 Finding the repo’s “important” files

analysis/architecture.py turns the call-edge facts into a per-file importance score.

The logic, simplified:

A file gets points for every internal call it makes (it’s doing things)
A file gets more points when other files call into it (it’s depended on)
A file gets points whenever its calls match one of the category patterns above (it touches storage, network, LLMs, etc.)

file_scores[caller_file] += len(internal_callees) + len(external_callees)
...
file_edge_counts[(caller_file, callee_file)] += 1
file_scores[callee_file] += 2

Sort by score, and you get a ranked list of “central files”, a cheap but effective proxy for architectural importance.

5.3 Finding runtime flows: graph traversal, not guesswork

This is the most conceptually interesting part of the repo.

analysis/flows.py runs a breadth-first search (BFS) starting from each entry-point candidate, walking forward through the call graph, and scoring every path it finds:

entry_point
   -> calls method A
        -> calls method B  (touches "storage_db")
             -> calls method C  (touches "llm")

queue = deque([[entry]])
while queue and seen_paths < 80:
    path = queue.popleft()
    if len(path) >= 3:
        score, signals = path_score(path, method_to_file)
        if signals:
            candidates.append(runtime_candidate(...))
    if len(path) >= 5:
        continue
    for next_method in graph.get(path[-1], [])[:12]:
        if next_method not in path:
            queue.append(path + [next_method])

Each path’s score (analysis/graph.py) rewards length and, much more heavily, rewards touching “important” categories like api_web, storage_db, or llm:

score = len(path) + 4 * len(set(categories))
important = {"api_web", "storage_db", "filesystem", "llm", "network", "auth", "queue_worker"}
score += 5 * len(important.intersection(categories))

In plain English:

A path that goes from an entry point all the way to a database call or an LLM call is more “interesting” than a path that just bounces between two utility functions.

That’s a simple but effective heuristic.

find_data_flows is the mirror image: instead of starting from entry points, it starts from calls that look like data sources (request, input, argv, env, ...) and BFS-searches forward until it reaches calls that look like data sinks (write, save, insert, chat, post, ...).

source: read user input
    |
    v
  [ some processing methods ]
    |
    v
sink: save to DB / send to LLM

Important nuance the README states explicitly and the code backs up: these are graph-derived candidates, not proven runtime traces.

Joern is doing static analysis, it never executes the code.

A BFS path through the call graph is a plausible flow, not a guaranteed one.

6. Stage 4 : Compacting everything into one fact sheet

All of the analysis above gets assembled in summarization/facts_builder.py into a single summary_facts dictionary:

def build_summary_facts(repo_path: Path, facts: dict) -> dict:
    repo_map = build_repo_map(facts.get("files", []))
    architecture = derive_architecture(facts)

    return {
            "repo_name": repo_path.name,
            "repo_map": repo_map,
            "architecture_signals": architecture,
            "entry_points": facts.get("entry_candidates", [])[:40],
            "critical_runtime_flow_candidates": find_runtime_flows(facts),
            "critical_data_flow_candidates": find_data_flows(facts),
            "important_symbols": important_symbols(facts, architecture),
            "limits": {"note": "This is static analysis. Runtime/data flows are graph-derived candidates, not guaranteed actual production traces."},
        }

Notice what’s not in here:

raw source code,

full call lists,

every method in the repo.

important_symbols is deliberately filtered down to only the methods/types that live in the already-identified “central files”, another compression step that keeps the eventual LLM prompt small and focused.

This dictionary, not the repo itself, is what the LLM will actually see.

7. Stage 5 : Writing the prompt like a contract, not a suggestion

summarization/prompts.py

It builds the final prompt, and it’s worth reading closely because it shows how to constrain an LLM rather than just hope it behaves:

return f"""
You are generating a repository summary using Joern Code Property Graph facts.

Use only the supplied graph facts.
Do not invent files, folders, APIs, tests, classes, functions, runtime flows, or data flows.
Separate detected facts from inferred conclusions.
For runtime flows and data flows, include only the critical ones, not every path.
If something is weakly supported, say "likely".
If something is not supported, say "not detected".

Return Markdown with exactly these sections:
# Repository Summary
## 1. Repository Purpose
## 2. Repository Map
## 3. Architecture
## 4. Critical Runtime Flows
## 5. Critical Data Flows
## 6. Important Files
## 7. Important Symbols
## 8. Not Detected / Unknown
...
"""

A few important details:

Closed-world instruction: “use only the supplied facts” + “do not invent X, Y, Z” is the single highest-leverage anti-hallucination instruction you can give a model. It can’t promise zero hallucination, but combined with a small, accurate fact-sheet, it dramatically narrows the model’s room to wander.
Calibrated language is mandated, not optional: forcing the model to say “likely” or “not detected” instead of asserting things flatly turns confidence into a visible, checkable signal for the reader.
A fixed output schema: by naming the exact eight sections, the output is predictable and easy to parse, render, or diff across repos. Useful if you ever want to compare summaries over time or build a UI on top of this.
An explicit “Not Detected / Unknown” section: most prompts ask a model what it knows; this one also asks it to state what it doesn’t know. That’s a small change with an outsized effect on trustworthiness.

llm/client.py

llm/client.py then does the boring-but-important part: it’s a thin OpenAI-SDK wrapper that works with any OpenAI-compatible endpoint: Groq, OpenRouter, Gemini’s OpenAI-compatible endpoint, or Cerebras, controlled purely through .env config:

LLM_PROVIDER=groq
LLM_API_KEY=your_api_key_here
LLM_MODEL=llama-3.3-70b-versatile

def generate_repo_summary(summary_facts: dict, config: LLMConfig) -> str:
    client = make_client(config)
    response = client.chat.completions.create(
        model=config.model,
        temperature=config.temperature,
        max_tokens=config.max_tokens,
        messages=[
            {"role": "system", "content": "You are a precise static-analysis repo summarizer. You must not hallucinate unsupported repo facts."},
            {"role": "user", "content": build_summary_prompt(summary_facts)},
        ],
    )
    return response.choices[0].message.content or ""

8. Putting it all together: what actually happens when you run it

uv run code-graph-ai-summarizer /path/to/local/repo

Walking through run() in cli/main.py function end to end:

Make the output directory outputs/<repo-name>/.
JoernRunner.import_repo(...): Joern parses your repo into a CPG.
joern.collect_facts(max_items) : the six CPGQL queries (+ the optional data-flow query) run, each wrapped in its own try/except.
build_summary_facts(...) : repo map, architecture signals, runtime/data flow candidates, important symbols all get derived and compacted.
generate_repo_summary(...) : the compact fact JSON goes into the locked-down prompt, and the LLM writes the final Markdown.
Three files land in outputs/<repo-name>/ . First, joern_facts.json: the raw graph extraction (for debugging / inspection). Second, summary_facts.json : the compacted, ranked fact-sheet (what the LLM actually saw).

Third, repo_summary.md: the final human-readable summary.

outputs/<repo-name>/
├── joern_facts.json     <- raw, large, exact
├── summary_facts.json   <- compact, ranked, curated
└── repo_summary.md      <- narrated, by the LLM

9. Why this design generalizes beyond “summarizing a repo”

The specific use case here is repo summarization, but the underlying pattern is broadly applicable to anyone building tools on top of LLMs:

Don’t hand an LLM a haystack and ask it to find the needle. Use a purpose-built tool (a parser, a graph engine, a database, a search index) to find candidate needles first.
Rank and compress before you prompt. The smaller and more relevant the context, the less room there is for hallucination, and the cheaper/faster the call.
Keep every intermediate artifact. joern_facts.json and summary_facts.json aren’t just debug exhaust, they’re what let you trust, or distrust, the final output with evidence rather than vibes.
Fail softly at each stage. One bad query or one weird file shouldn’t take down the whole pipeline.

10. Trying it yourself

git clone <your-repo-url>
cd code-graph-ai-summarizer
uv sync
cp .env.example .env
# edit .env: set LLM_PROVIDER, LLM_API_KEY, LLM_MODEL

# in a separate terminal
joern --server

# in a separate terminal (if using ollama)
ollama serve

# back in your main terminal
uv run code-graph-ai-summarizer /path/to/any/local/repo

Point it at a small repo first, and get repo_summary.md.

11. A closing note:

Full credit where it’s due: I didn’t write this by hand, line by line, heroically, at 2 AM, fueled by coffee.

I played as an orchestrator, pointing ChatGPT and Claude at the problem, arguing with them when they hallucinated a function that didn’t exist, and stitching their outputs into something that actually runs and is a useful application. They wrote the code, I supplied the opinions, the rejections, and the “no, that’s not what I meant” loop until it converged.

So consider this repo a small case study in human + AI pair programming, minus the part where the AI gets annoyed at my code review comments.

Don’t Burn Claude Tokens: A Free, Local, Secure Way to Explore Your Code First

Shakti Wadekar — Mon, 22 Jun 2026 07:25:50 +0000

Every time you point Claude or ChatGPT at an unfamiliar codebase and ask “where is the auth logic?”, you’re spending real money and context just to get oriented.

The model has to read files it’s never seen, guess at structure, and you’re paying API tokens (or burning your message limit) for what is essentially a Google-Maps-level question, not a hard reasoning question.

talk-to-your-code is a neat fix for this. It's a small local app that lets you index a codebase on your own machine and ask it plain-English questions, using a small local model through Ollama instead of a cloud API.

The idea isn't to replace Claude, but to do the cheap, repetitive "let me look around first" work locally and for free, so that by the time you do talk to Claude, you already know which files matter and can ask a sharp, narrow question.

The big picture: what actually happens

At a high level, the app does four things in sequence:

It walks your repo and breaks files into chunks,
It stores those chunks in a local database,
It lets an LLM figure out which chunks are relevant to your question. (Context building)
It lets the LLM answer using only those chunks.

> Ingests any repository

This happens once, when you click “Ingest.” After that, everything else like the actual conversation, runs against this local database, not against the raw files.

That matters: it means each question is fast and cheap, because the app isn’t re-reading your whole repo every time you ask something.

> What happens when you actually ask a question

Here’s what each box is doing in practice:

Build a query plan. Your raw question gets turned into something a retrieval system can actually use: keywords to search for, files it suspects are relevant, and what kind of question this is (explain vs debug vs locate).
Hybrid retrieval. “Hybrid” here means combining different ways of finding relevant chunks: keyword, symbol matching (good for “find the function named authenticate") and embedding similarity search (good for "find code that does something like this," even without exact wording). Using both catches more relevant code than either alone.
Context builder with a budget. This is the unglamorous but critical step. You can’t just hand the LLM every chunk that matched. The context windows are finite, and stuffing in irrelevant code wastes tokens and confuses the model. So the builder ranks chunks by relevance and packs in as many as fit under a character/token limit you control (that’s the “context length slider” in the UI).
Generate a structured answer. The final LLM call gets the user’s question plus the packed context, and is asked to generate answer with a certain schema. [Example]

class StructuredAnswer(BaseModel):
    summary: str
    relevant_files: list[str]
    confidence: str

That’s it, two structured LLM calls, with a plain retrieval step sandwiched in between. No agents looping indefinitely, no unbounded tool calls. It’s a deliberately small, predictable pipeline.

Important detail:

When you type a question like “where is authentication handled?”, the app doesn’t just dump your code into a prompt and hope for the best.

It runs the LLM twice, each time forcing a specific output shape using structured generation (also called constrained decoding), so you get a guaranteed JSON object back instead of free text you’d have to parse and hope is valid.

In Python, that schema is typically just a Pydantic model, something like this: [Example]

from pydantic import BaseModel
class QueryPlan(BaseModel):
    keywords: list[str]
    target_files: list[str]
    intent: str  # e.g. "explain", "debug", "locate"

You hand this structured-data to the LLM call and the API guarantees the response matches it. No more “please respond in JSON” prompts that occasionally come back with extra commentary attached.

Summary:

A few things this repo demonstrates cleanly that show up everywhere once you start building with LLMs:

Structured generation beats prompting for JSON. Defining a Pydantic schema and binding it to the call is more reliable than asking nicely in the prompt and parsing the response with regex.
Separate “finding the right context” from “answering the question.” Bundling retrieval and generation into a single giant prompt is how you get hallucinated answers about code that isn’t relevant.

Splitting it into a plan → retrieve → answer pipeline keeps each step’s job small and checkable — you can literally inspect the query plan in the UI before the final answer is generated.

Context budgets aren’t optional. Whether you’re calling a 7B local model or Claude through the API, “how much do I actually send” is a real engineering decision, not an afterthought.
Local-first has a real use case. It’s not about avoiding Claude, it’s about not exposing a private codebase to an external API for the boring first-pass questions, and not paying API costs for what a free local model can already answer.

Once this tool/application helps you find the right files and understand them, you can ask Claude about the actual fix instead of spending tokens on repo exploration.

AI Agents in Production: Structured Generation for AI Workflows

Shakti Wadekar — Sun, 14 Jun 2026 17:56:55 +0000

Structured generation is one of the most important steps in moving AI agents from demos to production systems. In real applications, an agent is not just writing text for a user, it is passing decisions, tool arguments, routing outputs, validation results, and workflow states to other parts of a software pipeline. In this article, we will look at how vLLM helps enforce this structure during generation.

📚 Content

🚀 1. Motivation

🏭 2. Production Reality

⚙️ 3. Structured Generation in vLLM

🧩 3.1 JSON Schema-Constrained Generation

🏗️ 3.2 Pydantic Model → JSON Schema Conversion

🚀 1. Motivation

Why Structured Generation?

Imagine you built an AI customer support agent. A user sends: “I want to return my order #4821.” Your agent needs to call an internal API to look up the order. That API expects a clean JSON payload:

{ "order_id": "4821", "action": "return", "reason": null }

But your LLM, without any constraints, might output:

Sure! I can help with that. Here is the return request:
{ "order_id": 4821, "action": "return", "reason": "not specified" }
Let me know if you need anything else!

Three problems in that one response:

Extra text wrapped around the JSON,

order_id is a number instead of a string,
reason is "not specified" instead of null.
Your json.loads() will either crash or your API will reject the payload.

In a demo, you’d just fix this with a try/except and with more prompting.

In production, the same issue can happen thousands of times a day across multiple agents, tools, and workflows. At that scale, even a 2% formatting failure rate is no longer a small bug, it becomes broken automations, failed handoffs, and real customer impact.

The core problem:

LLMs are probabilistic text generators. They predict the most likely next token, they do not inherently “know” that your downstream system needs a strictly-typed JSON object. Even after prompting it with JSON requirements, it might still fail to produce exact required format.

The solution: Structured generation

Structured generation guides the model to produce outputs that follow a predefined format, such as JSON, a schema, or a set of allowed choices, so the response is easier for your code to validate and use reliably.

🏭 2. Production Reality

Production AI agents operate in pipelines. The LLM output is almost never the final product. The LLM output is fed into databases, APIs, other models, or UI components. Each handoff requires the output to conform to a format. Structured generation is how you enforce that format at the generation level.

Here is what structured generation unlocks:

Cleaner backend integration because the LLM output can map directly to typed application models, validation logic, APIs, and databases.
Cleaner agent pipelines and more reliable agent handoffs because each step can pass structured data to the next step without relying on messy text interpretation.
Fewer production failures because the model is constrained to return valid, expected outputs instead of unpredictable free text.
Lower retry and repair cost because the system spends less time fixing bad outputs and more time executing the actual workflow.

⚙️ 3. Structured Generation in vLLM

Introduction

vLLM is mainly known as a high-throughput inference and serving engine for LLMs, but it also provides built-in support for constraining model outputs into specific formats.

In vLLM, structured generation can be used in two common ways: through structured_outputs and StructuredOutputsParams for offline inference, or through response_format / extra_body={"structured_outputs": ...} when using the OpenAI-compatible API.

Setup

pip install -U openai vllm

If you again get the NumPy Inf error, run:

pip install "numpy<2"

Run the following command in the terminal to locally host the model with vLLM.

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --gpu-memory-utilization 0.85 \
  --max-model-len 4096 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

vllm serve starts vLLM as a model server. Instead of loading the model inside every notebook run, the model is loaded once in a terminal and kept running. Your notebook code then sends requests to this local server, just like it would send requests to the OpenAI API.

vLLM exposes an OpenAI-compatible API server, so the normal openai Python client can call it by changing only the base_url to http://localhost:8000/v1

Qwen/Qwen2.5-1.5B-Instruct
This is the Hugging Face model that vLLM will download/load and serve.

Run the following code. If this prints the model name, vLLM is running correctly.

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="unused",
)
print(client.models.list().data[0].id)

🧩 3.1 JSON Schema-Constrained Generation

The most common use case:

You define a JSON Schema, and vLLM constrains decoding so the generated text follows that schema.

Let’s understand this with an example:

Production Use Case: Support Ticket Triage System

from openai import OpenAI
import json

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY"
)

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"

triage_schema = {
    "type": "object",
    "properties": {
        "urgency": {"type": "string", "enum": ["low", "medium", "high", "critical"]},
        "category": {"type": "string", "enum": ["billing", "technical", "returns", "general"]},
        "customer_id": {"type": ["string", "null"]},
        "summary": {"type": "string", "maxLength": 200}
    },
    "required": ["urgency", "category", "summary"],
    "additionalProperties": False
}

email_text = """
Customer #C-4821 says: My payment was charged twice yesterday
and I still haven't received any confirmation. This is urgent!
"""

prompt = f"""Analyze this support email and return only JSON.

Email:
{email_text}
"""

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": prompt}],
    temperature=0.1,
    max_tokens=256,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "support_triage",
            "schema": triage_schema
        }
    }
)

text = response.choices[0].message.content
print(f"text: {text}")

triage = json.loads(text)
print(f"triage: {triage}")

print(f"Urgency: {triage['urgency']}")
print(f"Category: {triage['category']}")
print(f"Summary: {triage['summary']}")

Expected Output:

text: {"urgency":"high","category":"billing","customer_id":"C-4821","summary":"Customer was charged twice yesterday and has not received confirmation."}

triage: {'urgency': 'high', 'category': 'billing', 'customer_id': 'C-4821', 'summary': 'Customer was charged twice yesterday and has not received confirmation.'}

Urgency: high
Category: billing
Summary: Customer was charged twice yesterday and has not received confirmation.

The important part and what it does?

response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "support_triage",
            "schema": triage_schema
        }

vLLM uses a structured-output backend such as xgrammar or guidance to constrain decoding.

At each generation step invalid next tokens are masked from the model’s logits before sampling.

This makes the model generate text that follows the required structure, such as a JSON schema, regex, choice list, or grammar.

Important nuance:

Structured generation only guarantees structural validity, not that the extracted values are semantically correct.

🏗️ 3.2 Pydantic Model → JSON Schema Conversion

Writing raw JSON Schema objects is tedious and error-prone. Hence use Pydantic.

from openai import OpenAI
import json

from pydantic import BaseModel, Field, ConfigDict
from typing import Literal, Optional

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="EMPTY"
)

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"

# 1. Define output structure using Pydantic
class TriageOutput(BaseModel):
    model_config = ConfigDict(extra="forbid")

    urgency: Literal["low", "medium", "high", "critical"]
    category: Literal["billing", "technical", "returns", "general"]
    customer_id: Optional[str] = None
    summary: str = Field(max_length=200)

# 2. Convert Pydantic model to JSON Schema
triage_schema = TriageOutput.model_json_schema()

email_text = """
Customer #C-4821 says: My payment was charged twice yesterday
and I still haven't received any confirmation. This is urgent!
"""

prompt = f"""Analyze this support email and return only JSON.

Email:
{email_text}
"""

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": prompt}],
    temperature=0.1,
    max_tokens=256,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "support_triage",
            "schema": triage_schema
        }
    }
)

text = response.choices[0].message.content
print(f"text: {text}")

triage = json.loads(text)
print(f"triage: {triage}")

print(f"Urgency: {triage['urgency']}")
print(f"Category: {triage['category']}")
print(f"Summary: {triage['summary']}")

Conclusion

vLLM also provides more structured generation options like Regex-Constrained Generation, Grammar-Constrained Generation and Custom Logits Processors.

An more extended version of this article covers these topics along with examples of Tool-Calling and Routing agent, in my following medium article published in Towards AI.

Link the article: AI Agents in Production: Why Structured Generation Matters More Than Prompt Engineering

I hope this article was useful in showing why structured generation is not just a formatting trick, but a practical requirement for production AI agents. When agents are part of real software pipelines, their outputs must be predictable, valid, and easy for downstream systems to use.