DEV Community: John Hu

Why AI Agents Are Rediscovering Grep: The Limits of Vector RAG in Real-World Engineering

John Hu — Tue, 27 Oct 2026 14:06:58 +0000

As AI agents move from demos to production, engineering teams are rediscovering an important lesson: reliable access to information matters more than sophisticated retrieval algorithms.

For the past few years, vector databases have been treated as an almost essential component of AI applications. Whether the goal was to build an internal knowledge assistant, a code-search tool, or an autonomous agent, the architectural pattern was remarkably consistent: split documents into chunks, generate embeddings, store them in a vector database, and retrieve the most semantically similar passages whenever the model needs information.

For many applications, this approach works well.

But as AI agents take on increasingly complex engineering tasks — particularly debugging, code modification, system administration, and long-running workflows — the limitations of vector-based retrieval are becoming harder to ignore.

In some of these environments, a simple ripgrep command can deliver more useful results than an elaborate retrieval pipeline.

This is not because semantic search is fundamentally flawed. It is because semantic similarity and factual precision solve different problems, and autonomous agents often need the latter far more than traditional RAG architectures were designed to provide.
The broader shift is not necessarily away from RAG itself, but toward a more deliberate division of responsibility: probabilistic models for reasoning and planning, deterministic tools for retrieving facts and interacting with the environment.

Semantic Relevance Is Not the Same as Precision Vector search is designed to find content that is similar in meaning, even when the words differ.

Consider an employee asking:
“What changed in our expense reimbursement policy last fall?”

The employee may not remember the exact policy name, relevant department, or original document. Semantic search is useful precisely because it can bridge that gap between a vague question and the documents likely to contain the answer.

But software engineering agents face a different kind of problem.

Imagine an agent investigating an application failure involving the error:
invalid_token_handler
The agent does not need a general explanation of authentication failures. It needs to identify exactly where that error is defined, raised, handled, or referenced in the codebase.

A vector retrieval system may return documentation about token expiration, failed login attempts, authentication middleware, or other conceptually related topics.

Those results might be semantically relevant, but they may have nothing to do with the actual defect.

Worse, the exact function or error constant the agent needs could be missing from the retrieved results entirely.

A simple text search takes a different approach:
rg -n "invalid_token_handler" .
This command returns the locations where the exact string appears, including file paths and line numbers.

No semantic interpretation is involved.

That distinction is critical.

When debugging, a plausible but incorrect result can be more dangerous than an empty result. An agent may treat loosely related documentation as evidence and construct an entire debugging strategy around a false assumption.

An exact search, by contrast, establishes a concrete starting point.
For engineering agents, the value of retrieval is not merely finding something relevant. It is finding the specific evidence needed to take the next action.

Vector Indexes Struggle with Rapidly Changing Environments Traditional RAG systems are often built around data that changes relatively slowly.

A collection of internal documents might be indexed periodically, then queried thousands of times before the next update.

Code-editing agents operate under very different conditions.

An autonomous coding agent might modify eight files, create two new modules, and delete several obsolete functions within a few minutes.

Every modification changes the information the agent needs to reason about.

This creates an important architectural question:

How do you keep the retrieval system synchronized with the actual state of the codebase?

There are two common options, and both introduce trade-offs.
Option A: Update the vector index immediately
Every file modification potentially triggers a pipeline:

Detect the changed content
Identify which chunks need to be updated
Recompute the affected embeddings
Update the vector database
Ensure the new index state is available to subsequent queries Incremental indexing can reduce the overhead, but it does not eliminate the challenges.

A seemingly small code change may affect chunk boundaries, dependencies, or the meaning of surrounding code.

More importantly, index updates introduce another consistency boundary between the agent and the filesystem.
Option B: Update the index asynchronously
Asynchronous indexing reduces the impact on the agent’s immediate execution time.

But it creates a different problem: stale retrieval results.

The agent may modify a function, search for it moments later, and receive an older version from the vector index.

It may then reason about code that no longer exists.

In a long-running workflow, this can lead to incorrect edits, repeated failures, or attempts to fix problems the agent has already resolved.

Filesystem-based search avoids much of this complexity.

For example:
rg -n "process_payment" src/
The command searches the current files directly. Assuming the changes have been saved and the search scope is correct, the results reflect the code currently available on disk.

There is no separate embedding index waiting to catch up.

There are still practical concerns — ignored files, unsaved editor buffers, generated code, and repository scope. But the consistency model is far simpler.
For agents that constantly modify their environment, reducing the distance between the source of truth and the retrieval tool is a major reliability advantage.

Chunking Can Destroy the Structure That Makes Code Meaningful One of the most heavily optimized parts of traditional RAG systems is document chunking.

Developers experiment with different chunk sizes, overlapping windows, paragraph boundaries, recursive splitting, and model-assisted segmentation.

These techniques are useful for many kinds of text.

But source code is not simply a sequence of independent paragraphs.

Its meaning depends on relationships.

A method may reference a variable declared hundreds of lines earlier. A class may inherit behavior from another module. A function’s actual purpose may only become clear once its callers and dependencies are examined.

Consider a large class with the following structure:
payment_service.py
├── Imports
├── Class definition
├── Constructor
├── Configuration
├── Authentication helpers
├── Payment validation
├── Transaction execution
└── Error handling
If this file is split into fixed-length chunks, the error-handling logic could become separated from the function signature, relevant configuration, and surrounding control flow.

A retrieval system might return the error-handling chunk without showing the code needed to understand it.

The model receives text that appears useful but lacks the structural context required to reason correctly.

Various approaches attempt to address this, including parent-child retrieval, hierarchical indexing, syntax-aware chunking, and reranking.

These can significantly improve retrieval quality.

However, they also introduce complexity and additional operational overhead.

A different approach is to let the agent locate the relevant code first, then retrieve the surrounding structure on demand.

For example:
rg -n "validate_token" src/
Once the agent finds the symbol, it can inspect the relevant function, class, or module.

If necessary, it can follow imports, examine callers, or query the language server for references.

This is increasingly practical because modern language models can process substantially larger contexts than earlier generations.

Larger context windows do not guarantee perfect understanding, and reading entire repositories indiscriminately is rarely efficient.

But they make a more flexible retrieval strategy possible:
Find the exact location first. Expand the context only as much as the task requires.

Instead of forcing all information through a predefined chunking scheme, the agent can decide what constitutes a meaningful unit of context.

Agents Need Exploration, Not Just Retrieval Traditional RAG applications often follow a straightforward pipeline: User question ↓ Retrieve relevant documents ↓ Add documents to the prompt ↓ Generate an answer This works reasonably well for many question-answering tasks.

But autonomous agents operate through an iterative process:
Observe
↓
Form a hypothesis
↓
Choose an action
↓
Execute a tool
↓
Inspect the result
↓
Revise the hypothesis
↺
The difference is fundamental.

An agent investigating an unfamiliar codebase rarely knows exactly what to search for at the beginning.

It needs to explore.

Suppose the agent starts with:
rg -n "token" .
The command returns 3,000 matches.

That result is immediately informative: the search term is too broad.

The agent can narrow the scope:
rg -n "token" src/auth/ -g "*.py"
If that still produces too many results, it can search for a more specific identifier:
rg -n "invalid_token_handler" src/auth/
If there are no results, it can try related identifiers, inspect imported modules, or widen the directory scope.

Each tool response provides actionable feedback.

The agent can observe the number of matches, file locations, and error messages, then adjust its next step accordingly.

Semantic search can also support iterative exploration, especially when systems expose similarity scores, metadata filters, and multiple retrieval strategies.

However, its feedback is often less transparent.

When a vector query returns five loosely related documents, it may be difficult for the agent to determine whether the query was poorly phrased, the relevant content was never indexed, or the embedding model simply failed to rank it highly enough.

Deterministic search tools provide a more explicit relationship between the query and the result.

That makes them particularly well suited to agents that need to formulate and test hypotheses.
The important capability is not a single perfect retrieval call. It is the ability to search, inspect, learn, and search again.

The Best Retrieval Tool Often Depends on the Question Exact text search is powerful, but it is not sufficient for every engineering task.

Consider three different questions.
Question A: Where does this exact error string appear?
rg -n "invalid_token_handler" .
A text-search tool is an excellent fit.
Question B: Which functions call validate_token()?
A language server or compiler-backed reference lookup is often more reliable than searching for the function name as plain text.

Text matching may find comments, unrelated methods with the same name, or references that cannot be resolved without understanding the programming language.
Question C: Where is the business logic that prevents expired sessions from being reused?
The relevant implementation might use different terminology entirely.

It may involve session timestamps, middleware, cache invalidation, or database state.

In this situation, semantic search can help discover potential entry points, especially when the agent has little knowledge of the repository.

These examples point toward a broader retrieval architecture:

Exact strings and error codes → ripgrep, regular expressions

File paths and configuration names → path matching, glob patterns

Symbol definitions and references → LSP, compiler indexes

Code structure and syntax → AST parsers

Unfamiliar functionality → semantic search, repository exploration

Cross-document concepts → vector search, hybrid retrieval

Relationships between components → dependency graphs, call graphs

The right solution is rarely to force every type of information through a single retrieval mechanism.

Instead, agents should be able to choose the appropriate tool based on the nature of the question.

This is especially important for complex engineering tasks, where the agent may move repeatedly between exact lookup, structural analysis, and semantic understanding.

Deterministic Tools Make Probabilistic Agents More Reliable Large language models are probabilistic systems.

Even when their outputs appear confident, their reasoning and predictions are not guarantees of factual correctness.

That uncertainty is manageable when the surrounding system provides reliable ways to verify assumptions.

Problems arise when every layer introduces additional uncertainty.

Consider an agent architecture in which:

A semantic retrieval system may fail to surface the correct evidence
The language model may misinterpret the retrieved context
The agent may then execute an action based on that interpretation Errors introduced during retrieval can propagate through reasoning and eventually affect the external system.

One way to reduce this risk is to make the agent’s observations as verifiable as possible.

Rather than asking a model to guess whether a function exists, give it a tool that can check.

Rather than asking it to infer where a method is referenced, give it access to language-server functionality.

Rather than relying on a potentially stale vector index to understand the current repository, let it inspect the actual files.

This does not mean the entire system becomes deterministic.

An agent can still choose the wrong search query, misunderstand a correct result, or make an invalid code change.

Even compiler-backed tools can be limited by incomplete project configuration or dynamic language features.

But deterministic tools narrow the scope of uncertainty.

They give the model concrete observations against which its hypotheses can be evaluated.

A useful engineering principle emerges:
Use probabilistic intelligence to decide what to investigate, and deterministic mechanisms to establish what is actually true.

The model should handle interpretation, planning, prioritization, and decision-making.

Tools should handle exact lookup, state inspection, validation, and execution wherever possible.

Bigger Context Windows Change How Agents Should Retrieve Information Early RAG systems were designed during a period when model context windows were relatively small.

Breaking documents into chunks was often a practical necessity.

Sending a complete large file, let alone an entire repository, was either impossible or prohibitively expensive.

That constraint has changed.

Modern models can handle much longer inputs, making it feasible to inspect complete functions, classes, modules, or substantial portions of a repository when necessary.

This changes the economics of retrieval.

Instead of optimizing exclusively for how much information can be compressed into a few retrieved chunks, we can optimize for whether the retrieved material preserves the relationships needed for the task.

An agent might first use ripgrep to find a suspicious function.

Then it can read the relevant module.

Next, it can inspect callers and dependencies.

Finally, it can examine tests to understand the expected behavior.

The result is a context assembled through purposeful investigation rather than a single similarity-ranking operation.

Still, longer context windows are not unlimited memory.

Large prompts introduce costs, latency, distraction, and the possibility that important details receive insufficient attention.

For this reason, targeted retrieval remains essential.

The objective is not to replace small chunks with enormous prompts.
It is to give agents control over which information to retrieve, how much context to expand, and when to stop exploring.

This Is Not the End of RAG. It Is a More Mature Approach to Retrieval It would be a mistake to conclude that vector databases have no place in agent systems.

Semantic search remains valuable for many use cases:

Enterprise knowledge discovery
Research across large document collections
Questions expressed in unfamiliar terminology
Finding conceptually related material
Exploring repositories without knowing exact symbols
Matching natural-language requirements to possible implementations These tasks are genuinely difficult to solve using exact text matching alone.

The problem begins when vector search is treated as a universal abstraction for accessing knowledge.

A database containing embeddings does not automatically provide accurate memory, complete understanding, or reliable access to the current state of a system.

Those capabilities require deliberate system design.

For a coding agent, that might mean direct filesystem access, language-server integration, syntax-aware tools, and selective semantic retrieval.

For a business knowledge agent, it might mean combining vector search with SQL, metadata filtering, keyword search, and authoritative source references.

For an operations agent, it might mean querying live infrastructure APIs and observability systems instead of relying on embedded documentation describing how those systems behaved in the past.

In each case, the same principle applies:
Retrieve facts from the most authoritative and appropriate source available.

Semantic search is one tool in that architecture, not the architecture itself.

The Rediscovery of Unix Philosophy There is something revealing about the growing importance of tools such as grep, ripgrep, compiler services, and filesystem utilities in modern AI engineering.

These tools are not new.

Their underlying principles have existed for decades: do one thing well, expose predictable behavior, and compose small tools into larger workflows.

What has changed is the interface.

Previously, a human engineer had to know which commands to run and how to interpret their output.

Now a language model can decide when to use those commands, adapt its queries based on the results, and incorporate what it discovers into a broader plan.

This creates a powerful combination.

The model contributes flexible reasoning.

The tools contribute grounded observations and controlled interactions with the environment.

Neither needs to replace the other.

In fact, the more capable the reasoning model becomes, the more useful a collection of simple, reliable tools can be.

A sophisticated agent does not necessarily require a sophisticated retrieval pipeline at every step.

Sometimes, the most effective architecture is a model with access to a shell, a language server, a filesystem, and a small number of well-designed APIs.

That simplicity has tangible benefits: fewer moving parts, easier debugging, lower operational overhead, and clearer failure modes.
Conclusion: Reliability Matters More Than Architectural Novelty
The evolution of AI agents is forcing engineering teams to reconsider an assumption that became popular during the early RAG boom: that more advanced retrieval technology necessarily produces more capable systems.

It does not.

Vector retrieval, keyword search, structural code analysis, and direct API access each solve different problems.

As agents become responsible for longer-running and more consequential workflows, choosing the right information source becomes just as important as improving the model itself.

For many engineering tasks, the best first step is not to retrieve the five most semantically similar chunks.

It is to find the exact symbol, inspect the actual file, follow the real dependency, or query the live system.

Then let the model reason over the evidence.
The deeper lesson is not that grep is better than RAG.

It is that reliable agents need the freedom to interact with reality directly, rather than being confined to a preprocessed approximation of it.

The future of agent engineering is unlikely to belong exclusively to either semantic retrieval or deterministic tools.

It will belong to systems that combine them intelligently — and know when not to use one.

Sometimes, progress in AI engineering means adding a new capability.

And sometimes, it means realizing that a decades-old Unix tool was exactly what you needed all along.

I Built an Agent Loop by Hand. Then I Tried to Migrate It to the OpenAI Agents SDK

John Hu — Tue, 27 Oct 2026 12:23:25 +0000

A few weeks ago I wrote a small Kubernetes operations agent without using any agent framework. It is one Python for loop that talks to an LLM, calls cluster tools through MCP (Model Context Protocol), and writes a health report.

Then I read the example in the OpenAI Agents SDK repo. It wraps everything I had written by hand in a few clean abstractions: Agent, Runner, tools, tracing. The obvious question followed: should I just migrate?

I didn’t want to answer that from the docs alone, so I treated it as an engineering evaluation. I listed every behavior my agent has today, checked each one against the SDK source, and wrote a proof-of-concept script to measure the cases I couldn’t settle by reading.

Short answer: the migration is feasible, and it went better than my first estimate. But preserving my guardrails means spreading my loop logic across three different places. Along the way I found a real philosophical difference between my code and the SDK, and that difference is what this post is about.

All measurements below are against openai-agents 0.24.0, openai 3.35.1, mcp 1.30.0, and Python 3.13.

The loop I wrote by hand

The core of the agent fits on one screen:

for step in range(1, max_steps + 1):
    resp = await llm.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
        tool_choice="required" if step == 1 else "auto",
    )

    msg = resp.choices[0].message
    messages.append(msg)

    if not msg.tool_calls:
        return msg.content  # after evidence checks

    for tc in msg.tool_calls:
        args = json.loads(tc.function.arguments)
        result = await mcp.call_tool(tc.function.name, args)
        messages.append({
            "role": "tool",
            "tool_call_id": tc.id,
            "content": result,
        })

That is the entire idea of an agent: ask the model, run the tools it asks for, feed the results back, repeat until it stops asking.

The interesting part is everything I had to add after the first version met a real model. Each guardrail exists because the model did something wrong at least once, and my git history reads like a list of failures:

It answered without investigating. So step 1 forces tool_choice="required", and if the model still answers without calling any tool, the loop pushes back once.
It called tools that don’t exist. My favorite was delete_pod. The loop replies with a tool message listing the tools that do exist.
It sent arguments of the wrong type. Every call is validated against the tool’s JSON Schema before it reaches the MCP server, and the model is told exactly which field to fix.
It called the same tool with the same arguments over and over. Duplicates return the cached result plus a nudge, and too many duplicates force a summary.
It invented pod names in the final report. Every pod name in the report must appear in some tool result; otherwise the loop pushes back once and names the invented pods.
It never converged. After MAX_STEPS, the loop removes the tools and asks for a summary of what it already has.

Each of these has a test. The tests use a scripted fake LLM, a tiny local HTTP server that plays a misbehaving model, against a real MCP mock server. Six tests, no API key needed.

Those six tests became the most important artifact in this whole evaluation.

Learning the SDK by mapping it onto my own code

The fastest way I found to learn a framework is to stop reading it as new concepts and read it as a set of names for things I already built.

Agent is my configuration. Instructions, tools, model, model settings. It doesn't run anything.

Runner is my for loop. Runner.run(agent, input, max_turns=10) calls the model, executes tool calls, appends results, and stops when the model produces a final answer. max_turns is my MAX_STEPS.

reset_tool_choice is my step-1 trick. This was the moment the SDK clicked for me. My loop forces tool_choice="required" on step 1 and relaxes it afterwards. The SDK has a flag for exactly that, and its docstring says it “ensures that the agent doesn't enter an infinite loop of tool usage.” Someone on the SDK team hit the same problem I did.

Tools are my MCP glue. @function_tool builds a JSON Schema from a Python signature. For lower-level control there is a raw FunctionTool, which receives the model's arguments as a JSON string. MCP servers can also be attached directly.

Tracing is my _save_trace, grown up. I save each run as a JSONL file of messages. The SDK records a tree of spans, one per LLM call, tool call, guardrail, and handoff, each with timing.

Guardrails and handoffs are things I hadn’t built yet. Output guardrails check the final answer. Handoffs let one agent transfer the conversation to another.

Having built the loop by hand first, almost every SDK parameter answered a question I had already asked myself. I don’t think I would have understood reset_tool_choice or tool_use_behavior nearly as quickly otherwise.

The method: tests as a migration contract

“Can I migrate?” is too vague to answer. “Can the SDK pass my six guardrail tests?” is not.

So the six tests became the contract, and every behavior in my agent got one of three verdicts: moves cleanly, moves with a redesign, or doesn’t move.

One practical detail before anything else worked: the SDK uses the OpenAI Responses API by default. My agent supports any OpenAI-compatible endpoint (DeepSeek, Qwen, Ollama), and my fake LLM speaks Chat Completions. The fix is to use the Chat Completions model class explicitly:

from openai import AsyncOpenAI
from agents import OpenAIChatCompletionsModel, set_tracing_disabled

set_tracing_disabled(True)  # otherwise traces try to upload to OpenAI

model = OpenAIChatCompletionsModel(
    model="fake-model",
    openai_client=AsyncOpenAI(
        api_key="test",
        base_url=fake_llm_url,
    ),
)

With that in place, I reused the same fake-server pattern from my tests and wrote a PoC script for the four behaviors I couldn’t settle by reading source.

What I measured

1. `reset_tool_choice` doesn’t send `"auto"`. It sends nothing.

I expected request #2 to carry tool_choice: "auto". Here is what the fake server actually received:

request 1: tool_choice = "required"
request 2: tool_choice = <field absent>

The source resets tool_choice to None, and the Chat Completions converter omits a None value. For OpenAI that is equivalent to auto. But my test asserted the literal string "auto", so it would fail. The behavior matches; the assertion has to change.

Small, but this is exactly the kind of difference that only shows up when you check the request body.

2. Invalid JSON arguments: the decorator hides the error from the model

I made the fake model send {not-json as tool arguments, then tried both ways of defining a tool.

With @function_tool, my function never ran, and the model received this as the tool result:

An error occurred while running the tool. Please try again.

That’s a safe default for production, since it leaks nothing. But it’s useless for self-correction: the model has no idea what it did wrong.

With a raw FunctionTool, my handler received the raw string and could return a precise message, identical to what my hand-written loop sends:

Tool arguments are not valid JSON: Expecting property name enclosed
in double quotes: line 1 column 2 (char 1); raw arguments: {not-json

This decided the architecture: I would wrap my existing MCP client as raw FunctionTools instead of using the decorator or the native MCP integration.

3. Unknown tools: the default fails hard, and 0.24.0 has an escape hatch

When the fake model called delete_pod, the default behavior was:

ModelBehaviorError: Tool delete_pod not found in agent poc-unknown

The whole run aborts. Every AgentsException carries a run_data attribute with the run history. In my test the failure happened on the first turn, so there was nothing to recover, but even in principle catching the exception and rebuilding the conversation is clumsy.

My first draft of this evaluation filed unknown tools under “cannot migrate.” Then I read RunConfig more carefully and found this:

from agents import RunConfig

def whitelist(args):
    if args.kind == "tool_not_found":
        return (
            f"Unknown tool '{args.tool_name}'. "
            "Available tools: list_pods."
        )
    return None

run_config = RunConfig(
    tool_not_found_behavior="return_error_to_model",
    tool_error_formatter=whitelist,
)

With that config, the model received a normal tool output and the run continued:

Unknown tool 'delete_pod'. Available tools: list_pods.
Tool 'delete_pod' not found.

That is almost exactly my hand-written behavior.

Lesson learned: the examples show you the happy path; the config classes show you the edge cases.

4. Forced summaries: a sentinel works better than an exception

My loop has two ways to force a summary: too many duplicate calls, or too many steps. The SDK gives two candidate mechanisms.

max_turns raises MaxTurnsExceeded. Unlike the unknown-tool case, here run_data carried the full history in my test: four items (two tool calls, two tool outputs) and two raw model responses. So it is recoverable. I also noticed that 0.24.0 has run error handlers for the max_turns case, which I haven't tested yet.

tool_use_behavior turned out to be cleaner. It is a callback that runs after each round of tool calls and can declare the run finished:

from agents import ToolsToFinalOutputResult

FORCE_SUMMARY = "__FORCE_SUMMARY__"

async def stop_when_stuck(ctx, tool_results):
    if budget_exhausted():
        return ToolsToFinalOutputResult(
            is_final_output=True,
            final_output=FORCE_SUMMARY,
        )

    return ToolsToFinalOutputResult(is_final_output=False)

In my PoC, returning the sentinel ended the run with final_output == "__FORCE_SUMMARY__", and result.to_input_list() still contained the tool call and its output. From there, a forced summary is one more run of a tool-less clone of the agent.

The migration scorecard

Moves cleanly

The core loop, which becomes Runner.run.
Forcing tools on step 1, via tool_choice="required" plus the default reset_tool_choice (one test assertion changes).
OpenAI-compatible providers, via OpenAIChatCompletionsModel.
Logging, via RunHooks.
Unknown tool feedback, via tool_not_found_behavior="return_error_to_model".

Moves with a redesign

JSON errors, schema validation, duplicate caching. These all move into a FunctionTool wrapper around my existing MCP client:

def wrap_mcp_tool(mcp, spec, validator, seen):
    async def invoke(ctx, raw_args: str) -> str:
        try:
            args = json.loads(raw_args or "{}")
        except json.JSONDecodeError as exc:
            return f"Tool arguments are not valid JSON: {exc}"

        errors = list(validator.iter_errors(args))
        if errors:
            return (
                f"Invalid arguments for {spec['name']}: "
                f"{errors[0].message}"
            )

        sig = f"{spec['name']}:{json.dumps(args, sort_keys=True)}"

        if sig in seen:
            return "[Duplicate call] Result unchanged:\n" + seen[sig]

        seen[sig] = await mcp.call_tool(spec["name"], args)
        return seen[sig]

    return FunctionTool(
        name=spec["name"],
        description=spec["description"],
        params_json_schema=spec["parameters"],
        on_invoke_tool=invoke,
        strict_json_schema=False,  # keep the MCP schema untouched
    )

Forced summaries. These move into tool_use_behavior plus a follow-up run.

Pushback on lazy answers and invented pods. These are the ones that change shape the most, and they deserve their own section.

Changes format

Traces. to_input_list() returns SDK items like function_call and function_call_output, not Chat Completions messages, so my existing JSONL replays would need a new format, or a custom trace processor.

The real difference: feedback vs. exceptions

Here is the pattern that kept coming up.

My hand-written loop treats almost every model mistake as feedback: tell the model what went wrong, append it to the conversation, let it try again. Wrong arguments, unknown tool, invented pod name — they’re all just more messages.

The SDK’s defaults treat model mistakes as exceptions: a ModelBehaviorError for unknown tools, a generic error string for bad arguments, a tripwire exception when an output guardrail fires. The run stops, and the developer decides what happens next.

Neither is wrong. They are designed for different failure models.

Fail-fast is the right default for production systems. A model calling a tool that doesn’t exist might be a prompt injection, a version mismatch, or a bug, and you probably want to know rather than have the agent quietly improvise. Hiding exception details from the model also avoids leaking internals.
Self-correction is the right choice when you expect the model to be wrong often and cheaply fixable, which is the everyday reality with smaller or open-weight models.

This difference shows up most clearly in my two pushback guardrails. In my loop, “you answered without investigating” and “these pods never appeared in any tool result” are just user messages appended mid-loop. Inside Runner, a plain-text answer is the final output; there is no hook to reject it and continue. Output guardrails can detect the problem, but they end the run with an exception rather than giving the model a second chance.

So pushback moves outside the Runner, into an outer loop:

result = await Runner.run(
    agent,
    question,
    max_turns=budget,
    run_config=run_config,
)

problem = check_report(result)  # no tools used? invented pods?

if problem:
    retry_input = result.to_input_list() + [
        {"role": "user", "content": problem}
    ]

    result = await Runner.run(
        agent,
        retry_input,
        max_turns=remaining,
        run_config=run_config,
    )

It works. But my single for loop has now become a tool wrapper, a tool_use_behavior callback, and an outer loop that has to split the step budget across multiple runs. Each piece is simple; the whole is harder to read in one sitting.

The SDK didn’t make my guardrails impossible. It made me decide, for each one, whether the mistake should be the model’s problem or the developer’s.

My decision

I’m keeping the hand-written loop as the main implementation of this project, because its whole point is to show how an agent works with nothing hidden. Next to it, I’m adding an SDK version built on the design above, run against the same six tests.

I would reach for the SDK first when:

I need multiple agents with handoffs, where the SDK’s abstractions save real work.
I need tracing I can inspect visually rather than grepping JSONL files.
A team needs a shared vocabulary for agents instead of everyone’s own loop.
The deployment is production-facing, where fail-fast defaults are a feature.

I would keep a hand-written loop when the guardrails are the product: when the value lies in exactly how the agent recovers from a bad model output.

Three takeaways

Build it by hand once before adopting the framework. Every SDK parameter I understood quickly was one I had needed myself. The abstractions stop being magic and become design decisions you can evaluate.
Turn “can we migrate?” into tests. My guardrail tests turned a vague judgment call into a list of pass, fail, and needs-redesign verdicts. They also caught a difference ("auto" vs. absent) that I would never have noticed by reading documentation.
Measure, and pin the version. My first draft said unknown-tool handling couldn’t be migrated. Version 0.24.0 proved me wrong. Agent frameworks are moving fast; any claim about their behavior has an expiry date.

Tested with openai-agents 0.24.0, openai 3.35.1, mcp 1.30.0, and Python 3.13.

The hand-written agent, its guardrail tests, and the PoC script (poc_sdk_behaviors.py) are on GitHub:

View the code: agent_loop_demo_for_kubernetes

How I Stopped an LLM From Skipping the Investigation, Looping Forever, and Inventing Pods — Without LangChain

John Hu — Tue, 27 Oct 2026 07:46:34 +0000

An agent loop is just a for loop. The hard part is the invariants inside it.

LLMs misbehave in predictable ways: they skip steps, invent tools, loop forever, and fabricate evidence.
Every failure can be caught by a small, deterministic guardrail that replies with feedback instead of crashing.

3 a.m., and payment-service is down again

You know the routine. The pager fires, you open a terminal, and you start the ritual:

kubectl get pods -A
kubectl describe pod payment-service-6f7b8d9c0-ghi56 -n production
kubectl logs payment-service-6f7b8d9c0-ghi56 -n production --tail=50
kubectl get events -n production

Four commands, three context switches, and a lot of squinting before you spot the line that matters:

config validation failed: missing required field 'DB_PASSWORD'

The investigation follows a predictable shape, but it still takes a senior engineer’s instincts to run it efficiently. That combination — a repeatable process, with judgment calls at every step — is exactly what LLM agents should be good at.

So I built one. You ask it a question in plain English:

python agent.py "Which pods in the production namespace are unhealthy, and why?"

It decides which tools to call, digs through several rounds of evidence on its own, and returns a report like this:

payment-service-6f7b8d9c0-ghi56 — CrashLoopBackOff, 14 restarts
Evidence: logs show missing required field 'DB_PASSWORD'
Fix: restore the missing Secret entry, then roll out the deployment

inventory-service-59d8f7c6a-mno90 — ImagePullBackOff
Evidence: events show image v0.9.9-beta — manifest unknown
Fix: push the missing image or correct the tag

Every conclusion cites the tool result it came from.

One question in, a structured health report out — every step logged.

Getting a demo to produce that output once was easy. Getting it to produce that output reliably, even when the model misbehaves, was the real work. That is what this article is about.

Why I didn’t use a framework

An agent loop is, at its core, embarrassingly simple:

Send messages + tools to the LLM
Did the LLM return tool_calls?
- No → that text is the final answer, exit
- Yes → execute each tool, append results as role="tool", go to 1
Still not done after MAX_STEPS → force a summary

That fits in a single for loop. The hard part is not the loop itself.

The hard part is the invariants that have to hold inside the loop: every tool_call gets exactly one matching tool message, the loop always terminates, and the model never reports a fact it didn't observe.

Frameworks hide those invariants behind abstractions. That’s great — until production breaks and you need to know exactly what the model saw at step 4, and why it called list_pods for the third time.

So I wrote the loop by hand, using only the low-level openai and mcp SDKs. The entire agent fits in about 350 lines you can read in one sitting.

Architecture: three small pieces

Three files, three jobs:

agent.py — the loop and all of its guardrails.
mcp_client.py — a thin adapter. It launches the MCP server as a subprocess, converts MCP tool definitions to OpenAI’s format, and flattens tool results into strings.
mock_k8s_server.py — a FastMCP server simulating a 3-node, 12-pod cluster with planted failures: CrashLoopBackOff, ImagePullBackOff, and two Pending pods caused by resource pressure.

The key piece of glue is surprisingly small. The LLM only understands OpenAI’s tool format, and the server only speaks MCP:

{
    "type": "function",
    "function": {
        "name": tool.name,
        "description": tool.description or "",
        "parameters": tool.inputSchema,  # MCP JSON Schema -> OpenAI parameters
    },
}

Because the tool list is fetched from the server at runtime, switching from the mock to a real cluster is a single line in .env:

MCP_SERVER_COMMAND=npx -y mcp-server-kubernetes

agent.py doesn't change at all.

Seven ways the model misbehaves — and the guardrails that catch them

This is the heart of the project. Every guardrail below exists because a model actually did the thing it prevents.

1. Answering without investigating

Symptom: You ask for a health check, and the model confidently replies “All pods look healthy” — without calling a single tool. It has no idea what the cluster looks like. It is pattern-matching on what a health report usually says.

Fix: Force tool use on the first step only, then let the model decide freely after that:

tool_choice="required" if step == 1 else "auto",

Why only step 1? If you force tools on every step, the model can never stop and write the report. You’ve replaced “answers too early” with “never answers.”

The second layer: not every OpenAI-compatible provider honors tool_choice. So if the model still answers with zero tool calls, the harness pushes back:

if tool_calls_made == 0 and step < cfg["max_steps"]:
    messages.append({
        "role": "user",
        "content": "You have not called any tool yet. Investigate the "
                   "cluster with the provided tools before answering.",
    })
    continue

Lesson: don’t trust a single API parameter to enforce behavior across providers. Enforce it in your own code.

2. Hallucinating tools

Symptom: The model calls a tool that doesn’t exist, such as delete_pod.

That case is worth pausing on. A hallucinated tool name is not just a bug. In an ops context, it is a safety issue.

Fix: A whitelist built from what the server actually exposes. If the model asks for something else, the call is rejected — and the model is told what it can use:

if name not in known_tools:
    messages.append({
        "role": "tool",
        "tool_call_id": tc.id,
        "content": f"Unknown tool '{name}'. "
                   f"Available tools: {', '.join(sorted(known_tools))}.",
    })
    continue

3. Malformed arguments

Symptom: get_pod_logs({"name": "x", "tail": "fifty"}). A required namespace is missing, and tail is a string instead of an integer.

You could forward that to the server and let it fail. But that wastes a round trip, and server error messages are often vague.

Fix: Pre-flight validation. Every MCP tool already ships with a JSON Schema, so the harness compiles one Draft 2020–12 validator per tool at startup, and checks arguments before the call:

errors = sorted(
    validator.iter_errors(args),
    key=lambda e: [str(p) for p in e.absolute_path],
)
if errors:
    detail = "; ".join(
        f"{'.'.join(map(str, e.absolute_path)) or '<root>'}: {e.message}"
        for e in errors
    )
    # -> "Invalid arguments for get_pod_logs: <root>: 'namespace' is a
    #     required property; tail: 'fifty' is not of type 'integer'"

The model gets a precise, field-level correction it can act on in the next step.

One more detail: if a tool’s schema is itself invalid, the harness logs a warning and skips validation for that tool instead of crashing. A guardrail should never become a new single point of failure.

4. Looping forever

Symptom: The model calls list_pods(namespace="all"), gets the result, and calls list_pods(namespace="all") again. And again. Each loop burns tokens and moves nothing forward.

Fix: Signature-based duplicate detection. Each call is normalized to tool:json(args, sort_keys=True), so {"a": 1, "b": 2} and {"b": 2, "a": 1} count as the same call.

A duplicate is not executed again. The model gets the cached result plus a nudge:

[Duplicate call] You already called this tool with identical arguments;
the result has not changed. Base your next step on it, try a different
tool, or write the final report.

After MAX_REPEATS duplicates, the harness stops the investigation and forces a summary.

The protocol trap that bit me: my first instinct was to break as soon as the repeat limit was hit. That corrupts the conversation.

An assistant message can contain several tool_calls, and the API requires every one of them to have a matching role="tool" message. Break halfway through, and the next request is rejected outright.

So the harness sets a flag, finishes answering every call in the current round, and only then exits:

if repeat_count >= MAX_REPEATS:
    # Do not break mid-round: every tool_call in this assistant
    # message still needs a matching tool message, or the next
    # request would be rejected.
    force_summary = True

This is the kind of invariant a framework hides from you — until it doesn’t.

Break mid-round and the next request fails with a 400. Finish the round, then exit.

5. Inventing evidence

Symptom: The report is well written and confident — and it mentions ghost-app-12345-abcde, a pod that does not exist.

This is the most dangerous failure, because it looks correct. An on-call engineer may spend 20 minutes chasing a ghost.

Fix: A grounding check. Before accepting a final answer, the harness extracts every pod-like name from the report and verifies that it appeared in at least one tool result:

POD_NAME_RE = re.compile(r"\b[a-z0-9]+(?:-[a-z0-9]+){2,}\b")

def _invented_pods(answer: str, evidence: str) -> list[str]:
    return sorted(
        {n for n in POD_NAME_RE.findall(answer) if n not in evidence}
    )

If any are found, the model is told exactly which names to remove or correct.

The trade-offs, stated honestly:

The regex requires three or more hyphen-separated segments, matching Deployment-style names like payment-service-6f7b8d9c0-ghi56. That deliberately skips node-03 and kube-system to avoid false positives — which means node names aren't checked.
It only verifies names, not claims. “payment-service has 99 restarts” would pass.
The pushback happens at most once. A second wrong answer is accepted, because an unbounded correction loop is just a slower infinite loop.

That’s a narrow check. But it targets the highest-cost hallucination: sending a human after something that doesn’t exist.

6. Never converging

Symptom: The model keeps finding “one more thing” to check.

Fix: A hard MAX_STEPS budget (10 by default). When it runs out, the harness appends a final instruction and makes one more call with no tools parameter at all:

resp = await llm.chat.completions.create(
    model=cfg["model"], messages=messages
)

The model physically can’t call another tool. It has to summarize what it has already gathered. The loop is guaranteed to terminate.

7. Tool failures crashing the run

Symptom: The MCP subprocess hiccups, the arguments aren’t valid JSON, or the server returns isError. Naive code raises an exception, and the whole investigation is lost.

Fix: None of these raise. Each one becomes a tool result string sent back to the model:

except Exception as exc:
    return f"Tool call failed ({name}): {exc}"

The model can then decide to retry, pick another tool, or explain the gap in its report. A human engineer does the same thing when kubectl logs fails: you note it, and move on.

The principles underneath

The LLM proposes, the harness disposes.

Step back from the individual fixes, and a few principles emerge. I think they apply to almost any production agent.

1. The LLM proposes, the harness disposes

The model is a reasoning engine, not an executor. Every action it proposes passes through deterministic checks — whitelist, schema, deduplication, grounding — before it touches anything real.

2. Every guardrail is a feedback message, not an exception

None of the guards above crash the run. Each one rejects the action and tells the model how to fix it. Errors become observations, and observations are what the model reasons over. This turns a fragile pipeline into a self-correcting one.

3. Bound every correction

Every pushback has a budget: one evidence pushback, MAX_REPEATS duplicates, MAX_STEPS steps, and a final tool-less summary. Self-correction without a bound is a liability. With a bound, termination is guaranteed by construction — not by hoping the model behaves.

4. Tool descriptions are prompts

This one is easy to overlook. The docstrings on the MCP tools steer the model’s strategy:

The response already includes this pod’s events, so do not call get_events for the same pod; use get_pod_logs next if it is CrashLoopBackOff.

Do not use this for ImagePullBackOff or Pending pods: their containers never started, so no logs exist.

Those two sentences teach the model to skip wasted calls. If you own the MCP server, the tool documentation is your highest-leverage prompt.

5. Observability is not optional

Every run writes the complete conversation — system prompt, every tool call, every result, the final answer — to a JSONL trace. When the agent reaches a strange conclusion, you replay exactly what it saw. Trace-writing itself never raises, because diagnostics must not break the thing they’re diagnosing.

Testing a non-deterministic system deterministically

How do you write reliable tests for something built on top of an unpredictable LLM?

You don’t test the model. You test the harness against a model that misbehaves on purpose.

The test suite spins up a real local HTTP server that impersonates an OpenAI-compatible API. Each test scripts a “bad model,” response by response. The MCP side stays fully real: calls go through the actual client to the actual mock server.

A scripted fake LLM over real HTTP, a real agent loop, and a real MCP server — no API key required.

def test_unknown_tool_is_rejected(fake_llm):
    def script(body, n):
        if n == 1:
            return _completion(tool_calls=[
                _tool_call("c1", "delete_pod", {"name": "x"})
            ])
        joined = " ".join(_tool_contents(body))
        assert "Unknown tool 'delete_pod'" in joined
        assert "list_pods" in joined  # the whitelist is offered back
        return _completion(
            content="That tool does not exist; sticking to the whitelist."
        )

There are six tests, one per failure mode:

forced first-step tool use
pushback on a zero-tool answer
schema rejection
duplicate-driven forced summary
invented-pod pushback
unknown-tool rejection

None of them needs an API key, and all of them run in CI in seconds.

This is the part I’d most encourage other agent builders to copy: each guardrail should have a test that proves it fires.

Why this matters beyond a demo

Faster incident response. A four-command investigation with manual correlation becomes one question. The agent follows the same path a senior SRE would: overview first, then the unhealthy pods, then logs or events depending on the failure type.

Reports you can trust and audit. Every conclusion cites its source tool, the grounding check filters invented resources, and the full trace is on disk. When the report says “missing DB_PASSWORD”, you can see the exact log line it read.

Safe by default. The agent only has read-only tools, and the whitelist means it cannot call an action it wasn’t given — however confidently it hallucinates one.

No model lock-in. Any OpenAI-compatible endpoint works: OpenAI, DeepSeek, Qwen, or a local Ollama model for teams that can’t send cluster data to a third party. No data leaves your network unless you want it to.

Small enough to own. About 350 lines of agent code and no framework. Your team can read it in an afternoon, audit it before a security review, and change it without fighting abstractions.

A pattern, not just a tool. Nothing in the guardrail layer is specific to Kubernetes. Swap the MCP server, and the same loop can diagnose databases, CI pipelines, or cloud billing anomalies. Any read-only investigation over structured tools fits.

What it doesn’t do (yet)

Being honest about limits is part of building something people can trust:

The grounding check is narrow. It verifies pod names, not numbers, node names, or causal claims.
The cluster is simulated by default. The mock makes the demo reproducible. A real cluster is one config line away, but expect messier data.
Context grows with every step. For long investigations, history compression or summarization would be needed.
It’s read-only. Remediation actions like restarts or rollbacks would require human-in-the-loop approval, and I deliberately haven’t built that yet.
The tool list is simple. For example, if one container starts but fails to pass the health check (the process would be still running), the agent may not be able to detect it.

Next on the roadmap: schema-validated structured reports, an offline evaluation set built from saved traces, and approval-gated write actions.

Back to 3 a.m.

The pager still fires. But now the first thing you run is a single question — and in the time it takes to open your laptop, you have a report that names the broken pods, explains why each is broken, cites the evidence, and suggests a fix.

The LLM does the reasoning. A few hundred lines of deterministic code keep it honest.

rhuab / agent_loop_demo_for_kubernetes

agent_loop_demo_for_kubernetes

Diagnosing unhealthy Kubernetes Pods usually means manually chaining kubectl get pods, describe, logs, and get events until the root cause shows up. This project automates that investigation with a hand-written Agent Loop — no agent framework, just the low-level openai and mcp SDKs — so an LLM calls Kubernetes tools over MCP, digs in over multiple rounds on its own, and produces a structured cluster health report.

📦 Core Features

Hand-written Agent Loop, zero framework — no LangChain / AutoGen. The core loop in run_agent() (agent.py) is a single for loop: send messages + tools to the LLM, execute any tool_calls via MCP, append results as role="tool" messages, and repeat until the model answers in plain text.
Standard MCP tool calling — mcp_client.py launches the MCP Server as a stdio subprocess, converts MCP tool definitions into OpenAI's tools format (inputSchema → parameters)…

View on GitHub

Clone it, run python -m pytest with no API key, then point it at your own cluster. If it catches something interesting — or misbehaves in a way my guardrails don't cover — I'd love to hear about it.