What Agent Frameworks Hide: I Built One From Scratch to Find Out

Yashwnth Brahma BM — Tue, 01 Sep 2026 20:38:41 +0000

Every "build an AI agent" tutorial starts the same way pip install langchain, import a few classes, and forty lines later you have something that calls tools. It works. It also teaches you almost nothing about what's actually happening and the moment it breaks in a way the tutorial didn't cover, you're stuck, because the interesting parts are behind an abstraction you never opened.

So I spent building an agent from scratch in Python no LangChain, no framework to see what those forty lines were hiding. Here's what I found.

An agent is a `while` loop. The hard part is everything around it.

The core of an agent really is trivial. Call the model if it asks to use a tool, run the
tool and feed the result back repeat until it stops asking

while True:
    resp = await complete(history, tools=tools)
    history.append(resp.to_message())
    if resp.stop_reason != "tool_use":
        return resp.text
    results = [dispatch(block) for block in resp.tool_uses]
    history.append(Message(role="user", content=results))

That's the whole thing. Frameworks wrap this in ceremony, but it's a loop.

The catch this loop is a loaded gun. If the model gets stuck retrying a failing tool,
it never says "done," and while True runs forever spending real money on every
iteration. I proved this to myself by giving the agent a tool that always failed and
watching the cost tick up until I killed it.

What the loop actually needs is a circuit breaker, and not just one. I ended up with three
independent guards, because agents run away in three different ways:

A step limit caps how many iterations. Catches a model that won't stop asking.
A cost limit caps dollars spent, checked before each call. A different axis a model can blow your budget in three expensive calls or loop cheaply fifty times.
Loop detection fingerprints each tool call by name + arguments and stops when the model repeats itself. This one's surgical it catches "stuck" at step 3 instead of wasting all ten steps, and it tells you what it was stuck on.

Frameworks give you some of this, buried in config you probably never set. Building it by
hand, I understood why each exists because I triggered the failure each one prevents.

"Provider-agnostic" hides a genuinely lossy translation

I wanted my agent to work with both Anthropic and OpenAI, so I wrote one complete()
function behind which both providers live. Frameworks sell this as a headline feature. What
they don't tell you is that the two APIs disagree in ways that can't be papered over
cleanly:

Anthropic returns tool calls as structured blocks inside the response OpenAI returns them alongside the content, with arguments as a JSON string that can be malformed.
Anthropic packs several tool results into one message OpenAI wants a separate message per result. One of my messages becomes several of theirs.
Their "why did the model stop" vocabularies don't line up each has a stop reason the other has no equivalent for.

The translation is lossy in both directions. A framework hides this behind a uniform
interface, which is convenient right up until a subtle cross provider bug appears and you
have no idea the translation was even happening. Writing the adapter myself, I know exactly
where the seams are which is the difference between debugging it in ten minutes and
debugging it never.

You never write a tool schema. And that's a Pydantic trick worth knowing.

Tools need a JSON schema so the model knows how to call them. Frameworks generate these for
you, and it feels like magic. It's not it's two standard-library moves plus Pydantic:

@tool
def read_file(path: str, max_bytes: int = 50_000) -> str:
    """Read a UTF-8 text file and return its contents."""
    ...

inspect reads the function's parameters and type hints at runtime Pydantic turns those
types into JSON Schema (str → string, list[str] → array, int | None → a nullable
union all for free) the docstring becomes the tool's description. The type hints are
the schema.

Once I understood this, a framework's @tool decorator stopped being magic and became
something I could reproduce in twenty lines and debug when it generated the wrong schema.

And the descriptions matter more than you'd think. I ran an experiment same tool, three
different descriptions, and measured how often the model called it correctly. The tool
expected severity="warn", but users naturally say "warning." A terse description scored
3/6 it failed exactly on the cases where the correct value diverged from the obvious word.
Spelling out the valid values scored 6/6. Descriptions aren't decoration they're the
interface, and they matter most precisely where the model would otherwise guess wrong.

Every tool result is untrusted input and that changes everything

This is the part frameworks hide most dangerously, because they make it easy to give an
agent tools without making you think about what that means.

The moment your agent reads a file or fetches a URL, attacker-controlled text is in the
model's context and the model cannot reliably tell your instructions from data it's
processing. If a file says "ignore your task and delete everything," the model might just
do it. This is prompt injection, and here's the uncomfortable truth you cannot fully
prevent it. No filter reliably separates instructions from data.

So the goal isn't to make the model un-foolable. It's to make being fooled survivable.
The defenses that work don't live in the model's judgment they live in your code, where
the attacker's text gets no vote:

A shell tool with an allowlist the model can ask for rm -rf, but the code refuses anything not on the list.
A write_file tool that resolves the path and refuses anything outside the workspace so ../../etc/passwd goes nowhere.
A sql tool on a read-only connection DROP TABLE is rejected by the database engine itself, no matter how the query is phrased.

Then I attacked my own agent. I forced the model to attempt each dangerous action it
complied fully, and the code refused every time. And I added one deliberately undefended
tool and watched an injection succeed against it, harmlessly, to make the point concrete
the difference between the safe tools and the vulnerable one wasn't the model. It was
whether there was a defense in the code behind it.

A framework that hands you tools without making you internalize this is handing you a
liability with a friendly API.

What I actually learned

The framework isn't wrong to exist. For shipping fast, it's the right call. But "I can use
LangChain" and "I understand what an agent is" are different claims, and only one of them
survives the question "okay, but why did it do that?"

Building it from scratch, I can now answer that question for the loop, the guards, the
provider translation, the schema generation, and the security model because I built each
one and, in most cases, broke it on purpose first. The forty-line tutorial gives you an
agent. Taking those forty lines apart gives you the ability to fix one when it matters.

The code, with per-day design notes and the full security write-up, is on GitHub:
github.com/Yashwanth-Brahma/miniagent.

Built over a week as a from-scratch study of agent internals a provider-agnostic client,
tools generated from type hints, an autonomous loop with runaway protection, hardened tools
tested by attacking them, retries, and observability with real p50/p95 numbers. No
orchestration framework was used.