DEV Community: Shivansh Karan

What Actually Makes a Chatbot an Agent?

Shivansh Karan — Fri, 17 Jul 2026 03:39:00 +0000

About this series

Agentic AI from First Principles is a hands-on series where we build an AI agent from scratch, without relying on frameworks. Instead of starting with abstractions, every abstraction appears only after we've hit the problem it solves.

If you're joining from the beginning, welcome. If not, you can find the previous chapters in the series here.

Everyone seems to be building AI agents these days.

But nobody seems to agree on what an agent actually is.

Ask five engineers and you'll get five answers: "an LLM with tools," "a loop," "something with memory," "basically LangChain." All half right. None of them satisfying.

So instead of starting with a definition, I decided to start with the dumbest possible thing I could build, and see what I ran into.

I'll use the OpenAI SDK. Whether you're pointing it at OpenAI or a local Ollama model, the code looks almost identical.

from openai import AsyncOpenAI
import asyncio

client = AsyncOpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # can be any non-empty string
)


async def main():
    while True:
        user_message = input("User > ")

        response = await client.chat.completions.create(
            model="gemma4:e2b",
            messages=[{"role": "user", "content": user_message}],
            reasoning_effort="none",
        )

        print(f"Assistant > {response.choices[0].message.content}")


if __name__ == "__main__":
    asyncio.run(main())

I ran it, and it worked about like I expected:

User > Hi I am Shivansh
Assistant > Hi Shivansh! It's nice to meet you. How can I help you today? 😊

Twenty lines of code. A working chatbot. Kind of anticlimactic, honestly.

So I poked at it a little more.

User > What is my name?
Assistant > As an AI, I do not have access to your personal information, so I do not know your name.

Wait, what?

It knew my name ten seconds ago. Now it's acting like we've never met.

My first instinct was that it forgot. That's the natural way to describe it. But "forgot" implies it once knew something and then lost it, like walking out of a room and losing the thread. Did that actually happen here, or was something else going on?

I didn't want to guess, so I went and looked at exactly what I was sending.

response = await client.chat.completions.create(
    model="gemma4:e2b",
    messages=[{"role": "user", "content": user_message}],
)

And there it was. Every single time I called this, I built a brand-new messages list from scratch, containing nothing but whatever had just been typed. When I said "Hi I am Shivansh," the model received:

[{"role": "user", "content": "Hi I am Shivansh"}]

Ten seconds later, when I asked "What is my name?", the model received:

[{"role": "user", "content": "What is my name?"}]

I stared at that second payload for a second longer than I'd like to admit. The first message wasn't trimmed or summarized or forgotten in any active sense. It just wasn't there. I had never sent it a second time.

From the model's point of view, these weren't two turns in one conversation. They were two total strangers walking up and asking unrelated questions. There was no forgetting, because there was never anything to forget in the first place. Nothing carried over from one call to the next.

Which gave me my first real insight, and it landed harder than I expected for something this simple:

The model only knows what you send it.

This is what people mean when they say LLMs are stateless. The model isn't a person sitting there, quietly tracking your conversation between messages. It's a function. Call it with an input, get an output, and it has no idea it was ever called before.

So the obvious follow-up question in my head was: if the model has no memory, how does ChatGPT clearly remember what I said five messages ago?

The answer turned out to be almost anticlimactic once I saw it. The model isn't remembering anything. The application is. It's quietly collecting the conversation and re-sending the whole thing back to the model, every single time, as if for the first time.

I hadn't set out to build "memory." I was just trying to make the forgetting stop. Once I knew why it was forgetting though, the fix basically wrote itself. If the model only knows what I send it, I just need to send it more.

Instead of this:

[{"role": "user", "content": "What is my name?"}]

I sent this:

[
  {"role": "user", "content": "Hi I am Shivansh"},
  {"role": "assistant", "content": "Hi Shivansh! It's nice to meet you."},
  {"role": "user", "content": "What is my name?"}
]

Now the model could see the entire exchange, not just the latest fragment of it.

messages = []

while True:
    user_message = input("User > ")

    messages.append({"role": "user", "content": user_message})

    response = await client.chat.completions.create(
        model="gemma4:e2b",
        messages=messages,
    )

    assistant_message = response.choices[0].message.content
    messages.append({"role": "assistant", "content": assistant_message})

    print(f"Assistant > {assistant_message}")

User > Hi I am Shivansh
Assistant > Hi Shivansh!

User > What is my name?
Assistant > Your name is Shivansh.

There it is.

But notice what I didn't do. I didn't give the model memory. The model is exactly as stateless as it was thirty seconds ago, still a pure function, still forgetting everything the instant the response comes back.

What changed is that my application now keeps a growing transcript and replays it on every turn. The illusion of memory isn't happening inside the model at all. It's happening in a Python list sitting in my process.

I used to assume, without ever really examining it, that when ChatGPT "remembers" something from earlier in a conversation, the model is doing something clever. It isn't. Some code, somewhere, is just handing it the same conversation over and over, one message longer each time.

I didn't sit down that day to implement conversation memory. I sat down annoyed that my chatbot had amnesia. The memory system was just what was left over after I fixed that.

Okay. So now my chatbot has a memory of sorts.

Was I done? Did I have an agent yet?

Not even close. The next thing I tried made that painfully obvious.

The Chatbot Hits a Wall

I created a file in my project folder:

README.md

This project is a simple AI chatbot.

Then I asked my memory-having chatbot a completely reasonable question.

User > Summarize the contents of README.md
Assistant > Please provide the content of the README.md file.

Huh.

It's not that the model is bad at summarizing. It's genuinely very good at that. The problem was more basic, and once I saw it, kind of obvious. The model had never seen the file. It lives on my machine. The model lives wherever it lives. As far as it was concerned, README.md was just six characters that happened to appear in a sentence.

I had given it memory of the conversation. I hadn't given it anything about the world.

Every capability I take for granted with tools like Claude Code or ChatGPT plugins, reading files, running commands, checking the weather, has to come from somewhere. The model can't reach out and grab it. Somebody has to hand it over.

This is the wall every chatbot eventually hits. It can talk about anything, but it can't do anything.

Giving It Hands

If the model couldn't read the file itself, fine. I'd write a function that could, and describe that function to the model.

def read_file(path: str) -> str:
    """Read a UTF-8 text file and return its contents."""
    with open(path, "r", encoding="utf-8") as f:
        return f.read()

I also needed to tell the model this function exists, since it can't inspect my codebase either. I did that with a tool schema, basically a spec sheet describing the function's name, purpose, and arguments:

tools = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read a text file from disk.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {"type": "string", "description": "Path to the file"}
                },
                "required": ["path"],
            },
        },
    }
]

response = await client.chat.completions.create(
    model="gemma4:e2b",
    messages=messages,
    tools=tools,
)

I expected the model to just read the file. Run the function. Hand back the contents.

That is not what happened. Here's what actually came back:

{
  "content": "",
  "role": "assistant",
  "tool_calls": [
    {
      "id": "call_teaunoqm",
      "type": "function",
      "function": {
        "name": "read_file",
        "arguments": {"path": "README.md"}
      }
    }
  ]
}

I remember staring at this for a second, genuinely confused. There's no file content anywhere in here. The model didn't call read_file(). It didn't even attempt to. What it produced instead reads more like a memo: someone please call read_file("README.md") and get back to me.

This is where I ran into probably the single biggest misconception I had going into agent-building. The model never executes a tool. It only decides that a tool should be called, and with what arguments.

The actual execution, opening the file, running the command, hitting the API, has to happen entirely in my code. The model is more like a manager scribbling a request on a sticky note. My runtime is the one who actually gets up and does it.

So that's what I did next. Pulled the tool call out of the response, ran the real Python function, fed the result back in.

tool_fns = {"read_file": read_file}

async def main():
    messages = []
    while True:
        user_message = input("User > ")
        if user_message == "/bye":
            break

        messages.append({"role": "user", "content": user_message})

        response = await client.chat.completions.create(
            model="gemma4:e2b",
            messages=messages,
            tools=tools,
            reasoning_effort="none",
        )

        messages.append(response.choices[0].message)

        if response.choices[0].finish_reason == "tool_calls":
            for tc in response.choices[0].message.tool_calls:
                tool_fn = tool_fns[tc.function.name]
                tool_args = json.loads(tc.function.arguments)
                tool_result = await tool_fn(**tool_args)

                messages.append({"role": "tool", "content": tool_result})

                response = await client.chat.completions.create(
                    model="gemma4:e2b",
                    messages=messages,
                    tools=tools,
                    reasoning_effort="none",
                )

        print(f"Assistant > {response.choices[0].message.content}")

Now the whole thing actually worked. The model asked for the file, my code read it, I handed the contents back, and the model summarized it like it had the file all along.

I hadn't set out to build "tool calling" either. I was just trying to get one file into the model's hands. Somewhere along the way I'd built the exact mechanism every agent framework calls tool use.

At this point I had memory, tools, and tool execution. The model could inspect files, reason about what it found, and answer using information it never had at training time.

This felt like an agent. Genuinely close.

But there was a crack in the foundation, and it showed up the moment I asked for something slightly harder.

One Tool Call Isn't Enough

Say my folder didn't just have a README.md. It also had a CONTRIBUTING.md, a CHANGELOG.md, maybe a few more. Instead of asking about one file, I asked something more realistic:

Read the docs in this folder and write me a summary.

I ran it, and the model did exactly what I'd trained it to do in the previous section. It called a tool.

list_directory() → ["README.md", "CONTRIBUTING.md", "CHANGELOG.md"]

Great, I thought. Now it knows which files exist.

Except it doesn't know what's in any of them. It knows three filenames. That's it. To actually summarize anything, it needs to read each one, which means calling read_file() three more times.

And here's where I hit the wall for real. My runtime didn't let it do that.

Go look at the shape of the code I'd written:

Tool
  ↓
Answer

One shot. The model gets exactly one chance to call a tool. I run it, hand back the result, and then immediately force it to a final answer, ready or not.

So that's exactly what happened. The model called list_directory(), got its three filenames back, and then had nowhere to go. It wasn't allowed to act on what it had just learned. It had to answer right then, holding nothing but a list of names and zero idea what any of them contained.

I sat there looking at the output for a minute before it actually clicked. This wasn't a prompting problem. It wasn't a model-capability problem. The model had figured out exactly the right next step, go read those three files, and my architecture simply refused to let it take that step. The bottleneck wasn't the model anymore. It was the code I'd written around it.

The one-shot ceiling wasn't a bug I'd introduced by accident either. It was baked into the shape of the code. I had hardcoded one tool call, then answer, without ever asking whether one would be enough. Turns out, for almost anything interesting, it isn't.

The Loop

If the model needs to act, observe the result, and act again, as many times as the task actually requires, then the architecture can't be a straight line. It has to be a loop.

while True:
    response = call_model()

    if not tool_calls:
        return answer

    execute_tools()

That's genuinely the whole idea. Keep calling the model. If it wants to use a tool, run the tool, feed the result back, and let it decide what to do next. Including deciding it needs another tool, or that it finally has enough to answer.

I want to be honest about how I got here, because I think it matters. I didn't set out to build anything with a name. I wasn't chasing a pattern from a paper. I was solving one specific, annoying problem: the model needed to take a second action based on what the first one revealed, and my code didn't let it. The loop was just the shape that problem forced onto the code.

It was only in hindsight, looking at what I'd built, that I realized it already had a name. This is the ReAct pattern (Reason + Act, not to be confused with React.js), and it looks pretty close to what I'd just backed into:

Reason → Act → Observe → Reason → Act → Observe → ...

I didn't implement ReAct. I rediscovered it, one dead end at a time, which as far as I can tell is roughly how it got discovered the first time too. Nobody sits down and decides today I will invent a reasoning loop. You sit down annoyed that your chatbot can only do one thing, and a loop is what falls out.

That's been the pattern through this whole article, now that I look back at it. I never woke up wanting to build memory, or tools, or ReAct. I woke up with a chatbot that forgot my name, then one that couldn't read a file, then one that got stuck after a single tool call. Every piece of "architecture" here is really just scar tissue from a specific, annoying failure.

And with that, my twenty-line chatbot had quietly crossed a line. It's no longer just something that talks. It can decide what to do, do it, look at what happened, and decide again.

That's the difference between a chatbot and an agent, as far as I can tell. Not a framework, not a new model. A loop, and the willingness to let the model act more than once.

If you want to see the full implementation, including the parts trimmed out of this write-up for length, the code is here:

SpaceTesla / react-agent

A minimal ReAct agent built from first principles.

react-agent

A minimal ReAct agent built from first principles.

This repository is the companion project for the Agentic AI from First Principles series, where we build an AI agent step by step without relying on frameworks like LangChain or LangGraph.

Instead of starting with abstractions, we start with problems:

A chatbot that forgets everything

A chatbot that can't interact with the world

A chatbot that can only take a single action

And by solving those problems one at a time, we eventually arrive at a ReAct agent.

If you came here from the article, welcome 👋

This repository contains the exact implementation discussed in the series, along with the code needed to experiment, modify, and extend it yourself.

What Is This?

At its core, this project is a simple agent runtime built on top of the OpenAI-compatible API.

The agent can:

Maintain conversation history

Call tools

Execute tool requests

Observe…

View on GitHub

Here's the thing I keep coming back to though.

At the start of this article, my chatbot forgot everything. Every message, gone the moment it was sent. That was clearly broken, so I fixed it. I made it remember everything.

But remembering everything isn't actually the finish line. It just moves the problem somewhere else. Right now, every tool result my agent has ever produced gets appended to messages and never removed. Ask it to read ten files instead of three, and all ten now live in the conversation permanently, resent to the model on every single subsequent call whether they're still relevant or not.

A chatbot that forgets everything is broken. Turns out an agent that forgets nothing is also broken, just more slowly and more expensively. Context windows fill up. Costs climb. Latency climbs with them. Eventually the model has to reason around a pile of stuff it doesn't need anymore, which hurts the quality of its answers too.

So that's the next wall. Not "can it act more than once," we just solved that. It's "can it act more than once without drowning in its own history."

Underneath both of those specific problems, though, there's a bigger one, and it's the real reason I wanted to write this article the way I did.

I didn't sit down planning to build memory. I didn't sit down planning to build tool calling. I definitely didn't sit down planning to build a ReAct agent. Every piece of this architecture exists because something broke, and I fixed the thing that broke. Nothing more.

Memory showed up because the model forgot. Tools showed up because the model couldn't touch anything outside the conversation. The loop showed up because acting once wasn't enough.

Nobody handed me these abstractions. I didn't start with them. I started with problems, and the abstractions were just what was left standing after I solved them.

That's usually where the good ones come from.

Inside ChatGPT: What Really Happens Inside an LLM?

Shivansh Karan — Fri, 17 Jul 2026 02:39:00 +0000

About this series

Agentic AI from First Principles is a hands-on series where we build an AI agent from scratch, without relying on frameworks. Instead of starting with abstractions, every abstraction appears only after we've hit the problem it solves.

If you're joining from the beginning, welcome. If not, you can find the previous chapters in the series here.

You ask ChatGPT a question.

A few seconds pass.

And suddenly, a well-structured answer appears on your screen.

It feels almost magical.

The model seems to understand your question, reason about it, and respond intelligently.

But what actually happens inside?

Before we start building AI agents, we need to understand the engine that powers them: the Large Language Model (LLM).

Instead of diving into equations and research papers, let's follow a single prompt as it travels through the model.

Our prompt:

"Explain how a large language model actually works."

Let's see where it goes.

1. The Gate: Tokenization

Your prompt arrives at the entrance of the model.

But there's a problem.

LLMs don't understand text.

At least, not directly.

They only understand tokens.

A tokenizer sits at the gate and breaks your sentence into smaller pieces from the model's vocabulary.

For example:

Explain how a large language model actually works.

becomes:

["Explain", " how", " a", " large", " language",
 " model", " actually", " works", "."]

Each token is then converted into a numeric ID.

Token	Ex	plain	how	a	large	language	model	actually	works	.
Token ID	849	21435	1268	264	3544	4221	1646	3604	4375	13

The model never sees your original sentence again.

From this point onward, it only sees numbers.

Why tokenization exists

Gives the model a fixed vocabulary
Handles rare words efficiently
Converts messy human language into predictable units

At this stage, the model doesn't understand meaning.

It simply has a sequence of token IDs.

The journey continues.

2. The Embedding Hall

The token IDs now enter a massive chamber called the embedding layer.

Here, every token is transformed into a vector.

Something like:

"model"
↓
[0.18, -0.42, 0.07, ...]

Think of embeddings as coordinates in a giant semantic map.

Words with similar meanings tend to end up near each other.

But there's still a limitation.

The embedding for "model" doesn't yet know whether we're talking about:

a fashion model
a machine learning model
a mathematical model
a 3D model

The token has some notion of meaning.

But it doesn't yet know the meaning in this sentence.

For that, we need context.

And context lives inside the Transformer.

3. The Transformer Core

This is where the real magic happens.

Imagine a giant room where every token can look at every other token.

The token "model" starts asking questions:

Which other tokens matter to me?

It notices:

"large"
"language"
"actually"
"works"

Those surrounding words provide clues.

Suddenly, the meaning becomes clear.

We're talking about a large language model, not a fashion model.

This process is powered by something called self-attention.

You don't need to understand the math yet.

The intuition is enough:

Every token continuously gathers information from other relevant tokens.

As information flows between them, the sentence becomes clearer and clearer.

What the Transformer accomplishes

Resolves ambiguity
Understands relationships
Captures context
Builds a representation of meaning

After several transformer layers, the model has a rich understanding of your prompt.

Now it's ready to respond.

4. The Response Factory

This is the part that surprises most people.

The model does not generate an entire paragraph at once.

It generates one token at a time.

That's it.

The process looks like this:

Look at the prompt
Predict the most likely next token
Add it to the conversation
Repeat

For example:

"A"

becomes

"A large"

then

"A large language"

then

"A large language model"

and so on.

Every newly generated token is fed back into the model.

The model repeatedly asks:

Given everything I've seen so far, what token should come next?

Thousands of times per response.

This simple loop is responsible for everything:

explanations
stories
code
essays
conversations

Everything emerges from repeated next-token prediction.

5. The Exit Gate

Eventually the model decides it has finished.

Now we have a sequence of generated tokens:

["A", " large", " language", " model", " is", ...]

These tokens pass through the detokenizer.

The pieces are stitched back together into readable text.

The result becomes:

"A large language model is..."

And that's the answer you see on your screen.

6. The Entire Journey

Let's compress everything into a single flow:

Step 1: Tokenization

Text → Tokens

Step 2: Embeddings

Tokens → Vectors

Step 3: Transformer

Vectors → Contextual Understanding

Step 4: Generation

Predict the next token repeatedly

Step 5: Detokenization

Tokens → Human-readable text

Step 6: Final Response

Answer appears on your screen

What Most People Miss

At this point, something important should stand out.

Throughout this entire journey, we never gave the model:

memory
tools
a browser
a calculator
access to files
the ability to take actions

All it did was:

Process tokens
Understand context
Predict the next token

That's it.

An LLM is incredibly powerful.

But by itself, it's still just a next-token prediction engine.

This realization becomes extremely important once we start building AI agents.

Because many of the capabilities we associate with modern AI systems don't come from the LLM alone.

They come from the systems built around it.

Final Recap

If you remember only one sentence from this article, make it this one:

An LLM converts text into tokens, uses attention to understand context, and generates a response one token at a time.

Or even shorter:

Text → Tokens → Context → Prediction → Text

That's the core idea.

And now that we understand what an LLM actually does, we can start exploring what it doesn't do.

In the next article, we'll build a chatbot from scratch and discover a surprising limitation that every LLM has.

One that ultimately leads us toward agents.