DEV Community: Adi

how to do agentic evals

Adi — Sat, 18 Jul 2026 18:26:14 +0000

When your agent schedules an interview, sends a message to a candidate, or updates an application status, you can't evaluate it the way you test a function. The agent changes the world. Testing whether it changed the world correctly means building a world where it can safely do that.

Standard evals don't solve this. Static test environments miss how agents behave when conditions shift. Single runs miss non-determinism, even at temperature 0. Single-turn tests miss the steep performance drop when conversations go multi-turn.

In January, Anthropic published Demystifying Evals for AI Agents. I like their engineering posts, and this one gives a solid overview: terminology, grader types, an implementation roadmap. This article goes deeper into one corner: building the eval system for conversational agents that take real actions. I wrote about why model capability doesn't translate to agent reliability. This is about what it takes to get there.

evals

When most people say "evals," they mean a unit test for model behavior: give it an input, check the output. The checking can be deterministic: string matching or a rule. Or you can point another LLM, often called a judge, at the output and ask whether it's close enough to the expected answer. Platforms like Langfuse and Pydantic Logfire offer this as a playground: see a completion, tweak the input prompt, compare outputs.

We run these too. We use them to test safety: send a hundred off-topic or dangerous messages (politics, religion, protected characteristics) and verify the agent redirects every one. We use them to test FAQ accuracy: clients define how the agent should answer specific topics, and the eval tests whether it does across different phrasings. We use them to iterate on prompts: change the prompt, run the same batch, compare what changed in the completions.

These evals are useful, but they evaluate the surface. They tell you what the agent said. They don't tell you if it's true. An agent can say "your interview is confirmed for tomorrow" and sound perfectly helpful, while no interview was ever scheduled in the database. An agent that takes actions across multiple turns, changes state, and reasons internally about what to do next needs deeper instruments.

the harness

To know whether the agent actually scheduled that interview, you need to run it in a world where it can take actions safely. Not a mock, not a stub: a world that behaves like production without being production. That means the agent must run its full production logic. If you mock the agent, you're not testing the agent. If you mock the world too aggressively, you're testing behavior in a world that doesn't exist.

The agent doesn't need to know it's being tested. As long as the interfaces behave the same way, it doesn't matter what's behind them. A recent paper on agent evaluation calls this the contract principle: correctness is defined by the interface, not the implementation. Swap the providers: messages go to memory instead of the real messaging service, external API calls hit mocks that honor the same contracts, time is injected so the agent perceives the correct historical moment.

The harness should never run in production, but the data inside it should be as close to production as possible. That can mean exporting real context from production, enough for the agent to behave correctly, with all PII stripped. Or it can mean generating synthetic scenarios that are realistic enough to surface the same failures real users would. Either way, what the agent sees inside the harness should be indistinguishable from what it sees in production. Save this data as a frozen template: the starting scenario, the user, their conversation history, everything the agent needs as context.

Once you have a template, you need to make sure each run starts clean. If the agent schedules an interview during one run, the next run can't start with that interview already on the calendar. For each eval run, you clone the template into a disposable copy. The agent mutates the clone freely, and the clone is discarded after. Run it ten times, and all ten start from the same state.

If your agent has time-dependent logic, each clone also needs the correct historical moment injected. An agent making scheduling decisions needs to think it's March 3rd at 2pm, not whenever the eval runs. And anything that would reach the outside world, notifications, webhooks, downstream processes, must be suppressed. Otherwise an eval run could send a real message or trigger a real workflow. If you run evals concurrently, each run also needs its own isolated providers so one run's state doesn't leak into another.

Anthropic's guide describes all of the above in one bullet: "build a robust eval harness." In practice, it's the most engineering-intensive part of the whole system. Once it exists, it unlocks deeper eval modes.

replay

The simplest thing you can do with the harness is replay a situation. Take a specific moment, a user, a message, the state around it, freeze it as a template, and run the agent against it. Did it do the right thing?

Now run it 3, 5, 8 times. Agents are non-deterministic, so running the same situation repeatedly is how you distinguish reliability from luck. Did the agent succeed at least once? That's pass@k. Did it succeed every single time? That's pass^k. For reliable agents in production, pass^k is what we should worry about.

I did this when testing role coherence under adversarial pressure: freeze a conversation turn, replay it fifty times, observe the variance. The same principle applies broadly. It works for failures and for happy paths. A passing result that holds across every run is a pattern you can trust. A failure that repeats is a pattern you can fix.

Replay is a simple and important tool for testing reliability under a specific situation. But real users don't send one message and stop. They have a conversation.

simulation

Simulation is a full multi-turn conversation where an LLM plays the user. The simulated user needs a persona, a scenario, and a goal. A study from Microsoft and Salesforce Research tested 15 models across over 200,000 simulated conversations and found that performance drops from around 90% in single-turn to 65% in multi-turn, a 25-point gap consistent from small open-source models to frontier. When LLMs take a wrong turn, they don't recover. Simulation is how you test whether your agent does.

The simulated user's persona matters as much as the task. Researchers at Collinear AI found 2 to 30% performance degradation when they varied user traits: impatience, incoherence, skepticism. Standard benchmarks use cooperative, patient users, but real users are often neither. An eager user surfaces different failures than a skeptical one.

Where do the scenarios come from? When I tested role coherence with Promptfoo, the red-teaming tool generated attack scenarios from a description of the agent's purpose. There are many commercial platforms that offer generation of simulation scenarios using your agent's system prompt as input. Frankly, you can achieve more with any reasoning model by providing the full picture: system prompt, tool definitions, domain context, and the specific constraints your agent operates under.

The goal of simulation can be structured or open-ended. A simulated user can take the conversation in directions no test script would. Fully open-ended simulation is the ideal, but in practice most systems sit somewhere in between: scenarios are predefined, personas configured in advance, evaluation against an expected outcome. My agents operate in a bounded domain with defined goals and a constrained conversation space. If yours do too, structured simulation with varied personas covers most of what you need to test. What it won't catch is what falls outside your scenario set, and pushing toward more open-ended exploration over time is the right direction.

the judge

Just as an agent in production should leave traces you can review, so should an agent in the harness. Not only the conversation, but how it changed the world and what it was thinking when it did.

Capture the full state before and after each run, and diff them. A recent paper calls these state-diff contracts. The diff is your ground truth: what actually changed, and what changed that shouldn't have. Some of this you can verify deterministically, without an LLM: was the status updated, was the event created, does the diff match what was expected? That's the most reliable evaluation you can build.

For more complex outcomes, you need a judge. The standard approach, LLM-as-a-Judge, gives an LLM the conversation and asks it to score. But that's evaluating the final result without seeing how the agent got there. A Meta and KAUST research group proposed Agent-as-a-Judge: give the evaluator the intermediate reasoning and the state changes, not only the conversation. They found 90% alignment with human consensus, versus 70% without. We do the same. The more context the judge has, the better it evaluates.

the flywheel

The judge evaluates a simulation run and reveals a failure. You see the state diff, you see the reasoning. Now what?

Promote that failure to a template. Trim the conversation to the critical turn, freeze the state. Now you can replay that exact situation with repetitions to confirm whether the failure is a pattern or a one-off. If it's a pattern, you fix it. After you fix it, the template stays. It becomes a regression test: every future change to the agent gets tested against this situation automatically.

This is how the system compounds: simulation discovers failures, the judge and human review explain them, and replay locks them down.

Over time, the library of situations your agent must handle grows with every failure you find and every fix you ship. Now you have the tools to ship reliably.

riding the wave of ai

Adi — Sat, 18 Jul 2026 18:18:10 +0000

I remember when coding interviews happened at a whiteboard, no computer, no internet, just a marker and whatever you could hold in your head. When AI tools arrived, using one in an interview was the red flag. Now the red flag is the candidate who doesn't use AI enough. In about two years, the same tool went from forbidden to expected.

And it only gets faster. Every few months another wave rolls in, a model out of some lab, a tool that does something it couldn't last year, and it resets what counts as normal. You ride it, or you let it break over you. Plenty of people imagine a third option, waiting on the beach with their arms crossed until the water goes calm, but the water never goes calm, and it was never going to wait for them.

all in

I've decided to ride it, in the most literal way I have. All of my code is written by AI now (is it mine at this point?), and somewhere along the way I stopped treating that as a threat to be managed and started treating it as leverage to be spent. I know the engineers who went the other way, who made a personality out of dismissing the tools, and the tools got better anyway while they just fell further behind.

Going all in wasn't one decision; it's one I make again every few weeks. A better model ships, and I rebuild a working agent on top of it instead of staying on the old version, because the result comes out better and cheaper. Relying on older models would have been easier, and most people do. Riding means doing that over and over, long after the novelty wears off.

I get why people stop trying to keep up. The volume is genuinely insane, more launches in a week than you could try in a month, and it isn't only engineers feeling it now, it's anyone whose work runs on a keyboard. People are overwhelmed and often giving up. But you don't have to keep up with everything; you just can't ignore it all.

keep the thinking

It means handing a lot of work to AI, and I hand over more every month. It writes code, drafts the first version of almost everything, reads what I don't have time to read. That part I'm glad to give away.

The part I won't give away is the thinking. The moment you let the model do your reasoning, make your calls, decide for you what's good, you stop being the one person in the loop who can tell whether it's right. You end up accepting whatever it decides.

Take the writing. You can write with AI for AI: text that fills the page, looks finished, yet no actual human would want to read it. Or you can use it to write something a person understands, which only works if you understood it yourself first. So optimize for human understanding, both yours and the reader.

the catch

Keeping the thinking is the easy part for me, because I already did the work that builds it. I know good from merely plausible because I spent years getting it wrong and seeing why. When the model hands me something confident and wrong, I can challenge it. That knowledge was expensive, and it's the only reason any of this works for me.

But the work that built that knowledge, the slow and repetitive part, doing it badly until you can tell the difference, is exactly the work the tool now does for people starting out. They get the output without the years of effort. They ship things that look right and feel nothing when they aren't, and it has nothing to do with how smart they are. The same wave that compounds for me, because I bring judgment to it, may quietly keep them from ever building their own.

The wave is working for me now. I know it eventually takes my whole job, not just the parts I hand it. I ride it anyway, while it still needs me, because betting against it is the worse bet.

The code stopped being mine a while ago, but the thinking hasn't. I keep doing that part myself even when the tools would let me skip it, the way Mario Zechner argued in slowing the fuck down. I'll keep riding because I learned how to slow down.

[Boost]

Adi — Tue, 01 Jul 2025 12:48:03 +0000

Adi

Jun 29 '25

a loop is all you need: building conversation ai agents

#ai #tutorial #startup #howto

10 min read

coding = writing text: approach it like an essay

Adi — Sun, 29 Jun 2025 17:27:21 +0000

Every person who writes code believes it is clean. What a fallacy!

I used to think the same until my colleague Manuel asked me to review his Pull Request (PR) for a project we were working on. As I reviewed his code, I realized that his code is so much cleaner than mine.

What made his code cleaner? Uff, I wouldn't be able to tell, but it felt better. It is like reading your own university essay and then reading one by Paul Graham or Malcom Gladwell - you just know there's a difference.

At first, I thought it was just my lack of experience with JavaScript/TypeScript (as a data scientist I mostly wrote Python before). But in reality, it was just me not paying enough attention to and not recognizing all patterns of clean code.

Manuel recommended me a book by Robert C. Martin and I bought it immediately. The book is called The Clean Code: A Handbook of Agile Software Craftsmanship. It's an extensive and detailed book that you should not read as a novel from start to finish within a day or two. Instead, it serves as a handbook - read a chapter, reflect on it, and ideally apply it in your work. While the code examples are in Java, most of the lessons are universal to other languages.

In this blog post I won't be summarizing or paraphrasing the entire book. First, honestly I don't think I internalized the entire wisdom written there to be able to share it effectively. Second, I encourage everyone to read the book and draw their own conclusions from original source. This advice applies to any topic - always go to the original source. Finally, the book has so many valuable insights that a single blog post can't capture them all.

Instead, in this blog, I'll share the first key takeaway I gained after reading and applying some of its principles.

write code like you're writing an essay, not just a script

Let me give one example of code and how to rewrite it if you think about the code as an essay.

code example

Suppose that we have a pipeline or a workflow that executes a set of checks. First it runs external_check. If this external check returns APPROVED, then it proceeds with the second step where the status is determined based on metrics.

Here is some dummy implementation of this workflow:

from enum import Enum
from some_file import BaseWorkflow
from typing import Any, Dict, Optional

class WorkflowStatus(Enum):
  CREATED = "CREATED"
  APPROVED = "APPROVED"
  REJECTED = "REJECTED"
  PENDING = "PENDING"

class CheckOutcome(Enum):
  CREATED = "CREATED"
  PASSED = "PASSED"
  FAILED = "FAILED"
  PENDING = "PENDING"

class Workflow(BaseWorkflow)
  def __init__(self, status):
    self.status = status

  async def external_check(self) -> Dict[str, Any]
    # some operation, returns { outcome: CheckOutcome }
    pass

  async def fetch_metrics(self) -> Dict[str, Any]:
     # Does some operations
     # trigger data processing if it was not triggered
     # convert preprocessed data to metrics
     # returns metrics
     pass

  async def run(self) -> WorkflowStatus:
    check = await external_check()
    status = get_status(outcome=check["outcome"])

    if status != WorkflowStatus.APPROVED:
      self.status = status
      return self.status

    if status == WorkflowStatus.CREATED:
      await fetch_metrics()
      self.status = WorkflowStatus.PENDING
    else:
      metrics = await fetch_metrics()
      self.status = get_status(metrics=metrics)
      return self.status

  @staticmethod
  def get_status(
    outcome: Optional[CheckOutcome] = None,
    metrics: Optional[Dict[str, Any]] = None
  ) -> WorkflowStatus:
    status = WorkflowStatus.PENDING

    if outcome is not None:
       # Logic based on outcome
       pass

    if metrics is not None:
       # Logic based on metrics

    return status

improvement #1: write from top to bottom

The main method in this class is .run() which is how this workflow will be used in other codebase. Imagine you see somewhere in the codebase workflow.run() and you want to know what it does.

You scroll down to the middle of the file to find .run() implementation. It references three other methods: .external_check(), fetch_metrics() and get_status(). So, you scroll up a bit to check implementation of the first two methods. Where is get_status()? Ahh you scroll down again, this time to the bottom of the file, to find its implementation.

Quite a lot of scrolling to follow the code. Imagine reading an essay where the introduction is halfway down the page, the main arguments are at start, and the resolution is in the bottom. We can structure our code better than that.

How about reorganizing the code so that as you follow it, you only need to scroll down? This way, everything flows naturally, just like reading a well-structured document.

class Workflow(BaseWorkflow)
  def __init__(self, status):
    self.status = status

  async def run(self) -> WorkflowStatus:
    # calls external_check()
    # calls fetch_metrics()
    # calls get_status()
    pass

  async def external_check(self) -> Dict[str, Any]
    pass

  async def fetch_metrics(self) -> Dict[str, Any]:
     pass

  @staticmethod
  def get_status(
    outcome: Optional[CheckOutcome] = None,
    metrics: Optional[Dict[str, Any]] = None
  ) -> WorkflowStatus:
    pass

improvement #2: group into chapters

The code inside .run() handles several tasks at varying levels of abstraction: it calls an external check, fetches metrics, and derives the status from both. However, at its core, it’s a two-step process. First, it performs an external_check (Step 1), and only if the first step is APPROVED does it proceed with the rest of the code (Step 2). If this were a book, these would be two distinct chapters: one for each step.

So, let's break .run() into two separate methods: execute_first_step() and execute_second_step(). This division allows us to enhance each step with more complex logic without altering the original .run() implementation.

class Workflow(BaseWorkflow)
  def __init__(self, status):
    self.status = status

  async def run(self) -> WorkflowStatus:
    first_step_status = await self.execute_first_step()

    if first_step_status != WorkflowStatus.APPROVED:
      self.status = first_step_status
      return self.status

    second_step_status = await self.execute_second_step()
    self.status = second_step_status
    return self.status

  async def execute_first_step(self) -> WorkflowStatus:
    check = await external_check()
    status = get_status(outcome=check["outcome"])
    return status

  async def execute_second_step(self) -> WorkflowStatus:
    if self.status == WorkflowStatus.CREATED:
      await fetch_metrics()
      self.status = WorkflowStatus.PENDING
    else:
      metrics = await fetch_metrics()
      status = get_status(metrics=metrics)
      return status

  async def external_check(self) -> Dict[str, Any]
    pass

  async def fetch_metrics(self) -> Dict[str, Any]:
     pass

  @staticmethod
  def get_status(
    outcome: Optional[CheckOutcome] = None,
    metrics: Optional[Dict[str, Any]] = None
  ) -> WorkflowStatus:
    pass

improvement #3: have intentional naming

Now let's take a closer look at the other methods: get_status() and fetch_metrics().

The get_status() method's name suggests it returns a status, which is clear and straightforward. However, we notice that get_status() takes in two unrelated arguments: outcome and metrics. To understand what this method does, how it uses each argument, and what arguments it actually accepts, you'll need to dive into its implementation—especially if you plan to use it elsewhere.

To make the code more intuitive, let's improve the naming and split it into two methods: get_status_from_outcome() and get_status_from_metrics(). This not only simplifies the functions and ensures each does only one thing, but also makes it clear what each method accepts and returns, just from their names:

get_status_from_outcome: accepts outcome and returns status
get_status_from_metrics: accepts metrics and returns status

Otherwise, instead of creating get_status_from_outcome, we could also consider get_status_from_check which would allow us to pass the entire check variable directly, rather than just check['outcome'].

Now let's take a closer look at execute_second_step().

This method uses fetch_metrics. From the name, you'd expect it to simply fetch metrics from somewhere. When the current status isn't CREATED, the method is used exactly that way: it fetches metrics, stores them in a metrics variable, and then the status is determined from these metrics.

However, when the status is CREATED, the function is called without saving its output. This suggests that fetch_metrics is doing something more than just fetching metrics, contrary to what the name implies.

To address this confusion, we need to dive into the implementation of fetch_metrics. Upon inspection, we find that it actually performs two tasks:

It triggers a data pre-processing job if it hasn't been triggered already.
It fetches the pre-processed data and converts it into metrics.

The solution is to refactor fetch_metrics so that the data pre-processing logic is separated into its own function, e.g. trigger_data_preprocessing(). This will make the code more understandable and prevent any misconceptions about what fetch_metrics is doing.

Ultimately, the names of functions and variables should reflect their purpose and intention clearly. By choosing meaningful names, you're essentially translating your plan—"first, I will trigger data processing, then I will fetch metrics, and finally, I will derive the status from these metrics"—into the code itself. The specific details, such as how processing or fetching is implemented, become secondary and abstracted away.

To conclude, the core plan remains clear and easy to follow, ensuring that anyone reading the code understands the overall structure and flow at a glance, much like skimming a book by chapters and then diving into each chapter by sections.

Does it look better? What else can be improved?

a loop is all you need: building conversation ai agents

Adi — Sun, 29 Jun 2025 16:52:44 +0000

After months of building AI agents, I've come to a counterintuitive conclusion: those fancy agent frameworks everyone seems to be using? You probably don't need them.

Let me explain how I got here.

essential complexity is not in tech

A few months ago, I joined an AI startup as founding engineer.

Coming from years of doing data science, I thought I knew what I was getting into. I'd worked with libraries like LangChain, CrewAI, AutoGen, Pydantic AI and taught these agent frameworks at AI engineering bootcamps. I had notebooks full of multi-agent systems, chain-of-thought prompting experiments, and RAG pipelines.

Instead, I found myself reading "Conversations with Things: UX Design for Chat and Voice" by Diana Deibel and Rebecca Evanhoe, published in 2021, long before ChatGPT and modern AI wave.

The book opened my eyes to something I'd been missing: building conversational AI agents isn't primarily a technical challenge. It's a conversation design challenge. And conversations are messy, cultural, and deeply human.

turn-taking is difficult

The first thing that hit me was how bad we developers are at teaching our systems the most basic human skill: knowing when to let the AI speak and when to wait.

In AI there's this concept called "turn-taking" - the invisible dance of who speaks when. Humans are masters at this. We pick up on tiny cues: a slight intake of breath, a change in posture, the way someone's voice drops at the end of a thought. We know instinctively when it's our turn.

But when you're building an agent, you have to explicitly program these decisions: has the user finished their thought? Should the system respond now or wait for more input?

the messaging problem

Modern messaging systems introduces its own weird dynamics. You know that typing indicator - "Adilet is typing..."? That little social contract that says "hold on, I'm not done." But AI agents interact through APIs. They can't see when someone's typing, and they don't trigger the typing indicator themselves. So you get situations like:

User: Hi I have a question
User: It's about my order  
User: The one from last week
AI: [Already responding to the first message]

Some users write novels in single messages. Others send thoughts like morse code - tap, tap, tap. After weeks of experimentation, I still don't have a perfect solution. Do you wait a few seconds after each message to see if more are coming? Do you analyze message length patterns? Do you look for linguistic cues that suggest completion?

cultural turn-taking

I noticed something fascinating from my own multilingual background. Growing up in Kazakhstan, but then studying and working abroad, I've experienced how differently cultures handle conversational pauses.

In Spanish conversations, people overlap constantly - it's not rude, it's engaged. In Kazakhstan, those longer pauses aren't awkward: they're respectful. Now imagine programming a system to handle both styles. A 3-second pause might signal "I'm done" in one culture and "I'm thinking" in another.

The best solution so far I've found combines multiple strategies:

Looking for linguistic completion markers
Waiting fixed amount of time before replying
When it inevitably responds too early, to handle interruptions gracefully

This focus on handling mistakes gracefully brings me to the most important lesson in building conversations AI agents.

the art of failing gracefully

"Conversations with Things" introduced me to the concept of "repair" or what I call recovery. Human conversations go off the rails constantly, but we fix them together almost unconsciously. We clarify, we backtrack, we laugh it off.

Your AI will fail. Not occasionally, but regularly. The question isn't how to prevent failures; it's how to recover from them.

Let me give you a theoretical example. Imagine an AI agent for a food delivery service. A customer says they ordered "the usual" but the system has no record of previous orders (for whatever reason). The agent could crash and burn with "I don't understand 'the usual'" or it could recover gracefully: "I'd love to get you your usual! Could you remind me what that is? I want to make sure I get your order exactly right."

The difference? One leaves the customer frustrated (and makes AI agent seem stupid), the other makes them feel heard while smoothly getting the information needed.

💡 People are surprisingly forgiving of AI mistakes if you handle the recovery well.

This isn't just about having a few canned "sorry" responses. It's about building recovery into every interaction:

Acknowledgment: Don't pretend the error didn't happen
Empathy: Show you understand the inconvenience
Humor (when appropriate): A little self-deprecation goes a long way
Action: Actually fix the problem or offer an alternative

Once you've mastered recovery, you need to think about who exactly is doing the recovering - and that's where personality comes in.

well-defined personality isn't optional

Building a conversational AI agent is not just merely using LLMs to generate correct responses. We all can notice the difference when we interact with Amazon Alexa, Apple Siri, ChatGPT, Yandex Alice (or even Claude Chat / Mistral LeChat – anyone?), but often we cannot explicitly tell what actual differences are. Yet, we all perceive that these assistants have different personalities, which are accurately constructed by developers and designers that make us, humans, feel in an unique way.

People infer personality from the set of things: word choices, voice (sound) and behavior (more about this in the next section). Here's something that surprised me: the personality you give your AI agent fundamentally changes how users interact with it. This isn't about making your bot "fun" or "quirky." It's about creating consistent, predictable interactions that users can adapt to.

This is called "accommodation" or "mirroring" - how people naturally adjust their communication style to match their conversation partner. I noticed users actually start mirroring the agent's communication style. Formal agents get formal responses. Friendly agents get friendly responses. It's like watching a dance where one partner subtly leads and the other follows.

behavior and intents are the key

The trick isn't writing vague instructions like "be friendly and professional." It's defining specific behaviors for specific situations, and from these behaviors emerge the intents your system needs to handle.

But first, what are intents? In conversational AI, an intent represents what the user wants to accomplish or what action the system should take. Instead of trying to handle infinite variations of user input, you define specific set of intents that capture the core actions your agent can perform.

from vague prompts to specific intents

Here's what many developers do wrong. They write prompts like:

You are a helpful customer support agent. Be empathetic and professional.

This is too vague. What does "empathetic" mean in practice? Instead, define specific behaviors that create empathy.
For example, when you want the agent to be "empathetic," what you actually want is the behavior of "always acknowledge the customer's emotion before providing solutions."

This behavior naturally leads to creating intents like

class AcknowledgeEmotion(BaseModel):
    """Recognize and validate customer feelings"""
    emotion_type: Literal["frustrated", "confused", "happy", "angry"]
    intensity: Literal["mild", "moderate", "severe"]

class RequestClarification(BaseModel):
    """Ask for more details when the issue isn't clear"""
    unclear_aspect: str
    suggested_questions: list[str]

These intents emerged from defining how the agent should behave in emotional situations. The desired behavior drives the technical implementation, not the other way around.

conditional intents

Intents can also be conditionally available based on context. For a customer support agent that handles both free and paid users, we can add intent of RedirectToHuman for paid users and CannotRedirectToHuman intent for free users.

if user.account_type == "paid":
    intents.append(RedirectToHuman)
else:
    intents.append(CannotRedirectToHuman)

This prevents the agent from promising human support to someone who hasn't bought anything yet, while still letting it explain why it can't help with certain requests.

useful intents to consider

Besides adding intents based on condition and context, there are some intents that you might consider adding always.

One useful intent might be to handle off-topics:

class RedirectOffTopic(BaseModel):
    """Handle conversations that drift from support issues"""
    topic_type: Literal["personal_chat", "competitor_questions", "philosophical"]
    redirect_strategy: Literal["gentle_reminder", "humor", "firm_boundary"]

This came from the behavior of maintaining professional boundaries while keeping customers engaged. The humor strategy is particularly effective - when someone's clearly testing the system with "What's the meaning of life?", a playful "That's above my pay grade! But I'm great at tracking orders 😊" works much better than a robotic rejection "I cannot talk about this, I can only talk about support" from LLM guardrails system.

Another useful intent might be needed to handle cases where agent doesn't know or cannot provide the answer. This intent will minimize hallucinations better than adding phrases like "Never invent information that is not in your context" to your prompt.

Now that we understand intents and behaviors, let's see how to implement them without complex frameworks.

understanding agents from first principles

Here's where I probably lose half of you: after all this work, I built everything from scratch using just Python and the OpenAI SDK. No LangChain, no CrewAI, no new fancy frameworks.

I've been heavily influenced by Anthropic's definition of agents and HumanLayer's 12-factor agent principles. Both essentially say the same thing: agents are just models using tools in a loop.

Anthropic's definition:

env = Environment()
tools = Tools(env)
system_prompt = "Goals, constraints, and how to act"

while True:
    action = llm.run(system_prompt + env.state)
    env.state = tools.run(action)

HumanLayer's definition:

initial_event = {"message": "..."}
context = [initial_event]
while True:
    next_step = await llm.determine_next_step(context)
    context.append(next_step)

    if next_step.intent == "done":
        return next_step.final_answer

    result = await execute_step(next_step)
    context.append(result)

It is the same pattern: observe context, decide action, execute, update state, repeat. That's it. Everything else is implementation details.

back to basics

There's something beautifully ironic about this journey. When I taught Python for data science at bootcamps, one of the first concepts was loops:

for item in items:
    process(item)

Simple, right? But then when you get to data analysis (pandas, numpy), you spend weeks learning to avoid loops. "Never iterate over DataFrame rows!" we'd say. "Use vectorized operations!"

Now with AI agents, we're back to the most basic pattern:

while not done:
    observe()
    think()
    act()

Sometimes the oldest patterns are the best patterns. There's a certain poetry to it - the most advanced AI technology we have, operating on the most fundamental programming construct.

why not frameworks?

When you use a framework to build an agent, you're adding layers of abstraction over this simple loop. These frameworks are built for maximum flexibility - they need to handle every possible use case from chatbots to autonomous research agents.
But when you only have a specific use case and specific needs, such as:

Fine-grained monitoring of every decision
Precise conversation flow control
Custom error handling for conversation-specific edge cases
Integration with your specific communication channels and your particular tech stack

... those abstractions become obstacles. You spend more time fighting the framework than solving actual problem.

According to HumanLayer's research, most companies building production agents end up rolling their own. After trying both approaches, I understand why. The complexity isn't in the agent loop - it's in your conversation design, intent and error handling, and integration needs. Frameworks can't abstract those away.

customer support agent using first principles

We have been talking about the customer support agent as an example previously.

From the first principles and conversation design ideas, here is the draft implementation:

continue = True
while continue:
    intents = llm.determine_intents(context)
    context.append(intents)

    action = process_intents(intents)

    if action.is_final = True:
        continue = False

    result = await execute_action(action)
    context.append(result)

example flows

Let's walk through two scenarios to see how this works:

Simple FAQ example - "How to cancel my subscription?"

In customer support there is always a set of questions that users ask the most, called FAQs. We can preload the FAQs into LLM prompt, so that agent can respond immediately.

Agent determines intent: ReplyToUser
Processes this intent which results in action of sending message. Action of sending message is final, so the loop will end after execution
Executes the action by sending this message via API
Updates context (conversational history) with this message

Complex example - "When my subscription renews?":

Agent determines intent: FetchUserSubscriptionData
Executes fetch for subscription data
Updates context with user's renewal date
Loop continues with context now containing user data
Agent determines intent: ReplyToUser
Generates and sends personalized response with exact date
Marks action as final, loop ends

The beauty is that more complex scenarios just mean more loop iterations, not fundamentally different code.

context is everything

The final piece in the conversational AI agent development is context engineering.

This term, coined by Dex at HumanLayer, has gained traction recently with endorsements from Shopify CEO Tobi Lütke and even Andrej Karpathy.

The same user message means completely different things depending on context.

User: "Yes".
Without context, this is meaningless. With context:

After "Would you like to see our menu?" → User wants to browse options
After "Is this your current address?" → User is confirming information
After "Should I cancel your order?" → User wants to cancel

Consider the following scenario where context engineering is more than just prompt engineering:

Someone from the customer support team tries to call the user about the issue, but the call fails. We can log that event and add it to our customer support agent's context.
Then our customer support agent sees this within own context and naturally acknowledges it with proper intent: "Hey, it seems like we tried calling you earlier but couldn't reach you. Is now a better time to call?".
After user confirmation, the human from the customer support team calls again and all conversation is transcribed and saved.
If we pass this transcript to customer support agent's context, then the agent can say things like "As you discussed on the phone yesterday with ..."

Crafting the context like that makes the experience feel cohesive.

key takeaways

I'm still early in this journey. Every day brings new edge cases, new failures to recover from, new patterns to recognize. But here is my learnings so far:

Conversation design > technical complexity: Understanding turn-taking, cultural differences, and recovery strategies matters more than the latest framework
Recovery > perfection: Users forgive mistakes if you handle them gracefully. Build recovery into every interaction.
Context > features: A simple agent with rich context beats a complex one without it
Simple loops > complex frameworks: Most production agents are just while loops with good intent handling
Behavior drives implementation: Define specific behaviors first, then derive the technical intents from them

The future of conversational AI isn't in complex frameworks or ever-larger models. It's in understanding the fundamentals of conversation, context, and human interaction. It's in building systems that fail gracefully, adapt to their users, and remember that at the end of the day, they're having a conversation with a human who just wants to be understood.
And sometimes, that's as simple as a while loop that knows when to listen.

These ideas are based on personal experience building production agents. All examples are illustrative, and any views are my own—not those of my employer. If you're building conversational AI, I'd love to hear your thoughts. Are you team framework or team from-scratch?