DEV Community: Zachary Huang

Reinforcement Learning: Multi-Armed Bandits

Zachary Huang — Sun, 10 Aug 2025 23:17:30 +0000

Part 1: The Big Idea - What's Your Casino Strategy?

Before we dive in, let's talk about the big idea that separates Reinforcement Learning (RL) from other types of machine learning.

Most of the time, when we teach a machine, we give it instructions. This is called supervised learning. It's like having a math teacher who shows you the correct answer (5 + 5 = 10) and tells you to memorize it. The feedback is instructive: "This is the right way to do it."

Reinforcement Learning is different. It learns from evaluation. It's like a critic watching you perform. After you take an action, the critic just tells you how good or bad that action was—a score. It doesn't tell you what you should have done. The feedback is evaluative: "That was a 7/10." This creates a problem: to find the best actions, you must actively search for them yourself.

This need to search for good behavior is what we're going to explore, using a classic problem that makes it crystal clear.

The k-Armed Bandit: A Casino Dilemma

Imagine you walk into a casino and see a row of k slot machines. In our lingo, this is a k-armed bandit. Each machine (or "arm") is an action you can take.

You have a limited number of tokens, say 1000.
Each machine has a different, hidden probability of paying out a jackpot. This hidden average payout is the machine's true value.
Your goal is simple: Walk away with the most money possible.

How do you play? This isn't just a brain teaser; it’s the perfect analogy for the most important trade-off in reinforcement learning.

Formalizing the Problem (The Simple Math)

Let's put some labels on our casino game. Don't worry, the math is just a way to be precise.

An action a is the choice of which of the k levers to pull.
The action you choose at time step t (e.g., your first pull, t=1) is called A_t.
The reward you get from that pull is R_t.
Each action a has a true mean reward, which we call its value, denoted as q*(a).

This value is the reward we expect to get on average from that machine. Formally, it's written as:

q*(a) = E[R_t | A_t = a]

In plain English, this means: "The true value of an action a is the expected (average) reward you'll get, given you've selected that action."

The catch? You don't know the true values q*(a)! You have to discover them by playing the game.

The Core Conflict: Exploration vs. Exploitation

This is where the dilemma hits. With every token you spend, you face a choice:

Exploitation: You've tried a few machines, and one of them seems to be paying out more than the others. Exploitation means you stick with that machine because, based on your current knowledge, it's your best bet to maximize your reward right now.
Exploration: You deliberately try a different machine—one that seems worse, or one you haven't even tried yet. Why? Because it might be better than your current favorite. You are exploring to improve your knowledge of the world.

The Conflict: You cannot explore and exploit with the same token. Every time you explore a potentially worse machine, you give up a guaranteed good-ish reward from your current favorite. But if you only ever exploit, you might get stuck on a decent machine, never discovering the true jackpot next to it.

This is the exploration-exploitation dilemma. It is arguably the most important foundational concept in reinforcement learning. Finding a good strategy to balance this trade-off is the key to creating intelligent agents.

In the next section, we'll look at a simple but flawed strategy for solving this problem.

Part 2: A Simple (But Flawed) Strategy - The "Greedy" Approach

So, how would most people play the slot machine game? The most straightforward strategy is to be "greedy."

A greedy strategy works in two phases:

Estimate: Keep a running average of the rewards you've gotten from each machine.
Exploit: Always pull the lever of the machine that has the highest average so far.

This sounds reasonable, right? You're using the data you've collected to make the most profitable choice at every step. Let's formalize this.

How to Estimate Action Values

Since we don't know the true value q*(a) of a machine, we have to estimate it. We'll call our estimate at time step t Q_t(a). The simplest way to do this is the sample-average method:

Q_t(a) = (sum of rewards when action a was taken before time t) / (number of times action a was taken before time t)

This is just a simple average. If you pulled lever 1 three times and got rewards of 5, 7, and 3, your estimated value for lever 1 would be (5+7+3)/3 = 5.

The Greedy Action Selection Rule

The greedy rule is to always select the action with the highest estimated value. We write this as:

A_t = argmax_a Q_t(a)

The argmax_a part looks fancy, but it just means "find the action a that maximizes the value of Q_t(a)." If two machines are tied for the best, you can just pick one of them randomly.

Why the Greedy Strategy Fails

The greedy method has a fatal flaw: it's too quick to judge and never looks back. It gets stuck on the first "good enough" option it finds, even if it's not the best option.

Let's see this in action with a minimal example.

The Setup:

A 3-armed bandit problem.
The true (hidden) values are:
- Machine 1: q*(1) = 1 (A dud)
- Machine 2: q*(2) = 5 (Pretty good)
- Machine 3: q*(3) = 10 (The real jackpot!)
To start, our agent needs some data, so let's say it tries each machine once.

The Game:

Pull 1: The agent tries Machine 1. It's a dud, and the reward is R_1 = 1.
- Our estimate is now Q(1) = 1.
Pull 2: The agent tries Machine 2. The reward is a lucky R_2 = 7.
- Our estimate is now Q(2) = 7.
Pull 3: The agent tries Machine 3 (the true jackpot). By pure bad luck, this one pull gives a disappointing reward of R_3 = 4.
- Our estimate is now Q(3) = 4.

The Trap:
After these three pulls, our agent's estimates are:

Q(1) = 1
Q(2) = 7
Q(3) = 4

Now, the greedy strategy kicks in. From this point forward, which machine will the agent choose? It will always choose the argmax, which is Machine 2.

The agent will pull the lever for Machine 2 forever. It will never go back to Machine 3, because based on its one unlucky experience, it "believes" Machine 2 is better. It got stuck exploiting a suboptimal action and will never discover the true jackpot machine.

This is why pure exploitation fails. We need a way to force the agent to keep exploring, just in case its initial estimates were wrong. That brings us to our first real solution.

Part 3: A Smarter Strategy - The ε-Greedy (Epsilon-Greedy) Method

The greedy strategy failed because it was too stubborn. Once it found a "good enough" option, it never looked back. The ε-Greedy (pronounced "epsilon-greedy") method fixes this with a very simple and clever rule.

The Big Idea: "Be greedy most of the time, but every once in a while, do something completely random."

Think of it like choosing a restaurant for dinner.

Exploitation (Greed): You go to your favorite pizza place because you know it's good.
Exploration (Randomness): Once a month, you ignore your favorites and just pick a random restaurant from Google Maps. You might end up at a terrible place, but you might also discover a new favorite!

This small dose of randomness is the key. It forces the agent to keep exploring all the options, preventing it from getting stuck.

The ε-Greedy Rule

Here's how it works. We pick a small probability, called epsilon (ε), usually a value like 0.1 (which means 10%).

At every time step, the agent does the following:

Generate a random number between 0 and 1.
If the number is greater than ε: Exploit. Choose the action with the highest estimated value, just like the greedy method.
- This happens with probability 1 - ε (e.g., 90% of the time).
If the number is less than or equal to ε: Explore. Choose an action completely at random from all available actions, with equal probability.
- This happens with probability ε (e.g., 10% of the time).

Why ε-Greedy Works

Let's revisit our "Greedy Trap" from the previous section. Our agent was stuck forever pulling the lever for Machine 2, never realizing Machine 3 was the true jackpot.

How would an ε-Greedy agent with ε = 0.1 handle this?

90% of the time, it would look at its estimates (Q(1)=1, Q(2)=7, Q(3)=4) and greedily choose Machine 2.
But 10% of the time, it would ignore its estimates and pick a machine at random. This means it has a chance of picking Machine 1, Machine 2, or Machine 3.

Eventually, that 10% chance will cause it to try Machine 3 again. And again. And again. As it gets more samples from Machine 3, its estimated value Q(3) will slowly climb from that unlucky 4 towards the true value of 10.

Once Q(3) becomes greater than Q(2), the agent's "greedy" choice will switch! Now, 90% of the time, it will exploit the correct jackpot machine.

The Guarantee

The advantage of this method is huge: in the long run, as the number of plays increases, every single machine will be sampled many, many times. Because of this, the Law of Large Numbers tells us that our estimated values Q_t(a) will eventually converge to the true values q*(a).

This guarantees that the agent will eventually figure out which action is best and will select it most of the time. It solves the "getting stuck" problem completely. Now, let's see exactly how this works with a step-by-step example.

Part 4: Let's Play! A Step-by-Step Walkthrough

Seeing is believing. We're going to simulate a few turns of an ε-Greedy agent to watch its "brain" update.

The Setup

The Game: A 3-armed bandit problem.
The Hidden Truth: The true average payouts (q*(a)) are:
- q*(1) = 2 (Dud)
- q*(2) = 6 (The Jackpot!)
- q*(3) = 4 (Decent)
- The agent does not know these numbers.
Our Agent's Strategy: ε-Greedy with ε = 0.1 (10% chance to explore).
Initial State: The agent starts with no knowledge. Its estimated values (Q) and pull counts (N) for each arm are all zero.
- Q(1)=0, Q(2)=0, Q(3)=0
- N(1)=0, N(2)=0, N(3)=0

The Incremental Update Formula

As we get new rewards, we need to update our Q values efficiently. We won't re-calculate the average from scratch every time. Instead, we use a simple incremental formula.

When we choose action A and get reward R:

First, increment the count for that action: N(A) = N(A) + 1
Then, update the value estimate with this formula:

Q(A) = Q(A) + (1/N(A)) * [R - Q(A)]

Let's break this down:

[R - Q(A)] is the error: the difference between the new reward and what we expected.
We take a "step" to correct this error, with the step size 1/N(A). Notice that as we sample an action more (N(A) gets bigger), the step size gets smaller. This means our estimates become more stable and less affected by single random rewards over time.

The Game Begins

Let's follow the agent for the first 7 pulls. We will track everything in a table.

Step (t)	Agent's Decision	Action (A_t)	Reward (R_t)	Agent's Updated Brain: N(a) and Q(a)
Start	---	---	---	`N=(0,0,0)`, `Q=(0,0,0)`
1	All Qs are 0, must pick randomly.	Arm 1	`R=1`	`N=(1,0,0)`, `Q=(1, 0, 0)` `Q(1) = 0 + 1/1 (1-0) = 1`*
2	`argmax` is Arm 1. Roll is > 0.1 -> EXPLOIT.	Arm 1	`R=3`	`N=(2,0,0)`, `Q=(2, 0, 0)` `Q(1) = 1 + 1/2 (3-1) = 2`*
3	`argmax` is Arm 1. Roll is < 0.1 -> EXPLORE. Picks randomly.	Arm 3	`R=5`	`N=(2,0,1)`, `Q=(2, 0, 5)` `Q(3) = 0 + 1/1 (5-0) = 5`*
4	`argmax` is now Arm 3. Roll > 0.1 -> EXPLOIT.	Arm 3	`R=3`	`N=(2,0,2)`, `Q=(2, 0, 4)` `Q(3) = 5 + 1/2 (3-5) = 4`*
5	`argmax` is still Arm 3. Roll > 0.1 -> EXPLOIT.	Arm 3	`R=6`	`N=(2,0,3)`, `Q=(2, 0, 4.67)` `Q(3) = 4 + 1/3 (6-4) = 4.67`*
6	`argmax` is still Arm 3. Roll < 0.1 -> EXPLORE. Picks randomly.	Arm 2	`R=8`	`N=(2,1,3)`, `Q=(2, 8, 4.67)` `Q(2) = 0 + 1/1 (8-0) = 8`*
7	`argmax` is now Arm 2! Roll > 0.1 -> EXPLOIT.	Arm 2	`R=5`	`N=(2,2,3)`, `Q=(2, 6.5, 4.67)` `Q(2) = 8 + 1/2 (5-8) = 6.5`*

Let's Analyze What Happened

This short sequence shows the power of ε-Greedy in action:

Initial Belief: After two pulls, the agent thought Arm 1 was best (Q(1)=2). A purely greedy agent would have gotten stuck here.
Discovery through Exploration: In Step 3, a random exploratory action forced the agent to try Arm 3. It got a good reward (R=5), and its belief about the best arm changed.
Another Discovery: The agent was happily exploiting Arm 3 until Step 6, when another random exploration forced it to try the last unknown, Arm 2. It got a very high reward (R=8), and its belief changed again!
Nearing the Truth: After only 7 pulls, the agent's estimates are Q=(2, 6.5, 4.67). These are getting much closer to the true values of q*=(2, 6, 4). Its greedy choice is now correctly focused on the best arm, Arm 2.

This is the learning process. The agent starts with no idea, forms a belief, and then uses exploration to challenge and refine that belief. Over thousands of steps, this simple mechanism allows it to zero in on the best actions in its environment.

Part 5: Two More "Clever Tricks" for Exploration

The ε-Greedy method is simple and effective, but its exploration is random. It doesn't care if it's exploring a machine it has tried 100 times or one it has never touched. Can we be smarter about how we explore? Yes.

Here are two popular techniques that add a bit more intelligence to the exploration process.

1. Optimistic Initial Values: The Power of Positive Thinking

This is a wonderfully simple trick that encourages a burst of exploration right at the start of learning.

The Idea: Instead of initializing your value estimates Q(a) to 0, initialize them to a "wildly optimistic" high number. For example, if you know the maximum possible reward from any machine is 10, you might set all your initial Q values to 20.

Q_1(a) = 20 for all actions a.

How it Works:

On the first step, all actions look equally amazing (Q=20). The agent picks one, let's say Arm 1.
It gets a real reward, say R=5.
The agent updates its estimate Q(1). The new Q(1) will now be a value much lower than 20.
Now, the agent looks at its options again. Arm 1 looks "disappointing" compared to all the other arms, which it still believes have a value of 20.
So, for its next turn, the greedy agent will naturally pick a different arm.

This process continues. Every time an arm is tried, its value drops from the optimistic high, making it look "disappointing" and encouraging the agent to try all the other arms it hasn't touched yet. It’s a self-correcting system that drives the agent to explore everything at least a few times before it starts to settle on the true best option.

Key Takeaway: By being optimistic, a purely greedy agent is tricked into exploring.

2. Upper-Confidence-Bound (UCB): The "Smart Exploration" Method

UCB is a more sophisticated approach. It addresses a key question: if we're going to explore, which arm is the most useful one to try?

The Idea: The best arm to explore is one that is both:

Potentially high-value (its current Q(a) is high).
Highly uncertain (we haven't tried it much, so Q(a) could be very wrong).

UCB combines these two factors into a single score. Instead of just picking the argmax of Q(t), it picks the argmax of a special formula:

A_t = argmax_a [ Q_t(a) + c * sqrt(ln(t) / N_t(a)) ]

Let's break that down without fear:

Q_t(a) is our standard value estimate. This is the exploitation part.
The second part, c * sqrt(ln(t) / N_t(a)), is the exploration bonus or uncertainty term.
- t is the total number of pulls so far. As t increases, this term slowly grows, encouraging exploration over time.
- N_t(a) is the number of times we've pulled arm a. This is the important part: as N_t(a) increases, the uncertainty bonus shrinks.
- c is a constant that controls how much you favor exploration. A bigger c means more exploring.

How it Works:

If an arm has a good Q value but has been tried many times (N_t(a) is large), its uncertainty bonus will be small. It's a known quantity.
If an arm has a mediocre Q value but has been tried only a few times (N_t(a) is small), its uncertainty bonus will be very large. This makes it an attractive candidate for exploration because its true value could be much higher than we think.

UCB naturally balances exploration and exploitation. It favors arms it is uncertain about, and as it tries them, its uncertainty decreases, and the Q value starts to matter more. It's a more directed and often more efficient way to explore than the random approach of ε-Greedy.

Part 6: So What? Why Bandits Matter

We’ve journeyed through the k-armed bandit problem, starting with a simple casino analogy and exploring several strategies to solve it. So, what’s the big takeaway?

Summary: The Heart of the Problem

The multi-armed bandit problem is not really about slot machines. It is a simplified, pure version of the core challenge in all of reinforcement learning: the exploration-exploitation dilemma.

We saw that simple strategies can have major flaws:

The Greedy method gets stuck, failing to find the best option because it never looks back.

And we saw how to fix it by intelligently balancing the trade-off:

ε-Greedy is a simple and robust solution: it acts greedily most of the time but takes a random exploratory action with a small probability ε, ensuring it never gets stuck.
Optimistic Initial Values is a clever trick that uses a purely greedy agent but encourages a natural burst of exploration at the beginning by assuming everything is amazing.
Upper-Confidence-Bound (UCB) is a more sophisticated method that explores strategically, prioritizing actions that are both promising and highly uncertain.

Each of these methods provides a way to gather information (explore) while trying to maximize rewards with the information you have (exploit).

The Bridge to Real-World Reinforcement Learning

This "bandit" framework is a fundamental building block. The same principles apply to much more complex problems, both in technology and in real life.

A/B Testing on Websites: Which version of a headline or button color (actions) will get the most clicks (reward)? A company can use bandit algorithms to automatically explore different versions and quickly exploit the one that works best, maximizing user engagement in real-time.
Clinical Trials: Doctors want to find the most effective treatment (action) for a disease. Each patient's outcome is a reward. Bandit algorithms can help balance giving patients the current best-known treatment (exploit) with trying new, experimental ones that might be even better (explore), potentially saving more lives in the long run.
The Full RL Problem: The problems we've discussed so far are "non-associative," meaning the best action is always the same. But what if the best action depends on the situation or context?
- For a self-driving car, the best action (steer, brake) depends on the situation (red light, green light, pedestrian).
- For a game-playing AI, the best move depends on the state of the board.

This is called associative search or contextual bandits, and it's the next step towards the full reinforcement learning problem. The agent must learn not just the best action overall, but the best action for each specific situation. The methods we learned here—like ε-greedy and UCB—are used as the core decision-making components inside these more advanced AIs.

By understanding the simple trade-off in the k-armed bandit problem, you have grasped the essential challenge that every reinforcement learning agent must solve to learn and act intelligently in a complex world.

LLM Agents & Prompt Engineering: A Warrior's Guide to Clear Commands

Zachary Huang — Mon, 28 Jul 2025 21:29:01 +0000

You've forged a legendary warrior with the perfect arsenal and taught them to navigate any dungeon. But what happens when your commands are as clear as mud? "Attack the thing!" you shout. Which thing? With what weapon? Your warrior stands confused. In this guide, you'll master the final art—giving crystal-clear battle commands that turn hesitation into decisive action.

1. Introduction: The Confused Warrior

Picture this: You've spent weeks creating the ultimate digital warrior. In previous posts, you've given them a razor-sharp sword (perfect actions). You've taught them to carry a lightweight but complete backpack (smart context management), and they're standing right in front of the treasure chest. Victory is one command away.

"Get the treasure!" you command confidently.

Your warrior looks at you. Looks at the chest. Looks back at you. Then proceeds to attack the chest with their sword, smashing it to pieces and destroying the treasure inside.

"What are you DOING?!" you scream.

The warrior shrugs. "You said 'get' the treasure. I thought you meant destroy the chest to get to it. Maybe you should have said 'carefully open the chest and retrieve the treasure inside.'"

This is the final challenge in building effective LLM agents. You can have the most sophisticated graph structure, the deadliest arsenal of tools, and the smartest context management—but if your commands are vague, your agent becomes a confused mess.

The good news? Prompt engineering isn't magic. It's not about finding secret incantations that "unlock" your LLM's hidden powers. It's simply the art of giving clear, unambiguous instructions. Think of it as the final sharpening of your blade—not the forging itself, but the careful honing that turns a good sword into a legendary one.

In this guide, we'll strip away the mystique and show you three simple laws that will transform your prompts from confusing mumbles into crystal-clear battle commands. By the end, your warrior won't just be powerful—they'll actually understand what you want them to do.

2. How Agents Work: The Battle Plan Recap

Before we sharpen our commands, let's quickly remind ourselves how our warrior actually operates. As we discovered in our previous adventures, every LLM agent—no matter how complex it appears—follows a devastatingly simple loop:

Assess → Strike → Repeat

In the PocketFlow framework, this loop manifests as a graph:

The DecideNode is our warrior's brain—the battle tactician that looks at the situation and chooses the next move. But here's the crucial part: how does it think?

The answer is simpler than you might expect. The entire "thinking" process is just a prompt. Here's the general structure inside any DecideNode:

prompt = f"""
### YOUR TASK
{task_instruction}

### CURRENT STATE
{context}

### AVAILABLE ACTIONS
{action_space}

### YOUR NEXT MOVE
Based on your task and current state, choose the best action.
Format your response as:

```yaml
thinking: |
    <your reasoning>
action: <chosen action>
parameters: <any required parameters>
```"""

This is the prompt we're going to improve.

Notice how this universal template has three main components:

Task Instruction: What the agent is trying to accomplish
Context: The current state (from the shared store we discussed)
Action Space: The available tools (the arsenal we forged)

We've already mastered forging the perfect action space (giving our warrior the right weapons) and managing context (keeping the backpack light and relevant). Now it's time to focus on that final piece: the task instruction—the actual commands we give our warrior.

Look at it carefully. Everything the agent knows comes from these three sections. And since we've already perfected the action space and context management in our previous guides, the quality of your agent now hinges on that task instruction—the specific commands and guidelines you provide.

3. The Myth of "Prompt Engineering"

First, let's clear the air. The term "prompt engineering" sounds intimidating, and the internet is full of hype about finding secret phrases that "unlock" an LLM's true potential. People share "magic prompts" like ancient spells, promising that adding "let's think step by step" or "you are an expert" will transform your results.

The reality is much simpler and, frankly, more empowering. Effective prompt engineering is not about tricking the LLM or finding secret incantations. It's about clarity and structure.

Your goal is simple: write an instruction manual so clear, so unambiguous, that the LLM can't possibly misinterpret it. It's not magic—it's just good communication.

Here's the most important principle for any agent builder:

The 90/10 Rule: Spend 90% of your time designing actions and context. Prompting is the final 10%.

Think of it this way:

Actions (The Arsenal): If your warrior doesn't have the right weapons, no amount of yelling will help
Context (The Backpack): If your warrior is carrying 500 pounds of junk, they can't move effectively
Prompts (The Commands): Only after the first two are solid should you polish your instructions

If your agent is failing, resist the urge to immediately tweak prompts. Instead, ask:

Does it have the right tools for this job?
Is its context clear and focused?
Only then: Are my instructions clear?

This is why prompt engineering is the "sharpening, not the forging." You can't sharpen a blade that doesn't exist. You can't polish a sword made of rubber. But once you have a well-forged weapon and a clear battlefield, a sharp command makes all the difference.

Let's learn how to sharpen that blade.

4. The 3 Sacred Laws of Command

Now that we understand prompt engineering is about clarity, not magic, let's dive into the three fundamental techniques that will transform your vague mumbles into crystal-clear battle commands.

4.1. Law #1: Demand the Battle Plan (Chain of Thought)

Imagine two different commanders:

Commander A: "Attack the fortress!"

Commander B: "First, tell me your plan for attacking the fortress. Then execute it."

Which warrior is more likely to succeed? The one who charges blindly, or the one who thinks through their approach first?

This is the power of Chain of Thought (CoT). By explicitly asking your agent to explain its reasoning BEFORE taking action, you dramatically improve its decision quality. It's like the difference between a warrior who swings wildly and one who calculates each strike.

Before (No Chain of Thought):

prompt = f"""### YOUR TASK
{task_instruction}

### YOUR NEXT MOVE
Choose an action and provide parameters."""

After (With Chain of Thought):

prompt = f"""### YOUR TASK
{task_instruction}

### YOUR NEXT MOVE
First, explain your step-by-step plan for completing this task.
Then choose an action and provide parameters.

Format your response as:
```yaml
thinking: |
    <your step-by-step reasoning process>
action: <chosen action>
parameters: <required parameters>
```"""

The difference is profound. With the second prompt, your agent might think:

response = """thinking: |
    1. I need to find grocery spending for last month
    2. First, I should identify what "last month" means (get current date)
    3. Then search for transactions in that date range
    4. Filter for grocery-related merchants
    5. Sum up the amounts
    Starting with getting the current date...
action: get_current_date
..."""

Without CoT, it might have jumped straight to searching transactions without properly defining the date range. Chain of Thought forces your warrior to strategize before striking.

Pro Tip: If you're using a thinking model (like o1 or Claude with thinking mode), you get this for free! These models automatically think through problems step-by-step without explicit prompting. It's like having a warrior who's already trained to strategize before every move.

4.2. Law #2: Speak Like a General, Not a Poet (Be Specific)

Your warrior stands before two doors. You could say:

Poetic: "Choose wisely, for one path leads to glory."

General: "Take the left door. It leads to the armory. Avoid the right door—it's trapped with poison darts."

Which instruction leads to success? The specific one, every time.

The biggest prompt engineering mistake is using vague, open-ended language when you actually have specific requirements. Let's look at common examples:

Vague → Specific Transformations:

❌ "Search for relevant information"
✅ "Search for peer-reviewed studies published after 2020 about renewable energy costs"
❌ "Summarize this appropriately"
✅ "Create a 3-paragraph summary with: (1) main finding, (2) methodology, (3) implications"
❌ "Handle errors gracefully"
✅ "If a search returns no results, try broadening the query by removing adjectives. If it still fails, return 'No information found' and explain what was searched"

The Power of Constraints:

Constraints aren't limitations—they're guideposts that lead to better results. Consider these improvements:

# Vague prompt
prompt_vague = f"""### TASK (Vague)
Research the topic and provide information"""

# Specific prompt
prompt_specific = f"""### TASK (Specific)
Research electric vehicles with these constraints:
- Focus on: battery technology and charging infrastructure  
- Time period: 2022-2024 developments only
- Sources: Prioritize industry reports and government data
- Length: Provide 3-5 key findings, each in 1-2 sentences
- Format: Bullet points with source citations"""

The specific version removes all ambiguity. Your warrior knows exactly what victory looks like.

4.3. Law #3: Show, Don't Just Tell (Few-Shot Examples)

Sometimes, even the clearest words aren't enough. This is when you need to demonstrate, not just describe. Think of it as the difference between explaining sword techniques with words versus actually showing the movements.

Few-shot examples are perfect when you need:

A specific writing style
A particular output format
Complex patterns that are hard to describe

Without Examples (Frustrating):

prompt = f"""### TASK
Write a customer support response in our company's friendly, solution-focused style

<context and customer message here>
..."""

Your agent thinks: "What does 'friendly' mean? How solution-focused? What tone?"

With Examples (Crystal Clear):

prompt = f"""### TASK
Write a customer support response following our style:

Example 1:
Customer: "My order hasn't arrived and it's been a week!"
Response: "I completely understand your frustration - a week is definitely too long to wait! Let me track that down for you right away. I can see your order #12345 left our warehouse on Monday. I'll contact the carrier now and have an update for you within 2 hours. In the meantime, I've credited your account with 20% off your next order for the inconvenience."

Example 2:
Customer: "This product broke after one day"
Response: "Oh no! That's definitely not the experience we want you to have. I'm starting a replacement order for you right now - it'll ship today with express delivery at no charge. No need to return the broken item. Your new one should arrive by Thursday. Is there anything else I can help make right?"

Now respond to this customer following the same style:
Customer: {customer_message}"""

Notice how the examples teach the style without explicit rules:

Start with empathy/acknowledgment
Take immediate action (no "we'll look into it")
Provide specific next steps and timelines
Offer something extra for the inconvenience
End with an open offer to help more

When to Use Examples:

Writing style: "Match our blog voice" → Show 2-3 actual blog excerpts
Response formats: "Format like our reports" → Show a complete sample
Tone matching: "Professional but warm" → Demonstrate with real examples
Edge cases: Show how to handle tricky situations

Examples are particularly powerful because they bypass the ambiguity of language. Instead of trying to define "friendly but professional," you simply show what it looks like in action.

6. Conclusion: From Confused Grunt to Elite Soldier

And so, our journey is complete. We've transformed a simple graph into a legendary warrior:

The Graph: Every agent is just Assess → Strike → Repeat
The Arsenal: Sharp, purposeful tools for every task
The Context: A light backpack with only what's needed
The Commands: Crystal-clear orders that eliminate confusion

Remember the 90/10 rule: If your agent fails, check the arsenal and context first. Prompt engineering is just the final polish. But when you need that polish, your three laws are:

Demand the battle plan (Chain of Thought)
Speak like a general (Be Specific)
Show, don't tell (Few-Shot Examples)

Your warrior no longer smashes treasure chests when you meant "open them carefully." They understand. They execute. They win.

The next time someone shares "secret prompt tricks," smile knowingly. You understand it's not about magic—it's about clarity.

Ready to build agents that actually understand? Check out PocketFlow on GitHub and start commanding your digital warriors!

LLM Agents & Context: A Warrior's Guide to Navigating the Dungeon

Zachary Huang — Thu, 03 Jul 2025 17:43:26 +0000

Your agent has a legendary sword and a powerful spellbook. But what good are weapons if your warrior is lost in a sprawling dungeon, unable to remember which rooms are cleared and which hold treasure? In this guide, you'll learn the three master navigation techniques of agent memory—the Scrying Spell, the Grand Strategy, and the Cautious Explorer's Path. It's time to teach our warrior not just how to fight, but how to think.

1. Introduction: The Warrior Enters the Dungeon

In our previous adventures, we learned the secret that all agents are just simple graphs and forged our warrior in LLM Agents are simply Graph — Tutorial For Dummies. Then, we equipped it with a deadly arsenal of actions in LLM Agents & Their Arsenal: A Beginner's Guide. But now, our warrior faces its greatest challenge yet: the environment itself.

The agent's battle isn't on an open field; it's in a dark, complex dungeon—a large codebase, a multi-step research task, or a complex dataset. Here, the biggest danger isn't the monsters (the individual tasks), but getting lost, forgetting where you've been, and losing sight of the treasure at the end.

This brings us to the most critical, and often botched, aspect of agent design: context management. Think of it as the warrior's Cognitive Backpack. In our PocketFlow framework, this is the simple shared dictionary that each Node reads from and writes to. The naive approach is to stuff everything inside. Imagine loading the entire dungeon map, every monster's stat block, every rumor of treasure, and the history of the last three adventurers who failed into the warrior's backpack right at the start. They'd collapse under the weight before they even took their first step.

Smart context management isn't about giving the agent more memory; it's about giving it the right memory at the right time. In this guide, we'll stop treating our agent's memory like a junk drawer and start treating it like a high-tech utility belt. We will learn three master-level navigation techniques to keep the backpack light, the warrior agile, and the path to victory clear.

2. The Warrior's Battle Plan: A Quick Recap

Before we teach our warrior new navigation tricks, let's refresh our memory of the battle plan. As we learned, every agent, no matter how complex, follows a simple, relentless loop: Assess -> Strike -> Repeat.

In the PocketFlow framework, this elegant loop is visualized as a graph:

The DecideNode is the warrior's brain—the battle tactician. It assesses the situation and chooses the next move. The ActionNodes are the specialist soldiers who carry out a specific command. And the loop back is the report, bringing new information from the battlefield back to the tactician.

But how does the DecideNode actually think? How does it "assess" the situation? This isn't magic; it's a carefully crafted prompt. The node's entire worldview comes from the shared dictionary, which is formatted and injected directly into its brain.

The prompt inside our DecideNode looks something like this:

### CURRENT SITUATION
You are a research assistant. Here is what you know right now:
{context_from_shared_store}

### AVAILABLE ACTIONS
[1] search_web(query: str)
  Description: Search for new information online.
[2] write_file(filename: str, content: str)
  Description: Save information to a file.
[3] finish_task(reason: str)
  Description: Complete the mission because the goal is met.

## YOUR NEXT MOVE
Based **only** on the CURRENT SITUATION, choose the single best action to take next and provide the required parameters.

The content of our shared dictionary—our cognitive backpack—is dropped directly into the {context_from_shared_store} placeholder. This is the only thing the LLM sees. Its entire universe of knowledge for making a decision is contained within that block.

This reveals a critical truth: the quality of the agent's decisions is 100% dependent on the quality of the information in the shared store. This simple dictionary is the most important part of the agent's "brain." So, the central question becomes: what is the right way to manage it?

3. The Overwhelmed Warrior: Why Dumping All Context Fails

Imagine our warrior at the dungeon entrance. We, as the benevolent master, decide to "help" by giving them everything. We cram the entire 500-page dungeon history, every blueprint, every monster's family tree, and a transcript of every conversation ever had about the dungeon into their backpack. "Good luck!" we say, as the warrior stumbles forward, unable to even lift their sword under the crushing weight.

They enter the first room, which has a simple pressure plate on the floor. To solve it, they need to find the small, one-ounce stone they picked up just a moment ago. But to find it, they have to rummage through the 200-pound bag of useless junk we gave them. They get distracted by a map of a different dungeon wing, start reading about the goblin king's third cousin, and forget about the pressure plate entirely. They are paralyzed by information overload.

This is exactly what happens when you dump your entire shared history into an LLM's prompt on every turn. A cluttered backpack doesn't create a genius warrior; it creates an ineffective one. Here’s why:

Diluted Attention: LLMs have a finite attention span. When you give them a massive context, the critical piece of information—the "signal"—gets lost in a sea of irrelevant data—the "noise." The model might struggle to find the single most important fact ("the user just asked to search for this") when it's buried in ten pages of previous search results. This is often called the "lost in the middle" problem, where information in the center of a large prompt is frequently ignored.
Sky-High Costs & Latency: Every token in your prompt costs money and processing time. A cluttered backpack slows your warrior to a crawl and empties your coin purse. An agent that sends a 100,000-token context on every loop is not only breathtakingly expensive but also painfully slow, making any real-time interaction impossible.
Increased Hallucinations: When an LLM is given too much loosely related information, it starts to "cross the wires." It might grab a detail from an early, now-irrelevant step and incorrectly apply it to the current situation. It's the equivalent of our warrior trying to use a recipe for a health potion to disarm a magical trap—a confident but catastrophically wrong decision.

The takeaway is simple: a bigger context does not equal a smarter agent. Our goal is not to build the biggest backpack, but the most efficient one. We need to stop being hoarders and start being strategists, ensuring our warrior carries only what they need for the immediate fight.

4. Forging the Cognitive Backpack: Three Master Navigation Techniques

If stuffing everything into the backpack is the path to failure, what is the path to victory? The answer lies in transforming the backpack from a static, heavy burden into a dynamic, intelligent system. A master warrior doesn't carry every tool for every possible situation. They carry a few versatile tools that allow them to adapt to any situation.

Here are the three master techniques for forging your agent's cognitive backpack.

4.1. Technique #1: The Scrying Spell (Context On-Demand)

The Metaphor: Our warrior enters the dungeon with a nearly empty backpack. Instead of a map, they carry a magical "scrying orb." When they reach a fork in the path, they don't guess. They hold up the orb and ask, "What lies down the left corridor?" The orb shows them a brief vision of the next room. They now have a small, relevant piece of information. They add this "vision" to their mental map and then use the orb again to scout the right corridor. They build their map piece by piece, only gathering the information they need, precisely when they need it.

The Pattern: This is an exploratory, on-demand context strategy. Instead of feeding the agent a massive, pre-filled context, you give it the tools to build its own context. The agent's primary actions are not to solve the final problem, but to ask questions about the environment. The shared store starts small and grows incrementally, populated only by the answers to the agent's own, self-directed inquiries. It's a pull model of information gathering, not a push model.

Concrete Example: The AI Data Analyst

Imagine an agent tasked with: "Find the total revenue from our top 5 customers last quarter."

A naive approach would be to dump the entire database schema into the initial prompt. The Scrying Spell approach is far more elegant:

Loop 1: Assess the Landscape
- shared store: {"goal": "Find revenue of top 5 customers last quarter"}
- DecideNode thinks: "I have no idea what tables are in the database. I need to look."
- Action: It calls list_tables().
- Update shared: The list of tables (['customers', 'orders', 'products']) is added to the shared store.
Loop 2: Zoom in on a Clue
- shared store: Now contains the goal and the list of tables.
- DecideNode thinks: "Okay, 'orders' and 'customers' seem relevant. I need to know what columns are in the 'orders' table to find revenue and dates."
- Action: It calls get_table_schema(table_name='orders').
- Update shared: The schema for the orders table ({'order_id': 'INT', 'customer_id': 'INT', 'order_date': 'DATE', 'total_amount': 'DECIMAL'}) is added.
Loop 3: Formulate the Attack
- shared store: Contains the goal, table names, and the orders schema.
- DecideNode thinks: "Perfect. Now I have everything I need to write a precise SQL query to get the answer."
- Action: It calls execute_sql(query="SELECT ...").

Notice the difference. The context was never overwhelming. It was built intelligently, step-by-step, by the agent itself. We didn't give it a map; we gave it a scrying orb and trusted it to find its own way.

4.2. Technique #2: The Grand Strategy (Map-Reduce)

The Metaphor: The warrior now faces not a dungeon, but an entire fortress. Trying to map it room by room would take forever. Instead, they first send a hawk into the sky. The hawk returns with a high-level sketch of the fortress: the barracks, the keep, and the treasury (Map phase). The warrior decides to tackle the keep first. They leave the main map behind and take only the detailed blueprint of the keep with them (Subtask Execution). After conquering the keep and taking its treasure, they return to the starting point, drop off the loot, and then take only the blueprint for the treasury for their next mission. Once all wings are cleared, they have all the treasure in one place (Reduce phase).

The Pattern: This is the classic divide-and-conquer strategy, perfectly suited for tasks that are too large for a single context window. The process is:

Map: A high-level planning step where the agent breaks a large problem down into smaller, independent subtasks or "chapters."
Subtask Execution (in parallel or sequence): For each subtask, the agent is run with a hermetically sealed context. It is only given the information relevant to that one subtask, completely ignorant of the others. This keeps the context small and focused.
Reduce: A final step where the results from all the independent subtasks are gathered and synthesized into a final, coherent output.

Concrete Example: The AI Codebase Knowledge Builder

This is the exact strategy used in our Codebase Knowledge Builder project, which turns an entire GitHub repository into a friendly tutorial. Stuffing a whole codebase into a prompt is impossible. Here's how the Grand Strategy makes it work:

Map Phase: The Hawk's View
- The IdentifyAbstractions and OrderChapters nodes act as the hawk. They scan the file structure and code at a high level (without reading every line) to create a plan.
- shared store output: A list of core concepts and a recommended chapter order, like: ["1. BaseNode", "2. Flow", "3. SharedMemory"].
Subtask Execution Phase: Conquering the Keep
- The WriteChapters BatchNode in PocketFlow executes this phase perfectly. It iterates through the plan.
- For Chapter 1 ("BaseNode"): Its prep method intelligently scans the shared['codebase'] and gathers only the code files relevant to BaseNode. It then calls the LLM with a tiny, focused prompt: "Write a chapter on BaseNode using only this specific code." The LLM is completely unaware of the code for Flow or SharedMemory, preventing confusion.
- For Chapter 2 ("Flow"): The process repeats, but this time with a completely different, isolated context containing only the code relevant to Flow.
Reduce Phase: Gathering the Loot
- The CombineTutorial node acts as the final organizer. It takes all the individually written chapter outputs from the shared store (which now contains the completed text for each chapter) and assembles them into a single, polished tutorial document with a table of contents and navigation.

Without this strategy, the task would be impossible. With it, we can conquer a fortress of any size, one well-planned, focused assault at a time.

4.3. Technique #3: The Cautious Explorer (Backtracking with Verification)

The Metaphor: Our warrior enters a room with a suspicious-looking lever. A reckless warrior pulls it and hopes for the best. A cautious warrior pulls it (Apply Change), but keeps one foot in the doorway, ready to jump back. They listen intently. Do they hear the satisfying click of a hidden door opening, or the terrifying rumble of a ceiling collapse? (Verify). If it's the rumble, they immediately let go of the lever, which springs back into place (Revert Change), and they proceed to look for a different solution. They are allowed to make mistakes, as long as they can observe the consequences and undo them.

The Pattern: This is a trial-and-error with a safety net strategy, essential for agents that modify their environment, like coding agents. The flow is cyclical:

Apply Change: The agent performs an action that alters the state (e.g., writes to a file).
Verify: A special node captures the consequence of that action. This isn't just the action's output; it's an observation of the new world state (e.g., a git diff, linter output, or a failing test result).
Decide & Revert: This verification result is fed back to the DecideNode. The LLM is then prompted: "You tried X, and the result was Y. Was this successful? If not, should we revert and try something else?" If it decides to revert, a special action restores the previous state.

Concrete Example: The AI Coding Agent

Imagine an agent tasked with fixing a bug in a Python file.

Loop 1: The Attempt
- DecideNode thinks: "Based on the bug report, I think the error is on line 52. I will change x > 5 to x >= 5."
- Action: The write_file node modifies the Python file.
Loop 2: The Verification
- The post hook of the write_file node is a special verify_code function. It doesn't just return "success." It runs the project's linter and unit tests.
- Verification Result: The linter passes, but a unit test now fails with a new AssertionError. This full error message is the output.
- Update shared: The shared store is updated with: {"last_attempt": "Changed line 52 to x >= 5", "verification_log": "Linter: OK. Tests: FAILED - AssertionError: Test case for x=5 failed."}
Loop 3: The Reassessment
- The DecideNode sees the previous attempt and the failed test.
- DecideNode thinks: "My last change was wrong. It broke a different test case. I must revert the change and try a different approach. The logic must be more complex than a simple comparison."
- Action: It calls revert_last_change(), followed by a new write_file with a completely different solution.

This loop of Apply -> Verify -> Revert allows the agent to safely explore the solution space without permanently breaking things. It can make hypotheses, test them, and backtrack if they prove false—a much more robust and realistic way to solve complex problems.

5. Conclusion: A Smart Warrior Navigates, a Fool Memorizes

And so, the secrets of the dungeon are yours. We've moved beyond simply forging a powerful warrior; we've now taught it how to navigate the most treacherous and complex environments. You now understand that an agent's true intelligence isn't measured by the size of its brain (the LLM) or the sharpness of its weapons (the action space), but by the wisdom of its context management.

A cluttered, unmanaged shared store—our warrior's cognitive backpack—is a recipe for a slow, confused, and expensive agent. But a well-managed one is the key to a focused, efficient, and surprisingly clever digital warrior.

You've learned the three master navigation techniques, transforming you from a mere agent blacksmith into a grand strategist:

The Scrying Spell (Context On-Demand): The ultimate tool for exploration, allowing your agent to build its own map of the unknown, piece by piece, without ever getting overwhelmed.
The Grand Strategy (Map-Reduce): Your weapon against overwhelming complexity, enabling your agent to conquer massive challenges like entire codebases by breaking them down into small, focused, and manageable battles.
The Cautious Explorer (Backtracking with Verification): The safety net that empowers your agent to make bold moves and try new things, secure in the knowledge that it can observe the consequences and gracefully retreat from any dead ends.

The next time you build an agent, don't just ask, "What can it do?" Instead, ask, "How will it think? How will it manage its focus?" By thoughtfully designing your agent's cognitive backpack, you are no longer just coding a workflow; you are imparting wisdom. You are creating a smart warrior that doesn't just memorize the map, but navigates the dungeon with purpose, clarity, and skill.

Ready to forge your own intelligent navigators? Dive into the code, experiment with these context strategies, and see how a well-managed backpack can transform your agents. Check out PocketFlow on GitHub and start building smarter today!

I just wrote a tutorial on the most overlooked part of building powerful LLM agents: the Action Space. https://dev.to/zachary62/llm-agents-arsenal-a-beginners-guide-to-the-action-space-n75

Zachary Huang — Tue, 01 Jul 2025 05:00:27 +0000

LLM Agent's Arsenal: A Beginner's Guide to the Action Space

Zachary Huang — Tue, 01 Jul 2025 04:44:52 +0000

Ever sent your AI agent into the "battle" of a complex task, only to watch it fumble with a blunt sword or use the wrong weapon for the fight? When an agent fails, our first instinct is to blame its "brain" (the LLM). But the real culprit is often the arsenal we equipped it with—the collection of weapons was dull, confusing, or simply not right for the job.

In our previous tutorial, LLM Agents are simply Graph — Tutorial For Dummies, we revealed that every agent is like a warrior following a simple battle plan: Assess -> Strike -> Repeat. We showed how the 'assessing' happens in a decision node that plans the next move. Now, it's time to forge the weapons used for the Strike.

That 'Strike' is powered by the agent's Arsenal—the official set of weapons, tools, and spells it can draw upon. In technical terms, this is its Action Space. This isn't just a list of functions; it is the very soul of your agent's power. A well-forged arsenal, where every blade is sharp and serves a unique purpose, is the difference between an agent that is defeated by the first obstacle and one that conquers any challenge.

In this guide, you are the master blacksmith. Using the transparent and powerful PocketFlow framework as your forge, we will teach you how to craft an arsenal of actions that will turn your agent from a clumsy squire into a legendary warrior.

The Battle Tactician: How an Agent Chooses Its Weapon

So, we have an arsenal. But how does the agent, our digital warrior, know when to draw a longsword for a close-quarters fight versus firing a bow from a distance?

This critical decision happens in the DecideAction node—the agent's battle tactician. At its core, every agent is just a simple loop that consults its tactician, who then chooses an action from the arsenal. The chosen action is performed, and the results are reported back to the tactician to plan the next move.

Visually, the battle plan looks like this:

In this diagram:

DecideAction (The Tactician): This is the brain. It analyzes the battlefield (the user's request and current data).
The Arrows (The Commands): Based on its analysis, the tactician issues a command: search_web, write_file, or answer_question. This is the branch in the graph.
The Action Nodes (The Specialists): Each command goes to a specialist soldier who executes that one task.
The Loop Back (The Report): After the specialist completes their task, they report back to the tactician with new information, and the cycle begins again.

"But what magic happens inside that DecideAction node?" you ask. "How does it actually think?"

This is the most misunderstood part of agent design, and the secret is shockingly simple. It's just a prompt. There's no complex algorithm, just a carefully written set of instructions for the LLM.

The tactician's "brain" is a prompt that looks something like this:

### CONTEXT
You are a research assistant. Here is the current situation:
Question: {the user's original question}
Previous Actions: {a log of what has been done so far}
Current Information: {any data gathered from previous actions}

### ARSENAL (Available Actions)
Here are the weapons you can use. Choose one.

[1] search_web
  Description: Search the internet for up-to-date information.
  Parameters:
    - query (str): The specific topic to search for.

[2] write_file
  Description: Save text into a local file.
  Parameters:
    - filename (str): The name of the file to create.
    - content (str): The text content to write into the file.

[3] answer_question
  Description: Provide the final answer to the user.
  Parameters:
    - answer (str): The complete, final answer.

## YOUR NEXT COMMAND
Review the CONTEXT and choose the single best ACTION from your ARSENAL to proceed.
Format your response as a YAML block.

That's it! The agent's entire decision-making process boils down to this: the LLM reads the description of the situation and the "user manual" for every weapon in its arsenal, and then it picks the one that makes the most sense.

The quality of its choice is 100% dependent on how clearly you describe its weapons. A sharp, well-defined arsenal in your prompt leads to a smart, effective agent. A vague, confusing one leads to a warrior who brings a knife to a dragon fight.

Now, let's learn how to forge these weapons, from simple daggers to god-tier magic spells.

Level Up Your Arsenal: The Three Tiers of Weapon Complexity

As a master blacksmith, you wouldn't forge just one type of weapon. You need a full range, from simple daggers for quick jabs to powerful, enchanted swords for epic battles. The same is true for your agent's arsenal. Actions can be designed with varying levels of power and complexity. Let's explore the three tiers.

Level 1: The Simple Dagger (The "Button" Action)

A simple dagger is a no-frills weapon. You draw it, you use it. It does one thing, and it does it reliably. These are actions that require no parameters.

Think of them as on/off switches or simple commands.

In the Forge (Code):
An action like request_human_help or finish_task.

In the Arsenal (Prompt Description):

[1] request_human_help
  Description: If you are stuck or need clarification, use this action to pause and ask the human user for guidance.

When to Use It:
For clear, binary decisions. When the agent needs to signal a state change, like "I'm finished," "I'm stuck," or "I've failed." They are perfect for controlling the overall flow of the battle plan.

Level 2: The Sharpshooter's Bow (The Parameterized Tool)

A bow is useless without an arrow and a target. This weapon requires input to be effective. These are the most common and versatile actions in an agent's arsenal—actions that require specific parameters to function.

To use these weapons, the agent must not only choose the bow but also aim it by providing the correct inputs.

In the Forge (Code):
An action like search_web(query) or send_email(to, subject, body).

In the Arsenal (Prompt Description):

[2] search_web
  Description: Searches the public internet for a given text string.
  Parameters:
    - query (str): The precise search term to look up. Must be a focused string.

[3] send_email
  Description: Composes and sends an email to a recipient.
  Parameters:
    - to (str): The email address of the recipient.
    - subject (str): The subject line of the email.
    - body (str): The main content of the email.

The Crucial Link to Your Blacksmithing Skills:
How does the agent provide these parameters? This is where your skill in structured output becomes critical. As we covered in our guide, Structured Output for Beginners, you must instruct the LLM to format its response in a structured way (like YAML or JSON) so your program can easily parse the action and its parameters.

Without this skill, you've given your agent a powerful bow but no way to nock an arrow.

Level 3: The Spellbook of Creation (The Programmable Action)

This is the ultimate weapon: a spellbook that doesn't contain a list of spells but teaches the agent how to write its own. These are programmable actions where the agent generates code or complex instructions on the fly.

This gives the agent god-like flexibility to solve novel problems you never explicitly trained it for.

In the Forge (Code):
An action like execute_sql(query) or run_python_code(code).

In the Arsenal (Prompt Description):

[4] execute_sql
  Description: Write and run a SQL query against the company's sales database. The database contains tables named 'customers', 'orders', and 'products'.
  Parameters:
    - sql_query (str): A valid SQL query string to execute.

[5] run_python_code
  Description: Write and execute a sandboxed Python script for complex calculations, data manipulation, or interacting with APIs.
  Parameters:
    - code (str): A string containing the Python code to run.

The Power and the Peril:
A spellbook is the most powerful weapon in your arsenal, but it's also the most dangerous.

Power: Your agent can solve almost any problem that can be expressed in code. It's no longer limited to pre-defined tools.
Peril: It's much more likely to make a mistake (e.g., writing buggy code). More importantly, it opens up massive security risks if not handled carefully (e.g., executing malicious code like os.remove("important_file.txt")). Always run such code in a secure, sandboxed environment.

Mastering these three tiers allows you to build a balanced and effective arsenal, equipping your agent for any challenge it might face.

Forging the Perfect Arsenal: 3 Golden Rules for Your Weapon Inventory

A legendary warrior doesn't just carry a random assortment of weapons. Their arsenal is carefully curated—each item is perfectly crafted, serves a distinct purpose, and is instantly accessible. As the master blacksmith for your agent, you must apply the same discipline. Here are the three golden rules for forging a world-class action space.

Golden Rule #1: Engrave a Crystal-Clear User Manual (Clarity is King)

The descriptions for your actions and their parameters are not notes for yourself; they are the user manual for the LLM. If the manual is vague, the LLM will misuse the tool. Be painfully, relentlessly explicit.

A Dull Blade (Bad Description):

search: searches for stuff

The agent sees this and thinks, "What stuff? How? What do I provide?" The result is a wild guess, like search(query="who won the 2024 Nobel Prize in Physics and what were their contributions in detail and also list prior winners"), a query too broad to be effective.

A Sharpened Katana (Good Description):

search_web(query: str):
  Description: Searches the public internet for up-to-date information on a single, specific topic. Returns the top 5 text snippets.
  Parameters:
    - query (str): A simple and focused search query, typically 3-5 words long.

Now the agent understands its constraints. It knows the tool is for one topic and the query should be short. It will correctly generate a command like search_web(query: "2024 Nobel Prize Physics winner"), leading to a much better outcome.

Golden Rule #2: Don't Burden Your Warrior with a Junk Drawer (Keep it Concise)

A warrior grabbing a weapon in the heat of battle can't afford to sift through a hundred options. They need a small, elite set of choices. Overwhelming the LLM with too many actions leads to confusion, slower decision-making (more tokens to process), and a higher chance of picking the wrong tool.

The Blacksmith's Guideline: An arsenal of 10 weapons is formidable. An arsenal of 100 is a junk drawer.

If your action space is growing too large, it's a sign that your tools are too granular. Instead of creating read_json_file, read_csv_file, and read_text_file, forge a single, more powerful weapon: read_file(filename: str). Your code can handle the internal logic of parsing different file types. Keep the agent's choices clean and high-level.

Golden Rule #3: Make Every Weapon Unique (Slay Redundancy)

Every weapon in the arsenal should have a unique purpose. If the agent has two tools that do similar things, it will get confused about which one to use. This is called a lack of "orthogonality."

The Confusing Arsenal (Bad Design):

read_csv_from_disk(file_path: str): Reads customer data from a local CSV file.
query_database(sql: str): Queries the live customer database.

The agent is asked to "find the total sales for new customers from this quarter." Which tool should it use? The data might be in the CSV, or it might be in the database. The agent doesn't know and might make the wrong choice.

The Pro-Gamer Move: Simplify the Battlefield
A true master blacksmith doesn't just forge weapons; they shape the battlefield to their advantage. Instead of giving the agent two ambiguous tools, do the work for it behind the scenes.

The Decisive Arsenal (Good Design):
Before the agent even starts, run a script that loads the CSV data into a temporary table in the database.

Now, the agent's arsenal is clean and unambiguous:

query_database(sql: str): Queries the customer database, which contains all known customer data.

The ambiguity is gone. The agent has one, and only one, tool for retrieving customer data. You've eliminated redundancy and made the agent's decision trivial, guaranteeing it makes the right choice every time.

Conclusion: An Agent is Only as Sharp as its Arsenal

And so, the secrets of the forge are yours. You now understand that the true power of an LLM agent doesn't come from some mysterious, hidden algorithm. It comes from the thoughtful, disciplined, and creative process of crafting its Action Space.

You've learned that agents are just warriors in a loop with branches, making decisions based on a prompt that serves as their battle plan. And you've seen how to stock their arsenal for any challenge:

With Simple Daggers for quick, decisive commands.
With Sharpshooter's Bows for precise, targeted actions.
With reality-bending Spellbooks of Creation for ultimate flexibility.

Most importantly, you now hold the three golden rules of the master blacksmith:

Engrave a Clear Manual: Your descriptions are the agent's guide to victory.
Avoid the Junk Drawer: A curated, concise arsenal is deadlier than a cluttered one.
Slay Redundancy: Make every weapon unique to ensure the agent never hesitates.

The next time you see a complex agent framework with thousands of lines of code, you won't be intimidated. You'll know to look past the noise and ask the fundamental questions: "What's in the arsenal? How is it described? Is it sharp, concise, and unique?"

Armed with this knowledge, you are no longer just a coder; you are an agent blacksmith. You have the power to forge not just tools, but intelligent, reliable, and effective digital warriors.

Ready to light the forge? Dive into the code and explore these principles in action by checking out PocketFlow on GitHub!

I just wrote a tutorial showing how to build an AI chatbot for your website that just works. https://dev.to/zachary62/the-easiest-way-to-build-an-ai-chatbot-for-your-website-full-dev-tutorial-37kp

Zachary Huang — Thu, 19 Jun 2025 03:15:20 +0000

The Easiest Way to Build an AI Chatbot for Your Website (Full Dev Tutorial)

Zachary Huang — Thu, 19 Jun 2025 03:05:57 +0000

Want to build an AI chatbot for your website, but worried about the complexity? Are you picturing a maintenance nightmare of endless data updates and complex pipelines? Good news. This tutorial shows you how to build a lightweight AI chatbot that learns directly from your live website. No vector databases, no manual updates—just a chatbot that works. The project is open-sourced on GitHub.

1. That "Simple" Chatbot Project... Isn't

So, you want to build an AI chatbot for your website. It sounds easy enough. You call an API, write a clever prompt, and you're basically done, right?

Except for one tiny, soul-crushing detail: Your brand-new AI knows... nothing.

It has no idea what your company sells, what your return policy is, or who you are. It's just an empty brain in a box. To make it useful, you have to feed it knowledge. And that's where the "simple" project becomes a total nightmare.

The Old, Broken Way to Build a Chatbot's Brain

Here’s the standard, painful process everyone seems to follow:

The Scavenger Hunt. First, you go on a company-wide scavenger hunt, digging through folders and old emails to find every PDF, FAQ, and policy document you can.
The Data Janitor Job. Then, you become a data janitor. You write a bunch of tedious scripts to chop all that messy information into clean little "chunks" the AI can understand.
The Expensive Brain Surgery. Finally, you perform some expensive brain surgery. You set up a complicated (and often pricey) "vector database" and shove all those data chunks into it.

After all that, you finally have a chatbot that knows things. For about a day.

And Now... Your Chatbot Is a Liar

The moment your bot goes live, it starts to rot.

The marketing team updates the pricing page. The engineers release a new feature. Suddenly, your chatbot is confidently telling customers the wrong price. It's a walking, talking liability. You didn't build a smart AI assistant. You built a manual-syncing, high-maintenance chore that you have to babysit forever.

But what if this entire approach is wrong? What if the knowledge base wasn't some clunky database you have to constantly update? What if... the website itself was the brain? That’s the chatbot we’re building today. A bot so simple, it feels like cheating.

This project is powered by PocketFlow, a tiny but mighty AI framework that makes building this kind of intelligent, looping agent incredibly straightforward. Forget vector databases and manual updates. Let's build a chatbot that just works.

2. Our Solution: A "Dumb" Crawler That's Actually Smart

Let's throw that entire, complicated process in the trash. We are not going to hunt for documents, clean up data, or set up a single database.

Instead, our chatbot will get its information directly from the source: your live website. Think of it like this. The old way is like printing a map once a year and hoping the roads don't change. Our new way is like using Google Maps on your phone—it's always live, always current.

The Master Plan: Let the Bot Read

Our chatbot works like a very fast, very focused intern. When a user asks a question, the bot doesn't look up the answer in some dusty old database. Instead, it visits your website and starts reading, right then and there.

Let's imagine your website has a realistic structure. A user asks a question that requires information from multiple places: "How do I get a refund for Product A?"

The bot needs to be smart. It has to navigate the site to find all the relevant pieces of the puzzle. In the diagram below, the lines show all the possible links. The dashed line shows the exact path our bot takes to find the answer.

Here's a play-by-play of the bot's clever thought process:

It starts on the Homepage. It sees both "refund" and "Product A" in the question. It decides to find the product page first to confirm the product's details.
It navigates to the "Product A" page. It reads the content and finds key info, like a "30-day warranty," but it doesn't find the process for actually getting a refund.
It intelligently changes course. It realizes the refund steps aren't on the product page. So, it thinks like a human would: "Okay, I need to find the general company policies." It navigates back to the site's main "Support" section to find the official information. It doesn't need a direct link; it understands the site's structure.
It finds the final piece of the puzzle. On the Support page, it sees a link to "Shipping & Returns Policy," reads it, and learns the exact steps to submit a refund request.

Now, it combines the "30-day warranty" from the product page with the "how-to steps" from the returns policy to give a perfect, comprehensive answer.

Why This is So Much Better

The beauty of this approach is its simplicity.

Your Knowledge is Always Fresh: You change your pricing? The bot knows instantly. You update your team bio? The bot knows that too. There is no sync step. There is no "stale data." Ever.
There is Zero Maintenance: You never have to tell the bot about updates. Just update your website like you normally would, and the chatbot takes care of the rest.

But what stops it from wandering off your site and crawling the entire internet? Simple. We give it a leash. We provide a list of approved website domains (like yourwebsite.com) and tell it: "You are only allowed to visit links on these sites. Don't go anywhere else."

This all sounds great, but building an agent that can make decisions and get stuck in a loop sounds complicated, right? You'd think you need a massive, heavy framework to manage that kind of logic.

Actually, you don't. And that’s where PocketFlow comes in.

3. PocketFlow: The Tiny Engine That Powers Our Bot

You wouldn't use a bulldozer to plant a single flower. In the same way, we don't need a massive, heavyweight AI framework for our straightforward crawling task. We need something small, fast, and built for exactly this kind of job.

That's why we're using PocketFlow. PocketFlow is a minimalist AI framework that's just 100 lines of code. It has zero dependencies and zero fluff. Let's look at its three core ideas.

The Node: A Specialist Worker

In PocketFlow, each task is a Node. A Node is like a specialist worker who is a pro at one specific thing. Here’s what a Node looks like in the actual PocketFlow code:

class BaseNode:
    def __init__(self):
        self.params, self.successors = {}, {}
    def prep(self, shared): pass
    def exec(self, prep_res): pass
    def post(self, shared, prep_res, exec_res): pass
    def run(self, shared):
        p = self.prep(shared)
        e = self.exec(p)
        return self.post(shared, p, e)

Don't worry if __init__ or self look weird; they're just Python things! The important bit is the prep -> exec -> post cycle:

prep(shared): "Hey, I'm about to start. What info do I need from the shared whiteboard?"
exec(data_from_prep): "Okay, I have my info. Now I'll do my main job!" (Like calling an AI).
post(shared, ..., ...): "Job's done! I'll write my results to the shared whiteboard and tell the manager what to do next by returning a signal (like a keyword, e.g., "explore" or "answer")."

The Shared Store: The Central Whiteboard

This is just a plain old Python dictionary (we'll call it shared). All our Node workers can read from it and write to it. It's how they pass information—like the user's question or the list of URLs to visit—to each other.

For our chatbot, it might look like this initially:

shared = {
    "user_question": "How do I get a refund?",
    "urls_to_process": ["https://example.com"],
    "visited_urls": set(),
    "final_answer": None
}

As Nodes do their work, they'll update this shared dictionary.

The Flow: The Workshop Manager

A Flow object is the manager of your workshop. You tell it which Node to start with, and it handles the rest. When you run a Flow, it just keeps doing one thing over and over:

Run the current Node.
The Node finishes and returns a signal (just a string, like "explore").
The Flow looks at the Node's connections to see where that signal leads, and moves to the next Node.

Here's how tiny the Flow manager class actually is in PocketFlow:

class Flow(BaseNode):
    def __init__(self, start): 
        # ... a bit of init code ...
        self.start = start
    def orch(self, shared, params=None):
        curr = self.start
        while curr:
            # ... a bit of setup code ...
            signal = curr.run(shared)
            curr = curr.successors.get(signal or "default")

That's it! It starts a while loop, runs a node, gets a signal, and finds the next node. If there's no next node for that signal, the loop ends.

Tiny Math Example: PocketFlow in Action!

Let's build a super-tiny workflow: take a number, add 5, then multiply by 2.

Worker 1: The Adder Node

class AddFive(BaseNode):
    def prep(self, shared):
        return shared.get("number_to_process", 0) # Read from whiteboard

    def exec(self, current_number):
        return current_number + 5 # Do the work

    def post(self, shared, prep_res, addition_result):
        shared["intermediate_result"] = addition_result # Write to whiteboard
        print(f"AddFive Node: Added 5, result is {addition_result}")

Notice post doesn't return anything? PocketFlow automatically treats that as the signal "default".

Worker 2: The Multiplier Node

class MultiplyByTwo(BaseNode):
    def prep(self, shared):
        return shared["intermediate_result"] # Read from whiteboard

    def exec(self, current_number):
        return current_number * 2 # Do the work

    def post(self, shared, prep_res, multiplication_result):
        shared["final_answer"] = multiplication_result # Write to whiteboard
        print(f"MultiplyByTwo Node: Multiplied, final answer is {multiplication_result}")

Connecting the Workers and Running the Flow:

# Create our specialist worker Nodes
adder_node = AddFive()
multiplier_node = MultiplyByTwo()

# Connect them: after adder_node is done, run multiplier_node
adder_node.add_successor(multiplier_node) 

# Create the Flow manager
math_flow = Flow(start=adder_node)

# Create the whiteboard with our starting number
shared_math_data = {"number_to_process": 10}
print(f"Starting math game with: {shared_math_data}")

# Run the flow!
math_flow.run(shared_math_data)

print(f"Math game finished. Whiteboard looks like: {shared_math_data}")

If you run this, you get exactly what you'd expect:

Starting math game with: {'number_to_process': 10}
AddFive Node: Added 5, result is 15
MultiplyByTwo Node: Multiplied, final answer is 30
Math game finished. Whiteboard looks like: {'number_to_process': 10, 'intermediate_result': 15, 'final_answer': 30}

See? Each Node is simple. The shared dictionary carries the data. The Flow manager makes sure AddFive runs, then MultiplyByTwo.

Now, just swap our math workers for chatbot workers:

AddFive becomes CrawlAndExtract.
MultiplyByTwo becomes AgentDecision.
And instead of just a "default" signal, AgentDecision will return "explore" to loop back or "answer" to move forward.

The pattern is exactly the same. Now that we have our blueprint, let's build the three "workers" that make our chatbot come to life.

4. Building the Brain: A Look Under the Hood

Alright, theory's over. Let's look at the actual code that makes our chatbot's brain tick. By the end of this section, you'll understand the entire backend, from the high-level workflow down to the individual "workers."

(Note: We've simplified the code below to focus on the core ideas. For the complete, unabridged version, you can view the full code in the project on GitHub.)

The Game Plan

First, let's look at our workflow diagram. This is the entire brain of our operation: a simple loop.

The Assembly Line Instructions (`flow.py`)

Before we build the individual workers, let's look at the instructions that tell them how to work together. This is our flow.py file, and it's the "manager" that directs the assembly line.

# From flow.py
from pocketflow import Flow
from nodes import CrawlAndExtract, AgentDecision, DraftAnswer

# 1. Create an instance of each "worker" node
crawl_node = CrawlAndExtract()
agent_node = AgentDecision()
draft_answer_node = DraftAnswer()

# 2. Define the connections and signals
crawl_node >> agent_node                 # After crawling, always go to the agent to decide.
agent_node - "explore" >> crawl_node     # If the agent says "explore", loop back to the crawler.
agent_node - "answer" >> draft_answer_node # If the agent says "answer", move to the writer.

# 3. Create the final flow, telling it where to start
support_bot_flow = Flow(start=crawl_node)

That's the entire orchestration logic. It's a simple, readable blueprint for our agent's behavior.

The Shared Whiteboard (`shared` dictionary)

Next, our workers need a central place to read and write information. This is just a simple Python dictionary that holds the state of our operation.

shared = {
    "user_question": "How do I get a refund?",
    "urls_to_process": [0], # A "to-do" list of URL indices to crawl
    "visited_urls": set(),  # A set of URL indices it has already crawled
    "url_content": {},      # Where it stores the text from each URL
    "all_discovered_urls": ["https://example.com"], # Master list of every URL
    "final_answer": None
}

The Workers (`nodes.py`)

Now let's look at the simplified code for our three specialist nodes.

1. `CrawlAndExtract`: The Librarian

This BatchNode efficiently processes a list of URLs. Its job is to read a page and return its text and any new links it finds.

class CrawlAndExtract(BatchNode):
    def prep(self, shared):
        return shared.get("urls_to_process", [])

    def exec(self, url_index):
        url = shared["all_discovered_urls"][url_index]
        content, new_links = crawl_webpage(url)
        return {"url_index": url_index, "content": content, "new_links": new_links}

    def post(self, shared, prep_res, all_results):
        for result in all_results:
            idx = result["url_index"]
            shared["url_content"][idx] = result["content"]
            shared["visited_urls"].add(idx)

            for link_url in result["new_links"]:
                if link_url not in shared["all_discovered_urls"]:
                    shared["all_discovered_urls"].append(link_url)

        shared["urls_to_process"] = []

In English: It crawls each page on its to-do list, stores the content, and adds any new, unique links to the master URL list.

2. `AgentDecision`: The Brain

This node looks at what we've learned and decides what to do next, returning a signal to the Flow.

class AgentDecision(Node):
    def prep(self, shared):
        knowledge = "\n".join(shared["url_content"].values())
        unvisited_urls = [url for i, url in enumerate(shared["all_discovered_urls"]) 
                          if i not in shared["visited_urls"]]

        return {
            "question": shared["user_question"],
            "knowledge": knowledge,
            "unvisited_urls": unvisited_urls
        }

    def exec(self, prepared_data):
        prompt = f"""
        User Question: {prepared_data['question']}
        Knowledge I have: {prepared_data['knowledge']}
        URLs to explore next: {prepared_data['unvisited_urls']}

        Should I 'answer' or 'explore'? If exploring, which URLs are best?
        Respond in YAML:
        decision: [answer/explore]
        selected_urls: [...]
        """
        response_yaml = call_llm(prompt)
        return parse_yaml(response_yaml)

    def post(self, shared, prep_res, decision):
        if decision["decision"] == "explore":
            selected_indices = [shared["all_discovered_urls"].index(url) 
                                for url in decision["selected_urls"]]
            shared["urls_to_process"] = selected_indices
            return "explore" # This signal matches our flow.py instruction
        else:
            return "answer" # This signal also matches flow.py

In English: It asks the AI for a strategy (answer or explore) and returns that exact signal to the Flow, which knows what to do next.

3. `DraftAnswer`: The Writer

Once the Brain says "answer", this node crafts the final response.

class DraftAnswer(Node):
    def prep(self, shared):
        knowledge = "\n".join(shared["url_content"].values())
        return { "question": shared["user_question"], "knowledge": knowledge }

    def exec(self, prepared_data):
        prompt = f"""
        Answer this question: {prepared_data['question']}
        Using ONLY this information:
        {prepared_data['knowledge']}
        """
        return call_llm(prompt)

    def post(self, shared, prep_res, final_answer_from_ai):
        shared["final_answer"] = final_answer_from_ai

In English: It gathers all the text we found, gives it to the AI, and asks it to write a beautiful, helpful response.

And that's the core of the system. Three simple nodes, each with a clear job, passing data through a simple dictionary.

Now that the magic is revealed (and you see it's not so magical after all), let's give our chatbot a pretty face so you can put it on your website.

5. Giving Our Bot a Face: From Terminal to Website

Okay, we have a functional AI brain that runs in the terminal. That's a great start, but it's not very useful for your website visitors.

Let's connect that brain to a user-friendly chat bubble. This is a classic web development pattern with two simple parts: a backend (our Python script) and a frontend (the chat bubble on a website).

The Architecture: A Brain and a Face

Think of it like this:

The Backend (The Brain): This is our Python script, server.py. Its only job is to wait for a question, run our PocketFlow logic to find the answer, and send the answer back. It's the powerhouse that does all the heavy lifting.
The Frontend (The Face): This is a small piece of JavaScript, chatbot.js, that you add to your website. It creates the chat icon and the chat window. When a user types a question, the JavaScript simply sends it to our backend for processing.

They communicate over the network. The frontend asks a question, and the backend provides the answer.

Let's look at the minimal code that makes each part work.

The Backend: `server.py`

We use a lightweight Python framework called FastAPI to create a simple web server. Its job is to expose a single "endpoint" (like a URL) that the frontend can send questions to.

Here’s the core logic in server.py:

from fastapi import FastAPI
from flow import support_bot_flow # Our PocketFlow brain

app = FastAPI()

@app.post("/get-answer") # An endpoint to receive questions
def get_answer(data: dict):
    # 1. Get the question and URL from the frontend
    question = data.get("question")
    start_urls = data.get("urls")

    # 2. Set up the shared dictionary for our flow
    shared_state = {"user_question": question, "all_discovered_urls": start_urls, ...}

    # 3. Run the PocketFlow brain to find the answer
    support_bot_flow.run(shared_state)

    # 4. Return the final answer as a response
    return {"answer": shared_state.get("final_answer")}

In English: The server waits for a POST request at /get-answer. When it gets one, it runs the same PocketFlow we built before and sends the result back.

The Frontend: `chatbot.js`

This is the JavaScript that lives on your website. It listens for the user to click "send," then makes a simple web request to our Python backend.

Here's the simplified logic from chatbot.js:

async function handleSendClick() {
    const userInput = document.getElementById("chat-input").value;
    const siteUrls = ["https://your-site.com"]; // The URLs for the bot to crawl

    // 1. Send the user's question to our Python backend
    const response = await fetch("/get-answer", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ question: userInput, urls: siteUrls })
    });

    // 2. Get the answer back from the server
    const data = await response.json();
    const botAnswer = data.answer;

    // 3. Display the bot's answer in the chat window
    displayMessage(botAnswer);
}

In English: When the user sends a message, it packages up the question and sends it to the /get-answer endpoint on our server. When the server responds, it displays the answer.

Running It Live

Now the process is clear:

Start the Backend: First, you need to run the brain. In your terminal, run python server.py. This starts the web server and gets it ready to answer questions.
Add the Frontend to a Page: Next, you add the <script src="chatbot.js"></script> tag to your website's HTML. This makes the chat bubble appear.

To make testing easy, the project includes a sample static/chatbot.html file that already has the script included. Once your server is running, just open that file in your browser to see your live, interactive chatbot in action

6. Conclusion: Simple, Maintainable, and Live

Let's take a step back. We just built a fully-functional AI chatbot that can intelligently answer questions about any website.

And we did it without touching a single vector database, writing a complex data-syncing script, or worrying about our information ever going stale. Its brain is your live website, which means its knowledge is always up-to-date.

This isn't just another chatbot. This is a better, simpler way to build one. Here’s why this approach wins:

Always Up-to-Date. Your bot’s knowledge is never stale. When you update your website, you've instantly updated your chatbot. There is no sync step, ever.
Practically Zero-Maintenance. You can finally "set it and forget it." Your only job is to keep your website current—something you were already doing anyway.
Incredibly Simple Code. Because the entire system is built on PocketFlow and a few straightforward Python scripts, the logic is easy to read and modify. There are no black boxes to fight with.

The days of babysitting your AI are over. You now have the blueprint for a system that’s not only intelligent but also practical and sustainable.

Ready to add a real-time brain to your own website?

The complete, open-source code for this chatbot is waiting for you on GitHub. It's powered by the 100-line PocketFlow framework. Dive in, experiment, and see for yourself how easy building a truly smart chatbot can be! Get the AI Website Chatbot Code on GitHub

Build an LLM Web App in Python from Scratch: Part 4 (FastAPI, Background Tasks & SSE)

Zachary Huang — Thu, 12 Jun 2025 01:44:07 +0000

Ever asked an AI to write a whole blog post, only to stare at a loading spinner for five minutes, wondering if your browser crashed? We've all been there. Today, we're fixing that. We'll build an AI web app that takes on heavy-duty tasks—like writing a full article—without freezing up. You'll see live progress updates in real-time, so you always know what the AI is up to. Ready to make long-running AI tasks feel fast and interactive? Let's get building!

1. The Spinner of Doom: Why Long AI Tasks Break Your App 😫

Imagine your new AI app is a genius content writer. You ask it to "write an article about space exploration," hit enter, and... the dreaded loading spinner appears. The whole page is frozen. You can't click anything. After a minute, your browser might even give you a "This page is unresponsive" error. Yikes.

This is the classic problem with long-running tasks on the web. Standard web apps work like this:

You ask for something. (e.g., "Generate my article!")
The server works on it. (e.g., Calls the AI, which takes 2-3 minutes)
You wait... and wait... (The connection is held open, your browser is stuck)
Finally, you get the result. (Or a timeout error!)

This is a terrible user experience. Users need to know their request is being handled and see that progress is being made.

Our Solution: The Smart Restaurant Analogy

Think of it like ordering food at a restaurant:

The Bad Way (Standard Web Request): You order a steak. The waiter stands at your table, staring at you without moving, for the entire 15 minutes it takes to cook. Awkward, right? You can't even ask for more water.
The Good Way (Our Approach): You order a steak. The waiter says, "Excellent choice! I'll put that in with the chef," and walks away (Background Task). A few minutes later, they bring you some bread ("Making progress!"). A bit later, your drink arrives ("Almost ready!"). You're happy and informed while the main course is being prepared in the background.

That's exactly what we're building today. Our app will:

Instantly confirm the user's request.
Offload the heavy AI work to a background task.
Stream live progress updates back to the user with Server-Sent Events (SSE).

Our toolkit for this mission:

🔧 FastAPI BackgroundTasks: For running the AI job without freezing the app.
🔧 Server-Sent Events (SSE): A simple way to push live updates from the server to the browser.
🔧 PocketFlow: To organize our multi-step article writing process.

Let's dive into the tools that make this magic possible.

You can find the complete code for the app we're building today in the PocketFlow cookbook: FastAPI Background Jobs with Real-time Progress.

2. Our Tools for the Job: Background Tasks & Server-Sent Events (SSE) 🛠️

To build our responsive AI article writer, we need two key pieces of technology: one to handle the work behind the scenes and another to report on its progress.

FastAPI `BackgroundTasks`: The "Work in the Back" Crew

Think of BackgroundTasks as giving a job to a helper who works independently. You tell FastAPI, "Hey, after you tell the user I got their request, please run this other function in the background."

The magic is that the user gets an immediate response. They don't have to wait for the background work to finish.

It's like ordering from Amazon. You get the "Order Confirmed!" email instantly. The actual process of picking, packing, and shipping your item happens later, in the background.

Here's a simple example: sending a welcome email after a user signs up.

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

# This is our slow task (e.g., calling an email service)
def send_welcome_email(email: str):
    import time
    time.sleep(5) # Simulate a 5-second delay
    print(f"Email sent to {email}! (This prints in the server console)")

@app.post("/signup")
async def user_signup(email: str, background_tasks: BackgroundTasks):
    # Add the slow email task to the background
    background_tasks.add_task(send_welcome_email, email)

    # Return a response to the user IMMEDIATELY
    return {"message": f"Thanks for signing up, {email}! Check your inbox soon."}

What's happening?

When you send a request to /signup, FastAPI sees background_tasks.add_task(...).
It immediately sends back the {"message": "Thanks..."} response. Your browser is happy and responsive.
After sending the response, FastAPI runs send_welcome_email() in the background.

This is perfect for our AI article generator! We'll use it to run the entire AI writing process.

Server-Sent Events (SSE): Your Live Progress Ticker

Okay, so the work is happening in the background. But how do we tell the user what's going on? That's where Server-Sent Events (SSE) come in.

SSE is a super simple way for a server to push updates to a browser over a single, one-way connection. It's like a live news ticker: the server sends new headlines as they happen, and your browser just listens.

Why not use WebSockets again?
WebSockets (from Part 3) are awesome for two-way chat. But for just sending one-way progress updates, they're a bit like using a walkie-talkie when all you need is a pager. SSE is simpler, lighter, and designed for exactly this "server-to-client" push scenario.

Here's how simple an SSE endpoint is in FastAPI:

import asyncio, json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def progress_generator():
    for i in range(1, 6):
        # In a real app, this would be a progress update from our AI
        yield f"data: {json.dumps({'progress': i*20, 'step': f'Step {i} done'})}\n\n"
        await asyncio.sleep(1) # Wait 1 second

@app.get("/stream-progress")
async def stream_progress():
    return StreamingResponse(progress_generator(), media_type="text/event-stream")

And on the browser side, the JavaScript is just as easy:

<div id="progressStatus">Waiting for progress...</div>
<script>
    const progressStatus = document.getElementById('progressStatus');
    const eventSource = new EventSource("/stream-progress"); // Connect to our stream!

    eventSource.onmessage = (event) => {
        const data = JSON.parse(event.data);
        progressStatus.textContent = `Progress: ${data.progress}% - ${data.step}`;
    };
</script>

When you open this page, eventSource connects to our endpoint, and the progressStatus div will update every second. Simple and effective!

3. The AI's To-Do List: A PocketFlow Workflow 📝

So, how does our AI actually write an article? It doesn't happen in one go. We need to give it a step-by-step plan, like a recipe. This is where PocketFlow helps us. It breaks the big job of "write an article" into small, manageable Nodes (or steps).

The Central Hub: Our `shared` Dictionary

Before we dive into the nodes, let's look at the brain of our operation: a simple Python dictionary. All our nodes will read from and write to this central shared data hub. It's how they pass information to each other.

Here's what it looks like at the start of a job:

shared_data = {
    "topic": "The user's article topic",
    "sse_queue": None, # The "mailbox" for progress messages
    "sections": [],    # Will be filled by the Outline node
    "draft": "",       # Will be filled by the Content node
    "final_article": "" # The final result!
}

The sse_queue is our "mailbox." Each node will drop progress updates into it, and our FastAPI server will read from it to update the user.

Step 1: `GenerateOutline` Node

This node's only job is to create the article's structure. (We'll assume a call_llm function exists that talks to the AI).

class GenerateOutline(Node):
    def prep(self, shared):
        return shared["topic"]

    def exec(self, topic):
        prompt = f"Create 3 section titles for an article on '{topic}'."
        return call_llm(prompt) # e.g., "Intro,Main Points,Conclusion"

    def post(self, shared, outline_str):
        sections = outline_str.split(',')
        shared["sections"] = sections

        progress = {"step": "outline", "progress": 33, "data": sections}
        shared["sse_queue"].put_nowait(progress)

What's happening here:

prep grabs the user's topic from the shared dictionary.
exec takes that topic and asks the AI for an outline.
post saves the outline for the next step and—most importantly—drops a progress message into the mailbox. This tells the frontend, "Hey, I'm 33% done!"

Step 2: `WriteContent` Node

This node takes the outline and writes the content for each section.

class WriteContent(Node):
    def prep(self, shared):
        return shared["sections"], shared["sse_queue"]

    def exec(self, prep_result):
        sections, queue = prep_result
        full_draft = ""
        for i, section in enumerate(sections):
            prompt = f"Write a paragraph for the section: '{section}'."
            paragraph = call_llm(prompt)
            full_draft += f"<h2>{section}</h2>\n<p>{paragraph}</p>\n"

            progress = {"step": "writing", "progress": 33 + ((i+1)*20)}
            queue.put_nowait(progress)

        return full_draft

    def post(self, shared, full_draft):
        shared["draft"] = full_draft

Why this is cool:

Inside the exec loop, after each paragraph is written, we immediately send another progress update.
This means the user will see the progress bar jump forward multiple times during this step, making the app feel very responsive.

Step 3: `ApplyStyle` Node

This is the final touch. It takes the combined draft and asks the AI to polish it.

class ApplyStyle(Node):
    def prep(self, shared):
        return shared["draft"]

    def exec(self, draft):
        prompt = f"Rewrite this draft in an engaging style: {draft[:500]}"
        return call_llm(prompt)

    def post(self, shared, final_article):
        shared["final_article"] = final_article

        progress = {"step": "complete", "progress": 100, "data": final_article}
        shared["sse_queue"].put_nowait(progress)

The grand finale:

This node does the final rewrite.
Crucially, it sends the complete message to the mailbox with progress: 100. This tells our frontend that the job is finished and the final article is ready!

Tying It All Together with a `Flow`

Finally, we just need to tell PocketFlow the order of our to-do list.

from pocketflow import Flow

def create_article_flow():
    outline_node = GenerateOutline()
    content_node = WriteContent()
    style_node = ApplyStyle()

    # Define the sequence: outline -> write -> style
    outline_node >> content_node >> style_node

    return Flow(start_node=outline_node)

This is super readable: outline_node runs, then content_node, then style_node. This Flow object is what our FastAPI background task will ultimately run.

Here is a visual summary of the entire process:

With our AI's "to-do list" ready, let's connect it to our FastAPI backend.

4. Connecting the Dots: The FastAPI Backend 🔗

Okay, our AI has its "to-do list" from PocketFlow. Now, let's build the web server that acts as the project manager. It will take requests from users, give the work to our AI, and report on the progress.

We'll walk through the main main.py file piece by piece.

Part 1: The Job Center

First, we need a place to keep track of all the article-writing jobs that are currently running. A simple Python dictionary is perfect for this.

# A dictionary to hold our active jobs
# Key: A unique job_id (string)
# Value: The "mailbox" (asyncio.Queue) for that job's messages
active_jobs = {}

Think of active_jobs as the front desk of an office. When a new job comes in, we give it a ticket number (job_id) and a dedicated mailbox (asyncio.Queue) for all its internal memos.

Part 2: Kicking Off the Job

This is the first thing a user interacts with. They send their article topic to our /start-job endpoint, which kicks off the whole process.

@app.post("/start-job")
async def start_job(background_tasks: BackgroundTasks, topic: str = Form(...)):
    job_id = str(uuid.uuid4())

    # Create a new, empty mailbox for this specific job
    sse_queue = asyncio.Queue()
    active_jobs[job_id] = sse_queue

    # Tell FastAPI: "Run this function in the background"
    background_tasks.add_task(run_article_workflow, job_id, topic)

    # IMMEDIATELY send a response back to the user
    return {"job_id": job_id}

Let's break that down:

Get a Ticket Number: job_id = str(uuid.uuid4()) creates a unique ID for this request.
Create the Mailbox: sse_queue = asyncio.Queue() creates the message queue. We then store it in our active_jobs dictionary using the job_id.
Hand off the Work: background_tasks.add_task(...) is the magic. It tells FastAPI, "Don't wait! After you send the response, start running the run_article_workflow function."
Instant Reply: The return {"job_id": job_id} is sent back to the user's browser right away. The user's page doesn't freeze!

Part 3: The Background Worker

This is the function that runs behind the scenes. It's the project manager that actually runs our PocketFlow to-do list.

def run_article_workflow(job_id: str, topic: str):
    sse_queue = active_jobs[job_id]
    shared = {
        "topic": topic,
        "sse_queue": sse_queue, # Here's where we pass the mailbox in!
        "sections": [],
        "draft": "",
        "final_article": ""
    }

    flow = create_article_flow()
    flow.run(shared) # Start the PocketFlow!

Here's the crucial connection:

It gets the correct sse_queue (mailbox) for this job from our active_jobs dictionary.
It creates the shared data dictionary and puts the sse_queue inside it.
When flow.run(shared) is called, our PocketFlow nodes now have access to this queue and can drop their progress messages into it!

Part 4: Streaming the Progress Updates

While the background task is running, the user's browser connects to our /progress/{job_id} endpoint to listen for updates.

This function looks a bit complex, but it's just a loop that checks the mailbox.

@app.get("/progress/{job_id}")
async def get_progress(job_id: str):

    async def event_stream():
        # First, find the right mailbox for this job
        if job_id not in active_jobs:
            yield 'data: {"error": "Job not found"}\n\n'
            return

        sse_queue = active_jobs[job_id]

        while True:
            # Wait for a new message to arrive in the mailbox
            progress_msg = await sse_queue.get()
            yield f"data: {json.dumps(progress_msg)}\n\n"

            # If the message says "complete", we're done!
            if progress_msg.get("step") == "complete":
                del active_jobs[job_id] # Clean up the job
                break

    return StreamingResponse(event_stream(), media_type="text/event-stream")

The logic is simple:

event_stream is a special async generator that can send (yield) data over time.
It finds the correct mailbox (sse_queue) for the job_id.
The while True: loop starts.
await sse_queue.get() pauses and waits until a message appears in the mailbox.
As soon as a PocketFlow node drops a message in, this line wakes up, grabs the message, and sends it to the browser with yield.
It keeps doing this until it sees the "step": "complete" message, at which point it cleans up and closes the connection.

And that's the whole system! It's a clean loop: the user starts a job, a background worker runs it while dropping messages into a mailbox, and a streamer reads from that mailbox to keep the user updated.

5. Mission Complete! Your App Now Handles the Heavy Lifting 💪

You did it! You've successfully built a web app that can handle long, complex AI tasks without breaking a sweat or frustrating your users. They get an instant response and live progress updates, making the whole experience feel smooth, interactive, and professional.

No more dreaded loading spinners or "page unresponsive" errors. Your app now works like a modern, intelligent assistant: it acknowledges your request, works on it diligently in the background, and keeps you informed every step of the way.

What you conquered today:

✅ No More Freezing UIs: Used FastAPI's BackgroundTasks to offload heavy AI work so your app stays responsive.
✅ Live Progress Updates: Mastered Server-Sent Events (SSE) to stream status updates from the server to the browser in real-time.
✅ Clean, Organized Logic: Structured a complex, multi-step AI job with PocketFlow, keeping your AI logic separate from your web code.
✅ Tied It All Together: Used an asyncio.Queue as a simple "mailbox" to let your background task communicate with your web server.

This architecture is a game-changer for building serious AI applications. You now have the skills to create tools that can generate reports, analyze data, or perform any other time-intensive task, all while keeping your users happy and engaged.

Our "Build an LLM Web App" Journey is Complete!

Part 1: Command-line AI tools ✅
Part 2: Interactive web apps with Streamlit ✅
Part 3: Real-time streaming chat with WebSockets ✅
Part 4 (You just crushed it!): Background tasks for heavy AI work ✅

Ready to see it all in action? Grab the complete code, including the HTML and PocketFlow nodes, from our cookbook: FastAPI Background Jobs with Real-time Progress. You've leveled up your AI dev skills in a big way. Now go build something amazing! 🚀

Build an LLM Web App in Python from Scratch: Part 3 (FastAPI & WebSockets)

Zachary Huang — Sun, 08 Jun 2025 02:01:56 +0000

Ever watched ChatGPT type back to you word by word, like it's actually thinking out loud? That's streaming AI in action, and it makes web apps feel incredibly alive! Today, we're building exactly that: a real-time AI chatbot web app where responses flow in instantly. No more staring at loading spinners! We'll use FastAPI for lightning-fast backends, WebSockets for live chat magic, and PocketFlow to keep things organized. Ready to make your web app feel like a real conversation? You can find the complete code for this part in the FastAPI WebSocket Chat Cookbook.

1. Why Your AI Web App Should Stream (It's a Game Changer!) 🚀

Picture this: You ask an AI a question, then... you wait. And wait. Finally, BOOM – a wall of text appears all at once. Feels clunky, right?

Now imagine this instead: You ask your question, and the AI starts "typing" back immediately – word by word, just like texting with a friend. That's the magic of streaming for AI web apps.

Why streaming rocks: It feels lightning fast, keeps users engaged, and creates natural conversation flow. No more "is this thing broken?" moments!

We're creating a live AI chatbot web app that streams responses in real-time. You'll type a message, and watch the AI respond word by word, just like the pros do it.

Our toolkit:

🔧 FastAPI – Blazing fast Python web framework
🔧 WebSockets – The secret sauce for live, two-way chat
🔧 PocketFlow – Our LLM framework in 100 lines

Quick catch-up on our series:

Part 1: Built command-line AI tools ✅
Part 2: Created interactive web apps with Streamlit ✅
Part 3 (You are here!): Real-time streaming web apps 🚀
Part 4 (Coming next!): Background tasks for heavy AI work

Want to see streaming in action without the web complexity first? Check out our simpler guide: "Streaming LLM Responses — Tutorial For Dummies".

Ready to make your AI web app feel like magic? Let's dive in!

2. FastAPI + WebSockets = Real-Time Magic ⚡

To build our streaming chatbot, we need two key pieces: FastAPI for a blazing-fast backend and WebSockets for live, two-way chat.

FastAPI: Your Speed Demon Backend

FastAPI is like the sports car of Python web frameworks – fast, modern, and async-ready. Perfect for AI apps that need to handle multiple conversations at once.

Most web apps work like old-school mail: Browser sends request → Server processes → Sends back response → Done. Here's a basic FastAPI example:

from fastapi import FastAPI
app = FastAPI()

@app.get("/hello")
async def say_hello():
    return {"greeting": "Hi there!"}

What's happening here?

app = FastAPI() – Creates your web server
@app.get("/hello") – Says "when someone visits /hello, run the function below"
async def say_hello() – The function that handles the request
return {"greeting": "Hi there!"} – Sends back JSON data to the browser

When you visit http://localhost:8000/hello, you'll see {"greeting": "Hi there!"} in your browser!

Your First FastAPI App Flow:

Simple enough, but for chatbots we need something more interactive...

WebSockets: Live Chat Superpowers

WebSockets turn your web app into a live phone conversation. Instead of sending messages back and forth, you open a connection that stays live for instant back-and-forth chat.

Here's a simple echo server that repeats whatever you say:

from fastapi import FastAPI, WebSocket

app = FastAPI()

@app.websocket("/chat")
async def chat_endpoint(websocket: WebSocket):
    await websocket.accept() # Pick up the call!
    while True:
        message = await websocket.receive_text() # Listen
        await websocket.send_text(f"You said: {message}") # Reply

The browser side is just as simple:

<input id="messageInput" placeholder="Say something..."/>
<button onclick="sendMessage()">Send</button>
<div id="chatLog"></div>

<script>
    const ws = new WebSocket("ws://localhost:8000/chat");
    const chatLog = document.getElementById('chatLog');

    ws.onmessage = (event) => {
        chatLog.innerHTML += `<p>Server: ${event.data}</p>`;
    };

    function sendMessage() {
        const message = document.getElementById('messageInput').value;
        ws.send(message);
        chatLog.innerHTML += `<p>You: ${message}</p>`;
        document.getElementById('messageInput').value = '';
    }
</script>

WebSocket Chat Flow:

That's it! You now have live, real-time communication between browser and server. Perfect foundation for our streaming AI chatbot!

3. Adding AI to the Mix: Why Async Matters 🤖

Great! We have live chat working. But here's the thing: calling an AI like ChatGPT takes time (sometimes 3-5 seconds). If our server just sits there waiting, our whole web app freezes. Not good!

The problem: Normal code is like a single-lane road. When the AI is thinking, everything else stops.

The solution: Async code is like a highway with multiple lanes. While AI is thinking in one lane, other users can chat in other lanes!

PocketFlow Goes Async

Remember PocketFlow from our earlier tutorials? It helps break down complex tasks into simple steps. For web apps, we need the async version:

AsyncNode – Each step can wait for AI without blocking others
AsyncFlow – Manages the whole conversation workflow

Here's the magic difference:

# ❌ This blocks everything
def call_ai(message):
    response = openai.chat.completions.create(...)  # Everyone waits!
    return response

# ✅ This lets others keep chatting
async def call_ai_async(message):
    response = await openai.chat.completions.create(...)  # Just this task waits
    return response

Streaming Chat Node: The Star of the Show

Our StreamingChatNode does three things:

Prep: Add user message to chat history
Execute: Call AI and stream response word-by-word via WebSocket
Post: Save AI's complete response to history

class StreamingChatNode(AsyncNode):
    async def prep_async(self, shared):
        # Add user message to history
        history = shared.get("conversation_history", [])
        history.append({"role": "user", "content": shared["user_message"]})
        return history, shared["websocket"]

    async def exec_async(self, prep_result):
        messages, websocket = prep_result

        # Stream AI response word by word
        full_response = ""
        async for chunk in stream_llm(messages):
            full_response += chunk
            await websocket.send_text(json.dumps({"content": chunk}))

        return full_response

    async def post_async(self, shared, prep_res, exec_res):
        # Save complete AI response
        shared["conversation_history"].append({
            "role": "assistant", 
            "content": exec_res
        })

That's it! The node streams AI responses live while keeping chat history. Next, let's see how this all connects together!

4. Putting It All Together: The Complete Streaming Flow 🔄

Time to connect all the pieces! Here's how a user message flows through our streaming chatbot:

The Journey of a Message:

User sends message → FastAPI receives it → PocketFlow handles AI logic → Streams response back live!

The FastAPI WebSocket Handler

Here's the main FastAPI code that ties everything together:

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    chat_memory = {
        "websocket": websocket, 
        "conversation_history": []
    }

    try:
        while True:
            # Get user message
            user_data = await websocket.receive_text()
            message = json.loads(user_data)  # {"content": "Hello!"}
            chat_memory["user_message"] = message["content"]

            # Run our PocketFlow
            chat_flow = create_streaming_chat_flow()
            await chat_flow.run_async(chat_memory)

    except WebSocketDisconnect:
        print("User left the chat")

def create_streaming_chat_flow():
    return AsyncFlow(start_node=StreamingChatNode())

What happens:

Accept WebSocket connection
Wait for user messages in a loop
For each message, run our StreamingChatNode
The node handles AI calling + streaming automatically!

Note: Each WebSocket connection gets its own chat_memory dictionary with the live connection, latest message, and full conversation history. This lets each user have independent conversations while the AI remembers context.

Frontend: The Streaming Magic in JavaScript

On the browser side, we need just a few lines to make streaming work:

<div id="aiResponse"></div>
<input id="userInput" placeholder="Type your message..."/>
<button onclick="sendMessage()">Send</button>

<script>
const ws = new WebSocket("ws://localhost:8000/ws");
const aiResponse = document.getElementById("aiResponse");

// The magic: append each chunk as it arrives
ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    if (data.content) {
        aiResponse.textContent += data.content; // Stream word by word!
    }
};

function sendMessage() {
    const input = document.getElementById("userInput");
    aiResponse.textContent = ""; // Clear for new response
    ws.send(JSON.stringify({content: input.value}));
    input.value = "";
}
</script>

The streaming happens in ws.onmessage – each time the server sends a text chunk, we append it to the display. That's how you get the "typing" effect!

Pretty neat, right? You now have all the pieces for a real-time streaming AI chatbot!

5. Mission Accomplished! You Built a Real-Time AI Chatbot 🎉

Boom! You just built a streaming AI chatbot web app that feels like magic. No more waiting around – your AI responds word by word, just like the pros!

What you crushed today:

⚡ FastAPI + WebSockets – Live, two-way chat that never gets old
🔄 Async PocketFlow – AI calls that don't freeze your app
🚀 Streaming responses – Watch the AI "type" in real-time

You've officially joined the ranks of developers building modern, responsive AI web apps. Pretty cool, right?

What's next in our series:

Part 1: Command-line AI tools ✅
Part 2: Interactive web apps with Streamlit ✅
Part 3 (You just finished!): Real-time streaming ✅
Part 4 (Coming up!): Background tasks for heavy AI work

Ready for the big leagues? Part 4 will tackle those marathon AI tasks – think generating reports or complex analyses that take minutes, not seconds. We'll explore background processing and Server-Sent Events to keep users happy even during the heavy lifting.

Want to try this yourself? Grab the complete code from the PocketFlow cookbook: FastAPI WebSocket Chat Example You're building some serious AI web development skills! See you in Part 4! 🚀

I just wrote a simple tutorial showing how to build your own Python LLM web app using Streamlit from scratch! https://dev.to/zachary62/build-an-llm-web-app-in-python-from-scratch-part-2-streamlit-fsm-277g

Zachary Huang — Thu, 05 Jun 2025 07:31:44 +0000

Build an LLM Web App in Python from Scratch: Part 2 (Streamlit & FSM)

Zachary Huang — Thu, 05 Jun 2025 07:11:26 +0000

Ever wanted to create your own AI-powered image generator, where you call the shots on the final masterpiece? That's exactly what we're building today! We'll craft an interactive web application that lets users generate images from text prompts and then approve or regenerate them – all within a user-friendly interface. We'll use PocketFlow for workflow management, and you can check out the complete example we're building.

1. Want to Build Your Own AI Art Director? Let's Do It! 🎨

Imagine you're an artist, and you have a super-smart assistant (an AI) who can paint anything you describe. You tell it, "Paint a cat and an otter on a sofa!" The AI quickly paints a picture. But maybe the cat looks a bit grumpy, or the otter is on the floor instead of the sofa. Wouldn't it be great if you could tell the assistant, "Make the cat look friendlier," or "Put the otter on the sofa next to the cat," and it would repaint it?

That's where you come in as the director. You get to say "Nope, try again!" or "Perfect, I love it!" This back-and-forth between you and the AI is called Human-in-the-Loop (HITL), and it's exactly what we're building today.

We're creating a web app where you can: (1) Type what you want (like "a robot eating pizza on Mars"), (2) See what the AI creates, (3) Approve it or ask for a do-over, and (4) Keep the final masterpiece.

Don't worry – we're keeping it simple with just a few Python tools:

🔧 Streamlit – Turns your Python code into a web app. No HTML/CSS needed!
🔧 PocketFlow – Organizes our AI tasks like a recipe (we used this in Part 1)
🔧 Finite State Machine (FSM) – Keeps track of where we are in the process (typing → generating → reviewing → done)

This is part 2 of our 4-part journey of LLM web app tutorial:

Part 1: Built the basic HITL system in the command line ✅
Part 2 (You are here!): Making it a real web app with Streamlit 🚀
Part 3: Adding real-time chat features
Part 4: Handling long AI tasks in the background

Ready to turn your Python skills into a real web app? Let's dive in!

Want to see the final result? Check out the complete code example we're building.

2. Streamlit 101: A Web App in a Single Python File!

So we've picked Streamlit to build our AI Image Generator. Why is this perfect for Python folks who don't want to mess with HTML and CSS? Streamlit gives you a full package of UI components (buttons, text boxes, sliders, charts) right out of the box – no web design skills required! Plus, it has this brilliantly simple approach called the "rerun" model.

Streamlit's Big Idea: Just Rerun It! 🔄

Here's the wild part – and it's dumb simple: Every time a user does something (clicks a button, types text), Streamlit runs your entire Python script from top to bottom. Again. Every single time.

"Wait, the whole script? Isn't that... slow?" you might think. Nope! Streamlit is crazy fast at this, and it actually makes everything simpler. Think of it like a chef who makes your entire meal fresh every time you ask for extra salt – except this chef works at lightning speed.

Why this "rerun everything" approach rocks:

Feels natural: You write normal Python code, step by step
No complicated UI updates: Streamlit automatically redraws everything based on your latest script run
Easy to understand: Your code flows from top to bottom, just like you'd expect

Your First Streamlit App: "Hello, Clicks!" 👋

Let's see this in action with a simple click counter. Create a file called hello_clicks.py:

import streamlit as st

st.title("Hello, Clicks!")

# Think of st.session_state as Streamlit's memory notebook
if 'click_count' not in st.session_state:
    st.session_state.click_count = 0

if st.button("Click Me!"):
    st.session_state.click_count += 1

st.write(f"Button has been clicked: {st.session_state.click_count} times")

Run it with: streamlit run hello_clicks.py

What Happens When You Click? (The Rerun Dance!)

Let's follow what happens when you click the button for the first time, like steps in a dance:

First Time: Setting the Stage (Grey Box):

You open the app in your Web Page (Browser)
The Python Script (hello_clicks.py – our play) runs for the very first time
It looks at the SessionState (our whiteboard) and sees 'click_count' isn't there
So, it writes st.session_state.click_count = 0 on the whiteboard
The script then tells the Web Page to show the button and the text "Count: 0"

User Clicks Button:

The User clicks the "Click Me!" button on the web page

Button Click: Perform the Play Again! (Blue Box):

Streamlit sees the click and tells the Script (the play) to run all over again, from the first line to the last
This time, when the script checks the SessionState (whiteboard), it finds 'click_count' (its value is 0)
The line if st.button("Click Me!"): is true (because this specific button click is what caused the rerun). So, the script updates st.session_state.click_count to 1 (0 + 1)
The script finishes, and the Web Page shows the updated UI with "Count: 1"

Every time you click again, the steps in the "Button Click Rerun" (blue box) repeat. The st.session_state.click_count keeps going up because SessionState (the whiteboard) remembers its value between each rerun of the play.

`st.session_state`: Your App's Memory Bank 🧠

Since your script reruns from scratch each time, regular Python variables would forget everything. That's where st.session_state comes in – it's like a personal notepad that Streamlit gives each user to remember important stuff.

What goes in this memory bank? Anything your app needs to remember: user inputs, calculation results, which screen you're on, and for our image generator – the user's prompt, the generated image, and where we are in the process.

Connecting to Part 1: Remember the shared_data dictionary from our command-line app? st.session_state is exactly that, but for web apps. Instead of passing data between PocketFlow nodes in an in-mem dictionary, we'll store it in st.session_state.

Now that you get how Streamlit works, let's plan out our image generation workflow!

3. Planning Our AI Image Generator (The PocketFlow Blueprint)

Time to plan our masterpiece! Before we start clicking buttons and typing prompts, let's map out what our AI Image Generator actually needs to do. Think of this like sketching before you paint – we want to get the big picture right first.

We'll use PocketFlow to organize our workflow. Remember from Part 1? It's our trusty tool for connecting different steps together in a logical sequence.

The Basic Journey: From Idea to Image

Our app has a simple but powerful flow: (1) User types what they want, (2) AI generates an image, (3) User reviews it and decides if it's good, and (4) Either keep it or try again. That's it! Here's how it looks:

PocketFlow Nodes: The Building Blocks

In PocketFlow, each step is a "Node" – think of them as LEGO blocks that do specific jobs. Every Node has three parts:

🔧 prep – Grabs what it needs from st.session_state (our memory bank)
⚡ exec – Does the actual work (like calling the AI or showing a button)
📝 post – Saves results back to st.session_state and signals what's next

Let's see these in action:

Node 1: Getting the User's Idea

class GetPromptNode(Node):
    def exec(self, prep_res):
        prompt = input("What do you want to see?")
        return prompt

    def post(self, shared, prep_res, exec_res_prompt):
        shared["task_input"] = exec_res_prompt  # Save to memory
        return "prompt_ready"  # Signal: we're ready for the next step!

Node 2: AI Creates the Magic

class GenerateImageNode(Node):
    def prep(self, shared):
        return shared.get("task_input")  # Get the prompt from memory

    def exec(self, prep_res_prompt):
        image_data = generate_image(prep_res_prompt)
        return image_data

    def post(self, shared, prep_res_prompt, exec_res_image_data):
        shared["generated_image"] = exec_res_image_data  # Save image
        return "image_generated"  # Signal: image is ready!

Node 3: User Reviews the Result

class ReviewImageNode(Node):
    def prep(self, shared):
        display(shared["generated_image"])

    def exec(self, prep_res):
        choice = input("Approve (a) or Regenerate (r)?")
        return choice

    def post(self, shared, prep_res, exec_res_choice):
        if exec_res_choice == 'a':
            shared["final_result"] = shared["generated_image"]
            return "approved"
        else:
            return "regenerate"  # Try again!

Node 4: Celebrating the Final Result

class ShowFinalNode(Node):
    def prep(self, shared):
        display(shared["generated_image"])

    def exec(self, prep_res):
        choice = input("Start Over (s)?")
        return choice

    def post(self, shared, prep_res, exec_res_choice):
        if exec_res_choice == 's':
            # Clear memory for fresh start
            for key in ["task_input", "generated_image", "final_result"]:
                shared.pop(key, None)
            return "start_over"

Connecting the Dots

PocketFlow makes connecting these steps super clean:

def create_image_generator_flow():
    # Create our building blocks
    get_prompt = GetPromptNode()
    generate_image = GenerateImageNode()
    review_image = ReviewImageNode()
    show_final = ShowFinalNode()

    # Connect them with signals
    get_prompt     - "prompt_ready"    >> generate_image
    generate_image - "image_generated" >> review_image
    review_image   - "approved"        >> show_final
    review_image   - "regenerate"      >> generate_image  # Loop back!
    show_final     - "start_over"      >> get_prompt      # Start fresh!

    return Flow(start_node=get_prompt)

The Web App Challenge: No More `input()`

Here's the thing: command-line apps can pause and wait with input("What's next?"). Web apps can't do that! A web server needs to stay responsive for all users, not freeze while waiting for one person to click a button.

That's where we need a different approach – and that's exactly what we'll solve with Finite State Machines in the next section!

4. FSM to the Rescue: Managing Interactive Web States

Remember the challenge we just hit? Command-line apps can pause and wait with input(), but web apps need to stay responsive. That's where Finite State Machines (FSMs) come to the rescue!

Think of an FSM like a roadmap for your app. Instead of getting lost wondering "where am I in the process?", your app always knows exactly which "room" it's in and what should happen next.

You can see the complete working example at: PocketFlow Streamlit FSM Example.

What's a Finite State Machine? 🤔

An FSM is just a fancy way to organize your app into different "modes" or "screens":

🏠 States – Different rooms your app can be in:

initial_input: "Hey, tell me what you want to see!"
user_feedback: "Here's your image. Like it or want me to try again?"
final: "Awesome! Here's your approved masterpiece!"

🚪 Transitions – How you move between rooms:

User clicks "Generate" → move from initial_input to user_feedback
User clicks "Approve" → move from user_feedback to final
User clicks "Start Over" → move from final back to initial_input

Visualizing Our App's States

Here's how our image generator flows between states:

How FSM + Streamlit + PocketFlow Work Together

Here's the beautiful part: st.session_state becomes our master control center. It tracks both which state we're in AND all our PocketFlow data.

# Initialize our app's memory
if 'state' not in st.session_state:
    st.session_state.state = "initial_input"  # Start here
    st.session_state.task_input = ""
    st.session_state.generated_image = ""
    st.session_state.final_result = ""

Then we use simple if/elif blocks to show different screens based on the current state:

The Magic: State-Based UI

# State 1: Getting user input
if st.session_state.state == "initial_input":
    st.header("🎨 What do you want to see?")
    prompt = st.text_area("Describe your image:")

    if st.button("Generate Image"):
        st.session_state.task_input = prompt

        # Run PocketFlow node
        image_node = GenerateImageNode()
        image_node.run(st.session_state)

        # Move to next state
        st.session_state.state = "user_feedback"
        st.rerun()

# State 2: User reviews the image
elif st.session_state.state == "user_feedback":
    st.header("🖼️ Here's your image!")
    st.image(st.session_state.generated_image)

    col1, col2 = st.columns(2)
    with col1:
        if st.button("👍 Approve"):
            st.session_state.final_result = st.session_state.generated_image
            st.session_state.state = "final"
            st.rerun()

    with col2:
        if st.button("🔄 Try Again"):
            # Run PocketFlow node again
            image_node = GenerateImageNode()
            image_node.run(st.session_state)
            # Stay in same state, just refresh
            st.rerun()

# State 3: Show final approved image
elif st.session_state.state == "final":
    st.header("🎉 Your Masterpiece!")
    st.success("Image approved!")
    st.image(st.session_state.final_result)

    if st.button("🆕 Start Over"):
        # Reset everything
        st.session_state.task_input = ""
        st.session_state.generated_image = ""
        st.session_state.final_result = ""
        st.session_state.state = "initial_input"
        st.rerun()

The Power of This Approach

🔄 No More Confusion: Your app always knows exactly where it is and what should happen next.
🎯 Clean Code: Each state handles its own UI and logic – no messy spaghetti code!
🚀 Streamlit Friendly: Works perfectly with Streamlit's rerun model.
🔧 PocketFlow Integration: Nodes run smoothly within state transitions.
👥 User-Friendly: Each state shows exactly what the user needs to see and do.

This FSM approach transforms our PocketFlow workflow from a linear command-line script into an interactive web experience. Users can bounce between states naturally, and your code stays organized and predictable.

5. Mission Accomplished! What's Next? 🚀

Boom! We just built a fully interactive AI Image Generator web app. Look what we achieved: (1) Streamlit handles the UI magic, (2) FSM keeps our app states organized, (3) PocketFlow manages our AI workflow, and (4) Users get a smooth, intuitive experience.

What we learned:

FSMs make interactive web apps way easier to manage
st.session_state is perfect for both FSM states and PocketFlow data
Streamlit + FSM + PocketFlow = a powerful combo for AI apps

Our journey so far:

Part 1: Built HITL logic with PocketFlow (command line) ✅
Part 2 (This part!): Created an interactive web app with Streamlit + FSM ✅
Part 3 (Coming up!): Real-time features with FastAPI & WebSockets
Part 4: Background processing and progress updates

Ready to add real-time superpowers to your AI apps? Part 3 will show you how FastAPI and WebSockets can create instant, live interactions – think real-time chat with AI or live image generation updates!

Want to try this out yourself? You can find the complete, runnable code in the PocketFlow cookbook here: PocketFlow Streamlit FSM Example

Build an LLM Web App in Python from Scratch: Part 1 (Local CLI)

Zachary Huang — Sun, 01 Jun 2025 22:33:45 +0000

Ever thought about sprinkling some AI magic into your web app? This is Part 1 of our journey from a basic command-line tool to a full-blown AI web app. Today, we're building a "Human-in-the-Loop" (HITL) chatbot. Think of it as an AI that politely asks for your "OK" before it says anything. We'll use a tiny, super-simple tool called PocketFlow – it's just 100 lines of code!

So, You Want to Add AI to Your Web App?

Adding AI (especially those smart Large Language Models or LLMs) to web apps can make them super powerful. Imagine smart chatbots, instant content creation, or even coding help. But hold on, it's not just "plug and play." You'll bump into questions like:

Where does the AI brain actually live? In the user's browser? On your server?
How do you handle AI tasks that need multiple steps?
How can users tell the AI what to do or give feedback?
What's the deal with remembering conversation history?
Most importantly: How do you stop the AI from saying something totally weird or wrong?

This tutorial series is your guide to building LLM web apps in Python, and we'll tackle these questions head-on! Here's our 4-part adventure:

Part 1 (That's Today!): We'll build a basic Human-in-the-Loop (HITL) chatbot that runs in your computer's command line. We'll use PocketFlow to nail down the core logic, no fancy UI to distract us.
Part 2: Let's get this HITL bot on the web! We'll build a user interface using Streamlit (or maybe Flask), and learn how to manage user interactions smoothly.
Part 3: Time for real-time action! We'll upgrade our web app with WebSockets (using FastAPI) for instant messages and a slicker chat feel.
Part 4: What about AI tasks that take a while? We'll set up background processing and use Server-Sent Events (SSE) with FastAPI. This lets us show users live progress updates, making our app feel truly pro.

To make this journey smooth, we'll use PocketFlow. It's not just for simple AI calls; PocketFlow helps you build complex AI workflows that you can actually control. And because it's so small (seriously, just 100 lines for the core framework), you'll always know what's going on. No black boxes!

The secret sauce to control and quality? A pattern called Human-in-the-Loop (HITL). With PocketFlow, HITL is easy. It means your AI is like a helpful assistant: it drafts stuff, but you (the human co-pilot) get the final say. Approve, edit, or reject – you're in control before anything goes live. Total quality control, phew!

You can try out the code for the command-line HITL bot discussed in this part at: PocketFlow Command-Line HITL Example.

Your First HITL App: The "Are You Sure?" Bot

Let's build the most basic HITL app: a chatbot where you approve every single AI response. Imagine it's an AI trying to tell jokes. AI can be funny, but sometimes its jokes might be... well, not funny, or maybe even a little weird or inappropriate. That's where you come in!

The Basic Idea (It's Super Simple!)

Think of a tiny script. The AI suggests a joke, and you, the human, give it a thumbs-up or thumbs-down before it's told. That's HITL in action! If the AI suggests, "Why did the chicken cross the playground? To get to the other slide!" and you think it's a winner, you say "yes!" If it suggests something that makes no sense, or isn't right for your audience, you say "nope!"

user_topic_request = "Tell me a joke about cats."
ai_suggested_joke = call_llm_for_joke(user_topic_request)

# You get to approve!
approval = input(f"AI suggests: '{ai_suggested_joke}'. Tell this joke? (y/n): ")

if approval.lower() == 'y':
    print(f"To Audience: {ai_suggested_joke}")
else:
    print("Human said: Nope! That joke isn't hitting the mark.")

That's the core! AI suggests a joke, human (input()) approves, then it's (maybe) told. This simple check is crucial for joke quality and appropriateness. Everything else we build is about making this core idea robust for more complex apps. Visually, it's a little loop, especially for our joke bot:

Let's See It Go!

You: Tell me a programming joke.
AI suggests: 'Why do programmers prefer dark mode? Because light attracts bugs!'
Approve? (y/n): y
To Audience: Why do programmers prefer dark mode? Because light attracts bugs!

You: Tell me a joke about vegetables.
AI suggests: 'Why did the tomato turn red? Because it saw the salad dressing! But also, all vegetables are boring.'
Approve? (y/n): n
Human said: Nope! That joke isn't hitting the mark. (And the comment about vegetables is a bit much!)
Regenerating... (Imagine we have logic for this)
AI suggests: 'What do you call a sad strawberry? A blueberry!'
Approve? (y/n): y
To Audience: What do you call a sad strawberry? A blueberry!

See? The human caught that slightly off-kilter joke and the unnecessary comment. That's HITL making sure the AI comedian doesn't bomb too hard!

Why Just Plain Python Gets Tangled Fast

When you try to build something more than a quick demo with plain Python while loops and if statements, things can turn into a bowl of spaghetti code REAL quick. The main headaches are:

🍝 Spaghetti Code: Add features like conversation history, letting users edit, or trying different AI prompts, and your simple loop becomes a monster of nested if/else blocks. It's tough to read and a nightmare to fix.
🧱 Not Very Mix-and-Match: Your logic for getting input, calling the AI, getting approval, and sending the response all gets jumbled together. Want to test just one piece? Or reuse your "AI calling" bit in another app? Good luck untangling that!
🕸️ Hard to Change or Grow: Want to add a new step, like checking for bad words before the human sees it? Or offer three ways to react instead of just "yes/no"? In plain Python, these changes mean carefully rewiring everything, and you'll probably break something.

These problems make it super hard to build AI workflows that are robust and ready for real users.

Enter PocketFlow - Think of it as LEGO blocks for your AI workflows.

PocketFlow: HITL Workflows, Super Simple!

Trying to build complex AI steps with plain Python is like building a giant LEGO castle without instructions – you'll end up with a wobbly mess. PocketFlow is your friendly instruction manual and a set of perfectly fitting LEGO bricks. It helps you build AI workflows in just 100 lines of actual framework code!

Imagine PocketFlow as running a little workshop:

You have Nodes: These are your specialist workers, each good at one job.
You have a Shared Store: This is like a central whiteboard where everyone shares notes.
You have a Flow: This is the manager who tells the workers what to do and in what order.

The Node Pattern: Your Specialist Workers

In PocketFlow, each main task is a Node. A Node is like a specialist worker who's a pro at one specific thing.

Here's what a Node looks like in PocketFlow:

class Node:
    def prep(self, shared): pass
    def exec(self, prep_res): pass
    def post(self, shared, prep_res, exec_res): pass
    def run(self,shared): p=self.prep(shared); e=self.exec(p); return self.post(shared,p,e)

Don't worry if __init__ or self look weird; they're just Python things! The important bit is the prep -> exec -> post cycle:

prep(shared): "Hey, I'm about to start. What info do I need from the shared whiteboard?"
exec(data_from_prep): "Okay, I have my info. Now I'll do my main job!" (Like calling an AI).
post(shared, prep_res, exec_res): "Job's done! I'll write my results back to the shared whiteboard and tell the manager (the Flow) what happened by returning a simple signal (like a keyword, e.g., "done" or "needs_approval")."

The Shared Store: The Central Whiteboard (Just a Python Dictionary!)

This is just a plain old Python dictionary (let's call it shared_store). All our Node workers can read from it and write to it. It's how they pass info—like the user's question, the AI's draft answer, or conversation history—to each other.

For a math problem, it might start like this:

shared_store = {
    "number_to_process": 0,
    "intermediate_result": None,
    "final_answer": None
}

As Nodes do their work, they'll update this shared_store.

The Flow: The Workshop Manager

A Flow object is like the manager of your workshop. You tell it which Node kicks things off. When you run a Flow:

It runs that first Node.
The Node finishes and returns a signal (just a text string, like "user_approves" or "try_again").
The Flow checks a little map on that Node (called successors) that says: "If the signal is "user_approves", go to the SendResponseNode next. If it's "try_again", go back to the GenerateAINode."

You build this map with simple connections:

node_a >> node_b: This is a shortcut. If node_a finishes and gives the usual "all good" signal (PocketFlow calls this "default"), then node_b runs next.
node_a - "custom_signal" >> node_c: This means if node_a finishes and shouts "custom_signal", then node_c is up.

The Flow keeps this going: run a Node, get its signal, find the next Node. If it gets a signal and can't find a next step, the flow for that path just ends. Easy!
This lets you make workflows that branch off in different directions based on what happens. Like a choose-your-own-adventure for your AI!

Here's how tiny the Flow manager class actually is in PocketFlow:

class Flow:
    def __init__(self, start_node): self.start_node = start_node
    def run(self, shared_store):
        current_node = self.start_node
        while current_node:
            signal = current_node.run(shared_store)
            current_node = current_node.successors.get(signal)

That's it! It just needs a start_node and then it keeps running nodes and following their signals until there's no next step defined for a signal.

Tiny Math Example: PocketFlow in Action!

Let's build a super-tiny workflow: take a number, add 5, then multiply by 2.

Worker 1: The Adder Node
This Node's job is to add 5.

class AddFive(Node):
    def prep(self, shared):
        return shared.get("number_to_process", 0)

    def exec(self, current_number):
        return current_number + 5

    def post(self, shared, prep_res, addition_result):
        shared["intermediate_result"] = addition_result
        print(f"AddFive Node: Added 5, result is {addition_result}")
        return "default" # Signal "all good, continue"

prep: Grabs "number_to_process" from shared_store.
exec: Adds 5.
post: Saves the new number as "intermediate_result" and says "default" (meaning "continue to the next step in line").

Worker 2: The Multiplier Node
This Node's job is to multiply by 2.

class MultiplyByTwo(Node):
    def prep(self, shared):
        return shared["intermediate_result"]

    def exec(self, current_number):
        return current_number * 2

    def post(self, shared, prep_res, multiplication_result):
        shared["final_answer"] = multiplication_result
        print(f"MultiplyByTwo Node: Multiplied, final answer is {multiplication_result}")
        return "done" # Signal "all finished with this path"

prep: Grabs "intermediate_result".
exec: Multiplies by 2.
post: Saves it as "final_answer" and says "done".

Connecting the Workers (The Assembly Line):
Now, let's tell PocketFlow how these Nodes connect.

# First, make our specialist worker Nodes
adder_node = AddFive()
multiplier_node = MultiplyByTwo()

adder_node >> multiplier_node

# Create the Flow manager
math_flow = Flow(start_node=adder_node)

# Let's get some data for them to work on
shared_store_for_math = {"number_to_process": 10}
print(f"\nStarting math game with: {shared_store_for_math}")

# Run the flow!
math_flow.run(shared_store_for_math)

print(f"Math game finished. Whiteboard looks like: {shared_store_for_math}")

And if you run this, you'd see:

Starting math game with: {'number_to_process': 10}
AddFive Node: Added 5, result is 15
MultiplyByTwo Node: Multiplied, final answer is 30
Math game finished. Whiteboard looks like: {'number_to_process': 10, 'intermediate_result': 15, 'final_answer': 30}

See? Each Node (worker) is simple. The shared_store (whiteboard) carries the number through. The Flow (manager) made sure AddFive ran, then MultiplyByTwo, because of the "default" signal. If MultiplyByTwo had other signals for different outcomes, the flow could branch off!

Now we know how PocketFlow uses Nodes, a Shared Store, and a Flow to handle steps and pass data. Let's use these exact same ideas for our HITL approval chatbot!

Building Your HITL "Approval Bot" Workflow

Alright, enough theory! Let's get our hands dirty and build that HITL workflow with PocketFlow for our joke bot. This time, we'll start by thinking about what information our Nodes will need to share.

First, Design the Shared Store (Our Whiteboard)

For our interactive joke generator, the shared_store dictionary (our central whiteboard) needs to keep track of a few key things as the conversation flows:

topic: What kind_of_joke the user wants (e.g., "cats", "programming"). This will be filled in by our first Node.
current_joke: The latest joke the AI cooked up. This will be updated by the joke generation Node.
disliked_jokes: A list of jokes about the current topic that the user already said "no" to. This helps the AI avoid telling the same bad joke twice. It will be updated by our feedback Node.
user_feedback: The user's latest decision (e.g., "approve" or "disapprove"). Also updated by the feedback Node.

Here's a peek at what it might look like while the bot is running:

shared_store = {
    "topic": "dogs",
    "current_joke": "What do you call a dog magician? A labracadabrador!",
    "disliked_jokes": ["Why did the dog cross the road? To get to the barking lot!"],
    "user_feedback": None
}

Each Node we design will read the information it needs from this shared_store and write its results back here for other Nodes to use. This way, everyone's on the same page!

The Three Core Nodes: Our Specialist Joke Crafters

We'll use three main Nodes for our joke-making machine:

GetTopicNode: Asks the user what they want a joke about.
GenerateJokeNode: Cooks up a joke using the AI.
GetFeedbackNode: Asks the user if the joke was a hit or a miss.

Let's build them one by one!

1. GetTopicNode - The Idea Catcher 🎤

This Node's only job is to ask the user for a joke topic.

class GetTopicNode(Node):
    def prep(self, shared):
        shared["topic"] = None
        shared["current_joke"] = None
        shared["disliked_jokes"] = [] 
        shared["user_feedback"] = None
        return None

First, the prep method. Think of this as setting the table before a meal. It just cleans up any old information from a previous joke in our shared whiteboard so we start fresh for the new topic.

    def exec(self, _prep_res): 
        return input("What topic shall I jest about today? ")

Next, exec does the main work: it simply asks the user for a topic using input() and returns whatever they type.

    def post(self, shared, _prep_res, topic_input):
        shared["topic"] = topic_input
        print(f"Alright, a joke about '{topic_input}'! Coming right up...")
        return "generate_joke"

Finally, post takes the topic_input from exec, saves it to our shared whiteboard under the key "topic", prints a little message, and then returns the signal "generate_joke". This signal tells the Flow manager, "Okay, we have a topic, now go to the node that generates jokes!"

2. GenerateJokeNode - The AI Comedy Chef 🤖

This Node grabs the topic (and any jokes the user didn't like about it) and tells the AI to whip up a new joke.

class GenerateJokeNode(Node):
    def prep(self, shared):
        topic = shared.get("topic", "something random")
        disliked = shared.get("disliked_jokes", [])

        prompt = f"Tell me a short, funny joke about: {topic}."
        if disliked:
            avoid_these = "; ".join(disliked)
            prompt += f" Please try to make it different from these: [{avoid_these}]."
        return prompt

In prep, this Node looks at the shared whiteboard for the "topic" and any "disliked_jokes". It then crafts a prompt (a set of instructions) for our AI. If there are disliked jokes, it cleverly tells the AI, "Hey, avoid jokes like these ones the user didn't like!"

    def exec(self, joke_prompt):
        return call_llm(joke_prompt)

Then, exec is where the AI magic would happen. We take the joke_prompt from prep and send it to our call_llm function. This function would be responsible for talking to a real AI service (like OpenAI, Anthropic, etc.) and returning its response.

    def post(self, shared, _prep_res, ai_joke):
        shared["current_joke"] = ai_joke
        print(f"🤖 AI Suggests: {ai_joke}")
        return "get_feedback"

And in post, we save the ai_joke to our shared whiteboard as "current_joke", print it out for the user to see, and then return the signal "get_feedback". This tells the Flow manager, "Joke's ready! Go to the node that gets the user's opinion."

3. GetFeedbackNode - The Joke Judge 🤔

This Node shows the joke and asks the user: thumbs up or thumbs down? Based on their answer, it decides if we should try another joke on the same topic or if we're done.

class GetFeedbackNode(Node):
    def prep(self, shared):
        return None

prep is super simple here. GenerateJokeNode already showed the joke, so there's nothing to set up. We just pass None.

    def exec(self, _prep_res):
        while True: 
            decision = input("Did you like this joke? (yes/no): ").strip().lower()
            if decision in ["yes", "y", "no", "n"]:
                return decision 
            print("Hmm, I didn't catch that. Please type 'yes' or 'no'.")

exec asks the user if they liked the joke. It waits for a clear "yes" (or "y") or "no" (or "n") before moving on.

    def post(self, shared, _prep_res, user_decision):
        if user_decision in ["yes", "y"]:
            print("🎉 Hooray! Glad you liked it!")
            shared["user_feedback"] = "approve"
            return "joke_approved"
        else: 
            print("😅 Oops! My circuits must be crossed. Let me try again...")
            shared["user_feedback"] = "disapprove"
            current_joke = shared.get("current_joke")
            if current_joke: 
                shared.setdefault("disliked_jokes", []).append(current_joke)
            return "regenerate_joke"

Finally, post looks at the user_decision:

If it's a "yes", we celebrate, store "approve" in shared["user_feedback"], and return the signal "joke_approved". This means we're done with this joke topic.
If it's a "no", we apologize, store "disapprove", add the failed current_joke to our shared["disliked_jokes"] list (so GenerateJokeNode knows not to repeat it), and return the signal "regenerate_joke". This tells the Flow manager: "Back to the joke drawing board (the GenerateJokeNode) for the same topic!"

Connecting the Flow: Drawing the Joke Map & Running It! 🗺️

Now we tell PocketFlow how to get from one Node to another using the signals we just defined, and then we'll run it.

# 1. Make our specialist Node workers
get_topic_node = GetTopicNode()
generate_joke_node = GenerateJokeNode()
get_feedback_node = GetFeedbackNode()

# 2. Draw the paths for the Flow manager
get_topic_node - "generate_joke" >> generate_joke_node
generate_joke_node - "get_feedback" >> get_feedback_node
get_feedback_node - "regenerate_joke" >> generate_joke_node

# 3. Let's Run Our HITL Joke Bot!
shared = {
    "topic": None, "current_joke": None,
    "disliked_jokes": [], "user_feedback": None
}
hitl_joke_flow = Flow(start_node=get_topic_node)
hitl_joke_flow.run(shared)

print(f"\n🎤 Joke session over! Here's what happened: {shared}")

This is like drawing a map for our Flow manager and then telling it to start the journey:

We create our three Node workers.
We use the node - "signal" >> next_node pattern to define the paths.
We set up our shared whiteboard.
We create our Flow manager (using the Flow class we saw earlier), telling it to kick things off with get_topic_node.
We call hitl_joke_flow.run(shared). The Flow manager now takes over! It runs the Nodes, listens for their signals, and follows the map. The shared dictionary gets updated live.
When the flow naturally ends (because "joke_approved" has no next step), the run method finishes, and we print out the final state of our whiteboard.

And that's it! You've built a Human-in-the-Loop chatbot using PocketFlow. Each piece is small, understandable, and they all work together to create a flexible workflow where the human is always in control.

From CLI to Web App: What's Next & Key Takeaways

You've built a cool command-line bot where you're the boss! But most people don't hang out in command prompts, right? They use web apps! The great news is that the HITL logic you've built with PocketFlow is the engine that can power a web UI too.

The Journey Ahead:

Part 2 (Web UI): We'd take this exact HITL flow and hook it up to something like Streamlit or Flask. Instead of input(), users would click buttons (st.button("👍 Approve")). The underlying PocketFlow logic remains largely the same!
Part 3 (Real-Time): Time for real-time action! We'll upgrade our web app with WebSockets (using FastAPI) for instant messages and a slicker chat feel. The core HITL decision points (AI drafts, human approves) are still there.
Part 4 (Background Tasks & Progress Updates): What about AI tasks that take a while? We'll set up background processing and use Server-Sent Events (SSE) with FastAPI. This lets us show users live progress updates, making our app feel truly pro. Your PocketFlow Flow is a key part of each user's session.

In Part 2, we'll take our HITL bot to the web using Streamlit! This means building a proper user interface where users can interact with buttons and see responses. We'll even explore how such a UI could handle more than just text, like displaying images, making our AI interactions richer.

Want to try out this exact command-line HITL bot yourself? You can find the complete, runnable code in the PocketFlow cookbook here: PocketFlow Command-Line HITL Example. Go ahead, build your own HITL apps!

DEV Community: Zachary Huang

Reinforcement Learning: Multi-Armed Bandits

Part 1: The Big Idea - What's Your Casino Strategy?

The k-Armed Bandit: A Casino Dilemma

Formalizing the Problem (The Simple Math)

The Core Conflict: Exploration vs. Exploitation

Part 2: A Simple (But Flawed) Strategy - The "Greedy" Approach

How to Estimate Action Values

The Greedy Action Selection Rule

Why the Greedy Strategy Fails

Part 3: A Smarter Strategy - The ε-Greedy (Epsilon-Greedy) Method

The ε-Greedy Rule

Why ε-Greedy Works

The Guarantee

Part 4: Let's Play! A Step-by-Step Walkthrough

The Setup

The Incremental Update Formula

The Game Begins

Let's Analyze What Happened

Part 5: Two More "Clever Tricks" for Exploration

1. Optimistic Initial Values: The Power of Positive Thinking

2. Upper-Confidence-Bound (UCB): The "Smart Exploration" Method

Part 6: So What? Why Bandits Matter

Summary: The Heart of the Problem

The Bridge to Real-World Reinforcement Learning

LLM Agents & Prompt Engineering: A Warrior's Guide to Clear Commands

1. Introduction: The Confused Warrior

2. How Agents Work: The Battle Plan Recap

3. The Myth of "Prompt Engineering"

The 90/10 Rule: Spend 90% of your time designing actions and context. Prompting is the final 10%.

4. The 3 Sacred Laws of Command

4.1. Law #1: Demand the Battle Plan (Chain of Thought)

4.2. Law #2: Speak Like a General, Not a Poet (Be Specific)

4.3. Law #3: Show, Don't Just Tell (Few-Shot Examples)

6. Conclusion: From Confused Grunt to Elite Soldier

LLM Agents & Context: A Warrior's Guide to Navigating the Dungeon

1. Introduction: The Warrior Enters the Dungeon

2. The Warrior's Battle Plan: A Quick Recap

3. The Overwhelmed Warrior: Why Dumping All Context Fails

4. Forging the Cognitive Backpack: Three Master Navigation Techniques

4.1. Technique #1: The Scrying Spell (Context On-Demand)

4.2. Technique #2: The Grand Strategy (Map-Reduce)

4.3. Technique #3: The Cautious Explorer (Backtracking with Verification)

5. Conclusion: A Smart Warrior Navigates, a Fool Memorizes

I just wrote a tutorial on the most overlooked part of building powerful LLM agents: the Action Space. https://dev.to/zachary62/llm-agents-arsenal-a-beginners-guide-to-the-action-space-n75

LLM Agent's Arsenal: A Beginner's Guide to the Action Space

The Battle Tactician: How an Agent Chooses Its Weapon

Level Up Your Arsenal: The Three Tiers of Weapon Complexity

Level 1: The Simple Dagger (The "Button" Action)

Level 2: The Sharpshooter's Bow (The Parameterized Tool)

Level 3: The Spellbook of Creation (The Programmable Action)

Forging the Perfect Arsenal: 3 Golden Rules for Your Weapon Inventory

Golden Rule #1: Engrave a Crystal-Clear User Manual (Clarity is King)

Golden Rule #2: Don't Burden Your Warrior with a Junk Drawer (Keep it Concise)

Golden Rule #3: Make Every Weapon Unique (Slay Redundancy)

Conclusion: An Agent is Only as Sharp as its Arsenal

I just wrote a tutorial showing how to build an AI chatbot for your website that just works. https://dev.to/zachary62/the-easiest-way-to-build-an-ai-chatbot-for-your-website-full-dev-tutorial-37kp

The Easiest Way to Build an AI Chatbot for Your Website (Full Dev Tutorial)

1. That "Simple" Chatbot Project... Isn't

The Old, Broken Way to Build a Chatbot's Brain

And Now... Your Chatbot Is a Liar

2. Our Solution: A "Dumb" Crawler That's Actually Smart

The Master Plan: Let the Bot Read

Why This is So Much Better

3. PocketFlow: The Tiny Engine That Powers Our Bot

The Node: A Specialist Worker

The Shared Store: The Central Whiteboard

The Flow: The Workshop Manager

Tiny Math Example: PocketFlow in Action!

4. Building the Brain: A Look Under the Hood

The Game Plan

The Assembly Line Instructions (flow.py)

The Shared Whiteboard (shared dictionary)

The Workers (nodes.py)

1. CrawlAndExtract: The Librarian

2. AgentDecision: The Brain

3. DraftAnswer: The Writer

5. Giving Our Bot a Face: From Terminal to Website

The Architecture: A Brain and a Face

The Backend: server.py

The Assembly Line Instructions (`flow.py`)

The Shared Whiteboard (`shared` dictionary)

The Workers (`nodes.py`)

1. `CrawlAndExtract`: The Librarian

2. `AgentDecision`: The Brain

3. `DraftAnswer`: The Writer

The Backend: `server.py`

The Frontend: `chatbot.js`

FastAPI `BackgroundTasks`: The "Work in the Back" Crew

The Central Hub: Our `shared` Dictionary

Step 1: `GenerateOutline` Node

Step 2: `WriteContent` Node

Step 3: `ApplyStyle` Node

Tying It All Together with a `Flow`

`st.session_state`: Your App's Memory Bank 🧠

The Web App Challenge: No More `input()`