Why Your AI Works in a Notebook But Fails in Production — LLM Zoomcamp Module 3

#rag #agents #kestra #datatalksclub

Every developer has seen it: an AI prototype that impresses in demos but breaks the moment real users touch it. The problem is rarely the model. It's everything around the model, how context is prepared, how tools are called, how failures are handled, how costs are tracked.

Module 3 of LLM Zoomcamp 2026 addresses this gap head-on. Taught by Will Russell from Kestra, this module shifts the conversation from "how do I call an LLM" to "how do I run AI reliably at scale."

Here's what I learned and what most LLM tutorials never teach you.

The Experiment That Changed How I Think About AI

Try this yourself. Open ChatGPT and ask it to generate a Kestra workflow that loads CSV data into BigQuery. You'll get YAML that looks completely reasonable and doesn't work. Wrong plugin types, outdated parameters, invented task names.

Now try the same prompt in Kestra's AI Copilot. It generates a flow that runs.

Same model family. Same prompt. Completely different outcome.

The reason: Kestra's AI Copilot retrieves current plugin documentation before generating. The model has accurate, up-to-date context. ChatGPT is guessing from training data that's months or years old.

This is Context Engineering, the discipline of giving a model exactly the right information at the right time. It's more impactful than model size, fine-tuning, or prompt tricks.

RAG vs No RAG: Measured, Not Assumed

The module includes a concrete experiment with two pre-built flows:

Without RAG, the model invents plausible but fabricated Kestra 1.1 features with full confidence. It describes "Declarative I/O with Variables", "Enhanced Task Groups", and other features that sound real but aren't accurate to that release.

With RAG, the model retrieves the actual Kestra 1.1 release notes and returns real features: the No-Code Dashboard Editor, Multi-Agent AI Systems, Fix with AI, the Human Task for manual approvals, and improved air-gapped environment support.

One is a liability in production. The other is actually useful.

AI Agents: The LLM Takes the Wheel

The second half of the module moves from RAG to agents. The difference:

RAG pipeline: you decide when to search, what to search, how many results to retrieve
AI agent: the model decides all of that itself

In Kestra, agents are declared in YAML and the framework handles the agentic loop:

- id: research_agent
  type: io.kestra.plugin.ai.agent.AIAgent
  systemMessage: |
    You are a technical research assistant.
    Use your tools to find accurate information before answering.
  prompt: "{{ inputs.question }}"

The agent decides whether to call a tool, which tool, with what query, and when it has enough context to stop searching and answer. You get full observability, every tool call, every response, every token logged in the Kestra UI.

Tracking What Matters: Token Usage

Most tutorials skip this. The module doesn't.

Every agent task in Kestra exposes token usage on its output object. The module shows you how to log it explicitly:

- id: log_usage
  type: io.kestra.plugin.core.log.Log
  message: |
    Output tokens: {{ outputs.agent.tokenUsage.outputTokenCount }}

What I measured across runs:

Short summary: 63 output tokens
Long summary: 184 output tokens (~3x more)
3-sentence vs 1-sentence prompt: ~2x more tokens

Small wording changes in prompts have real cost implications when running thousands of executions. Monitoring from day one prevents surprises.

When Agents Are the Wrong Choice

This is the lesson most AI courses skip: agents are non-deterministic. The same input can produce different tool calls, different search strategies, different answers across runs.

For creative tasks, research, and open-ended workflows that flexibility is a feature.

For financial reporting, compliance workflows, audit trails, and regulated industries it's a liability. You need deterministic, repeatable, fully auditable pipelines where every step is predictable and logged.

Kestra supports both. Knowing when to use which is the actual engineering judgment.