What Actually Happens During a Single LLM Call?

leo-wang — Tue, 04 Aug 2026 06:45:26 +0000

The first time I called the OpenAI API with bare curl, I stared at the JSON response for about thirty seconds. That's it? A POST request, a messages array, back comes a string of text. LangChain wraps it in seven or eight layers and the tutorials make it sound mystical — but at the very bottom it's just an ordinary HTTP request, so ordinary it can't get any more ordinary.

Here's the conclusion that reshaped how I build agents: a so-called "Agent" is, in essence, just an ordinary backend program that calls a stateless LLM API in a loop. Its complexity doesn't live in the model — it lives in how your code manages the messages array. That's why the engineering effort in a production-grade Agent lands overwhelmingly on the memory, orchestration, security, and evaluation layers, not the model layer (which is actually the simplest).

My take: if you can treat every LLM call as an unreliable remote service — timeouts, retries, jitter, cost accounting — you've already outpaced most teams still chasing "magic prompt" tricks. The model layer is the part nobody should be afraid of.

Full article → https://leo76868.substack.com/p/what-actually-happens-during-a-single

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe

You Think You're Building AI Agents — You're Just Running a Demo

leo-wang — Mon, 03 Aug 2026 09:16:22 +0000

Last year a friend's company shipped an internal "smart customer service Agent." Two weeks to a demo. The boss asked it questions, it answered every one. Applause. Ship it.

Day three in production, it fell apart — rate-limit meltdowns, a leaked expense record, a prompt injection that dumped the system prompt, ¥47,000 in tokens burned in a month, and a fabricated approval chain that seized up finance.

The code looked exactly like the tutorials. RAG? Checked. Function Calling? Checked. The problem: it was only ever a demo.

Most agent tutorials cover the three least important layers — the model, the tools, the basic loop. Production-grade agents live in the layers nobody teaches: error handling under load, cost blowup, data permissions, prompt injection, observability, and evaluation.

My take: learning frameworks is already late. Learning the engineering system — how to wrap an unstable probabilistic model into something safe to ship — is right on time. That half-life is measured in years, not months. That's what this series is about.

Full article — the 7-layer production skeleton + the full post-mortem → https://leo76868.substack.com/p/you-think-youre-building-ai-agents

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe

DEV Community: leo-wang

What Actually Happens During a Single LLM Call?

You Think You're Building AI Agents — You're Just Running a Demo