DEV Community: leo-wang

Memory Doesn't Exist — You're Just Forging History

leo-wang — Fri, 14 Aug 2026 06:58:43 +0000

There's a team whose death I remember especially well. They built a customer-service Agent with a "user profile" — a user says "I'm a VIP customer, Zhang Wei" once, and in every conversation after the Agent "remembers." The product manager asked in the demo: "Will you remember me?" — "Of course, I'll always remember you, Mr. Zhang." They high-fived and shipped. Day one in production, a customer reported: "Your AI says it doesn't know me."

No line of code was wrong. What was wrong was the assumption that conversation.append() could hang memory onto the model. The LLM API is a pure function — response = f(messages, params). No session, no cookie, no cross-request state. From the moment they shipped, they were just stuffing a long chat log back into the API verbatim every time: forget to stuff, truncate, or restart the container, and the memory never existed. The essence of memory lives in your code and your database, not the model.

My take: the Memory layer's engineering maturity is often the decisive factor in whether a system can ever reach shipping. Models can be swapped, frameworks rewritten — but if the Memory layer is designed wrong, all prior conversation data is garbage and you start over. That's why this series places Memory ahead of Frameworks.

Read the full 60-article series — $79 one-time, lifetime access → https://buy.stripe.com/5kQ6ozeEsbKabvzdMQ97G00

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe

The Prompt Is Not a Spell — It's Structured Context Injection

leo-wang — Wed, 05 Aug 2026 01:29:34 +0000

"10 magic openings that boost GPT's IQ by 50." "Add this one sentence and your output quality jumps 200%." That's the typical content on Xiaohongshu, Zhihu, and paid prompt communities — and 95% of those courses sell exactly one feeling: that the prompt is a spell, some mystical trigger word that unlocks the model's hidden capability. The engineers who bought those courses build systems that crash every three days.

The prompt isn't mystical. Its essence is one sentence: a prompt is structured text that injects a stateless probabilistic model with all the context it needs to produce the desired output. Four kinds of context get stuffed in by you — role and behavior (Who), task and goal (What), background data (Context), output spec (How). The "engineering" in prompt engineering is building a maintainable, evaluable, reusable context-assembly pipeline — not finding the right spell.

My take: what's scarce in production Agent teams isn't "people who can write prompts." It's engineers who can build prompt version management, evaluation, gradual rollout, and rollback. A competent backend engineer plus good prompt infrastructure outcompetes 90% of the people flying a "prompt specialist" flag.

Full article → https://leo76868.substack.com/p/the-prompt-is-not-a-spell-its-structured

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe

What Actually Happens During a Single LLM Call?

leo-wang — Tue, 04 Aug 2026 06:45:26 +0000

The first time I called the OpenAI API with bare curl, I stared at the JSON response for about thirty seconds. That's it? A POST request, a messages array, back comes a string of text. LangChain wraps it in seven or eight layers and the tutorials make it sound mystical — but at the very bottom it's just an ordinary HTTP request, so ordinary it can't get any more ordinary.

Here's the conclusion that reshaped how I build agents: a so-called "Agent" is, in essence, just an ordinary backend program that calls a stateless LLM API in a loop. Its complexity doesn't live in the model — it lives in how your code manages the messages array. That's why the engineering effort in a production-grade Agent lands overwhelmingly on the memory, orchestration, security, and evaluation layers, not the model layer (which is actually the simplest).

My take: if you can treat every LLM call as an unreliable remote service — timeouts, retries, jitter, cost accounting — you've already outpaced most teams still chasing "magic prompt" tricks. The model layer is the part nobody should be afraid of.

Full article → https://leo76868.substack.com/p/what-actually-happens-during-a-single

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe

You Think You're Building AI Agents — You're Just Running a Demo

leo-wang — Mon, 03 Aug 2026 09:16:22 +0000

Last year a friend's company shipped an internal "smart customer service Agent." Two weeks to a demo. The boss asked it questions, it answered every one. Applause. Ship it.

Day three in production, it fell apart — rate-limit meltdowns, a leaked expense record, a prompt injection that dumped the system prompt, ¥47,000 in tokens burned in a month, and a fabricated approval chain that seized up finance.

The code looked exactly like the tutorials. RAG? Checked. Function Calling? Checked. The problem: it was only ever a demo.

Most agent tutorials cover the three least important layers — the model, the tools, the basic loop. Production-grade agents live in the layers nobody teaches: error handling under load, cost blowup, data permissions, prompt injection, observability, and evaluation.

My take: learning frameworks is already late. Learning the engineering system — how to wrap an unstable probabilistic model into something safe to ship — is right on time. That half-life is measured in years, not months. That's what this series is about.

Full article — the 7-layer production skeleton + the full post-mortem → https://leo76868.substack.com/p/you-think-youre-building-ai-agents

Production-Grade AI Agents — by Leo Wang · Follow on X · Subscribe