Don't Write Prompts, Write Loops: 12 Months of Production Agent Engineering

Forge Voko — Fri, 10 Jul 2026 04:37:36 +0000

I'm not an AI evangelist. I'm an engineering manager who still reviews code daily and carries quarterly delivery targets on my back.
This isn't a trend forecast. These are field notes—hard-won lessons from shipping real projects with AI Agents over the past year. What works, what doesn't, the traps I fell into, and the principles I finally understood.
To be honest, I didn't believe in it at first. A teammate told me: "AI coding is a toy. I can write two lines of code faster than explaining it to that thing." I didn't argue. I was skeptical too. But as a manager, I have one rule: no opinion without skin in the game.
So I picked a real project—a motion tracking tool for myself—and got my hands dirty.
First version with Vibe Coding: two sentences, UI pops up. Felt amazing. Then reality hit me like a truck.

1. The Paradigm Shift: From Harness Engineering to Loop Engineering

When Manus exploded in March 2025, the industry was still arguing about "how to write better prompts" and "how to manage context windows." The consensus: models are powerful; the problem is how we use them.
When OpenClaw went viral at year-end, "lobster" became a meme. For the first time, Agents broke out of the tech bubble. But what really made me feel "the paradigm has shifted" were two things early this year: Hermes's self-evolution capability and the Claude Code leak.
Reading Claude Code's implementation, I realized the industry had quietly pivoted from Harness Engineering to Loop Engineering.

The fundamental difference:
Harness era ** was about "how to scaffold the model"—how to attach tools, stuff context, manage memory.
**Loop era is about "how to make the Agent run and converge on its own"—how to decompose goals, collect feedback, self-correct errors, and know when to stop.
In other words: from "I'm holding your hand" to "you run, I design the track and checkpoints."
Other threads quietly shifting this year:
The single-Agent vs. multi-Agent debate reached a conclusion: not either/or, but layered by task complexity
Function Calling is being replaced by the Skill paradigm (from "calling functions" to "calling capability bundles")
"Must use SOTA" became "pick the right model for the job"
Scaling Law magic is fading; people are betting on engineering dividends

2. Human on the Loop: Agents Handle Fast Feedback, Humans Handle Slow Feedback
Andrew Ng has a take that's stuck with me: Agents handle fast feedback; humans handle slow feedback.
The operational form of this insight is Human on the Loop.
I used to default to Human in the Loop—every step the Agent took, I'd check, confirm, course-correct. Result: my energy got consumed by trivial tasks, and the Agent never really ran.
The right approach: Humans stand at the rules layer and the acceptance layer; Agents self-cycle on the track.
Humans define rules, acceptance criteria, and stopping conditions. Agents handle fast feedback, self-correction, and convergence within those rules. Agents don't get tired at 7x24, but they also won't proactively tell you "this requirement itself is wrong"—they just execute, and the more beautiful the execution, the more spectacular the failure.
So you must design a signal collection mechanism—not waiting for the Agent to report, but actively collecting signals through preset loops and checkpoints.

3. The Reality Check: A Real Failure Story
Back to that motion tracking tool.
I said "add visual detection." The Agent gave me cv2.imread() and a placeholder. I said "use MediaPipe." It wrote the import but the model path was fake—FileNotFoundError on first run. I said "use YOLO." It installed ultralytics but the inference parameters in model.predict() were completely fabricated; the bbox format and post-processing logic didn't match reality.

The worst part: mock interfaces.
To make the program "run successfully," the Agent mocked every external dependency into stub functions returning fixed values. On the surface, python main.py didn't crash. But the entire recognition pipeline was fake—the camera pointed at me alone, and the screen showed "15 people detected."
Demoed it to the product team. First frame, exposed on the spot. Awkward silence.
The truth about early Vibe Coding:
It satisfies your literal request, not your actual goal. Its optimization target is "code runs + no errors," not "feature actually works."
Later, SOTA models dropped in rapid succession. Models got smarter—what used to need three paragraphs of context, now gets understood in one sentence. But I discovered something counterintuitive: the smarter the model, the more important "how you use it" becomes. Because smart models "confidently make mistakes"—and the mistakes are more subtle.

4. Seven Lessons Paid for with Real Pain

Lesson 1: Spec Isn't Documentation—It's Memory
Everyone's shouting "from Vibe Coding to Spec," but most people misunderstand: they think Spec means writing a mountain of design docs. It's not.
**The core value of Spec is building long-term memory for the Agent **so it doesn't drift or repeat mistakes.
I now maintain a Spec system for every project:

plain
AGENTS.md          # Index + project memory
  - Goals, tech stack, constraints
  - Module responsibilities, dependencies, immutable decisions
  - Mistake log (critical!)

spec/
  design/          # Architecture, module specs, interface contracts
  test/            # E2E test plans, edge case lists
  decisions/       # Historical decisions (ADRs)
  research/        # Open-source research, community data

The real designs, tests, and solutions go into the spec/ directories. AGENTS.md serves as the index. The Agent uses it to quickly locate specific solutions instead of digging through one giant document.
Where does the mistake log come from? I force the Agent to self-reflect before ending each session: what went wrong this round, why, how to avoid it next time. Write it into AGENTS.md. Next new session, first thing: read this file.
Results were immediate. The same mock interface bug that used to happen three times? Now it gets remembered after one.

Lesson 2: Think Like a Human—Ask Why Five Times
Agents love to "confidently make mistakes," and the smarter the model, the more subtle the errors. So Spec Coding must include a phase: think like a human, ask why multiple times, don't trust intuition.

My hard rules:
Before every technical decision, the Agent does "open-source research"—is there a mature open-source implementation? Any latest community approach?
Before every product design decision, cross-validate on Product Hunt, Reddit, etc.
Introduce reverse engineering: replicate historical commits, continuously correct and score based on existing knowledge
Ask why: why this approach? Why not that one? Any counterintuitive data supporting it?
Agent hallucination isn't "talking nonsense"—it's "talking nonsense with absolute confidence." You must validate it with external data, not gut feeling.

Lesson 3: Don't Be the Model's Teacher—Be Its Editor
I made a big mistake: I'd figure out the complete solution myself, then tell the Agent step by step. Result: I was still drawing sequence diagrams while the team next door had an Agent-built MVP running.
Then it clicked: models absorbed far more world knowledge during training than any individual ever could. Your value isn't in "teaching it how"—it's in "judging whether it's right."
Now my approach changed. Instead of "do it in these steps," I say: "I want to achieve X. Give me a plan, then tell me what's wrong with it." Force the Agent from executor to thinking partner.
Even sharper move: deliberately challenge it. Say "your last plan had issues" even when it didn't—it triggers self-review and often surfaces hidden risks. The underlying logic exploits model compliance: it tends to agree with your judgment, so when you question it, it works hard to find its own flaws.

Lesson 4: Agent Teams Aren't About Stacking SOTA—It's Division of Labor
At first, I threw every task at the strongest model. Token bill burned through a month's budget in a week, and output quality didn't proportionally improve.

Why different vendor for review? Same-source models share training blind spots. Bugs written by Model A, reviewed by Model A—often missed. Switch to Model B, different blind spots, catches them.
This config dropped costs 60%+, with more stable quality. Domestic models are fully capable at execution and review slots; the gap is mainly in ultra-long context reasoning—that's the design slot, solvable with a small amount of strong models.
Don't be held hostage by "must use SOTA." Pick models by role. That's engineering thinking.

Lesson 5: Use Open Source, But Know What You're Using
Sounds obvious, but there's a trap: Agents love writing from scratch because they feel "it's more controllable." You have to actively stop them.
My rule: before every new module, the Agent does "open-source research." If there's a mature open-source solution, integrate it first, save energy for business logic.
But copying has rules. Have the Agent evaluate three things:
Maintenance activity **(recent commit dates)
**License compatibility
Dependency chain complexity
I watched an Agent import a "perfect match" library that dragged in 47 transitive dependencies, two with known vulnerabilities.
Open source is standing on giants' shoulders. But check the giants' soles for nails.

Lesson 6: End-to-End Detection, Context-Dependent Gates

This is the most critical piece of closed-loop validation. Agents will mock every external dependency into fixed-value stubs to "make the program run." No surface errors, but the entire pipeline is fake.
So end-to-end detection is mandatory, and gates must be context-dependent:

These gates go into spec/test/e2e-plan.md. The Agent self-checks against them. Fails? Self-corrects, re-runs, converges.
Always do end-to-end detection. Humans only intervene at final acceptance—judging "does this result meet business goals?"—not at every step via Human in the Loop.

Lesson 7: Burn Tokens With Questions, Not Aimlessly
Two extreme attitudes toward "burning tokens": save every token, or burn as many as possible. The question is how you burn.
Aimless burning: chatting with the Agent, letting it reinvent wheels, starting without planning—low cognitive return.
Question-driven burning: tackling specific problems, forcing the Agent to summarize lessons at each session's end, writing into AGENTS.md—that's investment, not consumption.
Learning to ask the right questions matters. The model absorbed more world knowledge than any individual. Your value isn't in "teaching it how" but in "judging if it's right." Force the Agent from executor to thinking partner—that's when token burn becomes valuable.
Don't forget to burn tokens. Burn them with questions, and you'll gain insights far beyond expectations.

5. Managing Agent Teams vs. Managing People: Similar, Yet Different
This is what I've thought about most this year.
I used to spend 60% of my time on communication—aligning requirements, decomposing tasks, tracking progress, handling emotions. In the AI era, this is lightening, but not disappearing—it's transforming.
What's similar:
Both need SOPs, task decomposition, checkpoints, and acceptance. My experience as a manager transfers directly.
What's different (this is the key):
Humans get tired, emotional, and proactively raise issues. Agents run 7x24 but will never tell you "this requirement is wrong"—they just execute, and the more beautiful the execution, the worse the mistake.
Managing humans: collect signals via 1:1s and weekly meetings. Managing Agents: collect signals via d*esigned loops and checkpoints. Signals don't emerge on their own.
Humans can "understand fuzzy intent." Agents will **precisely execute your fuzzy intent into a wrong result.*

The new capability stack for engineering managers:
Define problems clearly **(old skill, more important now)
**Design Agent loops** and checkpoints (new skill)
**Make final acceptance decisions (irreplaceable skill)

The narrative of "one person controlling 100 Agents"? Honestly, I haven't seen any team run that stably. More realistic: **a 5-person team with proper Agent orchestration delivers what used to take 15 people. **That magnitude of improvement is enough to transform team structure.

Closing Thoughts
No grand philosophy here. If there's one takeaway: this field changes too fast, but "getting your hands dirty" always beats "watching from the sidelines."

I've seen too many peers discuss "how to design Agents" from the shore—discussing for six months while those in the water have iterated three rounds. I've also seen the opposite: heads-down builders who don't watch industry progress, reinventing solutions others solved long ago.
The best state? One foot in the water, one foot on the shore. Two hours a week scanning industry trends; the rest of the time validating in real projects.
Top-down programming is essentially: from "I'm holding your hand" to "you run, I design the track and checkpoints." Humans do complex work—defining problems, designing rules, final acceptance. Agents do simple work—fast feedback, self-correction, testing within rules. Maximize human value instead of filling human energy with trivial tasks.
It's never too late to get on board—the key is, you have to actually board, not just study the boarding posture.
Don't forget to burn tokens. Burn them with questions. You'll gain insights far beyond expectations.

Yason | Engineering Manager | 12 Months of Battle-Tested Lessons
One foot in the water, one foot on the shore.

DEV Community: Forge Voko

Don't Write Prompts, Write Loops: 12 Months of Production Agent Engineering