For the past year I've been a core engineer on TryCook.ai, an AI operating system that replaced a $3M/year fulfillment team and powers 348+ founders. That means agents doing real, billable work every day -- not demos. Here are the seven lessons that survived contact with production.
1. The agent is 20% of the system
The LLM call is the easy part. The other 80% is queues, retries, idempotency keys, state machines, audit logs, and rollback paths. If you architect the agent as a stateless function inside a boring, observable pipeline, failures become recoverable. If the agent is the pipeline, every hallucination is an outage.
2. Constrain outputs structurally, not rhetorically
Prompts that beg the model to "only respond with valid JSON" fail at scale. Schema-enforced outputs (tool calling, JSON mode, grammar constraints) fail approximately never. Every agent boundary in our stack is a typed contract -- if the model can't fill the schema, that's a retry, not a parse error downstream.
3. Design for the 2% failure case first
An agent that's 98% autonomous still fails dozens of times a day at volume. The difference between a product and a liability is what happens on failure: does the task land in a human review queue with full context, or does it silently vanish? Build the escalation path before you build the happy path.
4. Small models for routing, big models for judgment
Most agent steps are classification, extraction, or formatting -- cheap-model work. Reserve frontier models for the few steps that need actual judgment. This is the core of cost engineering for LLM products, and it routinely cuts spend 5-10x with no quality loss.
5. Memory is a database problem, not a prompt problem
Long-running agents need durable state: what they did, what they learned, what the customer prefers. Stuffing history into the context window doesn't scale and isn't queryable. Model agent memory as normal rows -- events, facts, preferences -- and retrieve selectively. Retrieval discipline is the same skill as building RAG that works.
6. Evals are your regression tests
Every prompt change ships against a fixed eval set of real (anonymized) cases with graded outputs. Without evals, prompt engineering is vibes; with them, it's engineering. Start with 30 cases and a pass/fail rubric -- you can grow sophistication later, but you can't retrofit confidence.
7. Humans are a feature, not a fallback
The systems that win don't remove humans -- they move humans up the stack. In a well-designed agentic system, one operator supervises the throughput that used to take a team, intervening only on flagged exceptions. Frame it that way and adoption stops being a fight.
Agents don't replace engineering discipline. They punish the lack of it, at machine speed.
Originally published at haiderfarooq.dev.
Top comments (0)