Agentic Workflows That Survive Real Inputs

Most agent demos work because the demo input is clean. Real inputs are not. They arrive half-formed, with missing fields, contradictory context, PDFs that are actually images, and users who change their mind mid-thread. The gap between a working demo and a workflow that runs unattended for six months is almost entirely about how you decompose the problem and where you put the guardrails.

I design multi-agent workflows daily at BizFlowAI, and the pattern below is the one I keep coming back to. It is not glamorous. It is the reason my content pipeline runs 24/7 without me babysitting it, and it is the reason my serverless AWS integrations hit SLA instead of paging me at 2am.

Start by writing the workflow as a deterministic pipeline, then decide where an agent earns its keep

Before I touch an LLM SDK, I write the workflow as if agents did not exist. A boring sequence of functions with typed inputs and outputs. This is the single most useful exercise in agentic design, and it is the one most teams skip.

Here is the decomposition I run on every new workflow:

1. What is the trigger? A webhook, a cron, a queue message, a human uploading a file. Write the exact payload shape.
2. What is the final artifact? A JSON object, a published post, a Zendesk ticket update, a database row. Write the exact schema.
3. What are the intermediate artifacts? Between trigger and final, list every object that has to exist. Each one gets a name and a schema.
4. For each transition between artifacts, ask: does this require judgment, or is it a rule? Rules go in code. Judgment goes to an LLM. That is the whole heuristic.

On my ContentStudio pipeline, the flow looks like this:

trigger (topic gap detected)
  -> ResearchBrief        [LLM: judgment on angle + gap]
  -> OutlineDraft         [LLM: judgment on structure]
  -> DraftPost            [LLM: generation]
  -> SEOAuditReport       [code: deterministic checks]
  -> RevisedPost          [LLM: targeted rewrites only]
  -> PublishPayload       [code: schema validation]
  -> published (WordPress API)

Notice that four steps are code, three are LLM. In an earlier version I had the LLM doing the SEO audit and the publish payload assembly. It hallucinated slugs, invented canonical URLs, and once tried to publish to a site that did not exist. Moving those to deterministic code cut error rate to near zero and made the whole pipeline debuggable.

The rule I follow: an agent should be responsible for a decision, not for the plumbing around the decision.

Pick tools like you're hiring, not shopping

Tool selection is where most agent projects quietly go wrong. Teams give the agent 15 tools because "more tools = more capable" and then wonder why it picks the wrong one 30% of the time.

My working numbers, from actual production agents:

Tools available	Correct tool selection rate
3-5 tools, tightly scoped	96-99%
6-10 tools	88-93%
11-20 tools	70-82%
20+ tools	often below 70%, highly prompt-dependent

These are directional numbers from my own pipelines, not a benchmark. But the pattern is real and it holds across model families. The mitigation is not a smarter model. It is tool routing: a small dispatcher agent that picks the sub-agent, and each sub-agent has 3-5 tools.

When I write a tool spec, I treat it like a job description:

• Name: verb-first, unambiguous. search_customer_by_email, not customer_lookup.
• Description: one sentence on what it does, one sentence on when NOT to use it. The "when not" line is what stops confident-wrong calls.
• Input schema: strict types, required fields marked. No optional-with-a-default fields hiding as strings.
• Output schema: the same shape every time, including error shape. Never let an agent parse a free-form error string.

If you cannot write the "when not to use it" sentence, the tool is too broad. Split it.

Guardrails belong at the boundaries, not in the prompt

Prompt-level guardrails ("do not make up customer names", "always validate the email") are advisory at best. In production I treat them as backup, not primary defense. The primary defense is at the boundary of every step.

There are four boundary types where I always put guardrails:

1. Input validation on tool calls. Every tool validates its input with a strict schema (I use Zod in TypeScript, Pydantic in Python) before doing anything else. If the agent hallucinates a field, the tool returns a structured error and the agent gets to retry with feedback. No exceptions, no silent coercion.

2. Output validation on LLM responses. Every LLM call that produces structured output goes through a validator. If the structured output is required, I use the provider's structured output mode (Claude tool use, OpenAI response_format) and then re-validate. Belt and suspenders. I have caught model regressions this way.

3. Cost and loop caps. Every workflow has a hard budget: max tokens, max tool calls, max wallclock. If it exceeds any of them, it fails loudly, writes to the dead-letter queue, and pages me. An agent that silently loops for 40 minutes and spends $80 is a bug you only find on the invoice.

4. Side-effect gates. Any tool that writes to the real world (send email, publish post, create ticket, charge card) has a separate authorization check that is NOT part of the LLM prompt. It reads from a config or a feature flag. This is how you sleep at night when you give an agent write access.

Here is a minimal side-effect gate pattern I use:

async function publishPost(input: PublishInput, ctx: AgentContext) {
  // 1. Schema validation
  const parsed = PublishSchema.parse(input);

  // 2. Authorization check (NOT in prompt)
  if (!ctx.permissions.canPublishTo(parsed.siteId)) {
    return { ok: false, error: "unauthorized_site" };
  }

  // 3. Idempotency: has this artifact already been published?
  const existing = await db.publishedPosts.findOne({
    contentHash: parsed.contentHash
  });
  if (existing) {
    return { ok: true, alreadyPublished: existing.url };
  }

  // 4. Actual side effect, wrapped in retry with backoff
  const result = await wpClient.publish(parsed);
  await db.publishedPosts.insert({ ...parsed, url: result.url });
  return { ok: true, url: result.url };
}

Four things happen before we ever hit the WordPress API. Every one of them has caught a real bug in production.

Observability is the difference between "it works" and "I can prove it works"

You cannot debug an agent workflow by reading logs after the fact if you did not instrument it up front. And you will need to debug it, because real inputs will do things you did not predict.

The minimum I ship with every workflow:

• A trace ID per workflow run, propagated through every tool call, LLM call, and DB write. One ID to grep on.
• Structured logs, not print statements. JSON with trace_id, step_name, duration_ms, token_in, token_out, cost_estimate, and outcome.
• Every LLM call persisted: prompt, response, model, temperature, tokens, latency. Storage is cheap; not being able to reproduce a bad run is expensive.
• Every tool call persisted: inputs, outputs, duration, error.
• A run summary record per workflow: trigger, final status, total cost, total duration, artifacts produced.

I use a simple Postgres schema for this. Not because it is fancy, but because I can write SQL against it when something breaks. When a run fails, I can pull the entire history in one query and replay it.

The metric I actually watch, across every workflow I run:

Signal	Why it matters
Success rate per workflow version	Regressions after prompt changes
p50 / p95 latency per step	Which step is the bottleneck
Cost per run (tokens + tools)	Unit economics, budget alerts
Tool call error rate by tool	Which tool spec is confusing the agent
Retry rate per step	Silent flakiness
Human-intervention rate	The honest measure of autonomy

The last one is the one nobody wants to look at. If a workflow needs a human 15% of the time, it is not autonomous, it is assisted. Fine, but call it what it is and price the ROI accordingly.

Design for failure modes you have already seen

After building enough of these, the failure modes rhyme. Design against them from day one.

Confident-wrong outputs. The agent produces something plausible but wrong. Mitigation: a validation step that checks the output against ground truth (a DB lookup, a rules check, a second-model critique for high-stakes outputs). For my content pipeline, every draft goes through a deterministic SEO/AEO audit before it can be published. If it fails, it goes back to a targeted revision agent with the specific failures listed.

Silent tool failures. The tool returns a 200 with an empty body, or a partial success that looks fine. Mitigation: tools must return an explicit { ok: boolean, ... } shape and the agent's tool-use loop must treat ok: false as a first-class case, not a string to interpret.

Runaway loops. The agent keeps calling the same tool with slightly different inputs. Mitigation: track recent tool calls in the loop and inject a system message when the same tool+input hash repeats. Also: the wallclock cap from earlier.

Context rot. Long conversations where the agent forgets or contradicts earlier decisions. Mitigation: don't run one long conversation. Break the workflow into steps with fresh context windows and pass forward only the artifact each step needs. This is the single biggest reliability win in multi-step agent design.

Model drift. The vendor updates the model and your prompts subtly break. Mitigation: pin model versions in code, and run a small eval set on every deploy. Even 20 representative cases will catch most regressions.

What I'd do if I were starting a new agentic workflow tomorrow

The order matters. This is what I actually do:

1. Write the workflow as a plain pipeline, no LLM calls. Get the schemas right.
2. Identify the 2-4 steps that genuinely need judgment. Everything else stays as code.
3. Write tool specs like job descriptions. If you have more than 5 per agent, split into sub-agents with a router.
4. Add strict input/output validation on every tool and every structured LLM call.
5. Add cost caps, loop caps, and side-effect gates before you connect it to anything that writes.
6. Instrument every call with a trace ID and persist the full history.
7. Build an eval set of 20-50 real inputs (not synthetic ones) before you ship.
8. Deploy behind a feature flag. Run in shadow mode against production traffic for a week.
9. Watch human-intervention rate. Iterate on the step with the highest rate first.
10. Only then, expand scope.

Steps 1-6 take about 60% of the total build time and prevent about 90% of the incidents. Skipping them is how you end up with a demo that crashes in week two.

The trending mistake I see repeatedly: teams pick a framework (LangGraph, CrewAI, whatever is fashionable this quarter) before they have written the pipeline as a boring function. The framework is not what makes it work. The decomposition is.

If you are building an agentic workflow that has to run against real production inputs, and you want a second set of eyes on the decomposition or the guardrails, I am open to a few consulting engagements this quarter. Reach out at lazar-milicevic.com/#contact, or read more on the blog for related posts on RAG evaluation, agentic context, and shipping agents that survive contact with users.