Calling an LLM Inside a Business Workflow: Validate, Retry, Fallback, Log

An LLM call inside a business workflow is an unreliable network call that returns text you then treat as data. Design it like one.

Demos call the model, parse the reply and move on. In production that step sits between an incoming email and a CRM record, or between a scanned invoice and a ledger. When the reply is malformed, slow or confidently wrong, the workflow needs a defined path, not an exception that someone finds in the logs a week later.

This post walks through a small TypeScript wrapper built on four steps: Validate, Retry, Fallback and Log. The example classifies an inbound enquiry email. It is a reference scenario for illustration, not a client system.

The contract: what the workflow actually needs

Start from the downstream consumer, not the prompt. The CRM step needs a category, a priority and a short summary. Express that as a schema:

import { z } from "zod";

export const EnquirySchema = z.object({
  category: z.enum(["sales", "support", "billing", "spam", "other"]),
  priority: z.enum(["low", "normal", "high"]),
  summary: z.string().min(10).max(280),
  confidence: z.number().min(0).max(1),
});

export type Enquiry = z.infer<typeof EnquirySchema>;

Two details matter. Closed enums stop the model inventing a category your routing does not handle. A confidence field gives you something to gate on, with the caveat that self-reported confidence is a weak signal; treat it as one input, not a guarantee.

Validate: never pass raw model output downstream

type StepResult<T> =
  | { ok: true; value: T }
  | { ok: false; reason: "invalid_json" | "schema" | "timeout" | "api_error"; detail: string };

function validate(raw: string): StepResult<Enquiry> {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch (e) {
    return { ok: false, reason: "invalid_json", detail: String(e) };
  }
  const result = EnquirySchema.safeParse(parsed);
  return result.success
    ? { ok: true, value: result.data }
    : { ok: false, reason: "schema", detail: result.error.message };
}

Use your provider's JSON or structured-output mode where it exists, and still validate. Structured modes reduce malformed output; they do not check business rules such as "billing enquiries are never low priority" if you need that. Add those as .refine() checks on the schema.

Retry: bounded, with the error fed back

Retries fix two different problems: transient API errors, and output that failed validation. Handle both, but cap attempts and total time.

async function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  let timer: NodeJS.Timeout;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(() => reject(new Error("timeout")), ms);
  });
  try {
    return await Promise.race([p, timeout]);
  } finally {
    clearTimeout(timer!);
  }
}

async function classify(
  email: string,
  callModel: (messages: Msg[]) => Promise<string>,
  opts = { maxAttempts: 3, timeoutMs: 15_000 }
): Promise<{ result: StepResult<Enquiry>; attempts: number }> {
  const messages: Msg[] = [
    { role: "system", content: SYSTEM_PROMPT },
    { role: "user", content: email },
  ];
  let last: StepResult<Enquiry> = { ok: false, reason: "api_error", detail: "not run" };

  for (let attempt = 1; attempt <= opts.maxAttempts; attempt++) {
    try {
      const raw = await withTimeout(callModel(messages), opts.timeoutMs);
      last = validate(raw);
      if (last.ok) return { result: last, attempts: attempt };
      // Feed the validation error back once, so the retry is informed.
      messages.push({ role: "assistant", content: raw });
      messages.push({
        role: "user",
        content: `Your reply failed validation: ${last.detail}. Reply with corrected JSON only.`,
      });
    } catch (e) {
      const msg = String(e);
      last = { ok: false, reason: msg.includes("timeout") ? "timeout" : "api_error", detail: msg };
      await sleep(backoffMs(attempt));
    }
  }
  return { result: last, attempts: opts.maxAttempts };
}

const backoffMs = (n: number) => Math.min(8_000, 500 * 2 ** n) + Math.random() * 250;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

Keep maxAttempts small. If a model fails validation three times on the same input, a fourth attempt rarely helps and costs latency and tokens. That input belongs with a person.

Fallback: a defined path when the model cannot be trusted

The fallback is the part most implementations skip. Decide it per workflow, before launch:

const CONFIDENCE_FLOOR = 0.7;

async function routeEnquiry(email: string, ctx: Ctx) {
  const { result, attempts } = await classify(email, ctx.callModel);

  if (result.ok && result.value.confidence >= CONFIDENCE_FLOOR && result.value.category !== "other") {
    await ctx.crm.createLead({ ...result.value, source: "llm", attempts });
    return "auto";
  }

  await ctx.reviewQueue.push({
    email,
    suggestion: result.ok ? result.value : null,
    failure: result.ok ? "low_confidence" : result.reason,
    attempts,
  });
  return "review";
}

Three rules make this hold up:

• The fallback writes nothing irreversible. The review queue holds the email and the model's suggestion; a person confirms before the CRM record exists.
• The suggestion is kept. Reviewers correct faster than they classify from scratch, and corrections become your evaluation set.
• The floor is a setting, not a constant buried in code. You will tune it once you see real review volume.

Count the review queue when you estimate time saved. Automation that sends a share of items to people still needs those people.

Log: one structured record per call

Every call should leave a record you can query later:

ctx.log.info("llm_step", {
  workflow: "enquiry_classify",
  model: ctx.modelName,
  promptVersion: "enquiry-v4",
  attempts,
  outcome: result.ok ? "valid" : result.reason,
  route,                         // "auto" | "review"
  confidence: result.ok ? result.value.confidence : null,
  latencyMs,
  inputTokens, outputTokens,
});

With promptVersion and outcome logged, you can answer the questions that matter after launch: did the new prompt reduce schema failures, what share goes to review, and which categories reviewers correct most. Avoid logging full customer emails in general-purpose logs; store a reference ID and keep content where your data-retention rules apply.

Testing the wrapper without calling a model

Inject callModel so tests can script replies:

test("invalid JSON twice, then valid, routes auto", async () => {
  const replies = ["not json", '{"category":"sales"}', VALID_HIGH_CONFIDENCE];
  const callModel = async () => replies.shift()!;
  const route = await routeEnquiry("Need a quote for 20 units", fakeCtx({ callModel }));
  expect(route).toBe("auto");
});

test("three schema failures go to review with no CRM write", async () => {
  const callModel = async () => '{"category":"unknown"}';
  const ctx = fakeCtx({ callModel });
  expect(await routeEnquiry("hello", ctx)).toBe("review");
  expect(ctx.crm.createLead).not.toHaveBeenCalled();
});

These tests pin the behaviour that protects your data: invalid output never reaches the CRM, and exhausted retries always land in review.

Checklist

• [ ] Schema defined from the downstream consumer, with closed enums
• [ ] Every reply validated, including business-rule refinements
• [ ] Retries capped by attempts and timeout, validation error fed back once
• [ ] Fallback writes nothing irreversible and keeps the model's suggestion
• [ ] Confidence floor configurable
• [ ] One structured log record per call with prompt version and outcome
• [ ] Tests for malformed, invalid and timeout paths using an injected model client

We design AI steps inside business workflows with these four controls from the first build: JC TechLabs AI automation.