DEV Community

Yuya AI Lab
Yuya AI Lab

Posted on Edited on AI-assisted

How to Build a Reliable 3-Step Chained AI Prompt Pipeline in Python (With Schema Validation and Zero-Cost Fallbacks)

A three-step prompt chain — plan, draft, review — works beautifully in a notebook and falls apart the first week it runs unattended. Not because the model is bad, but because nothing in the chain checks what each step hands to the next one, and nothing decides how much money a failure is allowed to spend before it starts.

This post is the small amount of structure that fixed that for me: three steps, a contract per step, two different recovery strategies, and a cost ceiling that is enforced before a request leaves the process. Everything is standard library. No SDK, no framework, no dependency to pin.

The four failure modes I kept hitting:

  1. Type drift. Step 1 returns "3", step 2 was written for 3, and step 3 silently renders nothing.
  2. Wrapped output. The model answers with a friendly sentence, then a fenced block, then JSON — and occasionally a trailing "Let me know if you want me to expand this!"
  3. Cost creep. A retry loop that retries a validation failure at the most expensive model in the list, four times, at 3 a.m.
  4. One provider, no plan. The first 429 ends the run, even though a free tier that is nobody's bottleneck was already configured.

Step 1: Give every hop a contract

A prompt is not a schema, and "please return JSON" is not validation. I describe each step's output with plain dataclasses — one Field per key — and validate the parsed reply against it. That is enough to catch everything on the list above, and it is ~50 lines you can read in one sitting.

from dataclasses import dataclass, field
from typing import Any, Mapping, Sequence
import json, time

class ValidationError(ValueError):
    """The model answered, but the answer is not what the step promised."""

@dataclass(frozen=True)
class Field:
    name: str
    kind: type
    min_len: int = 0
    max_len: int | None = None
    choices: tuple[str, ...] | None = None
    minimum: int | None = None
    maximum: int | None = None
    item: "Field | None" = None            # list of scalars
    of: tuple["Field", ...] | None = None  # list of objects

    def opts(self) -> dict:
        out = {}
        if self.min_len: out["min_len"] = self.min_len
        if self.max_len is not None: out["max_len"] = self.max_len
        if self.choices is not None: out["choices"] = self.choices
        if self.minimum is not None: out["minimum"] = self.minimum
        if self.maximum is not None: out["maximum"] = self.maximum
        return out

def _check(obj: Mapping[str, Any], f: Field) -> Any:
    if f.name not in obj:
        raise ValidationError(f"missing field {f.name!r}")
    v = obj[f.name]

    if f.kind is int:
        # bool is an int subclass in Python; reject it on purpose.
        if isinstance(v, bool) or not isinstance(v, int):
            raise ValidationError(f"{f.name!r} must be int, got {type(v).__name__}")
        if f.minimum is not None and v < f.minimum:
            raise ValidationError(f"{f.name!r} is below {f.minimum}")
        if f.maximum is not None and v > f.maximum:
            raise ValidationError(f"{f.name!r} is above {f.maximum}")
        return v

    if f.kind is list:
        if not isinstance(v, list):
            raise ValidationError(f"{f.name!r} must be a list")
        if f.max_len is not None and len(v) > f.max_len:
            raise ValidationError(f"{f.name!r} has more than {f.max_len} items")
        if f.item is not None:
            rows = []
            for i, it in enumerate(v):
                key = f"{f.name}[{i}]"
                rows.append(_check({key: it}, Field(key, f.item.kind, **f.item.opts())))
            return rows
        if f.of is not None:
            rows = []
            for i, it in enumerate(v):
                if not isinstance(it, dict):
                    raise ValidationError(f"{f.name}[{i}] must be an object")
                rows.append({sub.name: _check(it, sub) for sub in f.of})
            return rows
        return v

    if not isinstance(v, f.kind):
        raise ValidationError(f"{f.name!r} must be {f.kind.__name__}, got {type(v).__name__}")
    if isinstance(v, str):
        v = v.strip()
        if len(v) < f.min_len:
            raise ValidationError(f"{f.name!r} is shorter than {f.min_len} chars")
        if f.max_len is not None and len(v) > f.max_len:
            raise ValidationError(f"{f.name!r} is longer than {f.max_len} chars")
    if f.choices is not None and v not in f.choices:
        raise ValidationError(f"{f.name!r} must be one of {list(f.choices)}, got {v!r}")
    return v
Enter fullscreen mode Exit fullscreen mode

Three details earn their keep. bool is rejected where an int is required, because True will happily pass an isinstance(x, int) check and then break an arithmetic step three hops later. choices turns free text into an enum, which is the single cheapest guardrail against a model inventing "looks good" where you asked for "ship". And every error message names the field and the rule, because that message becomes the repair prompt in a moment.

A step is a name, a system prompt, its fields, and — this matters for cost — an explicit list of which upstream keys it is allowed to see.

@dataclass
class Step:
    name: str
    system: str
    fields: tuple[Field, ...] = ()
    needs: tuple[str, ...] = ()                        # upstream keys it sees
    max_repairs: int = 1
    clip: dict[str, int] = field(default_factory=dict)  # "key.subkey" -> chars

    def validate(self, payload: Any) -> dict[str, Any]:
        if not isinstance(payload, dict):
            raise ValidationError("top-level JSON must be an object")
        return {f.name: _check(payload, f) for f in self.fields}

    def shape(self) -> dict[str, Any]:
        out = {}
        for f in self.fields:
            if f.of is not None:
                out[f.name] = [{s.name: s.kind.__name__ for s in f.of}]
            elif f.item is not None:
                out[f.name] = [f.item.kind.__name__]
            else:
                out[f.name] = f.kind.__name__
        return out
Enter fullscreen mode Exit fullscreen mode

shape() also solves failure mode 2 at the source: the prompt shows the model the exact keys and Python types it must return, serialized with json.dumps(..., separators=(",", ":")) and no prose around it.

Step 2: Parse defensively, because the fence is coming

Even with a shape in the prompt, replies arrive fenced, prefixed, or suffixed. Find the outermost braces and parse only that span — and turn a parse failure into a ValidationError so it takes the same repair path as a wrong type.

FENCE = "`" * 3

def _load_json(raw: str) -> Any:
    """Models love wrapping JSON in prose or fences. Strip that before parsing."""
    t = raw.strip()
    if t.startswith(FENCE):
        t = t.split("\n", 1)[-1]
        t = t.rsplit(FENCE, 1)[0]
    start, end = t.find("{"), t.rfind("}")
    if start == -1 or end <= start:
        raise ValidationError("no JSON object found in the reply")
    try:
        return json.loads(t[start:end + 1])
    except json.JSONDecodeError as e:
        raise ValidationError(f"invalid JSON: {e.msg} at char {e.pos}") from e
Enter fullscreen mode Exit fullscreen mode

If you have ever written a regex to strip code fences, note that the version above handles a fence with a language tag and a closing fence on its own line, and nothing more — which is exactly as much cleverness as this deserves. FENCE is built as "`" * 3 so that this very article's code block does not contain the characters that would close it. Leftover edge cases land in the repair path, where they belong.

Step 3: Two recovery strategies, not one

This is the part I got wrong for a long time. A transport failure (429, timeout, 5xx) and a contract failure (valid HTTP, wrong shape) are not the same event, and retrying them identically is how you burn an evening of quota.

  • Transport failure → same step, same prompt, short backoff, then the next provider down the ladder.
  • Contract failure → same step, different prompt: the complaint, and nothing else. Re-sending two kilobytes of broken output to tell the model it was broken is the most expensive way to say "no".
@dataclass(frozen=True)
class Provider:
    name: str
    call: Callable[[str, str], str]   # (system, prompt) -> raw text
    tier: int = 0                     # 0 free/local, 1 free tier, 2 paid

class BudgetExceeded(RuntimeError):
    pass

class ChainRunner:
    def __init__(self, providers: Sequence[Provider], *, max_tier: int = 0,
                 max_prompt_tokens: int = 1500,
                 sleep: Callable[[float], None] = time.sleep,
                 backoff: float = 0.0) -> None:
        self.providers = list(providers)
        self.max_tier = max_tier
        self.max_prompt_tokens = max_prompt_tokens
        self.sleep = sleep
        self.backoff = backoff
        self.spent_tokens = 0

    @staticmethod
    def estimate_tokens(text: str) -> int:
        return max(1, len(text) // 4)   # rough but stable for English/JSON

    def _guard(self, system: str, prompt: str) -> None:
        n = self.estimate_tokens(system) + self.estimate_tokens(prompt)
        if n > self.max_prompt_tokens:
            raise BudgetExceeded(f"prompt is ~{n} tokens, over the {self.max_prompt_tokens} cap")

    def _ask(self, step: Step, prompt: str) -> tuple[str, str]:
        last = None
        for p in [p for p in self.providers if p.tier <= self.max_tier]:
            for attempt in range(2):          # one try, one retry, that is all
                try:
                    return p.call(step.system, prompt), p.name
                except Exception as e:        # 429 / 5xx / socket
                    last = e
                    if attempt == 0 and self.backoff:
                        self.sleep(self.backoff)
        raise RuntimeError(f"every provider at tier <= {self.max_tier} failed: {last!r}")

    def run(self, steps: Sequence[Step], seed: Mapping[str, Any]) -> dict[str, Any]:
        state = dict(seed)
        trace = []
        for step in steps:
            prompt = step.prompt(state)
            self._guard(step.system, prompt)        # before the network, not after
            repairs = 0
            while True:
                raw, provider = self._ask(step, prompt)
                self.spent_tokens += self.estimate_tokens(prompt) + self.estimate_tokens(raw)
                try:
                    out = step.validate(_load_json(raw))
                except ValidationError as e:
                    trace.append({"step": step.name, "provider": provider,
                                  "ok": False, "error": str(e)})
                    if repairs >= step.max_repairs:
                        raise
                    repairs += 1
                    # send back the complaint, not the broken output
                    prompt = step.prompt(state, note=str(e))
                    continue
                trace.append({"step": step.name, "provider": provider,
                              "ok": True, "repairs": repairs})
                state[step.name] = out
                break
        state["_trace"] = trace
        return state
Enter fullscreen mode Exit fullscreen mode

max_tier is the cost policy, and it is a filter on the provider list rather than a flag inside each provider — so a provider that costs money is not merely discouraged, it is unreachable when the ceiling is 0. _guard measures the prompt before _ask is called; a runaway context is a free, local, instant exception instead of a bill. And max_repairs=1 per step is deliberate: if the model cannot satisfy the contract in two tries, the prompt or the contract is wrong, and a third try is just a more expensive way to learn the same thing.

Putting the chain together

outline → draft → review, where review deliberately does not receive the whole draft.

OUTLINE = Step(
    name="outline",
    system="You are a technical editor. Plan short, concrete developer posts.",
    needs=("topic", "audience"),
    fields=(
        Field("promise", str, min_len=20, max_len=160),
        Field("sections", list, max_len=6, of=(
            Field("heading", str, min_len=3, max_len=80),
            Field("intent", str, min_len=10, max_len=200),
        )),
    ),
)

DRAFT = Step(
    name="draft",
    system="You are a senior engineer writing runnable, copy-pasteable code.",
    needs=("promise", "sections"),
    fields=(
        Field("title", str, min_len=15, max_len=100),
        Field("body_markdown", str, min_len=400),
    ),
)

REVIEW = Step(
    name="review",
    system="You are a strict reviewer. Reject what you cannot verify.",
    needs=("promise", "draft"),
    clip={"draft.body_markdown": 1800},   # head + tail, not the whole body
    fields=(
        Field("score", int, minimum=0, maximum=100),
        Field("verdict", str, choices=("ship", "revise")),
        Field("issues", list, max_len=5, item=Field("issue", str, min_len=5, max_len=200)),
    ),
)
Enter fullscreen mode Exit fullscreen mode

The reviewer gets needs=("promise", "draft") — not the topic, not the audience, not the outline. Its only question is "does this body deliver the promised thing", and the body is clipped to its first and last 900 characters with an explicit [... N chars omitted ...] marker. Head plus tail is where titles, openings, and conclusions live, and it is enough to judge structure without paying for the middle of a 9,000-character draft.

That clip alone is measurable: on a 9,000-character draft the review prompt drops from 9,277 characters to 2,089 — 78% smaller, roughly 2,300 tokens down to 520. The whole three-step chain, including the draft reply, came to about 2,900 estimated tokens.

Verify it with stubs, not with your API key

The point of separating providers from steps is that you can exercise every failure path with canned replies. No key, no network, no tokens spent.

GOOD_OUTLINE = json.dumps({
    "promise": "Ship a chained prompt pipeline that fails safely and costs nothing.",
    "sections": [
        {"heading": "The failure modes", "intent": "Show how step 2 poisons step 3."},
        {"heading": "Contracts", "intent": "Validate every hop with dataclasses."},
    ],
})
GOOD_DRAFT = json.dumps({"title": "Chaining prompts without trusting them",
                         "body_markdown": "x" * 9000})
GOOD_REVIEW = json.dumps({"score": 91, "verdict": "ship",
                          "issues": ["No benchmark numbers for the fallback ladder."]})

def stub(*replies: str):
    seen = []
    def call(system: str, prompt: str) -> str:
        seen.append(prompt)
        return replies[min(len(seen) - 1, len(replies) - 1)]
    return call, seen
Enter fullscreen mode Exit fullscreen mode

Running the checks I care about gives this:

1) happy path   : [('outline', True, 0), ('draft', True, 0), ('review', True, 0)]
2) repair loop  : [('outline', False, "'promise' must be str, got int"), ('outline', True, ''), ...]
3) fallback     : {'free-tier'}   # primary raised 429, next tier answered
4) ceiling      : paid provider calls = 0
5) budget guard : prompt is ~2026 tokens, over the 200 cap
6) prompt chars : clipped=2089 full=9277 (78% smaller)
7) rejected     : 'score' is above 100
7) rejected     : 'score' must be int, got bool
7) rejected     : 'verdict' must be one of ['ship', 'revise'], got 'maybe'
Enter fullscreen mode Exit fullscreen mode

Number 2 is the one that changed my mind about chains: the repair prompt is the original prompt plus Your previous reply was rejected: 'promise' must be str, got int. Fix it and reply again. — around 40 extra tokens instead of a full re-send with the bad output attached. Whether the second attempt lands depends on the model and the field; what the design guarantees is that a repair attempt is cheap, bounded, and visible. The _trace shows exactly which step needed repairing, which is what you actually want in a log line.

Note also number 5: the guard fired on a prompt I built by accident, and it fired locally. A pipeline that can only discover it overspent from a dashboard is a pipeline that will, eventually, overspend.

What I would add before trusting it in production

Three things, in order of how much pain they save:

  1. Persist every raw reply as a fixture, keyed by step and prompt hash. Days later, when a step starts failing, you can replay the exact reply through validate() instead of guessing — and your regression test becomes deterministic.
  2. An idempotency key per (run, step) in whatever queue you use. Retries are part of the design above, so a step that succeeded and then lost its acknowledgement must not run twice.
  3. Total-chain budget, not just per-prompt. spent_tokens is already accumulated in run(); a ceiling there stops the pathology where every step is individually legal and the chain is not.

None of this is specific to prompting. It is the ordinary discipline of a distributed call graph — typed boundaries, two recovery regimes, a cost ceiling at the edge — applied to a component that fails in a slightly more creative way than an HTTP service does. The chain did not get smarter. It got observable, and that turned out to be the same thing as reliable.

Further Resources

The full single-file module (contracts, runner, steps, and the demo checks above) is the skeleton I reuse for other chains, and I keep a cleaned-up version of it — plus the prompt templates and the free-tier provider setup I run it on — in my workflow pack: The Complete 2026 AI Side-Hustle Blueprint: Zero-Cost Prompts & Workflows.

Note: This post includes a link to my own prompt workflow pack. The code above is complete and runnable without it.

Top comments (0)