DEV Community

Cover image for AIchain: Connecting Steps into a Pipeline
YAIT
YAIT

Posted on

AIchain: Connecting Steps into a Pipeline

#programming #ai #llm #architecture

One LLM call solves a simple task. Real tasks look nothing like that. Real tasks are: fetch a page, extract the key points, rewrite them for a specific audience, translate the result into five languages. Four steps, each with its own job. Chain is the glue between them.

The Problem with "One Big Prompt"

When developers first start building with LLMs, there's a natural instinct to pack everything into a single elaborate prompt. "Read this URL, summarize it, rewrite it in a friendly tone, and translate it to Spanish." Sometimes it works. Often it doesn't — you've handed one model a vague, multi-part job with no clear boundaries, and when the output is wrong, you can't tell which sub-task failed. Was it the summary? The rewrite? The translation? Token limits compound the problem, and output formatting across sub-tasks becomes inconsistent with nothing to enforce boundaries.

A better pipeline is several simple steps, each with a clear boundary of responsibility. One step fetches. Another writes. Another edits. Another translates. Each step does one thing well, and the output of one becomes the input of the next.

That's what Chain does in yait_aichain.

How Chain Works

A Chain is an ordered sequence of steps. Each step is a Skill, a tool, or even an Agent. They all share one accumulated variable dictionary — outputs flow forward automatically. No manual variable shuffling, no temp files.

Here's the simplest version: two skills, two different providers.

from yait_aichain import Model, Skill, Chain

writer = Skill(
    model=Model(provider="openai", model="gpt-4o-mini"),
    prompt="Write a short paragraph about {topic}.",
)

reviewer = Skill(
    model=Model(provider="anthropic", model="claude-3-5-haiku-20241022"),
    prompt="Review this paragraph and suggest one improvement:\n\n{result}",
)

chain = Chain(steps=[writer, reviewer])
result = chain.run(variables={"topic": "the importance of clean code"})

print(result)
Enter fullscreen mode Exit fullscreen mode

Step 1 runs GPT-4o-mini. Its output is stored under the default key "result" in the accumulated variable dict. Step 2 picks up {result} in its prompt template and sends it to Claude. Two models, two providers, no plumbing.

Notice that each step uses its own model. You're not locked into one provider for the entire pipeline. Use GPT for generation, Claude for review, a cheap model for translation — whatever makes sense for each task.

Mixing Tools and Skills

Not every step needs an LLM. Sometimes the first step is just fetching data. Chain handles that too.

from yait_aichain import Model, Skill, Chain
from yait_aichain.tools import convertToMD

summariser = Skill(
    model=Model(provider="anthropic", model="claude-3-5-haiku-20241022"),
    prompt="Summarise this page in three bullet points:\n\n{result}",
)

chain = Chain(steps=[convertToMD, summariser])
result = chain.run(variables={"url": "https://en.wikipedia.org/wiki/Python_(programming_language)"})

print(result)
Enter fullscreen mode Exit fullscreen mode

convertToMD is a tool — it fetches a URL and converts the page to Markdown. No LLM involved. It stores its output under the default key "result", which is why the summariser's prompt references {result} directly. Tools and skills sit side by side in the same pipeline.

Named Keys and Variable Remapping

When pipelines get longer, the default "result" key isn't enough. You want each step's output to have a meaningful name so downstream steps can reference exactly what they need.

Pass a 2-tuple of (step, output_key) to give a step's output an explicit name:

chain = Chain(steps=[
    (summariser,  "summary"),
    (translator,  "final"),
])
Enter fullscreen mode Exit fullscreen mode

Now the summariser's output lives under accumulated["summary"], and the translator's prompt can reference {summary} directly. Every step has access to every previously accumulated variable — not just the one immediately before it.

If a skill's prompt expects a variable name that doesn't match what's in the accumulated dict, remap it with a 3-tuple of (step, output_key, remap_dict):

# Stored as "body_content" earlier in the pipeline,
# but this skill's prompt expects {current_section_content}
(summariser, "summary", {"current_section_content": "body_content"})
Enter fullscreen mode Exit fullscreen mode

The remap dict maps prompt_variable → accumulated_key. Here, Chain reads the value stored under "body_content" and passes it to summariser as current_section_content. Here's a minimal working example:

from yait_aichain import Model, Skill, Chain

extractor = Skill(
    model=Model(provider="openai", model="gpt-4o-mini"),
    prompt="Extract the main argument from this text:\n\n{text}",
)

summariser = Skill(
    model=Model(provider="openai", model="gpt-4o-mini"),
    prompt="Summarise this argument in one sentence:\n\n{current_section_content}",
)

chain = Chain(steps=[
    (extractor,  "body_content"),
    (summariser, "summary", {"current_section_content": "body_content"}),
])

result = chain.run(variables={"text": "Your source text here..."})
print(result)
Enter fullscreen mode Exit fullscreen mode

So steps come in three forms: a bare runner, a 2-tuple with an output key, or a 3-tuple with an output key and a remap dict. Pick whichever fits the complexity of that step.

The Real-World Pipeline

Here's the scenario from the introduction: fetch a source URL, write content from it, make it readable, translate it. Four steps, each with one job. Named keys keep every output identifiable.

from yait_aichain import Model, Skill, Chain
from yait_aichain.tools import convertToMD

writer = Skill(
    model=Model(provider="openai", model="gpt-4o"),
    prompt="Based on this source material, write a concise blog post:\n\n{result}",
)

editor = Skill(
    model=Model(provider="anthropic", model="claude-3-5-sonnet-20241022"),
    prompt="Make this text more readable and engaging. Keep it concise:\n\n{draft}",
)

translator = Skill(
    model=Model(provider="openai", model="gpt-4o-mini"),
    prompt="Translate the following into {language}:\n\n{edited}",
)

chain = Chain(steps=[
    convertToMD,
    (writer,     "draft"),
    (editor,     "edited",  {"draft": "draft"}),
    (translator, "final"),
])

result = chain.run(variables={
    "url": "https://example.com/source-article",
    "language": "French",
})

print(result)
Enter fullscreen mode Exit fullscreen mode

Four steps. Three different models — GPT-4o for writing, Claude Sonnet for editing, GPT-4o-mini for translation. One tool. To run this for multiple languages, loop over the language values:

for language in ["French", "German", "Japanese", "Spanish", "Portuguese"]:
    result = chain.run(variables={
        "url": "https://example.com/source-article",
        "language": language,
    })
    print(f"--- {language} ---")
    print(result)
Enter fullscreen mode Exit fullscreen mode

Could an Agent handle this? Sure. But why introduce probabilistic decision-making when the sequence is fixed? You don't need a planner deciding "what to do next" when the answer is always step 1, then 2, then 3, then 4. Chain gives you deterministic, repeatable execution — the same inputs produce the same sequence of calls every time.

Debugging with chain.history

When a four-step pipeline produces unexpected output, you need to know which step went sideways. After chain.run() completes, inspect chain.history:

print(chain.history)
Enter fullscreen mode Exit fullscreen mode

chain.history is a list of dicts, one per step. Here's what an entry looks like:

[
  {
    "step_index": 0,
    "runner": "convertToMD",
    "inputs": {"url": "https://example.com/source-article"},
    "output": "# Example Article\n\nMarkdown content here..."
  },
  {
    "step_index": 1,
    "runner": "Skill(gpt-4o)",
    "inputs": {"result": "# Example Article\n\nMarkdown content here..."},
    "output": "Here is a concise blog post based on the source material..."
  },
  ...
]
Enter fullscreen mode Exit fullscreen mode

You look at step 2's output, see the editor mangled your formatting, fix that one prompt, and re-run. No diagnostic print statements scattered through your code, no running the whole pipeline blind hoping something changes.

This is the same step-by-step visibility that visual workflow tools offer through node inspectors — but it's plain Python. No YAML config files, no browser-based editors. Define your pipeline in code, run it, read the history.

When to Use Chain vs. an Agent

Simple rule: if the sequence of steps is known in advance, use Chain. If the system needs to decide at runtime which tools to call and in what order based on an ambiguous goal, use an Agent.

Most production workflows are deterministic sequences. Content pipelines, data processing, report generation, ETL flows — fixed sequences with variable inputs. Chain was built for exactly this: a code-first alternative to visual workflow tools, without the overhead.

Keep steps simple, keep them separated, and let each one do exactly one thing. That's what makes a pipeline maintainable six months later when you've completely forgotten how it works.

Top comments (0)