Why Multi-Agent AI Pipelines Need Incremental Builds

arjun shukla — Sat, 05 Sep 2026 19:38:39 +0000

Multi-agent systems are getting more complex.

What starts as one agent calling a tool can quickly become a system with multiple agents, shared context, retrieval, prompts, model calls, evaluators, and several downstream steps.

The problem is that our development workflow hasn't really caught up.

A small change in one part of the system can cause us to rerun a large part of the pipeline — even when most of the work is still valid.

That becomes especially painful with RAG and multi-agent systems because some steps are expensive.

So I started thinking about a simple question:

Why can't AI pipelines work more like traditional build systems?

The problem with rebuilding AI pipelines

Consider a RAG system:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Index
    ↓
Retriever
    ↓
Agent
    ↓
Evaluation

Now imagine changing only the agent's system prompt.

The documents haven't changed.

The chunks haven't changed.

The embeddings haven't changed.

The vector index hasn't changed.

Yet depending on how the pipeline is implemented, it is easy to end up rerunning much more work than necessary.

Multi-agent systems make this even more obvious.

A system might look more like:

                    ┌── Research Agent ──┐
                    │                    │
User → Planner Agent ── Retrieval Agent ──→ Synthesizer
                    │                    │
                    └── Tool Agent ──────┘
                             ↓
                          Evaluator

Now add prompts, models, tools, datasets, retrieval configuration, evaluation criteria, and intermediate artifacts.

The system becomes a dependency graph.

But many workflows still treat it like a script.

AI systems are graphs

I think this is an important mental model.

An AI application isn't necessarily one program that runs from top to bottom.

It's often a graph of dependent artifacts and computations.

For example:

Dataset
   ↓
Preprocessing
   ↓
Embeddings
   ↓
Vector Index
   ↓
Retrieval
   ↓
Agent
   ↓
Evaluation
   ↓
Report

And some nodes may branch:

                    ┌── Embeddings ──→ Index
Dataset ──→ Clean ──┤
                    └── Metadata ────→ Evaluation

Once you think about the system this way, a useful property appears:

If a node hasn't changed, its output should usually be reusable.

That's the same basic idea behind build systems such as make.

What an incremental build actually means

An incremental build system doesn't ask:

Should I run the entire pipeline?

It asks:

What changed, and what depends on that change?

Suppose we have:

Dataset → Chunking → Embeddings → Index → Retrieval → Agent → Eval

If the dataset changes, a lot of the graph may need to be rebuilt.

But if only the agent prompt changes:

Dataset       cached
Chunking      cached
Embeddings    cached
Index         cached
Retrieval     cached
Agent         changed
Eval          rebuild

There is no reason to recompute the earlier stages.

This sounds obvious.

The difficult part is making the system know what actually changed.

Why timestamps aren't enough

Traditional build systems often use file timestamps to determine whether something needs to be rebuilt.

AI pipelines are more complicated.

An artifact can depend on:

a dataset
a configuration file
a prompt
model parameters
preprocessing code
another artifact
an evaluation configuration
an external model

The file itself might not have changed, but its contents or dependencies might have.

This is where content-based fingerprints become useful.

Instead of asking:

Did this file's timestamp change?

we can ask:

Did the content or dependency state that produced this artifact change?

A simple approach is to calculate a content hash.

Conceptually:

artifact_hash =
    hash(
        input_content
        + configuration
        + dependency_hashes
    )

If the resulting fingerprint is identical, the previous output may be reusable.

If it changes, downstream artifacts can be marked stale.

This becomes especially interesting for agents

Multi-agent systems introduce another layer of complexity.

An agent may depend on:

Model
Prompt
Tools
Tool schemas
Retrieved context
Memory
Agent configuration
Evaluation criteria

And an agent can produce outputs that become inputs to another agent.

So you can end up with:

Planner
   ↓
Researcher
   ↓
Retriever
   ↓
Analyst
   ↓
Writer
   ↓
Evaluator

Changing the writer's prompt shouldn't necessarily cause the researcher or retriever to run again.

Likewise, changing the retriever shouldn't necessarily require rebuilding an unrelated branch of the system.

This is where dependency-aware execution becomes useful.

There is another problem: evaluation

Evaluation is particularly interesting in agentic systems.

A traditional ML pipeline might evaluate a model against a dataset.

Agent evaluation can involve much more:

final output
tool calls
retrieval results
intermediate steps
context
latency
token usage
agent trajectories

Modern agent evaluation is increasingly concerned with the whole execution path rather than just the final answer.

That means evaluation itself becomes part of the dependency graph.

For example:

Agent
  ↓
Trace
  ↓
Evaluator
  ↓
Experiment
  ↓
Report

Change the evaluation criteria? You may only need to rerun the evaluator and downstream report.

Change the underlying agent? Now the evaluation needs new inputs.

This dependency relationship is something a build system can reason about.

Putting the idea into practice

This problem is what led me to build aimake.

The basic idea is to bring incremental build concepts to AI/ML and agent pipelines.

Instead of treating the pipeline as one large execution, aimake represents it as a dependency graph and uses content-based fingerprints to determine what needs to be rebuilt.

The goal isn't to create another agent framework.

It's the layer underneath the workflow — something closer to:

                 AI / Agent Application
                          ↓
                 Dependency Graph
                          ↓
              Incremental Build System
                          ↓
             Cache / Reuse / Execution

I'm experimenting with how far this model can go for RAG, multi-agent workflows, evaluation pipelines, and other AI systems.

Docs: aimake-doc.vercel.app

What I think is interesting about this

As AI applications get larger, the expensive part isn't always the model call itself.

It can be all the surrounding computation:

processing data
generating embeddings
building indexes
running retrieval
executing agent trajectories
running evaluations
generating reports
running experiments

If every small change invalidates everything, development becomes unnecessarily slow and expensive.

Incremental computation gives us another option:

Only recompute the part of the system whose inputs actually changed.

That's a very old idea in software engineering.

I don't think AI needs to reinvent it.

Open questions

There are still a lot of interesting problems here.

How should nondeterministic LLM calls be cached?

If the same prompt and model can produce different outputs, content hashing alone isn't enough.

What should happen when an external model changes?

If you're calling an API model, your local inputs may be identical while the provider's model has changed.

How should agent memory affect dependencies?

A memory update could invalidate part of an agent workflow without invalidating everything.

How do you handle stochastic evaluation?

Some evaluations themselves are nondeterministic.

How should distributed agent workflows be scheduled?

Once the dependency graph becomes large, you can start thinking about parallel execution and resource-aware scheduling.

These are the areas I'm interested in exploring.

The bigger idea

I don't think the future of AI development is going to be a collection of isolated LLM calls.

We're increasingly building systems composed of:

Models
Agents
Tools
Retrieval
Memory
Data
Prompts
Evaluations
Experiments

That looks less like a single application and more like a computational graph.

And computational graphs need infrastructure.

We've already spent decades building systems for incremental compilation, caching, dependency tracking, reproducibility, and distributed execution.

There is an opportunity to bring some of those ideas into AI engineering.

That's the problem I'm exploring with aimake.

If you're building RAG systems, multi-agent workflows, or large AI pipelines, I'm particularly interested in how you're currently handling incremental execution and caching.

What do you rebuild today that you wish you didn't have to?

DEV Community: arjun shukla