Multi-agent systems are getting more complex.
What starts as one agent calling a tool can quickly become a system with multiple agents, shared context, retrieval, prompts, model calls, evaluators, and several downstream steps.
The problem is that our development workflow hasn't really caught up.
A small change in one part of the system can cause us to rerun a large part of the pipeline — even when most of the work is still valid.
That becomes especially painful with RAG and multi-agent systems because some steps are expensive.
So I started thinking about a simple question:
Why can't AI pipelines work more like traditional build systems?
The problem with rebuilding AI pipelines
Consider a RAG system:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Index
↓
Retriever
↓
Agent
↓
Evaluation
Now imagine changing only the agent's system prompt.
The documents haven't changed.
The chunks haven't changed.
The embeddings haven't changed.
The vector index hasn't changed.
Yet depending on how the pipeline is implemented, it is easy to end up rerunning much more work than necessary.
Multi-agent systems make this even more obvious.
A system might look more like:
┌── Research Agent ──┐
│ │
User → Planner Agent ── Retrieval Agent ──→ Synthesizer
│ │
└── Tool Agent ──────┘
↓
Evaluator
Now add prompts, models, tools, datasets, retrieval configuration, evaluation criteria, and intermediate artifacts.
The system becomes a dependency graph.
But many workflows still treat it like a script.
AI systems are graphs
I think this is an important mental model.
An AI application isn't necessarily one program that runs from top to bottom.
It's often a graph of dependent artifacts and computations.
For example:
Dataset
↓
Preprocessing
↓
Embeddings
↓
Vector Index
↓
Retrieval
↓
Agent
↓
Evaluation
↓
Report
And some nodes may branch:
┌── Embeddings ──→ Index
Dataset ──→ Clean ──┤
└── Metadata ────→ Evaluation
Once you think about the system this way, a useful property appears:
If a node hasn't changed, its output should usually be reusable.
That's the same basic idea behind build systems such as make.
What an incremental build actually means
An incremental build system doesn't ask:
Should I run the entire pipeline?
It asks:
What changed, and what depends on that change?
Suppose we have:
Dataset → Chunking → Embeddings → Index → Retrieval → Agent → Eval
If the dataset changes, a lot of the graph may need to be rebuilt.
But if only the agent prompt changes:
Dataset cached
Chunking cached
Embeddings cached
Index cached
Retrieval cached
Agent changed
Eval rebuild
There is no reason to recompute the earlier stages.
This sounds obvious.
The difficult part is making the system know what actually changed.
Why timestamps aren't enough
Traditional build systems often use file timestamps to determine whether something needs to be rebuilt.
AI pipelines are more complicated.
An artifact can depend on:
- a dataset
- a configuration file
- a prompt
- model parameters
- preprocessing code
- another artifact
- an evaluation configuration
- an external model
The file itself might not have changed, but its contents or dependencies might have.
This is where content-based fingerprints become useful.
Instead of asking:
Did this file's timestamp change?
we can ask:
Did the content or dependency state that produced this artifact change?
A simple approach is to calculate a content hash.
Conceptually:
artifact_hash =
hash(
input_content
+ configuration
+ dependency_hashes
)
If the resulting fingerprint is identical, the previous output may be reusable.
If it changes, downstream artifacts can be marked stale.
This becomes especially interesting for agents
Multi-agent systems introduce another layer of complexity.
An agent may depend on:
- Model
- Prompt
- Tools
- Tool schemas
- Retrieved context
- Memory
- Agent configuration
- Evaluation criteria
And an agent can produce outputs that become inputs to another agent.
So you can end up with:
Planner
↓
Researcher
↓
Retriever
↓
Analyst
↓
Writer
↓
Evaluator
Changing the writer's prompt shouldn't necessarily cause the researcher or retriever to run again.
Likewise, changing the retriever shouldn't necessarily require rebuilding an unrelated branch of the system.
This is where dependency-aware execution becomes useful.
There is another problem: evaluation
Evaluation is particularly interesting in agentic systems.
A traditional ML pipeline might evaluate a model against a dataset.
Agent evaluation can involve much more:
- final output
- tool calls
- retrieval results
- intermediate steps
- context
- latency
- token usage
- agent trajectories
Modern agent evaluation is increasingly concerned with the whole execution path rather than just the final answer.
That means evaluation itself becomes part of the dependency graph.
For example:
Agent
↓
Trace
↓
Evaluator
↓
Experiment
↓
Report
Change the evaluation criteria? You may only need to rerun the evaluator and downstream report.
Change the underlying agent? Now the evaluation needs new inputs.
This dependency relationship is something a build system can reason about.
Putting the idea into practice
This problem is what led me to build aimake.
The basic idea is to bring incremental build concepts to AI/ML and agent pipelines.
Instead of treating the pipeline as one large execution, aimake represents it as a dependency graph and uses content-based fingerprints to determine what needs to be rebuilt.
The goal isn't to create another agent framework.
It's the layer underneath the workflow — something closer to:
AI / Agent Application
↓
Dependency Graph
↓
Incremental Build System
↓
Cache / Reuse / Execution
I'm experimenting with how far this model can go for RAG, multi-agent workflows, evaluation pipelines, and other AI systems.
Docs: aimake-doc.vercel.app
What I think is interesting about this
As AI applications get larger, the expensive part isn't always the model call itself.
It can be all the surrounding computation:
- processing data
- generating embeddings
- building indexes
- running retrieval
- executing agent trajectories
- running evaluations
- generating reports
- running experiments
If every small change invalidates everything, development becomes unnecessarily slow and expensive.
Incremental computation gives us another option:
Only recompute the part of the system whose inputs actually changed.
That's a very old idea in software engineering.
I don't think AI needs to reinvent it.
Open questions
There are still a lot of interesting problems here.
How should nondeterministic LLM calls be cached?
If the same prompt and model can produce different outputs, content hashing alone isn't enough.
What should happen when an external model changes?
If you're calling an API model, your local inputs may be identical while the provider's model has changed.
How should agent memory affect dependencies?
A memory update could invalidate part of an agent workflow without invalidating everything.
How do you handle stochastic evaluation?
Some evaluations themselves are nondeterministic.
How should distributed agent workflows be scheduled?
Once the dependency graph becomes large, you can start thinking about parallel execution and resource-aware scheduling.
These are the areas I'm interested in exploring.
The bigger idea
I don't think the future of AI development is going to be a collection of isolated LLM calls.
We're increasingly building systems composed of:
- Models
- Agents
- Tools
- Retrieval
- Memory
- Data
- Prompts
- Evaluations
- Experiments
That looks less like a single application and more like a computational graph.
And computational graphs need infrastructure.
We've already spent decades building systems for incremental compilation, caching, dependency tracking, reproducibility, and distributed execution.
There is an opportunity to bring some of those ideas into AI engineering.
That's the problem I'm exploring with aimake.
If you're building RAG systems, multi-agent workflows, or large AI pipelines, I'm particularly interested in how you're currently handling incremental execution and caching.
What do you rebuild today that you wish you didn't have to?
Top comments (0)